Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "No-one in my org puts money in their pension", published by tobyj on February 16, 2024 on The Effective Altruism Forum.

Epistemic status: the stories here are all as true as possible from memory, but my memory is so so.

This is going to be big

It's late Summer 2017. I am on a walk in the Mendip Hills. It's warm and sunny and the air feels fresh. With me are around 20 other people from the Effective Altruism London community. We've travelled west for a retreat to discuss how to help others more effectively with our donations and careers. As we cross cow field after cow field, I get talking to one of the people from the group I don't know yet. He seems smart, and cheerful. He tells me that he is an AI researcher at Google DeepMind.

He explains how he is thinking about how to make sure that any powerful AI system actually does what we want it to. I ask him if we are going to build artificial intelligence that can do anything that a human can do. "Yes, and soon," he says, "And it will be the most important thing that humanity has ever done."

I find this surprising. It would be very weird if humanity was on the cusp of the most important world changing invention ever, and so few people were seriously talking about it. I don't really believe him.

This is going to be bad

It is mid-Summer 2018 and I am cycling around Richmond Park in South West London. It's very hot and I am a little concerned that I am sweating off all my sun cream.

After having many other surprising conversations about AI, like the one I had in the Mendips, I have decided to read more about it. I am listening to an audiobook of Superintelligence by Nick Bostrom. As I cycle in loops around the park, I listen to Bostrom describe a world in which we have created superintelligent AI. He seems to think the risk that this will go wrong is very high. He explains how scarily counterintuitive the power of an entity that is vastly more intelligent than a human is.

He talks about the concept of "orthogonality"; the idea that there is no intrinsic reason that the intelligence of a system is related to its motivation to do things we want (e.g. not kill us). He talks about how power-seeking is useful for a very wide range of possible goals. He also talks through a long list of ways we might try to avoid it going very wrong. He then spends a lot of time describing why many of these ideas won't work. I wonder if this is all true.

It sounds like science fiction, so while I notice some vague discomfort with the ideas, I don't feel that concerned. I am still sweating, and am quite worried about getting sunburnt.

It's a long way off though

It's still Summer 2018 and I am in an Italian restaurant in West London. I am at an event for people working in policy who want to have more impact. I am talking to two other attendees about AI. Bostrom's arguments have now been swimming around my mind for several weeks. The book's subtitle is "Paths, Dangers, Strategies" and I have increasingly been feeling the weight of the middle one. The danger feels like a storm. It started as vague clouds on the horizon and is now closing in.

I am looking for shelter.

"I just don't understand how we are going to set policy to manage these things" I explain.

I feel confused and a little frightened.

No-one seems to have any concrete policy ideas. But my friend chimes in to say that while yeah there's a risk, it's probably pretty small and far away at this point.

"Experts thinks it'll take at least 40 more years to get really powerful AI" she explains, "there is plenty of time for us to figure this out".

I am not totally reassured, but the clouds retreat a little.

This is fine

It is late January 2020 and I am at after-work drinks in a pub in Westminster. I am talking to a few colleagues about the news. One of my colleagues, an accomplished government ec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective: PIBBSS Fellowship 2023, published by DusanDNesic on February 16, 2024 on The AI Alignment Forum.

Between June and September 2023, we (Nora and Dusan) ran the second iteration of the PIBBSS Summer Fellowship. In this post, we share some of our main reflections about how the program went, and what we learnt about running it.

We first provide some background information about (1) The theory of change behind the fellowship, and (2) A summary of key program design features. In the second part, we share our

reflections on (3) how the 2023 program went, and (4) what we learned form running it.

This post builds on an extensive internal report we produced back in September. We focus on information we think is most likely to be relevant to third parties, in particular:

People interested in forming opinions about the impact of the PIBBSS fellowship, or similar fellowship programs more generally

People interested in running similar programs, looking to learn from mistakes that others made or best practices they converged to

Also see

our reflections on the 2022 fellowship program. If you have thoughts on how we can improve, you can use

this name-optional feedback form.

Background

Fellowship Theory of Change

Before focusing on the fellowship specifically, we will give some context on PIBBSS as an organization.

PIBBSS overall

PIBBSS is a research initiative focused on leveraging insights and talent from fields that study intelligent behavior in natural systems to help make progress on questions in AI risk and safety. To this aim, we run several programs focusing on research, talent and field-building.

The focus of this post is our fellowship program - centrally a talent intervention. We ran the second iteration of the fellowship program in summer 2023, and are currently in the process of selecting fellows for the 2024 edition.

Since PIBBSS' inception, our guesses for what is most valuable to do have evolved. Since the latter half of 2023, we have started taking steps towards focusing on more concrete and more inside-view driven research directions. To this end, we started hosting several full-time research affiliates in January 2024. We are currently working on a more comprehensive update to our vision, strategy and plans, and will be sharing these developments in an upcoming post.

PIBBSS also pursues a range of other efforts aimed more broadly at field-building, including (co-)organizing a range of topic-specific AI safety workshops and hosting semi-regular

speaker events which feature research from a range of fields studying intelligent behavior and exploring their connections to the problem of AI Risk and Safety.

Zooming in on the fellowship

The Summer Research Fellowship pairs fellows (typically PhDs or Postdocs) from disciplines studying complex and intelligent behavior in natural and social systems, with mentors from AI alignment. Over the course of the 3-month long program, fellows and mentors work on a collaborative research project, and fellows are supported in developing proficiency in relevant skills relevant to AI safety research.

One of the driving rationales in our decision to run the program is that a) we believe that there are many areas of expertise (beyond computer science and machine learning) that have useful (if not critical) insight, perspectives and methods to contribute to mitigating AI risk and safety, and b) to the best of our knowledge, there does not exist other programs that specifically aim to provide an entry point into technical AI safety research for people from such fields.

What we think the program can offer:

To fellows

increased understanding of the AI risk problem, as well as potential avenues for reducing these risks.

the opportunity to explore how they can usefully apply their expertise, including identifying promising lines of ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Accelerate Your New EA Organization with Fiscal Sponsorship and Operations Support, published by KevinN on February 16, 2024 on The Effective Altruism Forum.

This post is primarily written for people who are considering, or are in the process of, starting a new nonprofit organization, especially those who are concerned about the operational aspects of running the project. Forming a new organization can be a daunting prospect, and finding the right fiscal sponsorship or operations support provider can significantly lower the bar for setting up a new org.

In my experience, there is limited awareness and understanding of these services within the EA community, which encouraged me to create a post like this. There are

many,

many great projects that are yet to be founded, and my hope is that increased awareness of the benefits of these services encourages more prospective founders to take the leap into entrepreneurship.

Summary

What these terms mean:

Fiscal sponsorship: Leverage another organization's existing legal and tax-exempt status (such as 501(c)(3) status in the US), rather than establishing a new tax-exempt entity. This is also called 'fiscal hosting' in the UK.

Operations support: Hire domain experts on a fractional, as-needed basis to perform operational services (compliance, finance, human resources, etc.) for an organization.

Key potential benefits and drawbacks:

Fiscal sponsorship = Instant initial infrastructure: Fiscal sponsorship significantly reduces the cost and time required to start doing object-level work by avoiding the need to create and operate a new tax-exempt entity.

Operations support = Ongoing support from specialists: Operations support provides additional organizational capacity and allows efficient access to a wide variety of skill sets.

Coordinated expert capacity - Hiring comprehensive operations support, or fiscal sponsorship plus operations support, can yield additional benefits when a service provider works cross-functionally in an efficient and synergistic way.

Potential Drawbacks - Costs and control: Entering into a fiscal sponsorship agreement and hiring operations support may incur significant service costs. Fiscal sponsorship, in particular, may slow down decision-making and restrict flexibility around key tasks such as financial management, hiring, and traveling, due to the fiscal sponsor's governance process and policies.

Due to the benefits and costs, fiscal sponsorship and operations support tend to have outsized benefits for newer and smaller organizations.

New organizations generally benefit more from the 'instant infrastructure' of fiscal sponsorship and find the limited flexibility less costly. For a new organization already facing a huge amount of other decisions, having some functional structure and process in place quickly and easily can be much more valuable than having it be perfect. On the other hand, organizations that already have an existing legal structure and tax-exempt status may not benefit as much from fiscal sponsorship.

Smaller organizations often benefit from operations support, where part-time contractors can provide expert capacity (or even coordinated expert capacity) across a variety of skill sets in a way that's more cost-efficient than a full-time operations employee. Conversely, a larger organization that requires more than several full-time operations staff may see less benefit from operations support as they are able to hire for more specialized operations roles internally.

New organizations may choose to start off utilizing fiscal sponsorship and operations support services to get up to speed quickly and then, over time, reduce their reliance on external support providers as they build internal capacity, scale, and expertise.

Disclosures: I have tried to provide a balanced view of fiscal s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summer Internships at Open Philanthropy - Global Catastrophic Risks (due March 4), published by Hazel Browne on February 16, 2024 on The Effective Altruism Forum.

We're excited to announce that the

Global Catastrophic Risks Cause Prioritization team at Open Philanthropy will be hiring several interns this summer to help us explore new causes and advance our research. We think this is a great way for us to grow our capacity, develop research talent, and expand our pipeline for future full-time roles. The key points are:

Applications are due at 11:59 PM Pacific on Monday, March 4, 2023.

Applicants must be currently enrolled in a degree program or working in a position that offers externship/secondment opportunities.

The internship runs from June 10 to August 16-30 (with limited adjustments based on academic calendars) and is paid ($2,000 per week) and fully remote.

We're open to a wide variety of backgrounds, but expect some of the strongest candidates to be enrolled in master's or doctoral programs.

We aim to employ people with many different experiences, perspectives and backgrounds who share our passion for accomplishing as much good as we can. We particularly encourage applications from people of color, self-identified women, non-binary individuals, and people from low and middle income countries.

Full details (and a link to the application)

are available here and are also copied below.

We hope that you'll apply and share the news with others!

About the internship

We're looking for students currently enrolled in degree programs (or non-students whose jobs offer externship/secondment opportunities) to apply for a research internship from June - August 2024 and help us investigate important questions and causes. We see the internship as a way to grow our capacity, develop promising research talent, and expand our hiring pipeline for full-time roles down the line.

We want to support interns as team members to work on our core priorities, while also showing them how Open Philanthropy works and helping them build skills important for cause prioritization research. As such, interns will directly increase our team's capacity to do research that informs our Global Catastrophic Risks strategy and grantmaking. Ultimately, this will help us get money to where it can have the most impact.

We anticipate that interns will collaborate closely with the team; at the same time, we expect a high degree of independence and encourage self-directed work.

Our internship tracks

We are hiring interns for either the Research or Strategy tracks. The responsibilities for these tracks largely overlap, and the two positions will be evaluated using the same application materials. The main difference is one of emphasis: while research track interns primarily focus on direct research, strategy track interns are sometimes tasked with working on non-research projects (such as helping run a contest or a request for proposals).

You will be asked to indicate which track you'd like to be considered for in the application.

Interns will work on multiple projects at different levels of depth, in the same way as full-time team members. They will report to an existing cause prioritization team member and participate in team meetings and discussions, including presenting their work to the team for feedback.

Specific projects will depend on the team's needs and the intern's skills, but will fall under the following core responsibilities:

Searching for new program areas. We believe there are promising giving opportunities that don't currently fall within the purview of our existing program areas. Finding them involves blending theoretical models with concrete investigation into the tractability of new interventions to reduce catastrophic risk. Much of this research is informed by conversations with relevant exp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Raising children on the eve of AI, published by juliawise on February 15, 2024 on LessWrong.

Cross-posted with light edits from Otherwise.

I think of us in some kind of twilight world as transformative AI looks more likely: things are about to change, and I don't know if it's about to get a lot darker or a lot brighter.

Increasingly this makes me wonder how I should be raising my kids differently.

What might the world look like

Most of my imaginings about my children's lives have them in pretty normal futures, where they go to college and have jobs and do normal human stuff, but with better phones.

It's hard for me to imagine the other versions:

A lot of us are killed or incapacitated by AI

More war, pandemics, and general chaos

Post-scarcity utopia, possibly with people living as uploads

Some other weird outcome I haven't imagined

Even in the world where change is slower, more like the speed of the industrial revolution, I feel a bit like we're preparing children to be good blacksmiths or shoemakers in 1750 when the factory is coming. The families around us are still very much focused on the track of do well in school > get into a good college > have a career > have a nice life. It seems really likely that chain will change a lot sometime in my children's lifetimes.

When?

Of course it would have been premature in 1750 to not teach your child blacksmithing or shoemaking, because the factory and the steam engine took a while to replace older forms of work. And history is full of millenialist groups who wrongly believed the world was about to end or radically change.

I don't want to be a crackpot who fails to prepare my children for the fairly normal future ahead of them because I wrongly believe something weird is about to happen. I may be entirely wrong, or I may be wrong about the timing.

Is it even ok to have kids?

Is it fair to the kids?

This question has been asked many times by people contemplating awful things in the world. My friend's parents asked their priest if it was ok to have a child in the 1980s given the risk of nuclear war. Fortunately for my friend, the priest said yes.

I find this very unintuitive, but I think the logic goes: it wouldn't be fair to create lives that will be cut short and never reach their potential. To me it feels pretty clear that if someone will have a reasonably happy life, it's better for them to live and have their life cut short than to never be born. When we asked them about this, our older kids said they're glad to be alive even if humans don't last much longer.

I'm not sure about babies, but to me it seems that by age 1 or so, most kids are having a pretty good time overall. There's not good data on children's happiness, maybe because it's hard to know how meaningful their answers are. But there sure seems to be a U-shaped curve that children are on one end of. This indicates to me that even if my children only get another 5 or 10 or 20 years, that's still very worthwhile for them.

This is all assuming that the worst case is death rather than some kind of dystopia or torture scenario. Maybe unsurprisingly, I haven't properly thought through the population ethics there. I find that very difficult to think about, and if you're on the fence you should think more about it.

What about the effects on your work?

If you're considering whether to have children, and you think your work can make a difference to what kind of outcomes we see from AI, that's a different question. Some approaches that seem valid to me:

"I'm allowed to make significant personal decisions how I want, even if it decreases my focus on work"

"I care more about this work going as well as it can than I do about fulfillment in my personal life"

There are some theories about how parenting will make you more productive or motivated, which I don't really buy (especi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CLR Summer Research Fellowship 2024, published by Center on Long-Term Risk on February 15, 2024 on The Effective Altruism Forum.

We, the

Center on Long-Term Risk, are looking for Summer Research Fellows to explore strategies for reducing suffering in the long-term future (

s-risks). For eight weeks, you will join our team at our office while working on your own research project. During this time, you will be in regular contact with our researchers and other fellows, and receive guidance from an experienced mentor.

You will work autonomously on challenging research questions relevant to reducing suffering. You will be integrated and collaborate with our team of intellectually curious, hard-working, and caring people, all of whom share a profound drive to make the biggest difference they can.

We worry that some people won't apply because they wrongly believe they are not a good fit for the program. While such a belief is sometimes true, it is often the result of underconfidence rather than an accurate assessment. We would therefore love to see your application even if you are not sure if you are qualified or otherwise competent enough for the positions listed.

We explicitly have no minimum requirements in terms of formal qualifications and many of the past summer research fellows have had no or little prior research experience. Being rejected this year will not reduce your chances of being accepted in future hiring rounds. If you have any doubts, please don't hesitate to reach out (see "Application process" > "Inquiries" below).

Purpose of the fellowship

The purpose of the fellowship varies from fellow to fellow. In the past, have we often had the following types of people take part in the fellowship:

People very early in their careers, e.g. in their undergraduate degree or even high school, who have a strong interest in s-risk and would like to learn more about research and test their fit.

People seriously considering changing their career to s-risk research who want to test their fit for such work.

People interested in s-risk who plan to pursue a research or research-adjacent career and who would like to gain a strong understanding of s-risk macrostrategy beforehand.

People with a fair amount of research experience, e.g. from a (partly or fully completed) PhD, whose research interests significantly overlap with CLR's and who want to work on their research project in collaboration with CLR researchers for a few months. This includes people who do not strongly prioritize s-risk themselves.

There might be many other good reasons for completing the fellowship. We encourage you to apply if you think you would benefit from the program, even if your reason is not listed above. In all cases, we will work with you to make the fellowship as valuable as possible given your strengths and needs. In many cases, this will mean focusing on learning and testing your fit for s-risk research, more than seeking to produce immediately valuable research output.

Activities

Carrying out a research project related to one of our priority areas below, or otherwise targeted at reducing s-risks. You will determine this project in collaboration with your mentor, who will meet with you every week and provide feedback on your work.

Attending team and Fellowship meetings, including giving occasional presentations on the state of your research.

What we look for in candidates

We don't require specific qualifications or experience for this program, but the following abilities and qualities are what we're looking for in candidates. We encourage you to apply if you think you may be a good fit, even if you are unsure whether you meet some of the criteria.

Curiosity and a drive to work on challenging and important problems;

Ability to answer complex research questions related to the long-term future;

Willi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coworking space in Mexico City, published by AmAristizabal on February 15, 2024 on The Effective Altruism Forum.

TL;DR: We want to know if people/orgs want to use a coworking space in Mexico City. Fill out

this form if this is of interest to you.

We are currently running the

AI Futures Fellowship in Mexico City. We have had a great experience with the current coworking space we are using,

Haab Project.

In light of the positive experience and our interest in being able to retain the office space for future iterations of the fellowship, we are considering opening up the office for teams or individuals keen on coworking from Mexico City, especially during times when we are unlikely to use the office space. If this interests you, please fill out

this form. We are currently in an exploratory phase, so completing the form is not a hard commitment. We encourage you to fill out this form even if you don't have high confidence that you will be able to join.

Given that this is still very much in the rough, we are unsure what this would look like funding-wise and what services we could offer. That said, here are some of the things we could likely provide:

We will have some office equipment like monitors and standing desks since we use these for the fellowship.

Haab has a very good restaurant/cafe with 15% discount office space holders (e.g. an americano will go for 45MXN/~2.5USD and you can get a meal for 153MXN/~9USD)

Depending on demand, we could secure the main office space on the 5th floor (which is the one we are using for our fellows). It fits ~17 people for coworking and ~35 for talks and events. It has a fantastic view of Amsterdam street (photos below).

This office is very popular among other teams in Mexico so this is why we want to know if we can secure it for us and collaborators.

Below are some images of the coworking space and the specific office that will be available.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Estimates on expected effects of movement/pressure group/field building?, published by jackva on February 15, 2024 on The Effective Altruism Forum.

It seems relatively uncontroversial within EA-grantmaking that field building (and the building of societal pressure groups?) is an effective strategy to induce long-run changes.

E.g. a resource frequently referenced in discussions is Teles's "The Rise of the Conversative Legal Movement" as an existence proof for a very large long-run impact of philanthropic money.

I am curious whether anyone has done systematic work on using this and other evidence to (1) estimate expected effects (2) base rates of success or (3) anything else of that sort that would inform how we can think about the average (a) tractability and (b) impact of such efforts?

Luke Muehlhauser's work on early-movement growth and field-building comes closest, reviewing historical case studies and generally giving the impression that intentional movement / field acceleration is (a) possible, (b) not rocket science (things one would expect to work do work), and (c) can be quite meaningful (playing a major role in shaping and/or accelerating fields).

But it doesn't offer much in terms of relative tractability or effectiveness vis-a-vis other interventions, such as funding existing think tanks to do Beltway-style policy advocacy or other surgical interventions that we do a lot of.

Broadly, I am trying to understand how to compare funding such work to more surgical interventions, so I am interested both in absolute estimates but also relative comparisons.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Social science research we'd like to see on global health and wellbeing [Open Philanthropy], published by Aaron Gertler on February 15, 2024 on The Effective Altruism Forum.

Open Philanthropy strives to help others as much as we can with the resources available to us. To find the best opportunities to help others, we rely heavily on scientific and social scientific research.

In some cases, we would find it helpful to have more research in order to evaluate a particular grant or cause area. Below, we've listed a set of social scientific questions for which we are actively seeking more evidence.[1] We believe the answers to these questions have the potential to impact our grantmaking. (See also our list of

research topics for animal welfare.)

If you know of any research that touches on these questions, we would welcome hearing from you. At this point, we are not actively making grants to further investigate these questions. It is possible we may do so in the future, though, so if you plan to research any of these, please

email us.

Land Use Reform

Open Philanthropy has been making grants in

land use reform since 2015. We believe that more permissive permitting and policy will encourage economic growth and allow people to access higher-paying jobs. However, we have a lot of uncertainty about which laws or policies would be most impactful (or

neglected/tractable relative to their impact) on housing production.

What are the legal changes that appear to spur the most housing? E.g. can we estimate the effects of removing parking mandates on housing production? How do those compare to the effects of higher

FAR or more allowable units?

Why we care: We think that permitting speed might be an important category to target, but have high uncertainty about this.

What we know: There are a number of different studies of the effects of changes in zoning/land use laws (e.g. see a summary

here in Appendix A), but we're not aware of studies that attempt to disentangle any specific changes or rank their importance. We suspect that talking to advocates (e.g. CA YIMBY) would be useful as a starting point.

Ideas for studying this: It seems unlikely that there have been "clean" changes that only affected a single part of the construction process, but from talking to advocates, it seems plausible that it would be possible to identify changes to zoning codes that primarily affect one parameter more than others. It also seems plausible that this is a topic where a systematic review, combining evidence from many other studies, would be unusually valuable.

What is the elasticity of construction with regards to factors like "the likelihood of acquiring permission to build" or "the length of an average permitting delay"?

Why we care: We are highly uncertain about how to best encourage more construction, and thus about where to target our grants.

What we know: there have been many recent changes to permitting requirements, such as the California ADU law that

requires cities to respond to permit requests within 60 days and a

new law in Florida that requires cities to respond to permit requests quickly or return permitting fees. This

blog post by Dan Bertolet at Sightline predates those changes, but is the best summary we've seen on the impacts of permitting requirements.

Ideas for studying this: one might compare projects that fall right below or above thresholds for permitting review (e.g. SEPA thresholds in Washington state), and try to understand how much extra delay projects faced as a result of qualifying for review. It could also be valuable to analyze the effects of the Florida law (e.g. a difference-in-difference design looking at housing construction in places that had long delays vs. short delays prior to the law passing).

Does the value of new housing (in terms of individual earnings gains ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Searching for Searching for Search, published by Rubi Hudson on February 14, 2024 on The AI Alignment Forum.

Thanks to Leo Gao, Nicholas Dupuis, Paul Colognese, Janus, and Andrei Alexandru for their thoughts. This post was mostly written in 2022, and pulled out of my drafts after recent conversations on the topic.

Searching for Search is the research direction that looks into how neural networks implement search algorithms to determine an action. The hope is that if we can find the search process, we can then determine which goal motivates it, which may otherwise be a much more difficult task. We may even be able to

retarget the search, specifying a new goal while keeping the capabilities of the model intact. Conclusively showing that search is not happening would also be useful, as it would imply

a lack of generalization ability that could make deception less likely. However, search can take many forms, and we do not have a widely accepted definition of search that neatly captures all the variants. Searching for search may fail either by using too narrow a definition and missing the specific type of search that a model actually implements, or by using too wide of definition to check for in practice.

This post analyzes the components of several algorithms for search, looking at commonalities and differences, and aims to provide a relatively tight definition of search that can guide the process of searching for search. Unfortunately, the definition arrived at here is so broad that it does not lead to useful clarification or application. Furthermore, many possible algorithms that fall under search do not suggest an overarching goal for the model.

Due to these issues, I instead suggest searching for substeps of search. Gathering evidence on how these substeps are compiled throughout the training process could shed light both on how to search for search and how agency and goals develop.

Search vs. Heuristics

What alternative is there to implementing a search process? Abram Demski splits the class of optimizers into

control and selection (which includes search). Selection processes can instantiate and evaluate arbitrary elements of the search space (though doing so may be costly) and then choose the best option, while control processes only instantiate one element and may not even evaluate it. Less formally, a control process can be thought of as a set of heuristics.

A standard example of one would be a thermostat, which raises or lowers the temperature according to a set rule depending on the temperature it detects.

The distinction between selection and control is not always clear though, and Abram describes it as more of a continuum. A search process may only be able to generate a subset of possible options for evaluation and may not be able to evaluate them perfectly, instead relying on heuristics to approximate it . A controller's heuristics to choose an option may instead be thought of as a limited search space, such as a thermostat searching over the options "up" and "down".

If search is important to identify, then so is a control process that imitates search.

Blurring the lines further is the fact that many search processes cannot be cleanly implemented in a neural net. The limited number of steps in a forward pass means that search processes with an unbounded duration must be approximated instead. Rather than implementing a search process directly, a neural network must approximate the search process using a series of heuristics.

This means that selection and control are both variations on a pile of heuristics, suggesting that instead of trying to identify holistic search processes in neural networks, it could be more feasible to split it into the heuristics that make up the substeps of search and look for those instead. Doing so also opens the possibility of stud...