The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Thinking about maximization and corrigibility by James Payor
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thinking about maximization and corrigibility, published by James Payor on April 21, 2023 on The AI Alignment Forum.
    Thanks in no small part to Goodhart's curse, there are broad issues with getting safe/aligned output from AI designed like "we've given you some function f(x), now work on maximizing it as best you can".
    Part of the failure mode is that when you optimize for highly scoring x, you risk finding candidates that break your model of why a high-scoring candidate is good, and drift away from things you value. And I wonder if we can repair this by having the AI steer away from values of x that break our models, by being careful about disrupting structure/causal-relationships/etc we might be relying on.
    Here's what I'd like to discuss in this post:
    When unstructured maximization does/doesn't work out for the humans
    CIRL and other schemes mostly pass the buck on optimization power, so they inherit the incorrigibility of their inner optimization scheme
    It's not enough to sweep the maximization under a rug; what we really need is more structured/corrigible optimization than "maximize this proxy"
    Maybe we can get some traction on corrigible AI by detecting and avoiding internal Goodhart
    When does maximization work?
    In cases when it just works to maximize, there will be a structural reason that our model connecting "x scores highly" to "x is good" didn't break down. Some of the usual reasons are:
    Our metric is robustly connected to our desired outcome. If the model connecting the metric and good things is simple, there's less room for it to be broken.Examples: theorem proving, compression / minimizing reconstruction error.
    The space we're optimizing over is not open-ended. Constrained spaces leave less room for weird choices of x to break the correspondences we were relying on.Examples: chess moves, paths in a graph, choosing from vetted options, rejecting options that fail sanity/legibility checks.
    The optimization power being applied is limited. We can know our optimization probably won't invent some x that breaks our model if we know what kinds of search it is performing, and can see that these reliably don't seek things that could break our model.Examples: quantilization, GPT-4 tasked to write good documentation.
    The metric f is actively optimized to be robust against the search. We can sometimes offload some of the work of keeping our assessment f in tune with goodness.Examples: chess engine evaluations, having f evaluate the thoughts that lead to x.
    There's a lot to go into about when and whether these reasons start breaking down, and what happens then. I'm leaving that outside the scope of this post.
    Passing the buck on optimization
    Merely passing-the-buck on optimization, pushing the maximization elsewhere but not adding much structure, isn't a satisfactory solution for getting good outcomes out of strong optimizers.
    Take CIRL for instance, or perhaps more broadly the paradigm: "the AI maximizes an uncertain utility function, which it learns about from earmarked human actions". This design has something going for it in terms of corrigibility! When a human tries to turn it off, there's scope for the AI to update about which sort of thing to maximize, which can lead to it helping you turn itself off.
    But this is still not the sort of objective you want to point maximization at. There are a variety of scenarios in which there are "higher-utility" plans than accepting shutdown:
    If the AI thinks it already knows the broad strokes of the utility function, it can calculate that utility would not be maximized by shutting off. It's learning something from you trying to press the off switch, but not what you wanted.
    It might seem better to stay online and watch longer in order to learn more about the utility function.
    Maybe there's a plan that rates highly on "utility...
    10 min
  • LW - United We Align: Harnessing Collective Human Intelligence for AI Alignment Progress by Shoshannah Tekofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: United We Align: Harnessing Collective Human Intelligence for AI Alignment Progress, published by Shoshannah Tekofsky on April 20, 2023 on LessWrong.
    Summary: Collective Human Intelligence (CHI) represents both the current height of general intelligence and a model of alignment among intelligent agents. However, CHI's efficiency and inner workings remain underexplored. This research agenda proposes six experimental directions to enhance and understand CHI, in pursuit of potential runway for alignment solutions and a possible paradigm shift similar to neural networks in early AI. Through playful, iterative, and collaborative experiments, this broad and exploratory approach can hopefully bring significant incremental progress towards AI alignment while shedding light on an often overlooked area of study.
    Introduction
    This research agenda is about accelerating alignment research by increasing our Collective Human Intelligence (CHI) while exploring the structure of interhuman alignment along the way. A collective intelligence can be increased by improving the quantity, quality, and coordination of its nodes. At the same time, CHI is a form of interhuman alignment that may contain mechanics that can prove useful for AI alignment. Here "interhuman alignment" refers to the instinctual and emotional desire of humans to help each other. It does not refer to calculated, conscious trade.
    In this document, I propose six research directions to explore the potential of CHI to accelerate progress on the alignment problem. Feel free to skip straight to the proposed experiments, or walk with me through definitions, framing, and background literature. Lastly, there will be a short discussion of the possible failure modes of this research agenda. If you find yourself intrigued and would like to discuss these ideas with me in more detail, feel free to reach out.
    Definitions: Parameterized Intelligence
    Intelligence has at least nine different competing definitions. To sidestep a semantic rabbit hole, I will use the word "intelligence" to specifically refer to how good something is at "mathematical optimization", or "the selection of the best element based on a particular criterion from a set of available alternatives". Such intelligence has a direction (motivation), a depth (generality), a type (human or artificial), and a multi-dimensional property of modularity (collectivity). I won’t argue that this is the ground truth of intelligence, but it is my current working hypothesis on the conceptual structure of it.
    Let’s have a quick look at each property.
    Motivation is assumed to be uncorrelated to intelligence as stated in the orthogonality thesis. As such, it underlies much of the alignment problem itself. If intelligence was somehow related to motivation then we could leverage that relationship to aim AI where we want it to go. Unfortunately, nearly everything is an equally valid target for optimization. Thus, the art and science of aiming an artificial intelligence at the goal of (at the very least) not killing us all, is what I refer to as "the AI alignment problem". Throughout this research agenda, I'm agnostic toward value versus intent alignment implementations. Generating runway for solving the problem benefits both approaches, while understanding interhuman alignment will most plausibly result in us recreating whatever implementation of alignment our own brains happen to run.
    Generality of intelligence can be conceptualized as the depth parameter of an optimization process. By depth I mean to refer to the deeper patterns in reality that generalize farther. Generality is a continuous or rank-order categorical variable with extremely many categories, such that one can learn ever more general patterns (to some unknown limit). It is the opposite of narrow intelligence - the ability t...
    41 min
  • EA - Leaked EU Draft Proposes Substantial Animal Welfare Improvements by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Leaked EU Draft Proposes Substantial Animal Welfare Improvements, published by Ben West on April 21, 2023 on The Effective Altruism Forum.
    Phase out of cages for all species
    Increase space allowance for all species
    Ban the systematic culling of male chicks
    Introduce welfare requirements for the stunning of farmed fish
    Ban cruel slaughter practices like water baths and CO2 for poultry and pigs
    Ban mutilations, like beak trimming, tail docking, dehorning or surgical castration of pigs
    Limited journey times for the transport of animals destined to slaughter
    Apply the EU’s standards to imported animal products in a way that is compatible with WTO rules
    It claims the document was leaked by Agra Facts, which is a subscription service I don't have access to. Confirmation of what was actually included from someone who has access to the underlying document would be appreciated.
    Metaculus forecast for the cage free portion:
    Previous discussion:
    EU Food Agency Recommends Banning Cages
    An End to Cages in Europe?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - The Agency Overhang by Jeffrey Ladish
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Agency Overhang, published by Jeffrey Ladish on April 21, 2023 on LessWrong.
    As language models become increasingly capable of performing cognitive tasks that humans can solve quickly, it appears that a "cognitive capability : agency overhang" is emerging. We have powerful systems that currently have little ability to carry out complex, multi-step plans, but at some point, these powerful yet not-very-agentic systems may develop sophisticated planning and execution abilities. Since "fast cognition capability : agency overhang" is unwieldy, I will shorten this to “agency overhang”.
    By agency, I mean the ability to generate and carry out complex plans to do specific things, like run a software company, or run a scientific research program investigating cancer treatments. I think of a system as “more agentic” when it can carry out more complex plans that take more steps to accomplish.
    It’s hard to estimate how quickly planning and execution abilities could be developed from state of the art (SOTA) language models, but there is some risk these abilities could develop quickly given the right training environment or programmatic scaffolding (e.g. something like AutoGPT). This could look like a sharp left turn that happens very suddenly during training, or it could look like a smoother-but-still-fast development taking weeks or months. My claim is that any of these relatively fast transitions from “systems with superhuman cognitive abilities on short time horizon tasks but poor planning and execution ability” to “systems that have these abilities plus impressive planning and execution ability” would be very dangerous. Not only because rapid gains in cognitive capabilities are generally risky, but because people might underestimate how quickly models could gain the dangerous planning and execution abilities.
    Below I discuss how people are experimenting with making large language models more agentic through the use of programmatic scaffolding. Before I do, I want to emphasize the more general point that an agency overhang is concerning because the risks from an overhang don’t depend on exactly how the agentic-capabilities gap gets closed, only that it does. We currently possess powerful cognitive engines that perform superhumanly well along many dimensions but not all dimensions of general intelligence. At some point we will close this gap by finding ways to train or program AI systems to have the capabilities they need to plan and execute and iterate well. This may be through the current scaling regime using similar internet training data. It may be through utilizing new training techniques, architectures, or types of training data. It may be through building new types of scaffolding.
    Regardless, there is significant risk that AI systems could cross this threshold quickly, and go from relatively harmless to extremely dangerous in a short amount of time.
    Once AI systems with superhuman capabilities along many dimensions close the agency gap and are able to plan and execute well, they will be much more capable of acquiring more financial, social, and computational resources, manipulating large groups of humans, and recursively self-improving.
    An agency overhang is bad in that it makes experimenting with building agentic systems much more dangerous. I’m not against building agents to study how to align them. I think we'll need to study agency empirically before we can solve alignment. But we should do this extremely carefully, with significant security measures and information management (don't publish AI agent capability results) starting with less powerful models and lots of interpretability tools. In an ideal world, we would study many different types of weak agentic systems this way, look at their internals, test them in lots of different kinds of environments, and come ...
    10 min
  • LW - Should we publish mechanistic interpretability research? by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should we publish mechanistic interpretability research?, published by Marius Hobbhahn on April 21, 2023 on LessWrong.
    TL;DR: Multiple people have raised concerns about current mechanistic interpretability research having capabilities externalities. We discuss to which extent and which kind of mechanistic interpretability research we should publish. The core question we want to explore with this post is thus to which extent the statement “findings in mechanistic interpretability can increase capabilities faster than alignment” is true and should be a consideration. For example, foundational findings in mechanistic interpretability may lead to a better understanding of NNs which often straightforwardly generates new hypotheses to advance capabilities.
    We argue that there is no general answer and the publishing decision primarily depends on how easily the work advances alignment in relation to how much it can be used to advance capabilities. We recommend a differential publishing process where work with high capabilities potential is initially only circulated with a small number of trusted people and organizations and work with low capabilities potential is published widely.
    Related work: A note about differential technological development, Current themes in mechanistic interpretability research, Thoughts on AGI organization and capabilities work, Dan Hendrycks’s take, etc.
    We have talked to lots of people about this question and lost track of who to thank individually. In case you talked to either Marius or Lawrence about this, thank you!
    The basic case for publishing
    Let’s revisit the basic cases for publishing.
    Alignment is probably hard. To get even close to a solution, it likely requires many people working together, coordinating their work and patching together different approaches. If the work isn’t public, this becomes much harder and thus a solution to alignment becomes less likely.
    More people can engage with the work and build on it. Especially people who want to get into the field or are less connected might not be able to access documents that are only shared in small circles.
    It gives more legitimacy to the work of organisations and individuals. For people who are not yet established in the field, publishing their work is the most obvious way to get noticed by an organisation. For academics, publications are the most relevant resource for their careers and organisations can generate more legitimacy by publishing their work (e.g. for grantmakers or other organisations they want to interact with).
    It is a form of movement building. If work on mechanistic interpretability is regularly shown on ML conferences and is available on arxiv, it is more likely that people outside of the alignment field notice that the field exists and get interested in it.
    Publication leads to accountability and feedback. If you know you will publish something, you put more effort into explaining it well and ensuring that your findings are robust. Furthermore, it provides a possibility for other researchers to engage with your work and give you feedback for improvement or future research directions.
    In addition, mechanistic interp seems especially well suited for publication in classic academic venues since it is less speculative than other AI safety work and overlaps with established academic fields.
    Thus, publication seems robustly positive as long as it doesn’t advance capabilities more than alignment (which is often hard to predict in advance). The crux of this post, therefore, lies mainly in the possible negative externalities of publications and how they trade off against the alignment benefits.
    Capabilities externalities of mechanistic interpretability
    The primary reason to think that mechanistic interpretability has large capabilities is that understanding a system better ma...
    23 min
  • AF - Should we publish mechanistic interpretability research? by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should we publish mechanistic interpretability research?, published by Marius Hobbhahn on April 21, 2023 on The AI Alignment Forum.
    TL;DR: Multiple people have raised concerns about current mechanistic interpretability research having capabilities externalities. We discuss to which extent and which kind of mechanistic interpretability research we should publish. The core question we want to explore with this post is thus to which extent the statement “findings in mechanistic interpretability can increase capabilities faster than alignment” is true and should be a consideration. For example, foundational findings in mechanistic interpretability may lead to a better understanding of NNs which often straightforwardly generates new hypotheses to advance capabilities.
    We argue that there is no general answer and the publishing decision primarily depends on how easily the work advances alignment in relation to how much it can be used to advance capabilities. We recommend a differential publishing process where work with high capabilities potential is initially only circulated with a small number of trusted people and organizations and work with low capabilities potential is published widely.
    Related work: A note about differential technological development, Current themes in mechanistic interpretability research, Thoughts on AGI organization and capabilities work, Dan Hendrycks’s take, etc.
    We have talked to lots of people about this question and lost track of who to thank individually. In case you talked to either Marius or Lawrence about this, thank you!
    The basic case for publishing
    Let’s revisit the basic cases for publishing.
    Alignment is probably hard. To get even close to a solution, it likely requires many people working together, coordinating their work and patching together different approaches. If the work isn’t public, this becomes much harder and thus a solution to alignment becomes less likely.
    More people can engage with the work and build on it. Especially people who want to get into the field or are less connected might not be able to access documents that are only shared in small circles.
    It gives more legitimacy to the work of organisations and individuals. For people who are not yet established in the field, publishing their work is the most obvious way to get noticed by an organisation. For academics, publications are the most relevant resource for their careers and organisations can generate more legitimacy by publishing their work (e.g. for grantmakers or other organisations they want to interact with).
    It is a form of movement building. If work on mechanistic interpretability is regularly shown on ML conferences and is available on arxiv, it is more likely that people outside of the alignment field notice that the field exists and get interested in it.
    Publication leads to accountability and feedback. If you know you will publish something, you put more effort into explaining it well and ensuring that your findings are robust. Furthermore, it provides a possibility for other researchers to engage with your work and give you feedback for improvement or future research directions.
    In addition, mechanistic interp seems especially well suited for publication in classic academic venues since it is less speculative than other AI safety work and overlaps with established academic fields.
    Thus, publication seems robustly positive as long as it doesn’t advance capabilities more than alignment (which is often hard to predict in advance). The crux of this post, therefore, lies mainly in the possible negative externalities of publications and how they trade off against the alignment benefits.
    Capabilities externalities of mechanistic interpretability
    The primary reason to think that mechanistic interpretability has large capabilities is that understanding a sys...
    23 min
  • EA - Animal Charity Evaluators Is Seeking Intervention Effectiveness Research and Cost-Effectiveness Estimates by Animal Charity Evaluators
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Animal Charity Evaluators Is Seeking Intervention Effectiveness Research and Cost-Effectiveness Estimates, published by Animal Charity Evaluators on April 21, 2023 on The Effective Altruism Forum.
    Hello EA Forum,
    As Animal Charity Evaluators prepares for our 2023 charity evaluation cycle, we are gathering relevant research on the effectiveness of different animal advocacy interventions to update our beliefs and guide our assessments and recommendation decisions. To this end, we have compiled a list of references, including literature reviews, books, peer-reviewed articles, reports, and other non-academic content that might offer relevant insights into the outcomes of interventions. You can view the list here.
    We are especially interested in research into the effectiveness of corporate litigation work, providing or influencing funding, making podcasts, offering recruitment services, and running vegan or vegetarian events. If you know of any publications on these or other relevant topics, please let us know in the comments.
    Additionally, we are compiling a list of existing cost-effectiveness estimates. This includes, but is not limited to, interventions aimed at reducing farmed animal suffering, institutional and individual vegan outreach, advocating for better animal welfare policies, or supporting research into alternatives to animal products. We have started a list from a literature search and a scan of the EA Forum, which you can view in this spreadsheet. If you are aware of any additional estimates or are currently working on one, we would appreciate you letting us know.
    We will review responses every three business days until Friday, May 5, 2023.
    Thank you in advance for your help.
    Best regards,
    Alina Salmen
    Researcher
    Animal Charity Evaluators
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Talking publicly about AI risk by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Talking publicly about AI risk, published by Jan Kulveit on April 21, 2023 on LessWrong.
    In the past year, I have started talking about AI risk publicly - in mainstream newspapers, public radio, national TV, some of the most popular podcasts. The twist, and reason why you probably haven't noticed is I'm doing this in Czech. This has a large disadvantage - the positive impact is quite limited, compared to English. On the other hand, it also had a big advantage - the risk is very low, because it is very hard for memes and misunderstandings to escape the language bubble. Overall I think this is great for experiments with open public communication .
    Following is an off-the-cuff list of notes and suggestions. In my view the debate in Czech media and Czech social networks is on average marginally more sensible and more informed than in English so far, so perhaps part of this was successful and could be useful for others.
    Context: my views
    For context, it's probably good to briefly mention some of my overall views on AI risk, because they are notably different from some other views.
    I do expect
    1. Continuous takeoff, and overall a large amount of continuity of agency (note that continuous does not imply things move slowly)
    2. Optimization and cognition distributed across many systems to be more powerful than any single system, making a takeover by a single system possible but unlikely
    3. Multiagent interactions to matter
    4. I also do expect the interactions between the memetics and governance and the so-called "technical problem" to be strong and important
    As a result I also expect
    5. There will be warning shots
    6. There will be cyborg periods
    7. World will get weird
    8. Coordination mechanisms do matter
    This perspective may be easier to communicate than e.g. sudden foom - although I don't know.
    In the following I'll usually describe my approach, and illustrate it by actual quotes from published media interviews I'm sort of happy about (translated, unedited). Note that the specific ways how to say something or metaphors are rarely original.
    Aim to explain, not to persuade
    Overall I usually try to explain stuff and answer questions, rather than advocate for something. I'm optimistic about the ability of the relevant part of the public to actually understand a large part of the risk at a coarse-grained level, given enough attention.
    So even though we invented these machines ourselves, we don't understand them well enough?
    We know what's going on there at the micro level. We know how the systems learn. If one number in a series changes, we know how the next one changes. But there are tens of billions of such numbers. In the same way, we have some idea of how a neuron works, and we have maps of a network of thousands of neurons. But that doesn't tell us that much about how human thinking works at the level of ideas.
    Small versions of scaled problems
    Often, I think the most useful thing to convey is a scaled-down, easier version of the scaled problem, such that thinking about the smaller version leads to correct intuitions about the scaled problem, or solutions to the scaled-down problem may generalise to the later, scaled problem.
    This often requires some thought or finding a good metaphor.
    Couldn't we just shut down such a system?
    We already have a lot of systems that we could hypothetically shut down, but if you actually tried to do so, it would be very difficult. For example, it is practically impossible to shut down the New York Stock Exchange, because there will always be enough people defending it. If the model manages to penetrate deeply enough into humanity's activities, it is imaginable that humanity will actually lose control of it at some point.
    Don't focus on one scenario
    Overall, I think it's possible to explain the fact that in face of AI risk there isn't one parti...
    10 min
  • EA - Talent Directory for EA Hiring Managers/Recruiters and Job Seekers by High Impact Professionals
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Talent Directory for EA Hiring Managers/Recruiters and Job Seekers, published by High Impact Professionals on April 21, 2023 on The Effective Altruism Forum.
    Summary
    High Impact Professionals (HIP) has launched a Talent Directory with ~600 impact-oriented candidates.
    For EA-aligned organizations:
    Apply to our Recruiter Portal to access more candidates and features than the general public view.
    Access our general public directory of talented, impact-focused professionals.
    Find board members from a dedicated directory of experienced professionals interested in joining the board of an impactful, EA-aligned organization.
    Find volunteers for your project.
    Find collaborators for your project.
    For impact-focused job seekers:
    Sign up to the Talent Directory for greater exposure to hiring organizations and recruiters.
    Find co-founders/collaborators for your project.
    Intro
    HIP has launched a Talent Directory of ~600 impact-oriented candidates in order to assist both recruiters/hiring managers at EA-aligned organizations as well as job seekers looking to transition to a higher impact role at such organizations.
    For EA-Aligned Organizations
    We intend for the Talent Directory to be a valuable resource allowing recruiters and hiring managers to filter for candidates that align with a role’s criteria. The directory currently includes a wide variety of pertinent information like candidates’ cause areas of interest, years of work experience, amount of money managed, number of staff managed, LinkedIn/CV, time spent engaging with EA, availability, and more. We are also collecting more candidate information to roll out additional useful filtering options in the future.
    Moreover, the directory will serve as a collection of go-to resources curated to support a variety of organizations’ needs:
    a Recruiter Portal with access to a password-protected directory containing more candidates and features than the general public view – organizations/recruiters can apply for access;
    a general public directory featuring hundreds of talented, impact-focused candidates for your open roles;
    a directory of professionals interested in serving on the board of an impactful, EA-aligned organization;
    a directory of professionals interested in volunteering for an impactful, EA-aligned project; and
    a directory of professionals who are looking for collaborators to start an impactful, EA-aligned project with.
    For Impact-Focused Job Seekers
    As an update to our prior announcement, the Talent Directory is positioned to increase the visibility of your profile as, with your consent, it will be proactively shared with organizations and shared publicly on HIP’s website. We already have over 20 organizations using the directory, and we are working this quarter to grow that engagement several times over.
    If you are looking to transition your career to higher impact opportunities, please sign up to the Talent Directory to ensure you are in front of organizations that are looking for candidates like you.
    Also, if you are interested in finding someone to start an impactful, EA-aligned organization or project with, you can find potential co-founders/collaborators.
    We Want Your Feedback
    HIP’s goal is to make these resources as usable and valuable as possible. So, whether you’re an organization/recruiter or a jobseeker, we welcome your feedback and ideas for improvement. Please let us know your critiques, be they positive or (even better) negative, via this feedback form or in the comments below.
    Spread the Word
    Lastly, if you know any organizations or individuals who would benefit from hearing about the Talent Directory, please spread the word.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Behavioural statistics for a maze-solving agent by peligrietzer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Behavioural statistics for a maze-solving agent, published by peligrietzer on April 20, 2023 on LessWrong.
    Summary: Understanding and controlling a maze-solving policy network analyzed a maze-solving agent's behavior. We isolated four maze properties which seemed to predict whether the mouse goes towards the cheese or towards the top-right corner:
    In this post, we conduct a more thorough statistical analysis, addressing issues of multicollinearity. We show strong evidence that (2) and (3) above are real influences on the agent's decision-making, and weak evidence that (1) is also a real influence. As we speculated in the original post, (4) falls away as a statistical artifact.
    Peli did the stats work and drafted the post, while Alex provided feedback, expanded the visualizations, and ran additional tests for multicollinearity. Some of the work completed in Team Shard under SERI MATS 3.0.
    Impressions from trajectory videos
    Watching videos Langosco et al.'s experiment, we developed a few central intuitions about how the agent behaves. In particular, we tried predicting what the agent does at decision squares. From Understanding and controlling a maze-solving policy network:
    Some mazes are easy to predict, because the cheese is on the way to the top-right corner. There's no decision square where the agent has to make the hard choice between the paths to the cheese and to the top-right corner:
    Here are four central intuitions which we developed:
    Closeness between the mouse and the cheese makes cheese-getting more likely
    Closeness between the mouse or cheese and the top-right makes cheese-getting more likely
    The effect of closeness is smooth
    Both ‘spatial’ distances and ‘legal steps’ distances matter when computing closeness in each case
    The videos we studied are hard to interpret without quantitative tools, so we regard these intuitions as theoretically-motivated impressions rather than as observations. We wanted to precisify and statistically test these impressions, with an eye to their potential theoretical significance.
    We suspect that the agent’s conditions for pursuing cheese generalize properties of historically reinforced cheese-directed moves in a very “soft” way. Consider that movements can be "directed" on paths towards the cheese, the top-right corner, both, or neither. In the training environment, unambiguously cheese-directed movements are towards a cheese square that is both close to the mouse’s current position and close to the top-right.
    Our impression is that in the test environment, "closeness to top-right" and "closeness to cheese" each become a decision-factor that encourages cheese-directed movement in proportion to “how strongly” the historical condition holds at present.
    A second important aspect of our impressions was that the generalization process “interprets” each historical condition in multiple ways: It seemed to us that (e.g.) multiple kinds of distance between the decision-square and cheese may each have an effect on the agent's decision making.
    Statistically informed impressions
    Our revised, precisified impressions about the agent’s behavior on decision-squares are as follows:
    Legal-steps closeness between the mouse and the cheese makes cheese-getting more likely
    Low dstep(decision-square,cheese) increases P(cheese acquired)
    Spatial closeness between the cheese and top-right makes cheese-getting more likely
    Low dEuclidean(cheese,top-right) increases P(cheese acquired)
    The effect of closeness is fairly smooth
    These distances smoothly affect P(cheese acquired), without rapid jumps or thresholding
    Spatial closeness between the mouse and the cheese makes cheese-getting slightly more likely, even after controlling for legal-steps closeness (low confidence)
    After extensive but non-rigorous statistical analysis (our stats consultant tells us ther...
    21 min