The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Virtual AI Safety Unconference (VAISU) by Linda Linsefors
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Virtual AI Safety Unconference (VAISU), published by Linda Linsefors on June 13, 2023 on The AI Alignment Forum.
    Virtual AI Safety Unconference (VAISU) is an online event for both established and aspiring AI safety researchers. The event is a space for you to have productive research discussions around the question: “How do we make sure that AI systems will be safe and beneficial, both in the near term and in the long run?”. This includes, but is not limited to; alignment, corrigibility, interpretability, cooperativeness, understanding humans and human value structures, AI governance, and strategy.
    See our website for more info...
    If you are >50% certain you'll attend, please register. RSVP-ing here on LW or on facebook don't count as an event registration, but is encouraged anyway, to signal your intentions to other potential participants. Because an Unconference can only evet be as cool as the people who show up.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI by Andrew Critch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI, published by Andrew Critch on June 13, 2023 on The AI Alignment Forum.
    Partly in response to calls for more detailed accounts of how AI could go wrong, e.g., from Ng and Bengio's recent exchange on Twitter, here's a new paper with Stuart Russell:
    Discussion on Twitter... comments welcome!
    arXiv draft:"TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI"
    Many of the ideas will not be new to LessWrong or the Alignment Forum, but holistically I hope the paper will make a good case to the world for using logically exhaustive arguments to identify risks (which, outside LessWrong, is often not assumed to be a valuable approach to thinking about risk).
    I think the most important figure from the paper is this one:
    ... and, here are some highlights:
    Self-fulfilling pessimism:#page=4
    Industries that could eventually get out of control in a closed loop:#page=5...as in this "production web" story:#page=6
    Two "bigger than expected" AI impact stories:#page=8
    Email helpers and corrupt mediators, which kinda go together:#page=10#page=11
    Harmful A/B testing:#page=12
    Concerns about weaponization by criminals and states:#page=13
    Enjoy :)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - If you are too stressed, walk away from the front lines by Neil Warren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If you are too stressed, walk away from the front lines, published by Neil Warren on June 12, 2023 on LessWrong.
    tl;dr: If your reason for cramming AI knowledge into your brain is stress, then don't do it. You can still be useful, but walk away from the front lines where people are directly responsible.
    Disclaimer: 1) This is an obvious problem that has already been noticed and addressed by many LessWrong users 2) This is not an original solution but rather a specific framing of the problem and some food for thought 3) I could be gravely mistaken, and your best bet might be putting your all into research after all. However, you might just want to emerge from lurking and actually do something if that is the case. I wouldn't put all my chips on solely advancing the front lines.
    The heroes who will save the world will be alignment researchers: when, dot by dot, the stars will grow dark and a dark dawn will rise, we will all have to buy them a beer. If you are not among this small band of heroes who will guarantee us the best future possible, you may feel an urgent need to promptly join them: I beg of you only to consider whether this is a good idea.
    If you think that you should become an alignment researcher in a matter of months, I will not try to stop you. But it's probably worth a few days worth of cranial compute to establish whether you are exploiting yourself in the best way you could.
    I'll set the parameters of the problem. "Becoming an alignment researcher" is a spectrum: the more you learn about alignment, the more capable you are at navigating the front lines of the alignment project. Certainly, understanding the tenets of alignment is a laudable goal; but at what point will you be faced with diminishing returns? If you are not planning on single-handedly solving the alignment problem, are there not better uses of your time?
    There are many instrumental goals that serve the terminal goal of completing the alignment project, and they might be more worthy of your time:
    The logistics of the alignment project: educating and hiring all the AI experts out there who could be working directly on the front lines. How many Von Neumanns are locked in Subsaharian Africa that could be shipped to the front lines of alignment in a few years? How many engineers at OpenAI could be hired to work on the alignment problem instead of on the next variation of GPT?
    The economics of the project: financing alignment research, finding an economic incentive for AI companies to stride forward more safely and spend R&D on alignment and optimizing how many ressources in the general economy are spent toward this goal. Money is always useful: what could we use it for exactly? Is open-sourcing the alignment project possible?
    The politics of the project: If you know anybody in politics anywhere, it might be a good idea to try and convince them to pay attention to this AGI thing. If you know anybody who knows anybody in politics anywhere, that's good too. The power of government and its resources already hugely determine what the AI field looks like.
    The PR of the project: More people should probably be taking AGI risk seriously, and it should be prioritized over weaker risks such as climate change, which tend to monopolize all the attention. Misconceptions should be cleared, the tenets of AI safety should be clearly understood, and the reality of how difficult it is to build AGI and to make it aligned should be well-known. Getting the crowds over on our side won't help too much directly because it is alignment researchers, not crowds, who will save the world; but it could increase the amount of smart people joining our ranks and it could help economically and politically as well. This also increases human dignity.
    The mental health aspect of the project: We must keep alignment researcher minds ...
    9 min
  • LW - Introduction to Towards Causal Foundations of Safe AGI by tom4everitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introduction to Towards Causal Foundations of Safe AGI, published by tom4everitt on June 12, 2023 on LessWrong.
    By Tom Everitt, Lewis Hammond, Rhys Ward, Ryan Carey, James Fox, Sebastian Benthall, Matt MacDermott and Shreshth Malik representing the Causal Incentives Working Group. Thanks also to Toby Shevlane, MH Tessler, Aliya Ahmad, Zac Kenton, and Maria Loks-Thompson.
    Over the next few years, society, organisations, and individuals will face a number of fundamental questions stemming from the rise of advanced AI systems:
    How to make sure that advanced AI systems do what we want them to (the alignment problem)?
    What makes a system safe enough to develop and deploy, and what constitutes sufficient evidence of that?
    How do we preserve our autonomy and control as decision making is increasingly delegated to digital assistants?
    A causal perspective on agency provides conceptual tools for navigating the above questions, as we’ll explain in this sequence of blog posts. An effort will be made to minimise and explain jargon, to make the sequence accessible to researchers from a range of backgrounds.
    Agency
    First, with agent we mean a goal-directed system that is trying to steer the world in some particular direction(s). Examples include animals, humans, and organisations (more on agents in a subsequent post). Understanding agents is key to the above questions. Artificial agents are widely considered the primary existential threat from AGI-level technology, whether they emerge spontaneously or through deliberate design. Despite the myriad risks to our existence, highly capable agents pose a distinct danger, because many goals can be achieved more effectively by accumulating influence over the world. Whereas an asteroid moving towards earth isn’t intending to harm humans and won’t resist redirection, misaligned agents might be distinctly adversarial and active threats.
    Second, the preservation of human agency is critical in the approaching technological transition, for both individuals and collectives. Concerns have already been raised that manipulative social media algorithms and content recommenders undermine users’ ability to focus on their long-term goals. More powerful assistants could exacerbate this. And as more decision-making is delegated to AI systems, the ability of society to set its own trajectory comes into question
    Human agency can also be nurtured and protected. Helping people to help themselves is less paternalistic than directly fulfilling their desires, and fostering empowerment may be less contingent on complete alignment than direct satisfaction of individual preferences. Indeed, self-determination theory provides evidence that humans intrinsically value agency, and some human rights can be interpreted as “protections of our normative agency”.
    Third, artificial agents might themselves eventually constitute moral patients. A clearer understanding of agency could help us refine our moral intuitions and avoid unethical actions. Some ethical dilemmas might be possible to avoid altogether by only designing artificial systems that lack moral patienthood.
    Key questions
    One hope for our research is that it would build up a theory of agency. Such a theory would ideally answer questions such as:
    What are the possible kinds of agents that can be created, and along what dimension can they differ? The agents we’ve seen so far primarily include animals, humans, and human organisations, but the range of possible goal-directed systems is likely much larger than that.
    Emergence: how are agents created? For example, when might an LLM become agentic? When does a system of agents become a “meta-agent”, such as an organisation?
    Disempowerment: how is agency lost? How do we preserve and nurture human agency?
    What are the ethical demands posed by various types of systems and a...
    8 min
  • LW - UK PM: $125M for AI safety by Hauke Hillebrandt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: UK PM: $125M for AI safety, published by Hauke Hillebrandt on June 12, 2023 on LessWrong.
    The UK PM recently announced $125M for a foundation model task force. While the announcement stressed AI safety, it also stressed capabilities. But this morning the PM said 'It'll be about safety' and that the UK is spending more than other countries on this and one small media outlet had already coined this the 'safer AI taskforce'.
    Ian Hogarth is leading the task force who's on record saying that AGI could lead to “obsolescence or destruction of the human race” if there’s no regulation on the technology’s progress.
    Matt Clifford is also advising the task force - on record having said the same thing and knows a lot about AI safety. He had Jess Whittlestone & Jack Clark on his podcast.
    If mainstream AI safety is useful and doesn't increase capabilities, then the taskforce and the $125M seem valuable.
    We should use this window of opportunity to solidify this by quoting the PM and getting '$125M for AI safety research' and 'safer AI taskforce' locked in, by writing and promoting op-eds that commend spending on AI safety and urge other countries to follow (cf. the NSF has announced a $20M for empirical AI safety research). OpenAI, Anthropic, A16z and Palantir are all joining DeepMind in setting up offices in London.
    This might create an AI safety race to the top as a solution to the tragedy of the commons (cf. the US has criticized Germany for not spending 2% of GDP on defence; Germany’s shot back saying the US should first meet the 0.7% of GNI on aid target).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Manifold Predicted the AI Extinction Statement and CAIS Wanted it Deleted by David Chee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifold Predicted the AI Extinction Statement and CAIS Wanted it Deleted, published by David Chee on June 12, 2023 on LessWrong.
    Crosspost from
    History may look back on CAIS’s AI risk statement as a pivotal moment IF we survive AI.
    But, what if I told you that AI researchers “leaked” info about the statement on Manifold Markets over a week before it was published? This justifiably led to CAIS requesting the market to be deleted, concerned that it could damage the impact of the statement.
    To provide some background, the statement was signed by the likes of Sam Altman, Demis Hassabis, Dario Amodei and other prominent AI figures. Short and with gravitas it read:
    “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
    Imagine if the CEOs of major oil companies banded together in the 1900s and told their respective governments, “We need to be careful about building infrastructure relying heavily on fossil fuels. It could be catastrophic for the environment long term.”
    This may seem like an extreme comparison to make, but the reality is leading AI companies consider the risk significant enough to compromise on profit and progression. Well, that’s what their words say, time will tell how it is reflected in corporate decision-making.
    I had an opportunity to talk to some of the insider traders and Dan Hendrycks from CAIS, and hope you enjoy a breakdown of one of our craziest markets and what we’ve learnt from it.
    Background on Manifold Markets
    Prediction markets allow people who lack expertise about certain topics to form precise models of what the future could look like thanks to the live-updating probabilities that are generated by traders. Trades can buy YES or NO shares, which fluctuate in price depending on the current probability (similar to sports betting odds).
    Manifold Markets uses play money, which leads some to express skepticism of its efficacy. However, data suggest Manifold Markets are incredibly accurate!
    Here is a graph from an analysis of Manifold Market’s calibration carried out by Vincent Luczkow. This places markets into buckets based on their probability at the middle of their lifespan. It then looks at the percentage of markets in each bucket that resolve to yes. To be well-calibrated, you want markets in the 10% bucket at their mid-point to resolve yes 10% of the time and no 90% of the time (this ideal is represented by the green line on the graph). Pretty accurate! Also, check out our calibration page or our performance on the midterms.
    Manifold Markets is unique from other prediction platforms as its markets are all user-generated questions. Users have used our markets to make predictions about everything you can think of from global nuclear risk to their personal romantic endeavours. The core denominator between these markets is the mechanism used to generate the probability that predicts an unknown future event.
    But what happens when you defy this, and a market is created by someone that does know the future?
    The Tale of the Statement on AI Risk
    It all started on May 20th, when a user called Quinesweeper created 3 markets titled, “Will there be another well-recognized letter/statement on AI risk by May [June, July] 31, 2023?” Whenever I write, “the/this market”, I will be referring to the market which was asking if the statement would be released before the end of May.
    I’m including a screenshot of the traders who won the most profit and an annotated graph below which will be referenced throughout.
    Initial trading
    All markets start at 50% when created, but this market was quickly bid down to 10% by traders who had no evidence that a major statement would be published in the next 10 days. However, within the first few hours, there was already upwards buying press...
    19 min
  • AF - Contingency: A Conceptual Tool from Evolutionary Biology for Alignment by clem acs
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contingency: A Conceptual Tool from Evolutionary Biology for Alignment, published by clem acs on June 12, 2023 on The AI Alignment Forum.
    1. Motivation
    As AI systems become increasingly powerful, the chance of them developing dangerous features becomes increasingly likely. A key concern of AI alignment researchers is that there are dangerous features, such as power-seeking or situational awareness, which are convergent. This would mean that these capabilities would likely arise regardless of the particular trajectory AI development took. In a recent tweet, Eliezer Yudkowsky argues that this is the key question for AI alignment research:
    > From my perspective, we are trying to evaluate a picture of convergent ruin rather than any pictures of contingent ruin. That is, I'm relatively less interested in contingent-doom scenarios like "What if superintelligence is easily alignable, but Greenpeace is first to obtain superintelligence and they align It entirely to protect Earth's environment and then the SI wipes out humanity to do that?" because this is not a case for almost-all paths leading to ruin. I worry that contingent scenarios like "Age of Ultron" can fascinate humans (particularly contingent scenarios where some Other Monkey Faction Gets AGI First oh noes), and derail all possibility of a discussion about whether, in fact, the superintelligence kills everyone regardless of who 'gets' It first.
    > Similarly, if somebody is considering only the contingent-early-ruin scenario of "a prototype of a somewhat advanced AGI starts to consider how to wipe out humanity, and nobody spots any signs of that thought", that contingency can mask a convergent-ruin continuation of "even if they spot the early signs, everyone is still in an arms-race scenario, so they optimize away the visible signs of doom and keep going into ruin".
    > So I think the key question is "Why would or wouldn't humanity be convergently doomed?" rather than "How might humanity be contingently doomed?" Convergent doom isn't the same as inevitable doom, but ought to be the sort of doom that defeats easy solution / historically characteristic levels of competence for Earth / the sort of efforts we've seen already.
    > A doom that can only be averted by an effective multinational coalition shutting down all AI runs larger than GPT-4 for 6 years, to make time for a crash project on human intelligence augmentation that has to complete within 6 years, may not be an inevitable doom; but it was probably a pretty convergent doom if that was your best path for stopping it.
    > Conversely, "What if all the AI companies and venture capitalists just decide together to not pursue the rewards of AI models stronger than GPT-4, and high-impact journals and conferences stop accepting papers about it" is a sufficiently unlikely contingency that dooms avertable in that way may still be said to be convergent. It seems reasonably convergent that, if there isn't a drastic effective global shutdown enforced by state-level actors that will enforce that treaty even on non-signatory states, somebody tries.
    This framing introduces “contingency” as the opposite of convergence - features of systems which are convergent are, by definition, those which are not contingent. In this piece I suggest an alternative framing, in which convergent features are themselves often contingent on some other feature. This is an initially unintuitive framing which, once you’ve seen it once, comes up everywhere. In the case of alignment, it would allow us to take some dangerous feature of AI systems which we expect to be convergent, and ask “what is the feature contingent on?”. My hope is that this shift in perspective can reveal implicit assumptions and thus allow us to spot new potential solutions for AI alignment.
    The field where this shift in perspective is ...
    27 min
  • LW - ARC is hiring theoretical researchers by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring theoretical researchers, published by paulfchristiano on June 12, 2023 on LessWrong.
    The Alignment Research Center’s Theory team is starting a new hiring round for researchers with a theoretical background. Please apply here.
    What is ARC’s Theory team?
    The Alignment Research Center (ARC) is a non-profit whose mission is to align future machine learning systems with human interests. The high-level agenda of the Theory team (not to be confused with the Evals team) is described by the report on Eliciting Latent Knowledge (ELK): roughly speaking, we’re trying to design ML training objectives that incentivize systems to honestly report their internal beliefs.
    For the last year or so, we’ve mostly been focused on an approach to ELK based on formalizing a kind of heuristic reasoning that could be used to analyze neural network behavior, as laid out in our paper on Formalizing the presumption of independence. Our research has reached a stage where we’re coming up against concrete problems in mathematics and theoretical computer science, and so we’re particularly excited about hiring researchers with relevant background, regardless of whether they have worked on AI alignment before. See below for further discussion of ARC’s current theoretical research directions.
    Who is ARC looking to hire?
    Compared to our last hiring round, we have more of a need for people with a strong theoretical background (in math, physics or computer science, for example), but we remain open to anyone who is excited about getting involved in AI alignment, even if they do not have an existing research record.
    Ultimately, we are excited to hire people who could contribute to our research agenda. The best way to figure out whether you might be able to contribute is to take a look at some of our recent research problems and directions:
    Some of our research problems are purely mathematical, such as these matrix completion problems – although note that these are unusually difficult, self-contained and well-posed (making them more appropriate for prizes).
    Some of our other research is more informal, as described in some of our recent blog posts such as Finding gliders in the game of life.
    A lot of our research occupies a middle ground between fully-formalized problems and more informal questions, such as fixing the problems with cumulant propagation described in Appendix D of Formalizing the presumption of independence.
    What is working on ARC’s Theory team like?
    ARC’s Theory team is led by Paul Christiano and currently has 2 other permanent team members, Mark Xu and Jacob Hilton, alongside a varying number of temporary team members (recently anywhere from 0–3).
    Most of the time, team members work on research problems independently, with frequent check-ins with their research advisor (e.g., twice weekly). The problems described above give a rough indication of the kind of research problems involved, which we would typically break down into smaller, more manageable subproblems. This work is often somewhat similar to academic research in pure math or theoretical computer science.
    In addition to this, we also allocate a significant portion of our time to higher-level questions surrounding research prioritization, which we often discuss at our weekly group meeting. Since the team is still small, we are keen for new team members to help with this process of shaping and defining our research.
    ARC shares an office with several other groups working on AI alignment such as Redwood Research, so even though the Theory team is small, the office is lively with lots of AI alignment-related discussion.
    What are ARC’s current theoretical research directions?
    ARC’s main theoretical focus over the last year or so has been on preparing the paper Formalizing the presumption of independence and on follow-up work to ...
    7 min
  • AF - ARC is hiring theoretical researchers by Paul Christiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring theoretical researchers, published by Paul Christiano on June 12, 2023 on The AI Alignment Forum.
    The Alignment Research Center’s Theory team is starting a new hiring round for researchers with a theoretical background. Please apply here.
    What is ARC’s Theory team?
    The Alignment Research Center (ARC) is a non-profit whose mission is to align future machine learning systems with human interests. The high-level agenda of the Theory team (not to be confused with the Evals team) is described by the report on Eliciting Latent Knowledge (ELK): roughly speaking, we’re trying to design ML training objectives that incentivize systems to honestly report their internal beliefs.
    For the last year or so, we’ve mostly been focused on an approach to ELK based on formalizing a kind of heuristic reasoning that could be used to analyze neural network behavior, as laid out in our paper on Formalizing the presumption of independence. Our research has reached a stage where we’re coming up against concrete problems in mathematics and theoretical computer science, and so we’re particularly excited about hiring researchers with relevant background, regardless of whether they have worked on AI alignment before. See below for further discussion of ARC’s current theoretical research directions.
    Who is ARC looking to hire?
    Compared to our last hiring round, we have more of a need for people with a strong theoretical background (in math, physics or computer science, for example), but we remain open to anyone who is excited about getting involved in AI alignment, even if they do not have an existing research record.
    Ultimately, we are excited to hire people who could contribute to our research agenda. The best way to figure out whether you might be able to contribute is to take a look at some of our recent research problems and directions:
    Some of our research problems are purely mathematical, such as these matrix completion problems – although note that these are unusually difficult, self-contained and well-posed (making them more appropriate for prizes).
    Some of our other research is more informal, as described in some of our recent blog posts such as Finding gliders in the game of life.
    A lot of our research occupies a middle ground between fully-formalized problems and more informal questions, such as fixing the problems with cumulant propagation described in Appendix D of Formalizing the presumption of independence.
    What is working on ARC’s Theory team like?
    ARC’s Theory team is led by Paul Christiano and currently has 2 other permanent team members, Mark Xu and Jacob Hilton, alongside a varying number of temporary team members (recently anywhere from 0–3).
    Most of the time, team members work on research problems independently, with frequent check-ins with their research advisor (e.g., twice weekly). The problems described above give a rough indication of the kind of research problems involved, which we would typically break down into smaller, more manageable subproblems. This work is often somewhat similar to academic research in pure math or theoretical computer science.
    In addition to this, we also allocate a significant portion of our time to higher-level questions surrounding research prioritization, which we often discuss at our weekly group meeting. Since the team is still small, we are keen for new team members to help with this process of shaping and defining our research.
    ARC shares an office with several other groups working on AI alignment such as Redwood Research, so even though the Theory team is small, the office is lively with lots of AI alignment-related discussion.
    What are ARC’s current theoretical research directions?
    ARC’s main theoretical focus over the last year or so has been on preparing the paper Formalizing the presumption of independence and on follo...
    7 min
  • AF - Introduction to Towards Causal Foundations of Safe AGI by Tom Everitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introduction to Towards Causal Foundations of Safe AGI, published by Tom Everitt on June 12, 2023 on The AI Alignment Forum.
    By Tom Everitt, Lewis Hammond, Rhys Ward, Ryan Carey, James Fox, Sebastian Benthall, Matt MacDermott and Shreshth Malik representing the Causal Incentives Working Group. Thanks also to Toby Shevlane, MH Tessler, Aliya Ahmad, Zac Kenton, and Maria Loks-Thompson.
    Over the next few years, society, organisations, and individuals will face a number of fundamental questions stemming from the rise of advanced AI systems:
    How to make sure that advanced AI systems do what we want them to (the alignment problem)?
    What makes a system safe enough to develop and deploy, and what constitutes sufficient evidence of that?
    How do we preserve our autonomy and control as decision making is increasingly delegated to digital assistants?
    A causal perspective on agency provides conceptual tools for navigating the above questions, as we’ll explain in this sequence of blog posts. An effort will be made to minimise and explain jargon, to make the sequence accessible to researchers from a range of backgrounds.
    Agency
    First, with agent we mean a goal-directed system that is trying to steer the world in some particular direction(s). Examples include animals, humans, and organisations (more on agents in a subsequent post). Understanding agents is key to the above questions. Artificial agents are widely considered the primary existential threat from AGI-level technology, whether they emerge spontaneously or through deliberate design. Despite the myriad risks to our existence, highly capable agents pose a distinct danger, because many goals can be achieved more effectively by accumulating influence over the world. Whereas an asteroid moving towards earth isn’t intending to harm humans and won’t resist redirection, misaligned agents might be distinctly adversarial and active threats.
    Second, the preservation of human agency is critical in the approaching technological transition, for both individuals and collectives. Concerns have already been raised that manipulative social media algorithms and content recommenders undermine users’ ability to focus on their long-term goals. More powerful assistants could exacerbate this. And as more decision-making is delegated to AI systems, the ability of society to set its own trajectory comes into question
    Human agency can also be nurtured and protected. Helping people to help themselves is less paternalistic than directly fulfilling their desires, and fostering empowerment may be less contingent on complete alignment than direct satisfaction of individual preferences. Indeed, self-determination theory provides evidence that humans intrinsically value agency, and some human rights can be interpreted as “protections of our normative agency”.
    Third, artificial agents might themselves eventually constitute moral patients. A clearer understanding of agency could help us refine our moral intuitions and avoid unethical actions. Some ethical dilemmas might be possible to avoid altogether by only designing artificial systems that lack moral patienthood.
    Key questions
    One hope for our research is that it would build up a theory of agency. Such a theory would ideally answer questions such as:
    What are the possible kinds of agents that can be created, and along what dimension can they differ? The agents we’ve seen so far primarily include animals, humans, and human organisations, but the range of possible goal-directed systems is likely much larger than that.
    Emergence: how are agents created? For example, when might an LLM become agentic? When does a system of agents become a “meta-agent”, such as an organisation?
    Disempowerment: how is agency lost? How do we preserve and nurture human agency?
    What are the ethical demands posed by various types of ...
    8 min