The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Celebrating EAGxLatAm and EAGxIndia by OllieBase
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Celebrating EAGxLatAm and EAGxIndia, published by OllieBase on January 26, 2023 on The Effective Altruism Forum.
    In the first two weeks of January, we had the first EA conferences to take place in Latin America and India: EAGxLatAm in Mexico City and EAGxIndia in Jaipur, Rajasthan.
    Lots of EAGx events happen every year, and we don’t usually post about each one. But I think these events are particularly special as two of the largest EA conferences to take place in LMICs (Low and Middle Income Countries) and the first ever in each region. So, I wanted to share a bit more about how they went.
    Tl;DR: they went well.
    EAGxLatinAmerica
    EAGxLatAm was organised by community builders from the Spanish-speaking EA community with support from CEA. Over 200 people attended the event: ~32% of attendees came from within Mexico, ~13% from the US, ~12% from Colombia, ~7% from Brazil, ~5% from Chile and ~5% from Spain. There were also attendees from several other countries including Argentina, Peru, the Philippines, Canada, and countries in Europe.
    The event hosted talks in English, Spanish, and Portuguese and attendees could mark on their name badges which languages they spoke. Sessions included:
    A talk on the high-impact careers service, Carreras con Impacto;
    A virtual Q&A with Toby Ord to discuss existential risks and the role of LMICs in tackling them;
    Racionalidad para principiantes (Rationality for beginners) by Laura González Salmerón;
    A panel on Riesgos Catastróficos Globales (Global Catastrophic Risks);
    A panel on AI safety with staff from OpenAI, Redwood Research, CSER and Epoch;
    A talk on Animal Welfare cause prioritisation by Daniela Romero Waldhorn (Rethink Priorities);
    A talk by JPAL staff on ‘Applying evidence to reduce intimate partner violence’;
    A talk on local cause prioritisation by Luis Mota (Global Priorities Institute);
    A panel on EA in LMICs with EA community builders from Brazil, the Spanish-speaking community, the Philippines, Nigeria, South Africa, and EA Anywhere;
    A talk by Vida Plena (a new organisation incubated by Charity Entrepreneurship focused on mental health in Latin America);
    Meetups for attendees from Mexico, Chile/Argentina, Colombia/Perú, Brazil and Spain.
    The event received a likelihood to recommend score of 9.07/10 and attendees made an average of 9.37 new connections. Some quotes from the feedback survey I enjoyed reading:
    “I had a 1-1 with someone else interested in nuclear security and nuclear winter research, and we're developing an online reading group curriculum to [ideally] launch in March!"
    “I got convinced on deciding to focus on [sic] direct work on ai safety by talking to [two other attendees].”
    “I met Rob Wiblin at lunch and I didn't recognize him."
    One attendee described the event to a CEA colleague as “the best weekend of their life”.
    The event wasn’t perfect, of course. Several attendees reported that there was too much content in English that some attendees couldn’t follow, and some of the snacks and beverages seemed low quality. We hope to do even better next time!
    EAGxIndia
    EAGxIndia was organised by community builders based in India with support from CEA. 163 people attended: ~70% of attendees came from within India, ~9% from the UK and ~9% from the US. The rest came from Singapore, Indonesia, the Netherlands, Germany, and Brazil.
    The sessions were all in English, and included:
    A talk on South Asian Air Quality by Santosh Harish (Open Philanthropy);
    An AI governance Q&A with Michael Aird (Rethink Priorities);
    An update from the Fish Welfare Initiative, who are based in India;
    A talk on AI safety research agendas and opportunities by Adam Gleave (FAR AI);
    Two sessions led by Charity Entrepreneurship;
    A workshop on what the EA community should look like in India, led by Indian community builders and CEA staff;
    A talk mapping the...
    7 min
  • EA - Overview of effective giving organisations by SjirH
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Overview of effective giving organisations, published by SjirH on January 26, 2023 on The Effective Altruism Forum.
    I've been keeping an overview of the public effective giving ecosystem that I thought would be worth sharing in its own post (I've previously referred to it here). I've noticed people often aren't aware of many of the initiatives in this space (>50!) and they could be missing out on some great funding opportunity recommendations, collaboration and job opportunities, and other useful connections and information.
    The list is meant to contain all organisations and projects that aim to identify publicly accessible philanthropic funding opportunities using an effective-altruism-inspired methodology (evaluators), and/or to fundraise for the funding opportunities that have already been identified (fundraisers). As I note on the sheet, inclusion in this list does not imply that the organisation or project's research and recommendations have been vetted for quality: it only implies self-association with the effective giving community.
    Please let me know if you think any organisation or project is missing! I aim to keep this list updated, and expect it to change quite a bit over the coming year (I know of a few more budding initiatives that may soon be added).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Have worries about EA? Want to chat? by Catherine Low
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Have worries about EA? Want to chat?, published by Catherine Low on January 26, 2023 on The Effective Altruism Forum.
    Feeling worried?
    Hey all. I’m Catherine, and I work on CEA’s Community Health and Special Projects Team. One of my roles is to act as a contact person for the EA community, alongside Julia Wise.
    I just wanted to let you know that I’m available if you want to share EA-related worries or frustrations with me (they can be specific or very vague). My email is [email protected], or you can reach the whole Community Health and Special Projects team on our form (which can be filled in anonymously if you wish). Sometimes I’ll have the time to jump on a video call to talk it through if you’d find that helpful (but it might not always be possible). Maybe I’ll be able to help in some way, or maybe not. I also read through all the posts on the Forum relating to worries about the community (and the comments).
    What I’m worried about right now
    Please feel free to reach out about any manner of worries relating to EA – whether they are related to my current worry or something entirely different.
    Personally, I’m currently feeling disappointed, worried, frustrated, and a bit angry (which isn’t an emotion I commonly feel) about a few things that I’ve read on the Forum recently. I’m also incredibly privileged in nearly all metrics of privilege one can imagine, so there is a high chance that what I’m feeling doesn’t scratch the surface of the feelings some of you are experiencing.
    But, I also notice that there are a lot of people in the wider EA community who don’t follow the online EA spaces. They’re just getting on with the job of learning, upskilling, or making the world a better place. I’ve been oscillating wildly between feeling that my worry is a really big signal I should pay attention to or mostly a function of me spending too much time online reading meta-EA things.
    One thing I tell myself when I get upset, angry or frustrated in an EA space is “This space ≠ EA”.
    If you’re feeling one EA space is bad for you, consider trying another EA space (for me, I sometimes get disillusioned when reading the Forum, but in-person meetings or video calls make me feel SO MUCH better about our community.)
    You might like to try rereading the texts that first brought you to EA (my personal choice for reinvigorating myself is On Caring - it’s not a happy read but for me it brings things back into focus).
    And, just a reminder, sometimes the right thing to do is to take a step back for a bit.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Have worries about EA? Want to chat? by Catherine Low
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Have worries about EA? Want to chat?, published by Catherine Low on January 26, 2023 on The Effective Altruism Forum.
    Feeling worried?
    Hey all. I’m Catherine, and I work on CEA’s Community Health and Special Projects Team. One of my roles is to act as a contact person for the EA community, alongside Julia Wise.
    I just wanted to let you know that I’m available if you want to share EA-related worries or frustrations with me (they can be specific or very vague). My email is [email protected], or you can reach the whole Community Health and Special Projects team on our form (which can be filled in anonymously if you wish). Sometimes I’ll have the time to jump on a video call to talk it through if you’d find that helpful (but it might not always be possible). Maybe I’ll be able to help in some way, or maybe not. I also read through all the posts on the Forum relating to worries about the community (and the comments).
    What I’m worried about right now
    Please feel free to reach out about any manner of worries relating to EA – whether they are related to my current worry or something entirely different.
    Personally, I’m currently feeling disappointed, worried, frustrated, and a bit angry (which isn’t an emotion I commonly feel) about a few things that I’ve read on the Forum recently. I’m also incredibly privileged in nearly all metrics of privilege one can imagine, so there is a high chance that what I’m feeling doesn’t scratch the surface of the feelings some of you are experiencing.
    But, I also notice that there are a lot of people in the wider EA community who don’t follow the online EA spaces. They’re just getting on with the job of learning, upskilling, or making the world a better place. I’ve been oscillating wildly between feeling that my worry is a really big signal I should pay attention to or mostly a function of me spending too much time online reading meta-EA things.
    One thing I tell myself when I get upset, angry or frustrated in an EA space is “This space ≠ EA”.
    If you’re feeling one EA space is bad for you, consider trying another EA space (for me, I sometimes get disillusioned when reading the Forum, but in-person meetings or video calls make me feel SO MUCH better about our community.)
    You might like to try rereading the texts that first brought you to EA (my personal choice for reinvigorating myself is On Caring - it’s not a happy read but for me it brings things back into focus).
    And, just a reminder, sometimes the right thing to do is to take a step back for a bit.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Spreading messages to help with the most important century by HoldenKarnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spreading messages to help with the most important century, published by HoldenKarnofsky on January 25, 2023 on LessWrong.
    In the most important century series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.
    In this more recent series, I’ve been trying to help answer this question: “So what? What can I do to help?”
    So far, I’ve just been trying to build a picture of some of the major risks we might face (especially the risk of misaligned AI that could defeat all of humanity), what might be challenging about these risks, and why we might succeed anyway. Now I’ve finally gotten to the part where I can start laying out tangible ideas for how to help (beyond the pretty lame suggestions I gave before).
    This piece is about one broad way to help: spreading messages that ought to be more widely understood.
    One reason I think this topic is worth a whole piece is that practically everyone can help with spreading messages at least some, via things like talking to friends; writing explanations of your own that will appeal to particular people; and, yes, posting to Facebook and Twitter and all of that. Call it slacktivism if you want, but I’d guess it can be a big deal: many extremely important AI-related ideas are understood by vanishingly small numbers of people, and a bit more awareness could snowball. Especially because these topics often feel too “weird” for people to feel comfortable talking about them! Engaging in credible, reasonable ways could contribute to an overall background sense that it’s OK to take these ideas seriously.
    And then there are a lot of potential readers who might have special opportunities to spread messages. Maybe they are professional communicators (journalists, bloggers, TV writers, novelists, TikTokers, etc.), maybe they’re non-professionals who still have sizable audiences (e.g., on Twitter), maybe they have unusual personal and professional networks, etc. Overall, the more you feel you are good at communicating with some important audience (even a small one), the more this post is for you.
    That said, I’m not excited about blasting around hyper-simplified messages. As I hope this series has shown, the challenges that could lie ahead of us are complex and daunting, and shouting stuff like “AI is the biggest deal ever!” or “AI development should be illegal!” could do more harm than good (if only by associating important ideas with being annoying). Relatedly, I think it’s generally not good enough to spread the most broad/relatable/easy-to-agree-to version of each key idea, like “AI systems could harm society.” Some of the unintuitive details are crucial.
    Instead, the gauntlet I’m throwing is: “find ways to help people understand the core parts of the challenges we might face, in as much detail as is feasible.” That is: the goal is to try to help people get to the point where they could maintain a reasonable position in a detailed back-and-forth, not just to get them to repeat a few words or nod along to a high-level take like “AI safety is important.” This is a lot harder than shouting “AI is the biggest deal ever!”, but I think it’s worth it, so I’m encouraging people to rise to the challenge and stretch their communication skills.
    Below, I will:
    Outline some general challenges of this sort of message-spreading.
    Go through some ideas I think it’s risky to spread too far, at least in isolation.
    Go through some of the ideas I’d be most excited to see spread.
    Talk a little bit about how to spread ideas - but this is mostly up to you.
    Challenges of AI-related messages
    Here’s a simplified story for how spreading message...
    28 min
  • LW - Spreading messages to help with the most important century by HoldenKarnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spreading messages to help with the most important century, published by HoldenKarnofsky on January 25, 2023 on LessWrong.
    In the most important century series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.
    In this more recent series, I’ve been trying to help answer this question: “So what? What can I do to help?”
    So far, I’ve just been trying to build a picture of some of the major risks we might face (especially the risk of misaligned AI that could defeat all of humanity), what might be challenging about these risks, and why we might succeed anyway. Now I’ve finally gotten to the part where I can start laying out tangible ideas for how to help (beyond the pretty lame suggestions I gave before).
    This piece is about one broad way to help: spreading messages that ought to be more widely understood.
    One reason I think this topic is worth a whole piece is that practically everyone can help with spreading messages at least some, via things like talking to friends; writing explanations of your own that will appeal to particular people; and, yes, posting to Facebook and Twitter and all of that. Call it slacktivism if you want, but I’d guess it can be a big deal: many extremely important AI-related ideas are understood by vanishingly small numbers of people, and a bit more awareness could snowball. Especially because these topics often feel too “weird” for people to feel comfortable talking about them! Engaging in credible, reasonable ways could contribute to an overall background sense that it’s OK to take these ideas seriously.
    And then there are a lot of potential readers who might have special opportunities to spread messages. Maybe they are professional communicators (journalists, bloggers, TV writers, novelists, TikTokers, etc.), maybe they’re non-professionals who still have sizable audiences (e.g., on Twitter), maybe they have unusual personal and professional networks, etc. Overall, the more you feel you are good at communicating with some important audience (even a small one), the more this post is for you.
    That said, I’m not excited about blasting around hyper-simplified messages. As I hope this series has shown, the challenges that could lie ahead of us are complex and daunting, and shouting stuff like “AI is the biggest deal ever!” or “AI development should be illegal!” could do more harm than good (if only by associating important ideas with being annoying). Relatedly, I think it’s generally not good enough to spread the most broad/relatable/easy-to-agree-to version of each key idea, like “AI systems could harm society.” Some of the unintuitive details are crucial.
    Instead, the gauntlet I’m throwing is: “find ways to help people understand the core parts of the challenges we might face, in as much detail as is feasible.” That is: the goal is to try to help people get to the point where they could maintain a reasonable position in a detailed back-and-forth, not just to get them to repeat a few words or nod along to a high-level take like “AI safety is important.” This is a lot harder than shouting “AI is the biggest deal ever!”, but I think it’s worth it, so I’m encouraging people to rise to the challenge and stretch their communication skills.
    Below, I will:
    Outline some general challenges of this sort of message-spreading.
    Go through some ideas I think it’s risky to spread too far, at least in isolation.
    Go through some of the ideas I’d be most excited to see spread.
    Talk a little bit about how to spread ideas - but this is mostly up to you.
    Challenges of AI-related messages
    Here’s a simplified story for how spreading message...
    28 min
  • EA - Spreading messages to help with the most important century by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spreading messages to help with the most important century, published by Holden Karnofsky on January 25, 2023 on The Effective Altruism Forum.
    In the most important century series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.
    In this more recent series, I’ve been trying to help answer this question: “So what? What can I do to help?”
    So far, I’ve just been trying to build a picture of some of the major risks we might face (especially the risk of misaligned AI that could defeat all of humanity), what might be challenging about these risks, and why we might succeed anyway. Now I’ve finally gotten to the part where I can start laying out tangible ideas for how to help (beyond the pretty lame suggestions I gave before).
    This piece is about one broad way to help: spreading messages that ought to be more widely understood.
    One reason I think this topic is worth a whole piece is that practically everyone can help with spreading messages at least some, via things like talking to friends; writing explanations of your own that will appeal to particular people; and, yes, posting to Facebook and Twitter and all of that. Call it slacktivism if you want, but I’d guess it can be a big deal: many extremely important AI-related ideas are understood by vanishingly small numbers of people, and a bit more awareness could snowball. Especially because these topics often feel too “weird” for people to feel comfortable talking about them! Engaging in credible, reasonable ways could contribute to an overall background sense that it’s OK to take these ideas seriously.
    And then there are a lot of potential readers who might have special opportunities to spread messages. Maybe they are professional communicators (journalists, bloggers, TV writers, novelists, TikTokers, etc.), maybe they’re non-professionals who still have sizable audiences (e.g., on Twitter), maybe they have unusual personal and professional networks, etc. Overall, the more you feel you are good at communicating with some important audience (even a small one), the more this post is for you.
    That said, I’m not excited about blasting around hyper-simplified messages. As I hope this series has shown, the challenges that could lie ahead of us are complex and daunting, and shouting stuff like “AI is the biggest deal ever!” or “AI development should be illegal!” could do more harm than good (if only by associating important ideas with being annoying). Relatedly, I think it’s generally not good enough to spread the most broad/relatable/easy-to-agree-to version of each key idea, like “AI systems could harm society.” Some of the unintuitive details are crucial.
    Instead, the gauntlet I’m throwing is: “find ways to help people understand the core parts of the challenges we might face, in as much detail as is feasible.” That is: the goal is to try to help people get to the point where they could maintain a reasonable position in a detailed back-and-forth, not just to get them to repeat a few words or nod along to a high-level take like “AI safety is important.” This is a lot harder than shouting “AI is the biggest deal ever!”, but I think it’s worth it, so I’m encouraging people to rise to the challenge and stretch their communication skills.
    Below, I will:
    Outline some general challenges of this sort of message-spreading.
    Go through some ideas I think it’s risky to spread too far, at least in isolation.
    Go through some of the ideas I’d be most excited to see spread.
    Talk a little bit about how to spread ideas - but this is mostly up to you.
    Challenges of AI-related messages
    Here’s a simplified story for h...
    28 min
  • LW - When Did EA Start? by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Did EA Start?, published by jefftk on January 25, 2023 on LessWrong.
    How old is the effective altruism movement? It depends when you count from, and it's hard to pick any specific day or event because it coalesced out of a lot of different strands. Plus, since this is a movement and not just an idea, it matters when it started gathering people which is naturally very fuzzy. Roughly, I'd describe the progression as something like:
    At the beginning of 2008 there wasn't an EA movement yet, while by
    2012 there was one. There wasn't an instantaneous change.
    Why this particular curve? Here are some historical points that anchor it to events:
    In March 2012, I wrote "there's a young movement that combines the idea of a duty to help others with the idea that you should maximize the impact of your actions," as part of trying to figure out what we should call
    EA. I'd say it 100% existed by this point, and I wasn't "calling it"—it existed before here.
    In November 2011, 80,000 Hours launched. It was mostly a way of spreading ideas that were already popular in the Oxford community and online, so I'd also say the EA movement existed before this. It's a bit less clear than when you have people literally calling it a movement, though.
    In December 2010, Roko Mijic ran a contest on LessWrong for the best explanation of EA, which Scott Alexander won with his essay, Efficient
    Charity: Do Unto Others.... Jonah Sinick also wrote a detailed entry, and Mass_Driver contributed one as well. This feels very movementy to me: a bunch of people working together to put some shared ideas into a form where they'll be able to spread better.
    In November 2009, the first 30 people joined Giving
    What We Can. A group of people pledging to do something seems like part of a movement. At the time GWWC was specific to global poverty, however, so I wouldn't fully count this.
    Over the course of 2009, discussion on the Felicifia forum (archive) started to feel like an early EA community. For example, see thread on charity choice and the applied ethics and philanthropy boards.
    In December 2006, the first GiveWell blog post comes out. Looking back at it, and their other early posts, you can tell that Holden and Elie had a lot of what has since become
    EA in mind. This also goes for a lot of other early writing like Nick
    Bostrom's 2003 post on astronomical waste, Anand's 2003 SL4
    "EffectiveAltruism" wiki entry, Brian Tomasik's 2006 post on earning to give, and Eliezer Yudkowsky's 2007 post on scope insensitivity (that coincidentally used the term "effective altruist" years before it was settled on). These are helpful for dating the ideas, but I'm looking for when the ideas started gathering people and they seem to predate most of that.
    One thing I'm not putting much weight on here is the founding dates of organizations. While many EA orgs have played a large role in the EA movement, the oldest became EA over time instead of starting out that way. For example, MIRI was founded in ~2000 as SIAI, but was initially wasn't something you'd recognize as EA and didn't have an associated community. It did grow in both of these directions over time, but to the extent that this helps us figure out the timeline of the movement we need to look at those later developments. Similarly, ACE could (but doesn't!) claim to have been founded in 2010 as JFA, but before merging with 80k's EAA it had a pretty different focus.
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - When Did EA Start? by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Did EA Start?, published by jefftk on January 25, 2023 on LessWrong.
    How old is the effective altruism movement? It depends when you count from, and it's hard to pick any specific day or event because it coalesced out of a lot of different strands. Plus, since this is a movement and not just an idea, it matters when it started gathering people which is naturally very fuzzy. Roughly, I'd describe the progression as something like:
    At the beginning of 2008 there wasn't an EA movement yet, while by
    2012 there was one. There wasn't an instantaneous change.
    Why this particular curve? Here are some historical points that anchor it to events:
    In March 2012, I wrote "there's a young movement that combines the idea of a duty to help others with the idea that you should maximize the impact of your actions," as part of trying to figure out what we should call
    EA. I'd say it 100% existed by this point, and I wasn't "calling it"—it existed before here.
    In November 2011, 80,000 Hours launched. It was mostly a way of spreading ideas that were already popular in the Oxford community and online, so I'd also say the EA movement existed before this. It's a bit less clear than when you have people literally calling it a movement, though.
    In December 2010, Roko Mijic ran a contest on LessWrong for the best explanation of EA, which Scott Alexander won with his essay, Efficient
    Charity: Do Unto Others.... Jonah Sinick also wrote a detailed entry, and Mass_Driver contributed one as well. This feels very movementy to me: a bunch of people working together to put some shared ideas into a form where they'll be able to spread better.
    In November 2009, the first 30 people joined Giving
    What We Can. A group of people pledging to do something seems like part of a movement. At the time GWWC was specific to global poverty, however, so I wouldn't fully count this.
    Over the course of 2009, discussion on the Felicifia forum (archive) started to feel like an early EA community. For example, see thread on charity choice and the applied ethics and philanthropy boards.
    In December 2006, the first GiveWell blog post comes out. Looking back at it, and their other early posts, you can tell that Holden and Elie had a lot of what has since become
    EA in mind. This also goes for a lot of other early writing like Nick
    Bostrom's 2003 post on astronomical waste, Anand's 2003 SL4
    "EffectiveAltruism" wiki entry, Brian Tomasik's 2006 post on earning to give, and Eliezer Yudkowsky's 2007 post on scope insensitivity (that coincidentally used the term "effective altruist" years before it was settled on). These are helpful for dating the ideas, but I'm looking for when the ideas started gathering people and they seem to predate most of that.
    One thing I'm not putting much weight on here is the founding dates of organizations. While many EA orgs have played a large role in the EA movement, the oldest became EA over time instead of starting out that way. For example, MIRI was founded in ~2000 as SIAI, but was initially wasn't something you'd recognize as EA and didn't have an associated community. It did grow in both of these directions over time, but to the extent that this helps us figure out the timeline of the movement we need to look at those later developments. Similarly, ACE could (but doesn't!) claim to have been founded in 2010 as JFA, but before merging with 80k's EAA it had a pretty different focus.
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - AGI will have learnt utility functions by Beren Millidge
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum.
    This post is part of the work done at Conjecture.
    Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post.
    There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function.
    Rather, the optimization process is assumed to be 'wrapped around' some core and unchanging utility function. The capabilities core of the agent is also totally modular and disjoint from the utility function such that arbitrary planners and utility functions can be composed so long as they have the right I/O interfaces. The core 'code' of an AIXI like agent is incredibly simple and, for instance, could be implemented in this Python pseudocode :
    def action_perception_loop(): while True: observation = self.sensors.get_observation() state = self.update_state(self.current_state, observation) all_action_plans = self.generate_action_plans(state) all_trajectories = self.world_model.generate_all_trajectories(all_action_plans, state) optimal_plan, optimal_utility = self.evaluate_trajectories(all_trajectories) self.execute(optimal_plan)
    There's a couple of central elements to this architecture which must be included in any AIXI-like architecture. The AGI needs some sensorimotor equipment to both sense the world and execute its action plans. It needs a Bayesian filtering component to be able to update its representation of the world state given new observations and its current state. It needs a world model that can generate sets of action plans and then generate 'rollouts' which are simulations of likely futures given an action plan. Finally, it needs a utility function that can calculate the utility of different simulated trajectories into the future and pick the best one. Let's zoom in on this component a little more and see how the evaluate_trajectories function might look inside. It might look like this:
    Essentially, the AIXI agent just takes all trajectories and ranks them according to its utility function and then picks the best one to execute. The fundamental problem with such an architecture, which is severely underappreciated, is that it implicitly assumes a utility oracle. That is, there exists some function self.utility_function() which is built into the agent from the beginning which can assign a consistent utility value to arbitrary world-states.
    While conceptually simple, my argument is that actually designing and building such a function into an agent to achieve a specific and complex goal in the external world is incredibly difficult or impossible for agents pursuing sufficiently complex goals and operating in sufficiently complex environments. This includes almost all goals humans are likely to want to program an AGI with. This means that in practice we cannot construct AIXI-like agents that optimize for arbitrary goals in the real world, and that any agent we do build must utilize some kind of learned utility model. Specifically, this is a utility (or reward) function uθ(x) where θ is some set of parameters and x is some kind of state, where the utility function is learned by some learning process (typically supervised learning) against a dataset of state, utility pairs that are provided either by the environment or by human designers.
    What this means is that, unlike a wrapper mind, the agent’s utility function can be influe...
    26 min