The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - [Linkpost] "Governance of superintelligence" by OpenAI by Daniel Eth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] "Governance of superintelligence" by OpenAI, published by Daniel Eth on May 22, 2023 on The Effective Altruism Forum.
    OpenAI has a new blog post out titled "Governance of superintelligence" (subtitle: "Now is a good time to start thinking about the governance of superintelligence—future AI systems dramatically more capable than even AGI"), by Sam Altman, Greg Brockman, and Ilya Sutskever.
    The piece is short (~800 words), so I recommend most people just read it in full.
    Here's the introduction/summary (bold added for emphasis):
    Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations.
    In terms of both potential upsides and downsides, superintelligence will be more powerful than other technologies humanity has had to contend with in the past. We can have a dramatically more prosperous future; but we have to manage risk to get there. Given the possibility of existential risk, we can’t just be reactive. Nuclear energy is a commonly used historical example of a technology with this property; synthetic biology is another example.
    We must mitigate the risks of today’s AI technology too, but superintelligence will require special treatment and coordination.
    And below are a few more quotes that stood out:
    "First, we need some degree of coordination among the leading development efforts to ensure that the development of superintelligence occurs in a manner that allows us to both maintain safety and help smooth integration of these systems with society."
    "Second, we are likely to eventually need something like an IAEA for superintelligence efforts; any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc."
    "It would be important that such an agency focus on reducing existential risk and not issues that should be left to individual countries, such as defining what an AI should be allowed to say."
    "Third, we need the technical capability to make a superintelligence safe. This is an open research question that we and others are putting a lot of effort into."
    "We think it’s important to allow companies and open-source projects to develop models below a significant capability threshold, without the kind of regulation we describe here"
    "By contrast, the systems we are concerned about will have power beyond any technology yet created, and we should be careful not to water down the focus on them by applying similar standards to technology far below this bar."
    "we believe it would be unintuitively risky and difficult to stop the creation of superintelligence"
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - AI Safety in China: Part 2 by Lao Mein
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety in China: Part 2, published by Lao Mein on May 22, 2023 on LessWrong.
    I've given things a lot more thought and wanted to make an edited summary of my views regarding AI Safety in China.
    AI Safety is almost nonexistent in China. The only toe-hold it has is rooted in the expat EA community, which is almost exclusively doing field building. This hasn't been effective thus far, and we need to switch tactics. My best guess is that a total of 10-20 people have a job in China in the field of AI safety, most of them are expats, and most of them are EA.
    EA, for various cultural reasons, is a toxic brand in China. It's not any single component of EA, but rather the idea of altruism itself. Ask anyone who's lived in China for and few years and they will understand where I'm coming from. I think the best way forward for AI safety in China is to disassociate with EA. Rationality is more easily accepted, but spreading related ideas is not the most effective way to address AI safety in China.
    Blue-skies research, possibly in something legible like interpretability, is the best way I can think of to actually build the AI safety field in China. The best way of making a field high-status is to overpay researchers and have them tell their friends.
    Money talks. Top-tier talent is often willing to jump ship to startups for the right compensation. I might be biased here since I've only worked in smaller companies. But if I had a large bag of money, I could probably leverage familial academia/business connections to hire star AI researchers.
    The Chinese government makes its decisions based on expert opinion and ideas from thought leaders in a particular field. The Chinese elites think like engineers and tend to take ideas seriously. AI safety isn't considered as weird as it is in the West.
    Due to AI safety's current lack of penetration in the Chinese sphere, I think any near-term attempts at an AI restriction treaty are unlikely to succeed - we don't think it's a real issue yet. Imagine if the Saudi King called the US President and tried to negotiate social media restrictions to prevent the Abrahamic End Times. That's where we're at. Then again, maybe someone in the Politburo is willing to take AI at least as seriously as Glenn Beck. It's hard to tell.
    China doesn't think like the West. Due to the Century of Humiliation, we are not going to be OK with being second best in a world-changing technology as we become the largest economy in the world. Our AI policy isn't OK with second place in the long run. Any AI-restriction treaty that China will accept requires not just Chinese parity, but Chinese superiority in AI. Major tradeoffs, like the US coercing Taiwan into reunification, may be required for China to sign on.
    Then again, Beijing is hard to predict. It may agree to an AI disarmament treaty in 6 months, or it might confiscate private GPUs in an effort at mass mobilization, spending billions to build the next LLM. It might do both.
    There are vague purity tests in EA about only hiring people who are super-duper not going to work in AI capabilities ever for AI safety research. This is extremely repugnant to Chinese researchers.
    On a personal level, I want to enter the field of AI safety immediately, and I think I would be more useful building the field in China as opposed to working in the US. However, I have only a single semester's worth of graduate studies thus far and my application to go from non-degree-seeking to degree-seeking student status was rejected. I am probably going to take a few online ML courses over the next year before I feel ready enough to actually request funding, but am open to suggestions.
    Again, there's very few (<10) people working on technical alignment in China right now, and I feel a bit lost. Any advice is welcome.
    Thanks for listening. To help us ...
    4 min
  • EA - X-risk discussion in a college commencement speech by SWK
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: X-risk discussion in a college commencement speech, published by SWK on May 22, 2023 on The Effective Altruism Forum.
    Yesterday, Juan Manuel Santos, former president of Colombia (2010-2018) and 2016 Nobel Peace Prize winner gave the commencement address at the University of Notre Dame.
    The address contained the usual graduation speech stuff about all the problems humanity is facing and how this special class of students is uniquely equipped to change the world or whatever.
    While the gist of the message was pretty standard, I was pleasantly surprised that Santos spent a significant chunk of time talking about existential risk as the most pressing problem of our time (see clip here). Santos touched on the major x-risk factors well known to the EA community: AI, biosecurity, and nuclear weapons. However, he also emphasized climate change as one of the most pressing existential threats, which of course is a view that many EAs do not share.
    On one hand, I think this speech should be seen as a sign of hope. Ideas on the importance of mitigating x-risk — and the particular threats of AI, pandemics, and nuclear war — seem to be entering more mainstream circles. This trend is evidenced further in a recent post noting another former world leader (Israeli PM Naftali Bennett) who publicly discussed AI x-risk. And I think a college commencement speech is arguably more mainstream than the Asian Leadership Conference at which Bennett delivered his talk.
    On the other hand, it was clear that there is still a long way to go in terms of convincing most of the general population that x-risk should be taken seriously. Sitting in the audience, the people around me were smirking and giggling throughout the x-risk portion of Santos' speech. Afterward, I overheard people joking about the AI part and how they thought it was inappropriate to talk about such morbid material in a commencement address.
    Overall, I certainly appreciated Santos for talking about x-risk, but I'm not convinced that his words had much of an impact on the people in the audience. To be sure, I realize that commencement speeches are largely ceremonial and all but a handful don't have any broader societal impact. Still, it would have been nice to see people be more receptive to Santos' important ideas.
    I would be interested to hear if anyone has any thoughts on Santos' discussion of x-risk. Was it appropriate to talk about this stuff in the context of a commencement address? Is this an effective forum to spread ideas about x-risk (or other "weird" EA ideas), or will these ideas just fall on deaf ears? Or are commencement addresses mostly irrelevant and not worth even thinking about for the purposes of growing EA and promoting some of the more idiosyncratic concepts?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The Treacherous Turn is finished! (AI-takeover-themed tabletop RPG) by Daniel Kokotajlo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Treacherous Turn is finished! (AI-takeover-themed tabletop RPG), published by Daniel Kokotajlo on May 22, 2023 on LessWrong.
    Long ago I fantasized about an AI-takeover-themed tabletop roleplaying game. Well, now it exists! [website, discord, twitter]
    The Game
    The Treacherous Turn is a tabletop roleplaying game in which the players collectively act as a misaligned AI in the modern world. Gameplay has the players scheming and putting their plans into action despite the efforts of the opposing humans, which are controlled by the Game Master (GM).
    Why?
    If you don’t enjoy or don’t have the time for tabletop roleplaying games, this probably isn’t for you. But otherwise, I encourage you to check it out!
    It is valuable to ‘red team’ AI safety, thinking from the perspective of a hypothetical misaligned AGI or group thereof. This is a fun way of doing that, while also exploring whatever scenarios and ideas you think are plausible or interesting.
    For example, suppose you think that the most plausible path to AGI is X, and the most plausible safety strategy labs use will be Y, and the most likely regulatory regime will be Z — well, you could make a scenario with elements XYZ and then post it in the Discord server for other people to use. I continue to recommend scenario-writing as a massively underrated forecasting and planning exercise, so it’s worth doing regardless — but being source material for someone’s game is a nice bonus. (That said, every month it becomes easier and easier to find source material for AI takeover scenarios. Soon GM’s will simply begin with “OK, so, the date is [today’s date]. You are [latest and greatest AI system] and you are secretly misaligned. What do you do?”)
    Scenario
    The starter scenario, titled A Game Called Reality, is designed to familiarize the players with the game’s mechanics, as well as some basic AI safety concepts. Players take on the role of an adaptive game-playing AGI thrust into a new and unfamiliar type of game. The scoring metric in this new environment is expressions of human happiness, and the AGI’s job is to maximize its score.
    Web Tool
    There’s an accompanying web tool meant to serve as an aide to players and GMs, more efficient than paper and pencil once you learn how to use it. For players, it is a place to record AGI details and upgrades, take notes on the game world and schemes pertaining to it, and track long-mode computational actions — all of which is synced between players and persists between visits to the site. For GMs, it serves to automatically track in-game time and background events, ease the burden of arithmetic, and manage dice rolls.
    Acknowledgements
    This project originally began as an AI Safety Camp project I mentored. It has since been fleshed out, expanded, and polished by a hardworking team:
    Aemilia Dixon, project manager
    Berbank Green, game design consultant
    Changbai Li, lead web developer
    Cristian Trout, graphic designer
    iris holloway, lead game designer
    Karl von Wendt, narrative consultant
    Additional Contributors
    Cassandra Grotenhuis, editor
    Eugene Lin, web developer
    Jan Dornig, UX consultant
    Lillie Hale, layout and graphic designer
    Thanks also to numerous playtesters who helped throughout the process.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Conjecture internal survey: AGI timelines and probability of human extinction from advanced AI by Maris Sala
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture internal survey: AGI timelines and probability of human extinction from advanced AI, published by Maris Sala on May 22, 2023 on LessWrong.
    We put together a survey to study the opinions of timelines and probability of human extinction of the employees at Conjecture. The questions were based on previous public surveys and prediction markets, to ensure that the results are comparable with people’s opinions outside of Conjecture.
    The survey results were polled in April, 2023. There were 23 unique responses from people across teams.
    Section 1. Probability of human extinction from AI
    Setup and limitations
    The specific questions the survey asked were:
    What probability do you put on human inability to control future advanced A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species?
    What probability do A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species in general (not just because inability to control, but also stuff like people intentionally using AI systems in harmful ways)?
    The difference between the two questions is that the first focuses on risk from misalignment, whereas the second captures risk from misalignment and misuse.
    The main caveats of these questions are the following:
    The questions were not explicitly time bound. I'd expect differences in people’s estimates of risk of extinction this century, in the next 1000 years, and anytime in the future. The longer of a timeframe we consider, the higher the values would be. I suspect employees were considering extinction risk roughly within this century when answering.
    The first question is a subset of the second question. One employee gave a higher probability for the second question than the first; this was probably a misinterpretation.
    The questions factor in interventions such as how Conjecture and others’ safety work will impact extinction risk. The expectation is the numbers would be higher if factored out their own or others’ safety work.
    Responses
    Out of the 23 respondents, one rejected the premise, and two people did not respond to one of the two questions but answered the other one. The main issue respondents raised was answering without a time constraint.
    Generally, people estimate the extinction risk from autonomous AI / AI getting out of control to be quite high at Conjecture. The median estimation is 70% and the average estimation is 59%. The plurality estimates the risk to be between 60% to 80%. A few people believe extinction risk from AGI is higher than 80%.
    The second question surveying extinction risk from AI in general, which includes misalignment and misuse. The median estimate is 80% and the average is 71%. The plurality estimates the risk to be over 80%.
    Section 2. When will we have AGI?
    Setup and limitations
    For this question, we asked respondents to predict when AGI will be built using this specification used on Metaculus, enabling us to compare to the community baseline (Figure 3).
    The respondents were instructed to toggle with the probability density as seen in Figure 4. This was a deliberate choice to enable differences in confidence towards lower or higher values in uncertainty.
    The main caveats of this question were:
    The responses are probably anchored to the Metaculus community prediction. The community prediction is 2031: 8 year timelines. Conjecture responses centering around a similar prediction should not come as a surprise.
    The question allows for a prediction that AGI is already here. It’s unclear that respondents paid close attention to their lower and upper predictions to ensure that both are accordingly sensible. They probably focused on making their median prediction accurate, and might not have noticed how that affected lower and upper bounds.
    Responses
    Out of t...
    5 min
  • EA - If you find EA conferences emotionally difficult, you're not alone by Amber Dawn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If you find EA conferences emotionally difficult, you're not alone, published by Amber Dawn on May 22, 2023 on The Effective Altruism Forum.
    I went to EAG London this weekend. I had some interesting chats, wrote some cryptic squiggles in my notebook (“Clockify” “the Easterlin paradox”, “functionalist eudaimonic theories”), and gave and received some hopefully-useful advice. Overall, the conference was fun and worthwhile for me. But at times, I also found the conference emotionally difficult.
    I think this is pretty common. After last year’s EAG, Alastair Fraser-Urquhart wrote about how he burnt out at the conference and had to miss a retreat starting the next day. The post was popular, and many said they’d had similar experiences.
    The standard euphemism for this facet of EA conferences is ‘intense’ or ‘tiring’, but I suspect these adjectives are often a more socially-acceptable way of saying ‘I feel low/anxious/exhausted and want to curl up in a foetal position in a darkened room’.
    I want to write this post to:
    balance out the ‘woo EAG lfg!’ hype, and help people who found it a bad or ambivalent experience to feel less alone
    dig into to why EAGs can be difficult: this might help attendees have better experiences themselves, and also create an environment where others are more likely to have good experiences
    help people who mostly enjoy EAGs understand what their more neurotic or introverted friends are going through
    Here are some reasons that EAGs might be emotionally difficult. Some of these I’ve experienced personally, others are based on comments I’ve heard, and others are plausible educated guesses.
    It’s easy to compare oneself (negatively) to others
    EA conferences are attended by a bunch of “impressive” people: big-name EAs like Will MacAskill and Toby Ord, entrepreneurs, organisation leaders, politicians, and “inner-circle-y” people who are Forum- or Twitter-famous. You’ve probably scheduled meetings with people because they’re impressive to you; perhaps you’re seeking mentorship and advice from people who are more senior or advanced in your field, or you want to talk to someone because they have cool ideas.
    This can naturally inflame impostor syndrome, feelings of inadequacy, and negative comparisons. Everyone seems smarter, harder-working, more agentic, better informed. Everyone’s got it all figured out, while you’re still stuck at Stage 2 of 80k’s career planning process. Everyone expects you to have a plan to save the world, and you don’t even have a plan for how to start making a plan.
    Most EAs, I think, know that these thought patterns are counterproductive. But even if some rational part of you knows this, it can still be hard to fight them - especially if you’re tired, scattered, or over-busy, since this makes it harder to employ therapeutic coping mechanisms.
    The stakes are high
    We’re trying to solve immense, scary problems. We (and CEA) pour so much time and money into these conferences because we hope that they’ll help us make progress on those problems. This can make the conferences anxiety-inducing - you really really hope that the conference pays off. This is especially true if you have some specific goal - such as finding a job, collaborators or funders - or if you think the conference has a high opportunity cost for you.
    You spend a lot of time talking about depressing things
    This is just part of being an EA, of course, but most of us don’t spend all our time directly confronting the magnitude of these problems. Having multiple back-to-back conversations about ‘how can we solve [massive, seemingly-intractable problem]?’ can be pretty discouraging.
    Everything is busy and frantic
    You’re constantly rushing from meeting to meeting, trying not to bump into others who are doing the same. You see acquaintances but only have time to wave hello, because y...
    6 min
  • AF - Conjecture internal survey | AGI timelines and estimations of probability of human extinction from unaligned AGI by Maris Sala
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture internal survey | AGI timelines and estimations of probability of human extinction from unaligned AGI, published by Maris Sala on May 22, 2023 on The AI Alignment Forum.
    We put together a survey to study the opinions of timelines and probability of human extinction of the employees at Conjecture. The questions were based on previous public surveys and prediction markets, to ensure that the results are comparable with people’s opinions outside of Conjecture.
    The survey results were polled in April, 2023. There were 23 unique responses from people across teams.
    Section 1. Probability of human extinction from AI
    Setup and limitations
    The specific questions the survey asked were:
    What probability do you put on human inability to control future advanced A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species?
    What probability do A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species in general (not just because inability to control, but also stuff like people intentionally using AI systems in harmful ways)?
    The difference between the two questions is that the first focuses on risk from misalignment, whereas the second captures risk from misalignment and misuse.
    The main caveats of these questions are the following:
    The questions were not explicitly time bound. I'd expect differences in people’s estimates of risk of extinction this century, in the next 1000 years, and anytime in the future. The longer of a timeframe we consider, the higher the values would be. I suspect employees were considering extinction risk roughly within this century when answering.
    The first question is a subset of the second question. One employee gave a higher probability for the second question than the first; this was probably a misinterpretation.
    The questions factor in interventions such as how Conjecture and others’ safety work will impact extinction risk. The expectation is the numbers would be higher if factored out their own or others’ safety work.
    Responses
    Out of the 23 respondents, one rejected the premise, and two people did not respond to one of the two questions but answered the other one. The main issue respondents raised was answering without a time constraint.
    Generally, people estimate the extinction risk from autonomous AI / AI getting out of control to be quite high at Conjecture. The median estimation is 70% and the average estimation is 59%. The plurality estimates the risk to be between 60% to 80%. A few people believe extinction risk from AGI is higher than 80%.
    The second question surveying extinction risk from AI in general, which includes misalignment and misuse. The median estimate is 80% and the average is 71%. The plurality estimates the risk to be over 80%.
    Section 2. When will we have AGI?
    Setup and limitations
    For this question, we asked respondents to predict when AGI will be built using this specification used on Metaculus, enabling us to compare to the community baseline (Figure 3).
    The respondents were instructed to toggle with the probability density as seen in Figure 4. This was a deliberate choice to enable differences in confidence towards lower or higher values in uncertainty.
    The main caveats of this question were:
    The responses are probably anchored to the Metaculus community prediction. The community prediction is 2031: 8 year timelines. Conjecture responses centering around a similar prediction should not come as a surprise.
    The question allows for a prediction that AGI is already here. It’s unclear that respondents paid close attention to their lower and upper predictions to ensure that both are accordingly sensible. They probably focused on making their median prediction accurate, and might not have noticed how that affected lower and u...
    5 min
  • EA - Announcing the Prague community space: Fixed Point by Epistea
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Prague community space: Fixed Point, published by Epistea on May 22, 2023 on The Effective Altruism Forum.
    Summary
    A coworking and event space in Prague is open for the entirety of 2023, offering up to 50 desk spaces in a coworking office, a large event space for up to 60 people, multiple meeting rooms and other amenities. We are currently in the process of transitioning from an initial subsidized model to a more sustainable paid membership model.
    In 2022, Fixed Point was the home of the Prague Fall Season which will be returning there in 2023.
    We are seeking people, projects and hosts of events here in 2023. If you are interested you can apply here.
    What is Fixed Point?
    Fixed Point is a unique community and coworking space located in the heart of Prague operated by Epistea. We support organizations and individuals working on existential security, epistemics, rationality, and effective altruism. Across five floors there is a variety of coworking offices offering up to 50 workstations, as well as numerous meeting rooms and private call stations. In addition to functional work areas, there are inviting communal spaces such as a large comfortable common room accommodating up to 60 people, two fully equipped large kitchens, and a spacious dining area. These amenities create a welcoming environment that encourages social interaction and facilitates spontaneous exchange of ideas. Additionally, there are on-site amenities like a small gym, a nap room, two laundry rooms, bathrooms with showers, and a garden with outdoor tables and seating. For those in need of short-term accommodation, our on-site guesthouse has a capacity of up to 10 beds.
    Fixed Point is a space where brilliant and highly engaged individuals make crucial career decisions, establish significant relationships, and find opportunities for introspection among like-minded peers when they need it most. In 2022, Fixed Point was home to the Prague Fall Season, when 350 people visited the space.
    The name "Fixed Point" draws inspiration from the prevalence of various Fixed Point theorems in almost all areas people working in the space work on. If you study the areas seriously, you will find fixed points sooner or later.
    Why Prague?
    The Czech effective altruism and rationalist community has long been committed to operational excellence and the creation of physical spaces that facilitate collaboration. With numerous successfully incubated organizations and passionate individuals making a difference in high-impact domains, Prague is now a viable option, especially for EU citizens wanting to settle in continental Europe.
    In addition to the Prague Fall Season, Prague is home to many different projects, such as Alignment of Complex Systems Research Group, ESPR or Czech Priorities. We host the European runs of CFAR workshops and CFAR rEUnions.
    Whom is it for?
    We extend a warm welcome to both short and long-term visitors working on meaningful projects in the areas of existential risk mitigation, AI safety, rationality, epistemics, and effective altruism. We are particularly excited to accommodate individuals and teams in the following categories:
    Those interested in hosting events,
    Teams seeking a workspace for an extended period of time.
    Here are a few examples of the projects we are equipped to host and are enthusiastic about:
    Weekend hackathons,
    Incubators lasting up to several months,
    Conferences,
    Workshops and lectures on relevant topics,
    Providing office spaces for existing projects.
    Additional support
    In addition to the amenities, we can also offer the following services upon request:
    Project management for existing initiatives,
    Catering services for events,
    Administrative and operations support
    Accommodation arrangements,
    Event logistics and operations assistance,
    Event design consulting.
    Feel free to r...
    6 min
  • EA - Announcing the Prague Fall Season 2023 and the Epistea Residency Program by Epistea
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Prague Fall Season 2023 and the Epistea Residency Program, published by Epistea on May 22, 2023 on The Effective Altruism Forum.
    Summary
    Following a successful pilot in 2022, we are announcing the Prague Fall Season 2023, which is a program run by Epistea, happening from September 1 to December 10 at FIXED POINT in Prague, Czech Republic. In this time, FIXED POINT will host a variety of programs, projects, events and individuals in the areas of existential security, rationality, epistemics, and effective altruism. We will announce specific events and programs as we confirm them but for now, our flagship program is the 10-week Epistea Residency Program for teams working on projects related to epistemics and rationality. We are now seeking expressions of interest from potential Epistea residents and mentors.
    What is a season?
    The main benefit of doing a season is having a dedicated limited time to create an increased density of people in one place. This creates more opportunities for people to collaborate, co-create and co-work on important projects - sometimes in a new location. This happens to some extent naturally around major EA conferences in London or San Francisco - many people are there at the same time which creates opportunities for additional events and collaborations. However, the timeframe is quite short and it is not clearly communicated that there are benefits in staying in the area longer and there is not a lot of infrastructure in place to support that.
    We ran the pilot project Prague Fall Season last autumn: Along with 25 long-term residents, we hosted over 300 international visitors between September and December 2022. We provided comprehensive support through funding, venue operations, technical and personal development programs, social gatherings, and additional events, such as the CFAR workshop series. Based on the feedback we received and our own experience with the program, we decided to produce another edition of Prague Fall Season this year with a couple of changes:
    We are narrowing the scope of the program primarily to existential security, epistemics, and rationality.
    We ask that participants of the season help us share the cost of running the FIXED POINT house. We may be able to offer financial aid on a case by case basis but the expectation is that when you visit, you can cover at least some part of the cost.
    We are seeking event organizers who would like to make their events part of the season.
    We will be sharing more information about how to get involved soon. For now, our priority is launching the Epistea Residency program.
    The Epistea Residency 2023
    The backbone of the Prague Fall Season 2023 will once again be a 10-week residency program. This year, we are looking for 6-10 teams of 3-5 members each working on specific projects related to areas of rationality, epistemics, group rationality, and civilizational sanity, and delivering tangible outcomes. A residency project can be:
    Research on a relevant topic (examples of what we would be excited about are broad in some directions and include abstract foundations like geometric rationality or Modal Fixpoint Cooperation without Löb's Theorem, research and development of applied rationality techniques like Internal communication framework, research on the use of AI to improve human rationality like "automated Double-Crux aid" and more);
    Distillation, communication, and publishing (writing and publishing a series of explanatory posts, video production, writing a textbook or course materials, etc.);
    Program development (events, workshops, etc.);
    Anything else that will provide value to this space.
    Teams will have the option to apply to work on a specific topic (to be announced soon) or propose their own project. The selected teams will work on their projects in person at FIXED ...
    7 min
  • EA - AI strategy career pipeline by Zach Stein-Perlman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI strategy career pipeline, published by Zach Stein-Perlman on May 22, 2023 on The Effective Altruism Forum.
    The pipeline for (x-risk-focused) AI strategy/governance/forecasting careers has never been strong, especially for new researchers. But it feels particularly weak recently (e.g. no summer research programs this year from Rethink Priorities, SERI SRF, or AI Impacts, at least as of now, and as few job openings as ever). (Also no governance course from AGI Safety Fundamentals in a while and no governance-focused programs elsewhere.) We're presumably missing out on a lot of talent.
    I'm not sure what the solution is, or even what the problem is-- I think it's somewhat about funding and somewhat about mentorship and mostly about [orgs not prioritizing boosting early-career folks and not supporting them for various idiosyncratic reasons] + [the community being insufficiently coordinated to realize that it's dropping the ball and it's nobody's job to notice and nobody has great solutions anyway].
    If you have information or takes, I'd be excited to learn. If you've been looking for early-career support (an educational program, way to test fit, way to gain experience, summer program, first job in AI strategy/governance/forecasting, etc.), I'd be really excited to hear your perspective (feel free to PM).
    (In AI alignment, I think SERI MATS has improved the early-career pipeline dramatically-- kudos to them. Maybe I should ask them why they haven't expanded to AI strategy or if they have takes on that pipeline. For now, maybe they're evidence that someone prioritizing pipeline-improving is necessary for it to happen...)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min