The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - I made a news site based on prediction markets by vandemonian
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I made a news site based on prediction markets, published by vandemonian on June 5, 2023 on The Effective Altruism Forum.
    Introduction
    “News through prediction markets”
    The Base Rate Times is a nascent news site that incorporates prediction markets prominently into its coverage.
    Please see current iteration: www.baseratetimes.com
    Twitter: www.twitter.com/base_rate_times
    What problem does it solve?
    Forecasts are underutilized by the media
    Prediction markets are more accurate than pundits, yet the media has made limited use of their forecasts. This is a big problem: one of the most rigorous information sources is being omitted from public discourse!
    The Base Rate Times creates prediction markets content, substituting for inferior news sources. This improves the epistemics of its audience.
    Forecasts are dispersed, generally inconvenient to consume
    Prediction markets are dispersed among many different platforms, fragmenting the information forecasters provide. For example, different platforms ask similar questions in different ways. Furthermore, platforms’ UX is orientated towards forecasters, not information consumers. Overall, trying to use prediction markets as ‘news replacement’ is cumbersome.
    There is value in aggregating and curating forecasts from various platforms. We need engaging ways of sharing prediction markets’ insights. The Base Rate Times aims to make prediction markets easily digestible to the general public.
    How does it work?
    News media (emotive narrative) vs Base Rate Times (actionable odds)
    For example, this is a real headline from a reputable newspaper: “Taiwan braces for China's fury over Pelosi visit”. Emotive and incendiary, it does not help you form an accurate model of the situation.
    By contrast, The Base Rate Times: “China-Taiwan conflict risk 14%, up 2x from 7% after Pelosi visit”. That's an actionable insight. It can inform your decision on whether to stay in Taiwan or to flee, for example.
    News aggregation, summarizing prediction markets
    Naturally, the probabilities in the example above come from prediction markets. The Base Rate Times presents what prediction markets are telling us about news in an engaging way.
    Stories that shift market odds are highlighted. And if a seemingly important story doesn’t shift market odds, that also tells you something.
    On The Base Rate Times, right now you can see the latest odds on:
    Putin staying in power
    Russian territorial gains in Ukraine
    Escalation risk of NATO involvement
    and more...
    By glancing at a few charts, you can form a more accurate model (in less time) of Russia-Ukraine than reading countless narrative-based news stories.
    Inspiration
    A key inspiration was Scott Alexander’s Prediction Market FAQ:
    I recently had to read many articles on Elon Musk’s takeover of Twitter, which all repeated that “rumors said” Twitter was about to go down because of his mass firing. Meanwhile, there were several prediction markets on whether this would happen, and they were all around 40%. If some journalist had thought to check the prediction markets and cite them in their article, they could have not only provided more value (a clear percent chance instead of just “there are some rumors saying this”), but also been right when everyone else was wrong.
    Also Scott’s 'Mantic Monday' posts and Zvi’s blog.
    This simple chart by @ClayGraubard was another inspiration. Wanted something like this, but for all major news stories. Couldn't find it, so making it myself. (Clay is making geopolitics videos and podcasts now, check it out.)
    Goals
    Like 538, but for prediction markets
    The Base Rate Times is a bet that forecasts can be popularized, as opinion polls have been, and improve society’s models of the world.
    Goal: Longshot probability of going mainstream, e.g. like 538.
    If highly successful in scaling, we’d be effectively run...
    7 min
  • LW - We Are Less Wrong than E. T. Jaynes on Loss Functions in Human Society by Zack M Davis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We Are Less Wrong than E. T. Jaynes on Loss Functions in Human Society, published by Zack M Davis on June 5, 2023 on LessWrong.
    These paragraphs from E. T. Jaynes's Probability Theory: The Logic of Science (in §13.12.2, "Loss functions in human society") are fascinating from the perspective of a regular reader of this website:
    We note the sharp contrast between the roles of prior probabilities and loss functions in human relations. People with similar prior proabilities get along well together, because they have about the same general view of the world and philosophy of life. People with radically different prior probabilities cannot get along—this has been the root cause of all the religious wars and most of the political repressions throughout history.
    Loss functions operate in just the opposite way. People with similar loss functions are after the same thing, and are in contention with each other. People with different loss functions get along well because each is willing to give something the other wants. Amicable trade or business transactions, advantageous to all, are possible only between parties with very different loss functions. We illustrated this by the example of insurance above.
    (Jaynes writes in terms of loss functions for which lower values are better, whereas we more often speak of utility functions for which higher values are better, but the choice of convention doesn't matter—as long as you're extremely sure which one you're using.)
    The passage is fascinating because the conclusion looks so self-evidently wrong from our perspective. Agents with the same goals are in contention with each other? Agents with different goals get along? What!?
    The disagreement stems from a clash of implicit assumptions. On this website, our prototypical agent is the superintelligent paperclip maximizer, with a utility function about the universe—specifically, the number of paperclips in it—not about itself. It doesn't care who makes the paperclips. It probably doesn't even need to trade with anyone.
    In contrast, although Probability Theory speaks of programming a robot to reason as a rhetorical device[1], this passage seems to suggest that Jaynes hadn't thought much about how ideal agents might differ from humans? Humans are built to be mostly selfish: we eat to satisfy our own hunger, not as part of some universe-spanning hunger-minimization scheme. Besides being favored by evolution, selfish goals do offer some conveniences of implementation: my own hunger can be computed as a much simpler function of my sense data than someone else's. If one assumes that all goals are like that, then one reaches Jaynes's conclusion: agents with similar goal specifications are in conflict, because the specified objective (for food, energy, status, whatever) binds to an agent's own state, not a world-model.
    But ... the assumption isn't true! Not even for humans, really—sometimes people have "similar loss functions" that point to goals outside of themselves, which benefit from more agents having those goals. Jaynes is being silly here.
    That said—and no offense—the people who read this website are not E. T. Jaynes; if we can get this one right where he failed, it's because our subculture happened to inherit an improved prior in at least this one area, not because of our innate brilliance or good sense. Which prompts the question: what other misconceptions might we be harboring, due to insufficiently general implicit assumptions?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Meta-conversation shouldn't be taboo by Adam Zerner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta-conversation shouldn't be taboo, published by Adam Zerner on June 5, 2023 on LessWrong.
    I often find myself in a conversation that feels like it's falling into some sort of failure mode. Some examples:
    A few years ago in some job as a programmer, I remember being in this meeting. We were trying to figure out the architecture of some feature that was important and we were behind on. It shouldn't have been that hard, but the people who were working on it just kept running into issues, so someone called a big meeting with maybe 12 people in it to just "get in a room together and figure it out". But even in that meeting we weren't figuring it out. I think a big reason why is because there were too many chefs in the kitchen.
    At work a few months ago I recall a meeting where Bob (let's call him), continuously interrupted Alice. In such a way where Alice wasn't really ever able to express her thoughts. And I wanted to hear her thoughts. Bob had objections to all of her thoughts, hence the interruptions, but I didn't think they were strong enough objections to justify the interruptions.
    A few years ago there was this conversation that has always stuck with me. I was visiting my mom with my girlfriend. The three of us were chatting. I'm not big on interrupting. I do it when it seems worth it, but try to avoid it. But they just constantly interrupted each other. Like, one of them would be about 80% finished saying what they were trying to say, and the other would interrupt and start talking. I couldn't get a word in.
    This one is hard to describe, but... ok. When you're talking to someone, they talk, they finish what they're saying, they pause, and then you talk. Some people pause for, what seems to me, a particularly long period of time, and then continue talking. And during that pause I assume they're done and I start talking, and we end up talking over each other.
    A few months ago I was at Enthea Teahouse, hanging out with some fellow rationalists. At first we were all sitting around the same table downstairs, but then as the group got larger, we split up, with half of us moving upstairs. Once we moved upstairs, two non-rationalists joined us. A girl and a guy who were boyfriend and girlfriend. For the next 2-3 hours, the girl dominated the conversation. She probably did about 80% of the talking in a group of maybe 7 of us. It was an energetic, fast-paced, and rushed type of talking. She'd also interrupt a lot. At one point she even mentioned that she gets social anxiety and responds to that by talking too much, and briefly apologized. After that I remember her boyfriend smiling and putting his hand on her leg in a way that said "you can take your foot off the gas pedal now". But she didn't.
    In each of these situations, I wish that I could have spoken up and initiated a conversation about the conversation. A meta-conversation. Something like this:
    "I feel like this isn't working out because there are too many chefs in the kitchen. Maybe it'd be better if we just had 3-4 of us figure this out? Or split into, say, four groups of three, each group comes up with a solution, and we then meet back up and compare the solutions?"
    "Hey Bob, I notice that you've been interrupting Alice. I know you probably are thinking that they're justified because your interruptions are steering the conversation in a more productive direction, but I actually suspect that they aren't and would like to at least test that theory by giving Alice a little more space to express her thoughts."
    In this situation I actually felt comfortable enough with the both of them where I did eventually speak up. I explained that they both just keep interrupting each other and I can't get a word in. They laughed and said there's nothing wrong with it and that's just how they talk and I should feel free to do the sa...
    7 min
  • EA - Abolishing factory farming in Switzerland: Postmortem by naoki
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Abolishing factory farming in Switzerland: Postmortem, published by naoki on June 5, 2023 on The Effective Altruism Forum.
    The Initiative to abolish factory farming was a nationwide ballot in Switzerland, instigated by Sentience Politics.
    The contents of the postmortem below were written by Philipp Ryf, then co-president at Sentience Politics and co-lead of the campaign, for the 2022 annual report of Sentience Politics. The author of this post is Naoki Peter, co-president at Sentience Politics.
    The initiative at a glance
    The initiative demanded the abolition of factory farming in Switzerland, granting a maximum transitional period of 25 years.
    It aimed at anchoring stricter animal welfare guidelines as a new minimum standard in the Swiss constitution. These standards would have granted cows, pigs and chickens regular access to the outdoors and considerably more space.
    On behalf of Swiss farmers the initiative included import regulations that take account of the new Swiss standards.
    The initiative was rejected by a 62.9% majority of the voters on in September 2022.
    For more information see the initiative text (in German, French and Italian) and the ballot results.
    Postmortem
    The living conditions of animals in agriculture have never been discussed so widely and publicly. Hundreds of thousands of people have engaged with the initiative beyond their usual scope, asking themselves the question: "What does my consumption mean for animals, people, and the environment?" The voting result has proven that the vision of a dignified, location-appropriate agriculture mobilises and moves people far beyond the base of the supporting parties.
    For the first time a broad alliance of Swiss animal protection, agricultural, and environmental organisations joined forces and stood up to the agricultural lobby to advocate for an animal welfare cause.
    Considering the high cost and resources required to launch an initiative, we aimed at including as many demands as possible. However, we now question whether a shorter, more targeted package of demands would have garnered more support. We will consider this for any future initiatives.
    In 2018 Sentience Politics launched the Initiative to abolish factory farming single-handedly. We should have prioritised alliance-building earlier. This would have enabled us to tap into the resources of more organisations.
    The opposition commanded resources which significantly exceeded our own. In order to have a realistic chance at the ballot box, any future initiative would likely require a substantially larger budget and campaign team.
    In order to inspire the majority of the population for a cause, it is essential to highlight the urgency of the matter and exclude any room for doubt. We could have done this better. For future campaigns, we must ensure that the actual conditions in animal farming are made transparent sooner. Possibilities include a greater focus on highlighting scandals, publishing our own investigations, and doing more to attract media exposure.
    Regional volunteer networks are very challenging to build. Here, we could have involved organisations that already have structures in place to manage regional leafleting, stand campaigns and billboards more effectively and at an earlier stage.
    We built up a large network of influential personalities from society. However, we lacked credible ambassadors from agriculture who could more readily be trusted as experts in their field until a late stage in the campaign. The clearest take-home message from the follow-up surveys was that policy change in agriculture does not work without the producers. In the future, we need to involve stakeholders within agriculture in our campaigns at an earlier stage.
    Alliance against factory farming
    The initiative to abolish factory farming was not successful at the ballo...
    5 min
  • LW - Wildfire of strategicness by TsviBT
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wildfire of strategicness, published by TsviBT on June 5, 2023 on LessWrong.
    It may not be feasible to make a mind that makes achievable many difficult goals in diverse domains, without the mind also itself having large and increasing effects on the world. That is, it may not be feasible to make a system that strongly possibilizes without strongly actualizing.
    But suppose that this is feasible, and there is a mind M that strongly possibilizes without strongly actualizing. What happens if some mental elements of M start to act strategically, selecting, across any available domain, actions predicted to push the long-term future toward some specific outcome?
    The growth of M is like a forest or prairie that accumulates dry grass and trees over time. At some point a spark ignites a wildfire that consumes all the accumulated matter.
    The spark of strategicness, if such a thing is possible, recruits the surrounding mental elements. Those surrounding mental elements, by hypothesis, make goals achievable. That means the wildfire can recruit these surrounding elements toward the wildfire's ultimate ends. By recruiting more dry matter to the wildfire, the wildfire burns hotter and spreads further.
    Also by hypothesis, the surrounding mental elements don't themselves push strongly for goals. Seemingly, that implies that they do not resist the wildfire, since resisting would constitute a strong push. We can at least say that, if the totality of the mental elements surrounding the wildfire is going to notice and suppress the wildfire, it would have to think at least strategically enough to notice and close off all the sneaky ways by which the wildfire might wax. This implies that the surrounding mental elements do a lot of thinking and have a lot of understanding relevant to strategic takeovers, which itself seemingly makes more available the knowledge needed for strategic takeovers.
    Capabilities overhang provides fuel for a wildfire of strategicness. That implies that it's not so easy to avoid wrapper-minds.
    This is very far from being a watertight argument, and it would be nice if the conclusion were false; how is it false? Maybe, as is sometimes suggested, minds selected to be creative are selected to keep themselves open to new ideas and therefore to new agency, implying that they're selected to prevent strategic takeovers, which would abhor new agency?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Wildfire of strategicness by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wildfire of strategicness, published by Tsvi Benson-Tilsen on June 5, 2023 on The AI Alignment Forum.
    It may not be feasible to make a mind that makes achievable many difficult goals in diverse domains, without the mind also itself having large and increasing effects on the world. That is, it may not be feasible to make a system that strongly possibilizes without strongly actualizing.
    But suppose that this is feasible, and there is a mind M that strongly possibilizes without strongly actualizing. What happens if some mental elements of M start to act strategically, selecting, across any available domain, actions predicted to push the long-term future toward some specific outcome?
    The growth of M is like a forest or prairie that accumulates dry grass and trees over time. At some point a spark ignites a wildfire that consumes all the accumulated matter.
    The spark of strategicness, if such a thing is possible, recruits the surrounding mental elements. Those surrounding mental elements, by hypothesis, make goals achievable. That means the wildfire can recruit these surrounding elements toward the wildfire's ultimate ends. By recruiting more dry matter to the wildfire, the wildfire burns hotter and spreads further.
    Also by hypothesis, the surrounding mental elements don't themselves push strongly for goals. Seemingly, that implies that they do not resist the wildfire, since resisting would constitute a strong push. We can at least say that, if the totality of the mental elements surrounding the wildfire is going to notice and suppress the wildfire, it would have to think at least strategically enough to notice and close off all the sneaky ways by which the wildfire might wax. This implies that the surrounding mental elements do a lot of thinking and have a lot of understanding relevant to strategic takeovers, which itself seemingly makes more available the knowledge needed for strategic takeovers.
    Capabilities overhang provides fuel for a wildfire of strategicness. That implies that it's not so easy to avoid wrapper-minds.
    This is very far from being a watertight argument, and it would be nice if the conclusion were false; how is it false? Maybe, as is sometimes suggested, minds selected to be creative are selected to keep themselves open to new ideas and therefore to new agency, implying that they're selected to prevent strategic takeovers, which would abhor new agency?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Ages Survey: Results by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ages Survey: Results, published by jefftk on June 5, 2023 on LessWrong.
    A few weeks ago after the
    community meeting with the Department of Children and Families I decided to run a survey to learn more about the range of ages at which people generally thought kids might be ready to do various activities without supervision. Stay home for a few hours? Play at the park alone? Take public transit? I put something together on
    Google Forms and shared it
    on
    Facebook and over email. Several friends shared it further, and over the next two weeks it gathered 219 responses. In reading the following please remember that this is not a representative sample of anything: it's heavily oversampling people geographically and socially similar to me!
    The main part of the survey was a series of questions about scenarios
    ("play in an unfenced backyard", "cross a medium-traffic street", "spend the night home alone") and asking "what age you'd guess the typical child in your area could learn to do it without supervision". Here are the average responses:
    The survey also asked "the ages you'd guess for an especially mature child (90th percentile) or an especially immature child (10th percentile)", assuming "the child doesn't have any relevant disabilities." Here's that chart again, with the addition of the estimates for especially mature or immature children:
    Taking medians is helpful for distilling each question down to a few numbers, but the wide range of responses are interesting. Here are CDF plots for each question. The way to read these is that at a given age (x-axis) they tell you what fraction of parents (y-axis) think a child this old is likely ready. You can think of the previous bar chart as a collection of the the y=50% slices from each of the charts below:
    I'm also interested in how people's backgrounds relate to their answers in these scenarios. First, let's look at what demographics we saw. The median respondent was 40 when they filled out the survey:
    Most respondents were female (83%), followed by male (13%), and non-binary (5%).
    Living in an urban area, most of the people I was able to survey also lived in an urban area:
    They overwhelmingly grew up in more suburban areas, though:
    Most respondents were parents (88%) and the most common number of children was 2 (45%):
    The median parent's oldest child was 8 and most of the parents had an oldest child under 18 (92%), but there were a few parents of grown children:
    So, how do these demographics interact with what sorts of ages people gave? Here is how the respondents differed from the mean, in years. For example, a dot at +1y indicates that this respondent on average gave answers one year larger than others. We can see that women generally gave ~1y higher ages than men, with non-binary respondents in between:
    The orange bar is median, green triangle is mean.
    Normally you'd use z-scores for this, where the x-axis is "standard deviations away from the mean" but in this case the standard deviations are similar between questions and I think it easier to think about with the x-units in years. This does mean that questions with more variance effectively have a bit more weight in determining how to classify each respondent.
    Here's the same chart, but by respondent ages:
    Ignoring the four responses from kids because of the small sample size, it looks like older people generally gave higher ages. I don't know if this is because they're farther from having kids in the relevant age ranges, have become more conservative over time, or were drawn from a somewhat different population than the younger people in the sample? If we ran this survey decades ago I'd expect to generally see younger ages, so this is a counter-intuitive result!
    Here's by location:
    It looks like people who live in exurban or rural areas, generally gave higher ages. This is...
    8 min
  • LW - The ants and the grasshopper by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The ants and the grasshopper, published by Richard Ngo on June 4, 2023 on LessWrong.
    One winter a grasshopper, starving and frail, approaches a colony of ants drying out their grain in the sun, to ask for food.
    “Did you not store up any food during the summer?” the ants ask.
    “No”, says the grasshopper. “I lost track of time, because I was singing and dancing all summer long.”
    The ants, disgusted, turn away and go back to work.
    One winter a grasshopper, starving and frail, approaches a colony of ants drying out their grain in the sun, to ask for food.
    “Did you not store up any food during the summer?” the ants ask.
    “No”, says the grasshopper. “I lost track of time, because I was singing and dancing all summer long.”
    The ants are sympathetic. “We wish we could help you”, they say, “but it sets up the wrong incentives. We need to conditionalize our philanthropy to avoid procrastination like yours leading to a shortfall of food.”
    And they turn away and go back to their work, with a renewed sense of purpose.
    ...And they turn away and go back to their work, with a flicker of pride kindling in their minds, for being the types of creatures that are too clever to help others when it would lead to bad long-term outcomes.
    ...And they turn away and go back to their work—all except for one, who brushes past the grasshopper and whispers “Meet me outside at dusk and I’ll bring you food. We can preserve the law and still forgive the deviation.”
    ...“Did you not store up any food during the summer?” the ants ask.
    “Of course I did”, the grasshopper says. “But it was all washed away by a flash flood, and now I have nothing.”
    The ants express their sympathy, and feed the grasshopper abundantly. The grasshopper rejoices, and tells others of the kindness and generosity shown to it. The ants start to receive dozens of requests for food, then hundreds, each accompanied by a compelling and tragic story of accidental loss. The ants cannot feed them all; they now have to assign additional workers to guard their doors and food supplies, and rue the day they ever gave food to the grasshopper.
    ...The ants start to receive dozens of requests for food, then hundreds, each accompanied by a compelling and tragic story of accidental loss—and while many are fraudulent, enough are real that they are moved to act. In order to set incentives correctly, the ants decide to only give food to those who can prove that they lost their food supplies through no fault of their own, and set up a system for vetting claims.
    This works well for a time—but as fraudsters grow more sophisticated, the ants’ bureaucratic requirements grow more onerous. In order to meet them, other creatures start to deposit their food in large group storehouses which can handle the administrative overhead. But now the food supply is exposed to systemic risk if the leaders of those storehouses make poor decisions, whether from carelessness or greed.
    One year several storehouses fail; in trying to fill the shortfall, the ants almost run out of food for themselves. To avoid that ever happening again, they set up stringent regulations and oversight of permissible storehouses, funded by taxes levied throughout the year. At first this takes only a small proportion of their labor—but as their regulatory apparatus inevitably grows, they need to oversee more and more aspects of the ecosystem, and are called upon to right more and more injustices.
    Eventually the ants—originally the most productive of all creatures—stop producing any food of their own, so busy are they in tending to the system they’ve created. They forget the mud and the muck of working the harvest, and are too preoccupied to hear feedback from those they're trying to help. And some, despite starting out with the best of intentions, are swept away by the heady rush of wielding power, ...
    9 min
  • LW - Announcing AISafety.info's Write-a-thon (June 16-18) and Second Distillation Fellowship (July 3-October 2) by steven0461
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing AISafety.info's Write-a-thon (June 16-18) and Second Distillation Fellowship (July 3-October 2), published by steven0461 on June 3, 2023 on LessWrong.
    Stampy’s AI Safety Info is an interactive FAQ started by Rob Miles that aims to be the best one-stop source of information about AI existential safety, gathering summaries and links on each of hundreds of subtopics.
    Now that the first fellowship is nearing an end, we’re preparing for a second distillation fellowship from July 3rd through October 2nd. Distillation fellows will be working with each other and with me to improve the site further by editing the existing answers and writing new ones. They’ll be paid $2500 per month for three months working full-time as remote contractors. (Partial fellowships will probably be possible as well.) We’ll have at least five spots for fellows, and maybe more.
    We’re also announcing a hackathon-style event, which we’re organizing together with Apart Research, on the 16th, 17th, and 18th of June. People are invited to write and edit articles during whichever parts of those days best fit their schedules. (If you’re unable to attend on those days, you can still participate by choosing another contiguous three-day period between now and the event—for example, next weekend—to write all your entries in.)
    Collaboration on the event will take place on Discord as well as on gather.town. I’ll be online for those three days to lead the event and answer any questions. We encourage people to work together on entries during that time, but we encourage one individual to be in charge of each entry.
    Submitting good articles and helping other people write their articles in the write-a-thon will be the main criteria we’ll use to select distillation fellows. There will also be prize money for the best entries: $1000, $600, $300, and $100 for the best four, and $200 for one randomly chosen (serious) entry.
    We will have a more specific list of suggested questions to use for entries later, but for now, you can look at the “Not started” part of Stampy’s list of questions, and take a look at existing articles and the style guidelines for what an article should look like. (You can also use a new question of your own devising; to make sure you’re not wasting your effort on a question that won't fit on the site, you can ask on Discord or check with me.)
    We wanted to put this announcement up without waiting longer, but we’re still finalizing some of the details, so check back in the coming days for further details on submission instructions, an application form, and the like.
    If you want to help out in some other way, AISafety.info welcomes donations (details soon), volunteer editors, and volunteer coders.
    If you have questions about the write-a-thon, the fellowship, or anything else, feel free to ask in the comments or message me here or on Discord (StevenK#3458).
    Keep track of who else was helpful; there will be a rank-ordering question in the application form for who you’d most like to work with. Making a joint entry with multiple people is also possible, but makes it harder to assign credit.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - The Control Problem: Unsolved or Unsolvable? by Remmelt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Control Problem: Unsolved or Unsolvable?, published by Remmelt on June 2, 2023 on LessWrong.
    td;lr No control method exists to safely contain the global feedback effects of self-sufficient learning machinery. What if this control problem turns out to be an unsolvable problem?
    Where are we two decades into resolving to solve a seemingly impossible problem?
    If something seems impossible. well, if you study it for a year or five, it may come to seem less impossible than in the moment of your snap initial judgment.
    Eliezer Yudkowsky, 2008
    A list of lethalities.we are not on course to solve in practice in time on the first critical try; none of it is meant to make a much stronger claim about things that are impossible in principle
    Eliezer Yudkowsky, 2022
    How do you interpret these two quotes, by a founding researcher, fourteen years apart?
    A. We indeed made comprehensive progress on the AGI control problem, and now at least the overall problem does not seem impossible anymore.
    B. The more we studied the overall problem, the more we uncovered complex sub-problems we'd need to solve as well, but so far can at best find partial solutions to.
    Which problems involving physical/information systems were not solved after two decades?
    Oh ye seekers after perpetual motion, how many vain chimeras have you pursued? Go and take your place with the alchemists.
    Leonardo da Vinci, 1494
    No mathematical proof or even rigorous argumentation has been published demonstrating that the A[G]I control problem may be solvable, even in principle, much less in practice.
    Roman Yampolskiy, 2021
    We cannot rely on the notion that if we try long enough, maybe AGI safety turns out possible after all.
    Historically, many researchers and engineers tried to solve problems that turned out impossible:
    perpetual motion machines that both conserve and disperse energy.
    uniting general relativity and quantum mechanics into some local variable theory.
    singular methods for 'squaring the circle', 'doubling the cube' or 'trisecting the angle'.
    distributed data stores where messages of data are consistent in their content, and also continuously available in a network that is also tolerant to partitions.
    formal axiomatic systems that are consistent, complete and decidable.
    Smart creative researchers of their generation came up with idealized problems. Problems that, if solved, would transform science, if not humanity. They plowed away at the problem for decades, if not millennia. Until some bright outsider proved by contradiction of the parts that the problem is unsolvable.
    Our community is smart and creative – but we cannot just rely on our resolve to align AI. We should never forsake our epistemic rationality, no matter how much something seems the instrumentally rational thing to do.
    Nor can we take comfort in the claim by a founder of this field that they still know it to be possible to control AGI to stay safe.
    Thirty years into running a program to secure the foundations of mathematics, David Hilbert declared “We must know. We will know!” By then, Kurt Gödel had constructed the first incompleteness theorem. Hilbert kept his declaration for his gravestone.
    Short of securing the foundations of safe AGI control – that is, through empirically-sound formal reasoning – we cannot rely on any researcher's pithy claim that "alignment is possible in principle".
    Going by historical cases, this problem could turn out solvable. Just really, really hard to solve. The flying machine seemed an impossible feat of engineering. Next, controlling a rocket’s trajectory to the moon seemed impossible.
    By the same reference class, ‘long-term safe AGI’ could turn out unsolvable – the perpetual motion machine of our time. It takes just one researcher to define the problem to be solved, reason from empirically sound premises, and arrive ...
    27 min