The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - High Status Eschews Quantification of Performance by niplav
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High Status Eschews Quantification of Performance, published by niplav on March 19, 2023 on LessWrong.
    In a recent episode of The Filan
    Cabinet, Oliver Habryka elaborated on a very interesting social pattern: If you have a community with high status people, and try to introduce clearer metrics of performance into that community, high status individuals in the community will strongly resist those metrics because they have an asymmetric downside: If they perform well on the metric, they stay in their position, but if they perform poorly, they might lose status. Since they are at least a little bit unsure about their performance on the metric relative to others, they can only lose.
    Daniel Filan: So let's go back to what you think on your bad days. So you mentioned that you had this sense that lots of things in the world were, I don't know, trying to distract you from things that are true or important. And that LessWrong did that somewhat less.
    Oliver Habryka: Yeah.
    Daniel Filan: Can you kind of flesh that out? What kinds of things are you thinking of?
    Oliver Habryka: I mean, the central dimension that I would often think about here is reputation management. As an example, the medical profession, which, you know, generally has the primary job of helping you with your medical problems and trying to heal you of diseases and various other things, also, at the same time, seems to have a very strong norm of mutual reputation protection. Where, if you try to run a study trying to figure out which doctors in the hospital are better or worse than other doctors in the hospital, quite quickly, the hospital will close its ranks and be like, “Sorry, we cannot gather data on [which doctors are better than the other doctors in this hospital].” Because that would, like, threaten the reputation arrangement we have. This would introduce additional data that might cause some of us to be judged and some others of us to not be judged.
    And my sense is the way that usually looks like from the inside is an actual intentional blinding to performance metrics in order to both maintain a sense of social peace, and often the case because... A very common pattern here [is] something like, you have a status hierarchy within a community or a local institution like a hospital. And generally, that status hierarchy, because of the way it works, has leadership of the status hierarchy be opposed to all changes to the status hierarchy. Because the current leadership is at the top of the status hierarchy, and so almost anything that we introduce into the system that involves changes to that hierarchy is a threat, and there isn't much to be gained, [at least in] the zero-sum status conflict that is present.
    And so my sense is, when you try to run these studies about comparative doctor performance, what happens is more that there's an existing status hierarchy, and lots of people feel a sense of uneasiness and a sense of wanting to protect the status quo, and therefore they push back on gathering relevant data here. And from the inside this often looks like an aversion to trying to understand what are actually the things that cause different doctors to be better than other doctors. Which is crazy, if you're, like, what is the primary job of a good medical institution and a good medical profession, it would be figuring out what makes people be better doctors and worse doctors. But [there are] all of the social dynamics that tend to be present in lots of different institutions that make it so that looking at relative performance [metrics] becomes a quite taboo topic and a topic that is quite scary.
    So that's one way [in which] I think many places try to actively... Many groups of people, when they try to orient and gather around a certain purpose, actually [have a harder time] or get blinded or in some sense get...
    9 min
  • AF - Probabilistic Payor Lemma? by Abram Demski
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Probabilistic Payor Lemma?, published by Abram Demski on March 19, 2023 on The AI Alignment Forum.
    Epistemic status: too good to be true? Please check my math.
    We've known for a while that Löb's theorem fails when proof is relaxed to probabilistic belief. This has pros and cons. On the pro side, it means there's no Löbian Obstacle to probabilistic self-trust. On the con side, it means that some Löb-derived insights for proof-based decision theory don't translate to probabilistic decision theory, at least not as directly as one might hope. In particular, it appeared to dash hopes for probabilistic generalizations of the "Löbian handshake" for cooperation.
    Recently, Andrew Critch wrote about the Payor Lemma, which allows for a very similar "modal handshake" without Löb's Theorem. The lemma was proved using the same modal assumptions as Löb's, so on the surface it may appear to be just a different method to achieve similar results, whose main advantage is that it is much easier to prove (and therefore explain and understand) than Löb's Theorem.
    But, a natural question arises: does Payor's Lemma have a suitable probabilistic version?
    I'll give an affirmative proof; but I haven't confirmed that the assumptions are reasonable to my satisfaction.
    Setup
    Let L be a language in first-order logic, expressive enough to represent its sentences s∈L as quoted terms ┌s┐, eg, through Gödel numbering; and with a probability function symbol on these terms, p(┌s┐), which can be equated with (some representation of) rational numbers, e.g. p(┌⊤┐)=1, p(┌s┐)=12, etc. I also assume the system can reason about these rational numbers in the basic ways you'd expect.
    For all a,b∈L and all r∈Q, we have:
    If ⊢a, then ⊢p(┌a┐)=1.
    If ⊢ab, then ⊢p(┌a┐)≤p(┌b┐).
    (These assumptions might look pretty minimal, but they aren't going to be true for every theory of self-referential truth; more on this later.)
    Let B(s) abbreviate the sentence p(┌s┐)>c for any s and some globally fixed constant c strictly between 0 and 1. This is our modal operator.
    Some important properties of B:
    Necessitation. If ⊢s, then ⊢B(s), for any s.
    Proof: Since ⊢s implies ⊢p(s)=1, and c∈(0,1), we have ⊢p(┌s┐)>c,, which is to say, ⊢B(s). [End proof.]
    Weak distrubitivity. If ⊢xy, then ⊢B(x)B(y).
    Proof: When ⊢xy, we have ⊢p(y)≥p(x), so ⊢p(x)>cp(y)>c. [End proof.]
    (Regular distributivity would say B(xy) implies B(x)B(y). The assumption ⊢xy is stronger than B(xy), so the above is a weaker form of distributivity.)
    Theorem Statement
    If ⊢B(B(x)x)x, then ⊢x.
    Proof
    ⊢x(B(x)x), by tautology (a(ba)).
    So ⊢B(x)B(B(x)x), from 1 by weak distributivity.
    Suppose ⊢B(B(x)x)x.
    ⊢B(x)x from 2 and 3.
    ⊢B(B(x)x) from 4 by necessitation.
    ⊢x from 4 and 1.[End proof.]
    Discussion
    Comparison to Original Proof
    The proof steps mirror Critch's treatment very closely. The key difference is step 2, IE, how I obtain a statement like ⊢□x□(□xx). Critch uses distributivity, which is not available to me:
    B(ab)(B(a)B(b))?
    Suppose B(ab), ie, p(┌ab┐)>c.
    Rewrite p(┌b∨¬a┐)>c.
    Now suppose B(a), that is, p(┌a┐)>c.
    p(┌¬a┐)<1−c.
    p(┌b∨¬a┐)≤p(┌b┐)+p(┌¬a┐)p(┌b∨¬a┐)−1+c>c−1+c.
    p(┌b┐)>2c−1.
    So we only get:
    Bc(ab)(Bc(a)Bd(b)),
    where Br(s) abbreviates p(┌s┐)>r and we have d=2c−1.
    So in general, attempted applications of distributivity create weakened belief operators, which would get in the way of the proof (very similar to how probabilistic Löb fails).
    However, the specific application we want happens to go through, due to a logical relationship between a and b; namely, that b is a weaker statement than a.
    This reveals a way in which the assumptions for Payor's Lemma are importantly weaker than those required for Löb to go through.
    So, the key observation I'm making is that weak distributility is all that's needed for Payor, and seems much more plaus...
    7 min
  • LW - Shell games by TsviBT
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shell games, published by TsviBT on March 19, 2023 on LessWrong.
    [Metadata: crossposted from. First completed November 18, 2022.]
    Shell game
    Here's the classic shell game: Youtube
    Screenshot from that video.
    The little ball is a phantom: when you look for it under a specific shell, it's not there, it's under a different shell.
    (This might be where the name "shell company" comes from: the business dealings are definitely somewhere, just not in this company you're looking at.)
    Perpetual motion machines
    Related: Perpetual motion beliefs
    Bhāskara's wheel is a proposed perpetual-motion machine from the Middle Ages:
    Here's another version:
    From this video.
    Someone could try arguing that this really is a perpetual motion machine:
    Q: How do the bars get lifted up? What does the work to lift them?
    A: By the bars on the other side pulling down.
    Q: How does the wheel keep turning? How do the bars pull more on their way down than on their way up?
    A: Because they're extended further from the center on the downward-moving side than on the upward-moving side, so they apply more torque to the wheel.
    Q: How do the bars extend further on the way down?
    A: Because the momentum of the wheel carries them into the vertical bar, flipping them over.
    Q: But when that happens, energy is expended to lift up the little weights; that energy comes out of the kinetic energy of the wheel.
    A: Ok, you're right, but that's not necessary to the design. All we need is that the torque on the downward side is greater than the torque on the upward side, so instead of flipping the weights up, we could tweak the mechanism to just shift them outward, straight to the side. That doesn't take any energy because it's just going straight sideways, from a resting position to another resting position.
    Q: Yeah... you can shift them sideways with nearly zero work... but that means the weights are attached to the wheel at a pivot, right? So they'll just fall back and won't provide more torque.
    A: They don't pivot, you fix them in place so they provide more torque.
    Q: Ok, but then when do you push the weights back inward?
    A: At the bottom.
    Q: When the weight is at the bottom? But then the slider isn't horizontal, so pushing the weight back towards the center is pushing it upward, which takes work.
    A: I meant, when the slider is at the bottom--when it's horizontal.
    Q: But if the sliders are fixed in place, by the time they're horizontal at the bottom, you've already lifted the weights back up some amount; they're strong-torquing the other way.
    A: At the bottom there's a guide ramp to lift the weights using normal force.
    Q: But the guide ramp is also torquing the wheel.
    And so on. The inventor can play hide the torque and hide the work.
    Shell games in alignment
    Some alignment schemes--schemes for structuring or training an AGI so that it can be transformatively useful and doesn't kill everyone--are prone to playing shell games. That is, there's some features of the scheme that don't seem to happen in a specific place; they happen somewhere other than where you're looking at the moment. Consider these questions:
    What sort of smarter-than-human work is supposed to be done by the AGI? When and how does it do that work--by what combination of parts across time?
    How does it become able to do that work? At what points does the AGI come to new understanding that it didn't have before?
    How does the AGI orchestrate it's thinking and actions to have large effects on the world? By what process, components, rules, or other elements?
    What determines the direction that the AGI's actions will push the world? Where did those determiners come from, and how exactly do they determine the direction?
    Where and how much do human operators have to make judgements? How much are those judgements being relied on to point to goodness, truth, aligned...
    7 min
  • LW - Wonder about the hard parts of the alignment problem by Mikhail Samin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wonder about the hard parts of the alignment problem, published by Mikhail Samin on March 18, 2023 on LessWrong.
    My p(doom) is pretty high and I found myself repeating the same words to explain some parts of the intuitions behind it. I think there are hard parts of the alignment problem that we’re not on track to solve in time. Alignment plans that I've heard fail for reasons connected to these hard parts of the problem, so I decided to attempt to write my thoughts in a short post.
    (Thanks to Theresa, Owen, Jonathan, and David for comments on a draft.)
    Modern machine learning uses a powerful search process to look for neural network parameters such that a neural network performs well on some function.
    There exist algorithms for general and powerful agents. At some point in the near future, there will be a training procedure with the gradient of the loss function(s) w.r.t. the parameters pointing towards neural networks implementing these algorithms.
    Increasingly context-aware and capable agents achieve a better score on a wide range of scoring functions than their neighbors and will, by default, attract gradient descent.
    Unfortunately, we haven’t solved agent foundations: we have these powerful search processes, and if you imagine the space of all possible AGIs (or possible neural networks, or possible minds), there are some areas that are aligned AGIs, but we have no idea how to define them, no idea how to look for them. We understand how all designs for a search process people came up with so far end up somewhere that’s not in an area of aligned AGI, and we also understand that some areas with aligned AGIs actively dispel many sorts of search processes. We can compare an area of aligned AGIs to the Moon. Imagine we’re trying to launch a rocket there, and if after the first take-off, it ends up somewhere that’s not the Moon (maybe after a rapid unplanned disassembly), we die. We have a bunch of explosives, but we don’t have equations for gravity, only maybe some initial understanding of acceleration.
    Also, actually, we don’t know where the Moon is in space; we don’t know how to specify it, we don’t know what kind of light we can look for that many other things wouldn’t emit, etc.; we imagine that the Moon must be nice, but we don’t have a notion of its niceness that we can use to design our rocket; we know that some specific designs definitely fail and end up somewhere that’s not the Moon, but that wouldn’t really help us to get to the Moon.
    If you launch anything capable and you don’t have good reasons to think it’s an aligned mind, it will not be an aligned mind. If you try to prevent specific failure modes- if you identify optimizations towards something different from what you want, or how exactly gradient descent diverges somewhere that’s certainly not aligned- you’re probably iteratively looking for training setups where you don’t understand failure modes instead of setups that actually produce something aligned. If you don’t know where you’re going, it’s not helpful enough not to go somewhere that’s definitely not where you want to end up; you have to differentiate paths towards the destination from all other paths, or you fail.
    When you get to a system capable enough to meaningfully help you, you need to have already solved this problem. I think not enough people understand what this problem is, and I think that if it is not solved in time, we die.
    I’ve heard many attempts to hide the hard problem in something outside of where our attention is directed: e.g., design a system out of many models overseeing each other, and get useful work out of the whole system while preventing specific models from staging a coup.
    I have intuitions for why these kinds of approaches fail, mostly along the lines of reasons for why, unless you already have something sufficiently ...
    11 min
  • AF - Shell games by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shell games, published by Tsvi Benson-Tilsen on March 19, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed November 18, 2022.]
    Shell game
    Here's the classic shell game: Youtube
    Screenshot from that video.
    The little ball is a phantom: when you look for it under a specific shell, it's not there, it's under a different shell.
    (This might be where the name "shell company" comes from: the business dealings are definitely somewhere, just not in this company you're looking at.)
    Perpetual motion machines
    Related: Perpetual motion beliefs
    Bhāskara's wheel is a proposed perpetual-motion machine from the Middle Ages:
    Here's another version:
    From this video.
    Someone could try arguing that this really is a perpetual motion machine:
    Q: How do the bars get lifted up? What does the work to lift them?
    A: By the bars on the other side pulling down.
    Q: How does the wheel keep turning? How do the bars pull more on their way down than on their way up?
    A: Because they're extended further from the center on the downward-moving side than on the upward-moving side, so they apply more torque to the wheel.
    Q: How do the bars extend further on the way down?
    A: Because the momentum of the wheel carries them into the vertical bar, flipping them over.
    Q: But when that happens, energy is expended to lift up the little weights; that energy comes out of the kinetic energy of the wheel.
    A: Ok, you're right, but that's not necessary to the design. All we need is that the torque on the downward side is greater than the torque on the upward side, so instead of flipping the weights up, we could tweak the mechanism to just shift them outward, straight to the side. That doesn't take any energy because it's just going straight sideways, from a resting position to another resting position.
    Q: Yeah... you can shift them sideways with nearly zero work... but that means the weights are attached to the wheel at a pivot, right? So they'll just fall back and won't provide more torque.
    A: They don't pivot, you fix them in place so they provide more torque.
    Q: Ok, but then when do you push the weights back inward?
    A: At the bottom.
    Q: When the weight is at the bottom? But then the slider isn't horizontal, so pushing the weight back towards the center is pushing it upward, which takes work.
    A: I meant, when the slider is at the bottom--when it's horizontal.
    Q: But if the sliders are fixed in place, by the time they're horizontal at the bottom, you've already lifted the weights back up some amount; they're strong-torquing the other way.
    A: At the bottom there's a guide ramp to lift the weights using normal force.
    Q: But the guide ramp is also torquing the wheel.
    And so on. The inventor can play hide the torque and hide the work.
    Shell games in alignment
    Some alignment schemes--schemes for structuring or training an AGI so that it can be transformatively useful and doesn't kill everyone--are prone to playing shell games. That is, there's some features of the scheme that don't seem to happen in a specific place; they happen somewhere other than where you're looking at the moment. Consider these questions:
    What sort of smarter-than-human work is supposed to be done by the AGI? When and how does it do that work--by what combination of parts across time?
    How does it become able to do that work? At what points does the AGI come to new understanding that it didn't have before?
    How does the AGI orchestrate it's thinking and actions to have large effects on the world? By what process, components, rules, or other elements?
    What determines the direction that the AGI's actions will push the world? Where did those determiners come from, and how exactly do they determine the direction?
    Where and how much do human operators have to make judgements? How much are those judgements being relied on to point to...
    7 min
  • LW - Against Deep Ideas by FayLadybug
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Deep Ideas, published by FayLadybug on March 19, 2023 on LessWrong.
    When discussing impactful research directions, it's tempting to get excited about ideas that seem deep and profoundly insightful. This seems especially true in areas that are theoretical and relatively new - such as AI Alignment Theory. Fascination with the concept of a research direction can leak into evaluations of the expected impact, most often through overestimating the likelihood of extremely impactful outcomes. As a result, we should a priori be more skeptical of research projects that we encounter that sound insightful and deep than of those that sound boring and incremental.
    This phenomenon can arise naturally from how ideas are generated and spread. If there are two research projects that are roughly equivalent, but one seems deep while the other seems boring, the deep one will garner more attention and interest. The spread and discovery of research ideas thus has a bias towards profound ideas, as profundity is more memetically fit than its absence. I believe that this bias is fairly strong in the AI alignment community, full as it is with researchers who love interesting intellectual challenges and ideas.
    Some researchers might think that profound ideas are likely necessary to solve AI Alignment. However, I'll note that even in such a scenario we should expect profound ideas to be given inordinate attention - as they will by default be selected over boring ideas that are as promising as the average profound approach to the problem. Unless exclusively profound ideas are promising, we should expect bias towards profound ideas to creep in.
    Even in a world where profound ideas are absolutely required for AI Alignment research, we should still expect that any given profound idea is very unlikely to succeed. Profound ideas very rarely yield significant results and the importance of solving a given problem should not affect our expectation that any given idea will be successful. In such a world I think exploration is much more important than exploitation - as the chances of success in any one direction are low.
    I'm particularly worried about profound research directions like Natural Abstractions or Heuristic Arguments being treated as more promising than they are and consuming a large amount of attention and resources. Both seem to have absorbed quite a lot of thought without yielding legible successes as of yet. Additionally, neither seems to me to be directed by feedback loops that rely on external validation of progress. I think researchers looking to start projects in theoretical alignment should keep these issues in mind, and not necessarily expect this status quo to change in the near future. It may be more promising to consider other directions.
    I don't think the way to deal with this is to completely stop working on profound ideas in fields like AI Alignment where we are often motivated by the expected impact of research. Instead, I think it's important to notice when a research direction seems deep and profound, acknowledge this, and have a healthy skepticism that expected impact is actually motivating excitement and attention about the idea - from both yourself and others.
    It’s perfectly valid to research things because you enjoy them. I do still think that it’s useful to be able to notice when this is happening.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - More information about the dangerous capability evaluations we did with GPT-4 and Claude. by Beth Barnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More information about the dangerous capability evaluations we did with GPT-4 and Claude., published by Beth Barnes on March 19, 2023 on LessWrong.
    [Written for more of a general-public audience than alignment-forum audience. We're working on a more thorough technical report.]We believe that capable enough AI systems could pose very large risks to the world. We don’t think today’s systems are capable enough to pose these sorts of risks, but we think that this situation could change quickly and it’s important to be monitoring the risks consistently. Because of this, ARC is partnering with leading AI labs such as Anthropic and OpenAI as a third-party evaluator to assess potentially dangerous capabilities of today’s state-of-the-art ML models. The dangerous capability we are focusing on is the ability to autonomously gain resources and evade human oversight.
    We attempt to elicit models’ capabilities in a controlled environment, with researchers in-the-loop for anything that could be dangerous, to understand what might go wrong before models are deployed. We think that future highly capable models should involve similar “red team” evaluations for dangerous capabilities before the models are deployed or scaled up, and we hope more teams building cutting-edge ML systems will adopt this approach. The testing we’ve done so far is insufficient for many reasons, but we hope that the rigor of evaluations will scale up as AI systems become more capable.
    As we expected going in, today’s models (while impressive) weren’t capable of autonomously making and carrying out the dangerous activities we tried to assess. But models are able to succeed at several of the necessary components. Given only the ability to write and run code, models have some success at simple tasks involving browsing the internet, getting humans to do things for them, and making long-term plans – even if they cannot yet execute on this reliably.
    As AI systems improve, it is becoming increasingly difficult to rule out that models might be able to autonomously gain resources and evade human oversight – so rigorous evaluation is essential. It is important to have systematic, controlled testing of these capabilities in place before models pose an imminent risk, so that labs can have advance warning when they’re getting close and know to stop scaling up models further until they have robust safety and security guarantees.
    This post will briefly lay out our motivation, methodology, an example task, and high-level conclusions. The information given here isn’t enough to give a full understanding of what we did or make our results replicable, and we won’t go into detail about results with specific models. We will publish more detail on our methods and results soon.
    Motivation
    Today’s AI systems can write convincing emails, give fairly useful instructions on how to carry out acts of terrorism, threaten users who have written negative things about them, and otherwise do things the world is not very ready for. Many people have tried using models to write and run code unsupervised, find vulnerabilities in code1, or carry out money-making schemes.
    Today’s models also have some serious limitations to their abilities. But the companies that have released today’s AI models are investing heavily in building more powerful, more capable ones.
    ARC is worried that future ML models may be able to autonomously act in the real world, doing things like “incorporate a company” or “exploit arbitrages in stock prices” or “design and synthesize DNA” without needing any human assistance or oversight. If models have the ability to act autonomously like this, this could pose major risks if they’re pursuing goals that are at odds with their human designers. They could make (or steal) money, impersonate humans, replicate themselves on unmonitored...
    13 min
  • AF - More information about the dangerous capability evaluations we did with GPT-4 and Claude. by Beth Barnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More information about the dangerous capability evaluations we did with GPT-4 and Claude., published by Beth Barnes on March 19, 2023 on The AI Alignment Forum.
    [Written for more of a general-public audience than alignment-forum audience. We're working on a more thorough technical report.]We believe that capable enough AI systems could pose very large risks to the world. We don’t think today’s systems are capable enough to pose these sorts of risks, but we think that this situation could change quickly and it’s important to be monitoring the risks consistently. Because of this, ARC is partnering with leading AI labs such as Anthropic and OpenAI as a third-party evaluator to assess potentially dangerous capabilities of today’s state-of-the-art ML models. The dangerous capability we are focusing on is the ability to autonomously gain resources and evade human oversight.
    We attempt to elicit models’ capabilities in a controlled environment, with researchers in-the-loop for anything that could be dangerous, to understand what might go wrong before models are deployed. We think that future highly capable models should involve similar “red team” evaluations for dangerous capabilities before the models are deployed or scaled up, and we hope more teams building cutting-edge ML systems will adopt this approach. The testing we’ve done so far is insufficient for many reasons, but we hope that the rigor of evaluations will scale up as AI systems become more capable.
    As we expected going in, today’s models (while impressive) weren’t capable of autonomously making and carrying out the dangerous activities we tried to assess. But models are able to succeed at several of the necessary components. Given only the ability to write and run code, models have some success at simple tasks involving browsing the internet, getting humans to do things for them, and making long-term plans – even if they cannot yet execute on this reliably.
    As AI systems improve, it is becoming increasingly difficult to rule out that models might be able to autonomously gain resources and evade human oversight – so rigorous evaluation is essential. It is important to have systematic, controlled testing of these capabilities in place before models pose an imminent risk, so that labs can have advance warning when they’re getting close and know to stop scaling up models further until they have robust safety and security guarantees.
    This post will briefly lay out our motivation, methodology, an example task, and high-level conclusions. The information given here isn’t enough to give a full understanding of what we did or make our results replicable, and we won’t go into detail about results with specific models. We will publish more detail on our methods and results soon.
    Motivation
    Today’s AI systems can write convincing emails, give fairly useful instructions on how to carry out acts of terrorism, threaten users who have written negative things about them, and otherwise do things the world is not very ready for. Many people have tried using models to write and run code unsupervised, find vulnerabilities in code1, or carry out money-making schemes.
    Today’s models also have some serious limitations to their abilities. But the companies that have released today’s AI models are investing heavily in building more powerful, more capable ones.
    ARC is worried that future ML models may be able to autonomously act in the real world, doing things like “incorporate a company” or “exploit arbitrages in stock prices” or “design and synthesize DNA” without needing any human assistance or oversight. If models have the ability to act autonomously like this, this could pose major risks if they’re pursuing goals that are at odds with their human designers. They could make (or steal) money, impersonate humans, replicate themselves o...
    14 min
  • EA - Potential employees have a unique lever to influence the behaviors of AI labs by oxalis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Potential employees have a unique lever to influence the behaviors of AI labs, published by oxalis on March 18, 2023 on The Effective Altruism Forum.
    (Cross posted from my personal blog)
    People who have received and are considering an offer from an AI lab are in a uniquely good spot to influence the actions of that lab.
    People who care about AI safety and alignment often have things they wish labs would do. These could be requests about prioritizing alignment and safety (eg. having a sufficiently staffed alignment team, having a public and credible safety and alignment plan), good governance (eg. having a mission, board structure, and entity structure that allows safety and alignment to be prioritized), information security, or similar. This post by Holden goes through some lab asks, but take this as illustrative, not exhaustive!
    So you probably have, or could generate, some practices or structures you wish labs would have in the realm of safety and alignment. Once you have received an offer to work for a lab, that lab suddenly cares about what you think far more than when you are someone who is just writing forum posts or tweeting at them.
    This post will go through some ways to potentially influence the lab in a positive direction after you have received your offer.
    Does this work? This is anecdata but I have seen offer holders win concessions, and I have heard recruiters talk about how these sorts of behaviors influence the lab’s strategy.
    We also have reason to expect this works given that hiring good ML and AI researchers is competitive, and that businesses have changed aspects about themselves in the past partially to help with recruitment. Some efforts for gender or ethnic diversity or environmental sustainability are taken so that hiring from groups who care about these things doesn’t become too difficult. One example is that Google changed its sexual harassment rules and did not renew its contract with the Pentagon over mass employee pushback. Of course some of this stuff they may have intrinsically cared about or done to appease the customers or the public at large, but it seems employees have a more direct lever and have successfully used it.
    The Strategy
    There are steps you can take at different stages of your hiring process. The best time to do this is when you have received an offer, because then you know they are interested in you and so will care about your opinion.
    Follow up call(s) or email just after receiving offer
    In the follow up call after your offer you can express any concerns before you join. This is a good time to make requests. I recommend being polite, grateful for the offer, and framing these as “Well, look I’m excited about the role but I just have some uncertainties or aspects that if they were addressed would make this is a really easy decision for me”
    Some example asks:
    I want the safety/alignment team to be larger
    I want to see more public comms about alignment strategy
    I would like to see coordination with other labs on safety standards and slower scaling, as well as independent auditing of safety and security efforts
    I want an empowered, independent board
    Theory of change:
    They might actually grant requests! I have seen this happen. If they don’t, they will still hear that information and if enough people say it, they may grant it in the future. This also sets you up for the next alternative which is.
    When you turn down an offer
    If you end up turning down the offer, either to work at another AI lab or some other entity, you should tell them why you did. If you partially turned them down because of concerns about their strategy or that they didn’t fulfill one of your asks, tell them!
    The most direct way to do this is to email your recruiter. Eg. write to the recruiter something like:
    “Thanks for this offer. I decided to turn it down...
    8 min
  • EA - Researching Priorities in Local Contexts by LuisMota
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Researching Priorities in Local Contexts, published by LuisMota on March 18, 2023 on The Effective Altruism Forum.
    Summary
    This post explores two ways in which EAs can adapt priorities to local contexts they face:
    Trying to do the most good at a global level, given specific local resources.
    Trying to do the most good at a local level.
    I argue that the best framing for EAs to use is the first of these. I also explore when doing good at the local level might be the best way to do the most good from a global perspective, and suggest a way to explore this possibility in practice.
    Introduction
    Effective Altruism is a global movement that aims to use resources as effectively as possible with the purpose of doing good. Members of this global community face different realities and challenges, which means that there is no one-size-fits-all path to making the world a better place. This requires local groups to adapt EA research and advice to their specific contexts.
    Currently, there is limited guidance on how to do this, and many approaches have been adopted. Research done with this purpose is known as local priorities research, and includes projects like local charity evaluation and local career advice. However, the exact goal of such an adaptation process has often been unclear, in a way that can come at the cost of doing the most good from a global perspective.
    This post seeks to improve the local group prioritization framework. I break down the current usage of local priorities research into two different approaches: one seeks to do the most good impartially in light of the local context, and the other aims to do the most good for the local region. I make the case that EA groups should focus on the first approach, and discuss various ways in which this could influence local group prioritization research.
    Existing concepts in priorities research
    To begin, it's useful to start this discussion with the definition of global priorities research (GPR). The definition I'll use throughout this post is the following, adapted from the definition of the term used by the Global Priorities Institute:
    Global Priorities Research is research that informs use of resources, seeking to do as much good as possible.
    “Resources” here includes things like talent, money, and social connections. The agents who have these resources can also vary; ranging from individuals trying to decide what to do with their careers, organizations defining which projects to work on, or community builders trying to figure out what the best directions for their group are.
    On the other hand, local priorities research (LPR) is the term frequently used to refer to research aimed at adapting priorities to local situations. The essential idea behind this concept is that, as one post puts it, it is “quite similar [to GPR], except that it’s narrowed down to a certain country”. That post defines it as follows.
    While GPR is about figuring out what are the most important global problems to work on, LPR is about figuring out what are the most important problems in a local context that can best maximise impact both locally and globally.
    This term is used to describe many research activities, including:
    Local cause area prioritization
    Charity evaluation
    High-impact local career pathway research
    Giving and philanthropy landscape research
    Some examples of projects within local priorities research include EA Singapore's cause prioritization report, which identifies AI safety and alternative proteins as Singapore's comparative advantages; the Brazilian charity Doebem, which aims to identify the best health and development charities in Brazil; and EA Philippines's cause prioritization report, which identifies 11 potential focus areas for work in the country, ranging from poverty alleviation in the Philippines to building the EA movem...
    15 min