The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - EA might systematically generate a scarcity mindset that produces low-integrity actors by Severin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA might systematically generate a scarcity mindset that produces low-integrity actors, published by Severin on April 25, 2023 on The Effective Altruism Forum.
    Epistemic status: Highly speculative quick Facebook post. Thanks to Anna Riedl for nudging me to share it here anyway.
    Something I've noticed recently is that people who are in a bad place in their lives tend to have a certain sticky sleazy black holey feel to them. Something around untrustworthiness, low integrity, optimizing for themselves regardless of the cost for the people around them. I've met people like that, and I think when others felt around me like my energy was subtly and indescribably off, it was due to me being sticky in that way, too.
    Game-theoretically, it makes total sense for people to be a bit untrustworthy while they are in a bad place in their life. If you're in a place of scarcity, it is entirely reasonable to be strategic about where you put your limited resources. Then, it's just reasonable to only be loyal to others as long as you can get something out of it yourself and to defect as soon as they don't offer obvious short-term gains. And similarly, it make sense for the people around you to be a bit weary of you when you are in that place.
    And now, a bit of a hot take: I think most if not all of Effective Altruism's recent scandals have been due to low-integrity sticky behavior. And, I think some properties of EA systematically make people sticky.
    We might want to invest some thought and effort into fixing them. So, here's some of EA's sticky-people-producing properties I can spontaneously think of, plus first thoughts on how to fix them that aren't supposed to be final solutions:
    1. Utilitarianism
    Yudkowsky wrote a thing that I think is true:
    "Go three-quarters of the way from deontology to utilitarianism and then stop. You are now in the right place. Stay there at least until you have become a god."
    Meanwhile, SBF and probably a bunch of other people in EA (including me at times) have gone all four quarters of the way. If there's no upper bound to when it is enough to make the numbers go up, you'll be in a place of scarcity no matter what, and will be incentivized to defect indefinitely.
    I think an explicit belief of "defecting is not a good utilitarian strategy" doesn't help here: Becoming sticky is not a decision, but a subtle shift in your cognition that happens when your animal instincts pick up that your prefrontal cortex thinks you are in a place of scarcity.
    Basically, I think Buddhism is what utilitarianism would be if it made sense and was human-brain-shaped: Optimizing for global optima, but from a place of compassion and felt oneness with all sentient beings, not from the standpoint of a technocratic puppet master.
    2. Ever-precarious salaries
    EA funders like to base their allocation of funds on evidence, and they like to be able to adjust course quickly as soon as there are higher expected-value opportunities. From the perspective of naive utilitarianism, this completely makes sense.
    From the perspective of grantees, however, it feels like permanently having to justify your existence. And that is a situation that makes you go funny in the head in a way that is not conducive to getting just about any job done, unless it's a job like fraud that inherently involves short-term thinking and defecting on society. Whether or not you treat people as trustworthy and competent, you'll tend to find that you are right.
    I don't know how to fix this. Especially at the place we are at now, where both the FTX collapse and funders' increased cautiousness made the precarity of EA funding even worse. Currently, I'm seeing two dimensions to at least partially solving this issue:
    Building healthier, more sustainable relationships between community members. That's why I'm building Authentic R...
    7 min
  • EA - Suggest candidates for CEA's next Executive Director by MaxDalton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Suggest candidates for CEA's next Executive Director, published by MaxDalton on April 26, 2023 on The Effective Altruism Forum.
    As previously mentioned on the Forum, the Centre for Effective Altruism (CEA) is searching for a new Executive Director.
    We (Claire Zabel, Max Dalton, and Michelle Hutchinson) have been appointed by the Effective Ventures boards to lead this search and make recommendations to the boards.
    We wanted to give you all a high-level update on the process, and the opportunity to suggest candidates and give input.
    What we’re looking for
    We’re looking for someone:
    With a compelling strategy for CEA,
    Who is dedicated to the mission set out in their strategy,
    Who has the management experience and leadership ability to execute on that vision,
    With strong judgement, high integrity, and clear communication skills.
    One thing to highlight is that we are both open to and enthusiastic about candidates who want to pursue significant changes to CEA. This might include:
    Spinning off or shutting down programs, or starting new programs,
    Focusing on specific cause areas, or on promoting general EA principles,
    Trying to build something more like a mass movement or trying to be more selective and focused,
    Significant staffing changes,
    Changing CEA’s name.
    We’re open to such significant changes because:
    We think that organizations tend to be more effective when they are led by people with a strong vision for the organization, and the leadership has leeway to execute on that vision.
    We think that much of CEA’s historical work has been highly effective, but we believe there are many alternative opportunities CEA could pursue (for instance, cause-area-specific community building, or focusing more on a subset of our programs), and that some of these could potentially have an even higher impact.
    However, we are also open to candidates who don’t make radical changes, but continue to build on CEA’s work to date.
    It is not a requirement that:
    The candidate has experience working in effective altruism (that might constrain our search space too far).
    They are an unalloyed fan of the effective altruism community as it exists; we think people who most notice the flaws in things may be best placed to help improve them.
    They are located in Oxford, or anywhere else in particular: we are a remote-first team. It is highly desirable for candidates to be able to have working hours that overlap with standard working hours between Pacific and British time though.
    Our process
    Advisors
    To increase diversity of viewpoints feeding into the search, and add people with additional experience running complicated executive hiring rounds, we have added advisors to the search committee, who we loop into our decision-making and ask for input.
    We invited the following people, who have all now accepted:
    James Snowden, who has worked in global health and wellbeing for a number of different organizations including GWWC and GiveWell, and is now a program officer at Open Philanthropy.
    An advisor who works outside effective altruism and doesn’t want to be publicly listed. They have corporate experience hiring and managing hundreds of people. This advisor has a “principles first” (relatively cause agnostic) approach to cause prioritization.
    Caitlin Elizondo, who is Head of People Operations at CEA. She has run a large number of hiring rounds and has a great understanding of CEA’s staff and history.
    Finding candidates
    We have reached out via email to about 100 people to ask for candidate recommendations and feedback on the process. We’ve also asked 80,000 Hours to help with headhunting for the role.
    We are also seeking candidate suggestions and feedback from Forum readers (you!).
    Assessing candidates
    At least one committee member will go through all the suggestions, and make a longlist. Then the committee and ...
    6 min
  • LW - Max Tegmark's new Time article on how we're in a Don't Look Up scenario [Linkpost] by Jonas Hallgren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Max Tegmark's new Time article on how we're in a Don't Look Up scenario [Linkpost], published by Jonas Hallgren on April 25, 2023 on LessWrong.
    Max Tegmark has posted a Time article on AI Safety and how we're in a "Don't Look Up" scenario.
    In a similar manner to Yudkowsky, Max went on Lex Fridman and has now posted a Time article on AI Safety. (I propose we get some more people into this pipeline)
    Max, however, portrays a more palatable view regarding societal standards. With his reference to Don't Look Up, I think this makes it one of my favourite pieces to send to people new to AI Risk, as I think it describes everything that your average joe needs to know quite well. (An asteroid with a 10% risk of killing humanity is bad)
    In terms of general memetics, it will be a lot harder for someone like LeCun to come up with a genius equivalence between asteroid safety and airplane safety with this framing. (Which might be a shame since it's one of the dumber counter-arguments I've heard.)
    But who knows? He might just claim that scientists know and have always known how to use nuclear bombs to shoot the asteroid away or something.
    What I wanted to say with the above is that I think Max is doing a great way of framing the problem, and with his respect from his earlier career as a physicist, I think it would be good to use his articles more in public discussions. I also did quite enjoy how he described alignment on the Lex Fridman podcast, and even though I don’t agree with all he says, it’s good enough.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Exploring the Lottery Ticket Hypothesis by Rauno Arike
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring the Lottery Ticket Hypothesis, published by Rauno Arike on April 25, 2023 on LessWrong.
    I have recently been fascinated by the breadth of important mysteries in deep learning, including deep double descent and phase changes, that could be explained by a curious conjectured property of neural networks called the lottery ticket hypothesis. Despite this explanatory potential, however, I haven't seen much discussion about the evidence behind and the implications of this hypothesis in the alignment community. Being confused about these things motivated me to conduct my own survey of the phenomenon, which resulted in this post.
    The Lottery Ticket Hypothesis, explained in one minute
    The lottery ticket hypothesis (LTH) was originally proposed in a paper by Frankle and Carbin (2018):
    A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.
    The authors call such subnetworks "winning lottery tickets". As the simplest example, they train a LeNet-300-100 model on the MNIST dataset and report that a network containing only 21.1% of the weights of the dense version reaches a higher test accuracy on less training iterations, while a network where only 3.6% of the weights remain performs almost identically to the dense network of the network.
    The lottery ticket hypothesis extends a long line of work in neural network pruning, a technique proposed in as early as 1990 by LeCun et al. Pruning simply means deleting some fraction of unimportant weights from the network after training to make inference more efficient. The vital insight of the lottery ticket hypothesis paper is that it may also be possible to prune the network before training to make both training and inference more efficient.
    In practice, the method that Frankle and Carbin used for finding winning tickets didn't yet eliminate the need to train the full network, instead just suggesting the possibility. The technique they used is iterative pruning, a procedure that roughly looks as follows:
    Train the full dense network on some classification task
    Prune out some fraction of the weights with the smallest magnitude
    Reinitialize the remaining weights to their original values
    Repeat the same procedure for a number of times
    This is computationally quite expensive, but as we'll see below, alternative approaches have been proposed later on.
    The paper also defined a stronger version of the hypothesis that they named the lottery ticket conjecture:
    We extend our hypothesis into an untested conjecture that SGD seeks out and trains a subset of well-initialized weights. Dense, randomly-initialized networks are easier to train than the sparse networks that result from pruning because there are more possible subnetworks from which training might recover a winning ticket.
    In the next section, I'll argue that making a distinction between the hypothesis and the conjecture appears to be quite important.
    Relevance to alignment
    Phase changes
    In his post about mechanistically interpreting grokking, Neel Nanda argues that the lottery ticket hypothesis may constitute the reason for why neural networks form sophisticated circuits.
    One may naively think that neural networks are optimized in a similar way to how linear regression classifiers are optimized: each weight slowly changes in a direction that marginally improves performance, and the result of these tiny individual improvements smoothly improve the performance of the entire ensemble. In practice, though, we observe the forming of sophisticated circuits like the induction circuit, which is comprised of a previous token head and an induction head. Either of those heads improves loss only in the case the other ...
    19 min
  • LW - How Many Bits Of Optimization Can One Bit Of Observation Unlock? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Many Bits Of Optimization Can One Bit Of Observation Unlock?, published by johnswentworth on April 26, 2023 on LessWrong.
    So there’s this thing where a system can perform more bits of optimization on its environment by observing some bits of information from its environment. Conjecture: observing an additional N bits of information can allow a system to perform at most N additional bits of optimization. I want a proof or disproof of this conjecture.
    I’ll operationalize “bits of optimization” in a similar way to channel capacity, so in more precise information-theoretic language, the conjecture can be stated as: if the sender (but NOT the receiver) observes N bits of information about the noise in a noisy channel, they can use that information to increase the bit-rate by at most N bits per usage.
    For once, I’m pretty confident that the operationalization is correct, so this is a concrete math question.
    Toy Example
    We have three variables, each one bit: Action (A), Observable (O), and outcome (Y). Our “environment” takes in the action and observable, and spits out the outcome, in this case via an xor function:
    Y=A⊕O
    We’ll assume the observable bit has a 50/50 distribution.
    If the action is independent of the observable, then the distribution of outcome Y is the same no matter what action is taken: it’s just 50/50. The actions can perform zero bits of optimization; they can’t change the distribution of outcomes at all.
    On the other hand, if the actions can be a function of O, then we can take either A=O or A=¯O (i.e. not-O), in which case Y will be deterministically 0 (if we take A=O), or deterministically 1 (for A=¯O). So, the actions can apply 1 bit of optimization to Y, steering Y deterministically into one half of its state space or the other half. By making the actions A a function of observable O, i.e. by “observing 1 bit”, 1 additional bit of optimization can be performed via the actions.
    Operationalization
    Operationalizing this problem is surprisingly tricky; at first glance the problem pattern-matches to various standard info-theoretic things, and those pattern-matches turn out to be misleading. (In particular, it’s not just conditional mutual information, since only the sender - not the receiver - observes the observable.) We have to start from relatively basic principles.
    The natural starting point is to operationalize “bits of optimization” in a similar way to info-theoretic channel capacity. We have 4 random variables:
    “Goal” G
    “Action” A
    “Observable” O
    “Outcome” Y
    Structurally:
    (This diagram is a Bayes net; it says that G and O are independent, A is calculated from G and O and maybe some additional noise, and Y is calculated from A and O and maybe some additional noise. So, P[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,O].) The generalized “channel capacity” is the maximum value of the mutual information I(G;Y), over distributions P[A|G,O].
    Intuitive story: the system will be assigned a random goal G, and then take actions A (as a function of observations O) to steer the outcome Y. The “number of bits of optimization” applied to Y is the amount of information one could gain about the goal G by observing the outcome Y.
    In information theoretic language:
    G is the original message to be sent
    A is the encoded message sent in to the channel
    O is noise on the channel
    Y is the output of the channel
    Then the generalized “channel capacity” is found by choosing the encoding P[A|G,O] to maximize I(G;Y).
    I’ll also import one more assumption from the standard info-theoretic setup: G is represented as an arbitrarily long string of independent 50/50 bits.
    So, fully written out, the conjecture says:
    Let G be an arbitrarily long string of independent 50/50 bits. Let A, O, and Y be finite random variables satisfying
    P[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,O]
    and define
    Δ:=(maxP[A|G,O]I(G;Y...
    5 min
  • EA - Growing the Animal Advocacy Community in China - Engaging Stakeholders in Research for Improved Effectiveness by Jack S
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Growing the Animal Advocacy Community in China - Engaging Stakeholders in Research for Improved Effectiveness, published by Jack S on April 25, 2023 on The Effective Altruism Forum.
    Summary
    This is the second in a series of blog/ forum posts that looks at the application of stakeholder-engaged research methods in EA-related cause areas. This post looks at how Good Growth has been applying these methods to supporting the community concerned about farmed animal welfare and alternative protein in China
    Improving research in the Chinese animal advocacy space should be a priority, in order to improve our theories of change and support the Chinese advocacy community
    We think that stakeholder-engaged research using mixed methods is particularly suitable for farmed animal welfare, community building, and alternative protein research in Asia, because there are large gaps in our theories of change, and a range of stakeholders whose input can provide value
    We describe two of our studies where we have used stakeholder-engaged methods to understand advocates and consumers in China
    These findings can help to refine the strategies and pathways of organisations focused on these cause areas
    We invite EAs to explore these methods. If you're interested in doing so reach out to us at [email protected] or DM us (Jack or Jah Ying) on the EA forum
    Introduction
    In the last post, we introduced our perspective on stakeholder-engaged research methods we think might be neglected in EA, and we put the spotlight on some of our past research in the community building/ meta-EA area.
    To recap, too much research is produced without engaging the people who might use or be informed by it, reducing both the quality of the research and limiting the potential for impact on the world. To resolve this issues, we should find creative ways to involve various stakeholders, such as implementing organisations, policymakers, and the general public, throughout the process of producing and disseminating research.
    This is not a new idea. A variety of terms are used to describe the idea of doing research with stakeholders playing a significant role - some of the more well-known terms include Mode 2 research, research co-creation and community-based participatory research. We use the term stakeholder-engaged research to encompass all of these approaches. Stakeholder-engaged research is often connected with various kinds of qualitative research, such as ethnography, participant observation, in-depth interviews (structured, semi-structured or unstructured), and focus groups, where engagement is “built-in” to the methodology, but it can encompass quantitative methodologies, natural science and engineering.
    In this post and the next, we’re going to look at Good Growth’s current area of focus - farmed animal welfare - and the related field of alternative protein. Our work is focused on supporting the broader community of people working on animal advocacy in Asia - this post will focus specifically on our China research. We’re using ‘animal advocacy/ animal advocates’ as an umbrella term to refer to all communities trying to help animals, regardless of focus on rights/ welfare or wild/ farmed animals.
    Why Animals in Asia/ China?
    Animal welfare in Asia is an important and neglected cause area: Asia is home to over 40% of farmed land animals and produces over 85% of farmed fish, the majority of which are in China. On top of the enormous welfare impacts, the growing animal industry in the region also contributes significantly to the global risks of climate change and zoonotic pandemic risk. Despite this, Asian advocates only receive an estimated 7% of global animal advocacy funding, and Chinese advocates are particularly neglected.
    As well as addressing welfare directly, developing and promoting alternative protein is an incr...
    35 min
  • LW - Should LW have an official list of norms? by Ruby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should LW have an official list of norms?, published by Ruby on April 25, 2023 on LessWrong.
    To get this written and shared quickly, I haven't polished it much and the English/explanation is a little rough. Seemed like the right tradeoff though.
    Recently, a few users have written their sense of norms for rationalist discourse, i.e. Basics of Rationalist Discourse and Elements of Rationalist Discourse. There've been a few calls to adopt something like these as site norms for LessWrong.
    Doing so seems like it'd provide at least the following benefits:
    It's a great onboarding tool for new users to help them understand the site's expectations and what sets it apart from other forums
    It provided a recognized standard that both moderators and other users can point to and uphold, e.g. by pointing out instances where someone is failing to live up to one of the norms
    Having it be official is a good reminder to all site users to live up to the best kind of discussion
    My current feeling is creating some lists as an onboarding tool seems good, but doing anything like declaring a list of Site Norms is fraught.
    The True Norm of LessWrong is that with each motion, you should aim towards truth. I think it's actually worth quoting the entire 12th virtue here (emphasis added).
    Before these eleven virtues is a virtue which is nameless.
    Miyamoto Musashi wrote, in The Book of Five Rings:
    The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.
    Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.
    If you fail to achieve a correct answer, it is futile to protest that you acted with propriety.
    How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think, “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.
    Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it. You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory.” But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.
    If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. Then you will see how all techniques are one technique, and you will move correctly without feeling constrained. Musashi wrote: “When you appreciate the power of nature, knowing the rhythm of any situation, you will be able to hit the enemy naturally and strike naturally. All this is the Way of the Void.”
    If we were to declare site norms, I'd want to do in a way that made it very clear to new users and everyone else that our true underlying commitment was to truth and good decisions, not a particular list of good things to do that we'd w...
    8 min
  • AF - How Many Bits Of Optimization Can One Bit Of Observation Unlock? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Many Bits Of Optimization Can One Bit Of Observation Unlock?, published by johnswentworth on April 26, 2023 on The AI Alignment Forum.
    So there’s this thing where a system can perform more bits of optimization on its environment by observing some bits of information from its environment. Conjecture: observing an additional N bits of information can allow a system to perform at most N additional bits of optimization. I want a proof or disproof of this conjecture.
    I’ll operationalize “bits of optimization” in a similar way to channel capacity, so in more precise information-theoretic language, the conjecture can be stated as: if the sender (but NOT the receiver) observes N bits of information about the noise in a noisy channel, they can use that information to increase the bit-rate by at most N bits per usage.
    For once, I’m pretty confident that the operationalization is correct, so this is a concrete math question.
    Toy Example
    We have three variables, each one bit: Action (A), Observable (O), and outcome (Y). Our “environment” takes in the action and observable, and spits out the outcome, in this case via an xor function:
    Y=A⊕O
    We’ll assume the observable bit has a 50/50 distribution.
    If the action is independent of the observable, then the distribution of outcome Y is the same no matter what action is taken: it’s just 50/50. The actions can perform zero bits of optimization; they can’t change the distribution of outcomes at all.
    On the other hand, if the actions can be a function of O, then we can take either A=O or A=¯O (i.e. not-O), in which case Y will be deterministically 0 (if we take A=O), or deterministically 1 (for A=¯O). So, the actions can apply 1 bit of optimization to Y, steering Y deterministically into one half of its state space or the other half. By making the actions A a function of observable O, i.e. by “observing 1 bit”, 1 additional bit of optimization can be performed via the actions.
    Operationalization
    Operationalizing this problem is surprisingly tricky; at first glance the problem pattern-matches to various standard info-theoretic things, and those pattern-matches turn out to be misleading. (In particular, it’s not just conditional mutual information, since only the sender - not the receiver - observes the observable.) We have to start from relatively basic principles.
    The natural starting point is to operationalize “bits of optimization” in a similar way to info-theoretic channel capacity. We have 4 random variables:
    “Goal” G
    “Action” A
    “Observable” O
    “Outcome” Y
    Structurally:
    (This diagram is a Bayes net; it says that G and O are independent, A is calculated from G and O and maybe some additional noise, and Y is calculated from A and O and maybe some additional noise. So, P[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,O].) The generalized “channel capacity” is the maximum value of the mutual information I(G;Y), over distributions P[A|G,O].
    Intuitive story: the system will be assigned a random goal G, and then take actions A (as a function of observations O) to steer the outcome Y. The “number of bits of optimization” applied to Y is the amount of information one could gain about the goal G by observing the outcome Y.
    In information theoretic language:
    G is the original message to be sent
    A is the encoded message sent in to the channel
    O is noise on the channel
    Y is the output of the channel
    Then the generalized “channel capacity” is found by choosing the encoding P[A|G,O] to maximize I(G;Y).
    I’ll also import one more assumption from the standard info-theoretic setup: G is represented as an arbitrarily long string of independent 50/50 bits.
    So, fully written out, the conjecture says:
    Let G be an arbitrarily long string of independent 50/50 bits. Let A, O, and Y be finite random variables satisfying
    P[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,G]
    and define
    Δ:=(max...
    5 min
  • EA - AI Safety Newsletter #3: AI policy proposals and a new challenger approaches by Oliver Z
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Newsletter #3: AI policy proposals and a new challenger approaches, published by Oliver Z on April 25, 2023 on The Effective Altruism Forum.
    Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.
    Subscribe here to receive future versions.
    Policy Proposals for AI Safety
    Critical industries rely on the government to protect consumer safety. The FAA approves new airplane designs, the FDA tests new drugs, and the SEC and CFPB regulate risky financial instruments. Currently, there is no analogous set of regulations for AI safety.
    This could soon change. President Biden and other members of Congress have recently been vocal about the risks of artificial intelligence and the need for policy solutions.
    From guiding principles to enforceable laws. Previous work on AI policy such as the White House Blueprint for an AI Bill of Rights and the NIST AI Risk Management Framework has articulated guiding principles like interpretability, robustness, and privacy. But these recommendations are not enforceable – AI developers can simply choose to ignore them.
    A solution with more teeth could be on its way. Axios reports that Senator Chuck Schumer has been circulating a draft framework for AI governance among experts over the last several weeks. To help inform policy making efforts, the Department of Commerce has issued a request for comments on how to effectively regulate AI.
    The European Union debates narrow vs. general AI regulation. In Europe, policy conversations are centering around the EU AI Act. The Act focuses on eight “high-risk” applications of AI, including hiring, biometrics, and criminal justice. But the rise of general purpose AI systems like ChatGPT calls into question the wisdom of regulating only a handful of specific applications.
    An open letter signed by over 50 AI experts, including CAIS’s director, argues that the Act should also govern general purpose AI systems, holding AI developers liable for harm caused by their systems. Several members from all political blocs of the EU parliament have publicly agreed that rules are necessary for “powerful General Purpose AI systems that can be easily adapted to a multitude of purposes.”
    Specific policy proposals for AI safety. With politicians promising that AI regulation is coming, the key question is which proposals they will choose to carry forward into law. Here is a brief compilation of several recent sets of policy proposals:
    Create an AI regulatory body. A national agency focused on AI could set and enforce standards, monitor the development of powerful new models, investigate AI failures, and publish information about how to develop AI safely.
    Clarify legal liability for AI harm. When ChatGPT falsely accused a law professor of sexual harassment, legal scholars argued that OpenAI should face legal liability for libel and defamatory statements made by its models. Others propose AI developers should be strictly liable for harm caused by AI, but questions remain about where to draw the line between an unsafe product versus deliberate misuse.
    Compute governance. AI regulations could be automatically enforced by software built into the cutting edge computer chips used to train AI systems.
    Nuclear command and control. Despite persistent problems with the security and reliability of AI systems, some military analysts advocate using AI in the process of launching nuclear weapons. A simple proposal: Don’t give AI influence over nuclear command and control.
    Fund safety research. Organizations promoting work on AI safety such as NIST and NSF could use more funding from federal sources.
    China proposes many AI regulations. Last week, China released its own set of AI regulations that go much further than current Western efforts. Under ...
    8 min
  • LW - Briefly how I've updated since ChatGPT by rime
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Briefly how I've updated since ChatGPT, published by rime on April 25, 2023 on LessWrong.
    I'm laying out my thoughts in order to get people thinking about these points and perhaps correct me. I definitely don't endorse deferring to anything I say, and I would write this differently if I thought people were likely to do so.
    OpenAI's model of "deploy as early as possible in order to extend the timeline between when the world takes it seriously to when humans are no longer in control" seems less crazy to me.
    I think ChatGPT has made it a lot easier for me personally to think concretely about the issue and identify exactly what the key bottlenecks are.
    To the counterargument "but they've spurred other companies to catch up," I would say that this was going to happen whenever an equivalent AI was released, and I'm unsure whether we're more doomed in the world where this happened now, versus later when there's a greater overhang of background technology and compute.
    I'm not advocating specifically for or against any deployment schedule, I just think it's important that this model be viewed as not crazy, so it's adequately considered in relevant discussions.
    Why will LLMs develop agency? My default explanation used to involve fancy causal stories about monotonically learning better and better search heuristics, and heuristics for searching over heuristics. While those concerns are still relevant, the much more likely path is simply that people will try their hardest to make the LLM into an agent as soon as possible, because agents with the ability to carry out long-term goals are much more useful.
    "The public" seems to be much more receptive than I previously thought, both wrt Eliezer and the idea that AI could be existentially dangerous. This is good! But we're at the beginning where we are seeing the response from the people who are most receptive to the idea, and we've not yet got to the inevitable stage of political polarisation.
    Why doom? Companies and the open source community will continue to experiment with recursive LLMs, and end up with better and better simulations of entire research societies (a network epistemologist's dream). This creates a "meta-architectures overhang" which will amplify the capabilities of any new releases of base-level LLMs. As these are open sourced or made available via API, somebody somewhere will plain tell them to recursively self-improve themselves, no complicated story about instrumental convergence needed.
    AI will not stay in a box (because humans didn't try to put it into one in the first place). AI will not become an agent by accident (because humans will make it into one first). And if AI destroys the world, it's as likely to be by human instruction as by instrumentally convergent reasons inherent to the AI itself. Oops.
    The recursive LLM thing is also something I'm exploring for alignment purposes. If the path towards extreme intelligence is to build up LLM-based research societies, we have the advantage that every part of it can be inspected. And you can automate this inspection to alert you of misaligned intentions at every step. It's much harder to deceive when successfwl attempts depend on coordination.
    Lastly, AIs may soon be sentient, and people will torture them because people like doing that.
    I think it's likely that there will be a window where some AIs are conscious (e.g. uploads), but not yet powerful enough to resist what a human might do to them.
    In that world, as long as those AIs are available worldwide, there's a non-trivial population of humans who would derive sadistic pleasure from anonymously torturing them. AIs process information extremely fast, and unlike with farm animals, you can torture them to death an arbitrary number of times.
    To prevent this, it seems imperative to make sure that the AIs that are mo...
    5 min