The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Ten Levels of AI Alignment Difficulty by Samuel Dylan Martin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten Levels of AI Alignment Difficulty, published by Samuel Dylan Martin on July 3, 2023 on The AI Alignment Forum.
    Chris Olah recently released a tweet thread describing how the Anthropic team thinks about AI alignment difficulty. On this view, there is a range of possible scenarios ranging from ‘alignment is very easy’ to ‘alignment is impossible’, and we can frame AI alignment research as a process of increasing the probability of beneficial outcomes by progressively addressing these scenarios. I think this framing is really useful, and here I have expanded on it by providing a more detailed scale of AI alignment difficulty and explaining some considerations that arise from it.
    The discourse around AI safety is dominated by detailed conceptions of potential AI systems and their failure modes, along with ways to ensure their safety. This article by the DeepMind safety team provides an overview of some of these failure modes. I believe that we can understand these various threat models through the lens of "alignment difficulty" - with varying sources of AI misalignment sorted from easy to address to very hard to address, and attempt to match up technical AI safety interventions with specific alignment failure mode scenarios. Making this uncertainty clearer makes some debates between alignment researchers easier to understand.
    An easier scenario could involve AI models generalising and learning goals in ways that fit with common sense. For example, it could be the case that LLMs of any level of complexity are best understood as generative frameworks over potential writers, with Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI (CAI) selecting only among potential writers. This is sometimes called ‘alignment by default’.
    A hard scenario could look like that outlined in ‘Deep Deceptiveness’, where systems rapidly and unpredictably generalise in ways that quickly obsolete previous alignment techniques, and they also learn deceptive reward-hacking strategies that look superficially identical to good behaviour according to external evaluations, red-teaming, adversarial testing or interpretability examinations.
    When addressing the spectrum of alignment difficulty, we should examine each segment separately.
    If we assume that transformative AI will be produced, then the misuse risk associated with aligned transformative AI does not depend on how difficult alignment is. Therefore, misuse risk is a relatively bigger problem the easier AI alignment is.
    Easy scenarios should therefore mean more resources should be allocated to issues like structural risk, economic implications, misuse, and geopolitical problems. On the ‘harder’ end of easy, where RLHF-trained systems typically end up honestly and accurately pursuing oversimplified proxies for what we want, like ‘improve reported life satisfaction’, or ‘raise the stock price of company X’, we also have to worry about scenarios like Production Web or What Failure looks like 1 which require a mix of technical and governance interventions to address.
    Intermediate scenarios are cases where behavioural safety isn’t good enough and the easiest ways to produce Transformative AI result in dangerous deceptive misalignment. This is when systems work against our interests but pretend to be useful and safe. This scenario requires us to push harder on alignment work and explore promising strategies like scalable oversight, AI assistance on alignment research and interpretability-based oversight processes. We should also focus on governance interventions to ensure the leading projects have the time they need to actually implement these solutions and then use them (in conjunction with governments and civil society) to change the overall strategic landscape and eliminate the risk of misaligned AI.
    In contrast, if alignment is...
    22 min
  • LW - Frames in context by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Frames in context, published by Richard Ngo on July 3, 2023 on LessWrong.
    In my previous post, I introduced meta-rationality and frames, and described some examples of frames and some of their properties. In this post I’ll outline some of the limitations of existing ways of thinking about cognition, and some of the dynamics that they can’t describe which meta-rationality can. This post (especially the second half) can be seen as a summary of the key ideas from the rest of the sequence; if you find it too dense, feel free to skip it and come back after reading the next five posts. To quickly list my main claims:
    Unlike logical propositions, frames can’t be evaluated as discretely true or false.
    Unlike Bayesian hypotheses, frames aren’t mutually exclusive, and can overlap with each other. This (along with point #1) means that we can’t define probability distributions of credences over frames.
    Unlike in critical rationalism, we evaluate frames (partly) in terms of how true they are (based on their predictions) rather than just whether they’ve been falsified or not.
    Unlike Garrabrant traders and Rational Inductive Agents, frames can output any combination of empirical content (e.g. predictions about the world) and normative content (e.g. evaluations of outcomes, or recommendations for how to act).
    Unlike model-based policies, policies composed of frames can’t be decomposed into modules with distinct functions, because each frame plays multiple roles.
    Unlike in multi-agent RL, frames don’t interact independently with their environment, but instead contribute towards choosing the actions of a single agent.
    I’ll now explain these points in more detail. Epistemology typically focuses on propositions which can (at least in principle) be judged true or false. Traditionally, truth and knowledge are both taken as binary criteria: each proposition is either true or false, and we either know which it is or we don’t. Intuitively speaking, though, this doesn’t match very well with our everyday experience. There are many propositions which are kinda true, or which we kinda know: cats are (mostly) carnivorous (I think); Bob is tall(ish, if I’m looking at the right person); America is beautiful (in some ways, by my current tastes).
    The most straightforward solution to the problem of uncertainty is to assign credences based on how much evidence we have for each proposition. This is the bayesian approach, which solves a number of “paradoxes” in epistemology. But there’s still the question: what are we assigning credences to, if not to the proposition being discretely true or false? You might think that we can treat propositions which are “kinda true” (aka fuzzily true) as edge cases—but they’re omnipresent not only in everyday life, but also when thinking about more complex topics. Consider a scientific theory like Darwinian evolution. Darwin got many crucial things right, when formulating his theory; but there were also many gaps and mistakes. So applying a binary standard of truth to the theory as a whole is futile: even though some parts of Darwin’s original theory were false or too vague to evaluate, the overall theory was much better than any other in that domain.
    The mental models which we often use in our daily lives (e.g. our implicit models of how bicycles work), and all the other examples of frames I listed at the beginning of this post, can also be seen as “kinda but not completely true”. (From now on I’ll use “models” as a broad term which encompasses both scientific theories and informal mental models.)
    Not being “completely true” isn’t just a limitation of our current models, but a more fundamental problem. Perhaps we can discover completely true theories in physics, mathematics, or theoretical CS. But in order to describe high-level features of the real world, it’s always ...
    11 min
  • EA - Getting over my fear that going vegan would make me weak and unhealthy by Drew Housman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting over my fear that going vegan would make me weak and unhealthy, published by Drew Housman on July 3, 2023 on The Effective Altruism Forum.
    I recently read a Richard Hanania article titled "Eating Animals and the Virtues of Honesty." It talks about the moral atrocity that is factory farming and how that relates to Hanania's personal dietary choices. I commend him for calling out how bad most animal agriculture is.
    Hanania understands that there is no excuse to torture so many sentient beings just because we like how they taste. He concedes it could very well be one of the worst crimes in human history. Given all that, he wishes he could be vegan.
    Unfortunately, he must continue to eat animals. If he stops, he will ruin his body composition:
    I just grant to the vegan that he has won the argument, and he is morally superior from a utilitarian perspective, but I want to be thin and have broad shoulders.
    His argument, as I understand it, boils down to the idea that he needs to eat animals in order be fit, strong, and healthy. His position made me think of these great tweets:I am tempted to poke fun at Hanania, but I used to have the exact same worry! So I am going to do the ethical thing and talk about my personal experience as a jacked vegan.
    I know it's easy to google “vegan weightlifter” or "vegan athlete" or visit r/veganfitness to find examples vegans with shoulders so broad they'd make Hanania weep.
    My goal with this post is to normalize the idea that even late 30’s, tech working, regular guy vegans can be muscular and healthy. Sorry if this comes across as bragging, or if it's cringey. It just struck me while reading the Hanania piece that more people might go vegan if they felt they could do so without withering away. I want Hanania to know he can have his cake and eat it too! Were he to internalize that, how many people amongst his large audience could he influence to change their diets? How much suffering could he reduce?
    I don't think it would be easy, possible, or desirable for every vegan to have big muscles. Nor do I think that a fully vegan diet is the healthiest choice for everyone. All I'm saying is that if what's holding you back from going vegan is a deep rooted fear that doing so will cause you to have small shoulders, I think you should reconsider.
    Going vegan, maintaining my strengthWhen I first considered going vegan, I was also worried I'd become thin and weak. I'd read Stephan Guyenet's account of going vegan for 6 months. He struggled a lot, and it really gave me pause.
    Old friends who came to visit during that period did repeatedly ask me if I was sick, because of the amount of weight I had lost-- largely muscle. I had grown paler as well.
    Health and strength are priorities for me. I didn't want to become pale and frail. But after I got my first puppy I decided that I could no longer tolerate the idea of eating other sentient beings, so I stopped.
    I went vegetarian for a month, then vegan. I figured I could always start eating meat again if my body fell apart.
    The whole getting pale and weak thing just.didn’t happen to me. At all. I kept expecting to lose weight, or at least lose muscle, but I never did.
    I have actually gained some muscle and weight over my 4 years as a vegan, while keeping to roughly the same workout routine. Maybe I'm just lucky. Or maybe the loudest voices online are the ones who either had a bad experience going vegan or are convinced they will if they try. Maybe it's not as hard as everyone makes it out to be.
    Here's me just before going vegan:
    And here I am after years of eating vegan:
    At 36, I can currently bench and squat more than when I was an avid meat eater in my early twenties.
    But am I destroying my health?
    To dispel the notion that I am only superficially healthy, I'll share some bloodwork. This section ca...
    9 min
  • EA - Recovering from Rejection (written for the In-Depth EA Program) by Aaron Gertler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Recovering from Rejection (written for the In-Depth EA Program), published by Aaron Gertler on July 3, 2023 on The Effective Altruism Forum.
    Formerly known as "Aaron's Epistemic Stories", which stops working as a title when it's on the Forum and people aren't required to read it.
    What is this post?
    A story about how I reacted poorly to my first few EA job rejections, and what I learned from reflecting on my mistakes.
    Context: When I worked at CEA, my colleague was working on EA Virtual Program curricula. She asked me to respond to this prompt:
    "What made you start caring about having good epistemics? What made you start trying to improve your epistemics? Why?"
    I wrote a meandering, stream-of-consciousness response and shared it. I assumed it would either be ignored or briefly summarized as part of a larger piece. Instead, it — went directly to the curriculum for the In-Depth Program?
    That was a surprise. It was a much bigger surprise when people started reaching out to tell me how much it had helped them: maybe a dozen times over the last two years. From the emails alone, it seems to be the most important thing I've written.
    So I'm sharing a lightly edited version on the Forum, in case it helps anyone else.
    Recovering from rejection
    Written in a bit of a rush. But I think that captures how it felt to be me in the throes of epistemic upheaval.
    After I graduated from college, I took the most profitable job I could find, at a company in a cheap city. I wanted to save money so I could be flexible later. So far, so good.
    I started an EA group at the company, which kept me thinking about effective altruism on a regular basis even without my college group. It wasn’t nearly as fun to run as the college group — people who work full-time jobs don't like extra meetings, and my co-organizers kept getting other jobs and leaving. But I still felt like “part of EA”.
    Eventually, I decided to move on from the company. So I applied to GiveWell, got to the very last step of the application process. and got rejected.
    Well, I thought, I guess it makes sense that I’m not qualified for an EA job. My grades weren’t great, and I was never a big researcher in college. Time to do something else.
    This is a story about a mistake. Do you see it?
    I moved to San Diego and spent the next 18 months as a freelance tutor and writer, feeling generally dissatisfied with my life. My local group met rarely and far away; I had no car, I was busy with family stuff, and I became less and less engaged with EA.
    Through an old connection, I was introduced to a couple who ran an EA-aligned foundation and lived nearby. I ended up doing part-time operations work for them — reading papers, emailing charities with questions, and other EA-flavored stuff.
    This boosted my confidence and led me to think harder about my career, though I kept running into limitations. For example, GiveDirectly’s CEO wanted to hire a research assistant for his lab at UCSD, but I’d totally forgotten my old R classes and wasn’t a good candidate, despite having a great connection from my operations work.
    There goes maybe the best opportunity I’ll ever get as a washed-up 24-year-old. Sigh.
    In early 2018, I got an email from someone at Open Philanthropy, inviting me to apply for a new research position. I was excited by the sudden opportunity and threw everything I had into the process. I made it to the last step. and got rejected.
    Well, I thought, I guess it makes sense that I’m still not qualified for an EA job. I’m not a kid with limitless potential anymore. I haven’t learned anything important since college. I guess it’s back to finding a coding bootcamp and trying to get a “real job”.
    Is the mistake standing out yet?
    This was a major setback; for a while, I was barely engaged in EA. But I did happen to see an 80,000 Hours page with a survey...
    18 min
  • EA - Mistakes in the moral mathematics of existential risk (Part 1: Introduction and cumulative risk) - Reflective altruism by BrownHairedEevee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mistakes in the moral mathematics of existential risk (Part 1: Introduction and cumulative risk) - Reflective altruism, published by BrownHairedEevee on July 3, 2023 on The Effective Altruism Forum.
    This is the first part of "Mistakes in the moral mathematics of existential risk", a series of blog posts by David Thorstad that aims to identify ways in which estimates of the value of reducing existential risk have been inflated. I've made this linkpost part of a sequence.
    Even if we use . conservative estimates, which entirely ignor[e] the possibility of space colonization and software minds, we find that the expected loss of an existential catastrophe is greater than the value of 1016 human lives. This implies that the expected value of reducing existential risk by a mere one millionth of one percentage point is at least a hundred times the value of a million human lives.
    Nick Bostrom, “Existential risk prevention as global priority”
    1. Introduction
    This is Part 1 of a series based on my paper, “Mistakes in the moral mathematics of existential risk.”
    (Almost) everyone agrees that human extinction would be a bad thing, and that actions which reduce the chance of human extinction have positive value. But some authors assign quite high value to extinction mitigation efforts. For example:
    Nick Bostrom argues that even on the most conservative assumptions, reducing existential risk by just one millionth of one percentage point would be as valuable as saving a hundred million lives today.
    Hilary Greaves and Will MacAskill estimate that early asteroid-detection efforts saved lives at an expected cost of fourteen cents per life.
    These numbers are a bit on the high side. If they are correct, then on many philosophical views the truth of longtermism will be (nearly) a foregone conclusion.
    I think that these, and other similar estimates, are inflated by many orders of magnitude. My paper and blog series “Existential risk pessimism and the time of perils” brought out one way in which these numbers may be too high: they will be overestimates unless the Time of Perils Hypothesis is true.
    My aim in this paper is to bring out three novel ways in which many leading estimates of the value of existential risk mitigation have been inflated. (The paper should be online as a working paper within a month.)
    I’ll introduce the mistakes in detail throughout the series, but it might be helpful to list them now.
    Mistake 1: Focusing on cumulative risk rather than per-unit risk.
    Mistake 2: Ignoring background risk.
    Mistake 3: Neglecting population dynamics.
    I show how many leading estimates make one, or often more than one of these mistakes.
    Correcting these mistakes in the moral mathematics of existential risk has two important implications.
    First, many debates have been mislocated, insofar as factors such as background risk and population dynamics are highly relevant to the value of existential risk mitigation, but these factors have rarely figured in recent debates.
    Second, many authors have overestimated the value of existential risk mitigation, often by many orders of magnitude.
    In this series, I review each mistake in turn. Then I consider implications of this discussion for current and future debates. Today, I look at the first mistake, focusing on cumulative rather than per-unit risk.
    2. Bostrom’s conservative scenario
    Nick Bostrom (2013) considers what he terms a conservative scenario in which humanity survives for a billion years on the planet Earth, at a stable population of one billion humans.
    We will see throughout this series that is far from a conservative scenario. Modeling background risk (correcting the second mistake) will put pressure on the likelihood of humanity surviving for a billion years. And modeling population dynamics (correcting the third mistake) will raise the possi...
    11 min
  • EA - Mistakes in the moral mathematics of existential risk (Part 2: Ignoring background risk) - Reflective altruism by BrownHairedEevee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mistakes in the moral mathematics of existential risk (Part 2: Ignoring background risk) - Reflective altruism, published by BrownHairedEevee on July 3, 2023 on The Effective Altruism Forum.
    This is the second part of "Mistakes in the moral mathematics of existential risk", a series of blog posts by David Thorstad that aims to identify ways in which estimates of the value of reducing existential risk have been inflated. I've made this linkpost part of a sequence.
    In the decades to come, advanced bioweapons could threaten human existence. Although the probability of human extinction from bioweapons may be low, the expected value of reducing the risk could still be large, since such risks jeopardize the existence of all future generations. We provide an overview of biotechnological extinction risk, make some rough initial estimates for how severe the risks might be, and compare the cost-effectiveness of reducing these extinction-level risks with existing biosecurity work. We find that reducing human extinction risk can be more cost-effective than reducing smaller-scale risks, even when using conservative estimates. This suggests that the risks are not low enough to ignore and that more ought to be done to prevent the worst-case scenarios.
    Millett and Snyder-Beattie, “Existential risk and cost-effective biosecurity”
    1. Introduction
    This is Part 2 of a series based on my paper “Mistakes in the moral mathematics of existential risk”.
    Part 1 introduced the series and discussed the first mistake: focusing on cumulative rather than per-unit risk. We saw how bringing the focus back to per-unit rather than cumulative risk was enough to change a claimed 'small’ risk reduction of one millionth of one percent into an astronomically large reduction that would drive risk to almost one in a million per century.
    Today, I want to focus on a second mistake: ignoring background risk. The importance of modeling background risk is one way to interpret the main lesson of my paper and blog series “Existential risk pessimism and the time of perils.” Indeed, in Part 7 of that series, I suggested just this interpretation.
    It turns out that blogging is sometimes a good way to write a paper. Today, I want to expand my discussion in Part 7 of the existential risk pessimism series to clarify the second mistake (ignoring background risk) and to show how it interacts with the first mistake (focusing on cumulative risk) in a leading discussion of cost-effective biosecurity.
    Some elements of this discussion are lifted verbatim from my earlier post. In my defense, I remind my readers that I am lazy.
    2. Snyder-Beattie and Millett on cost-effective biosecurity
    Andrew Snyder-Beattie holds a DPhil in Zoology from Oxford, and works as a Senior Program Officer at Open Philanthropy. Snyder-Beattie is widely considered to be among the very most influential voices on biosecurity within the effective altruist community.
    Piers Millett is a Senior Research Fellow at the Future of Humanity Institute. Millett holds advanced degrees in science policy, research methodology and international security, and has extensive industry experience in biosecurity.
    Millett and Snyder-Beattie’s paper, “Existential risk and cost-effective biosecurity”, is among the most-cited papers on biosecurity written by effective altruists. The paper argues that even very small reductions in existential risks in the biosecurity sector (henceforth, 'biorisks’) are cost-effective by standard metrics.
    Millett and Snyder-Beattie estimate the cost-effectiveness of an intervention as C/(NLR), where:
    C is the cost of the intervention.
    N is “the number of biothreats we expect to occur in 1 century”.
    L is “the number of life-years lost in such an event”.
    R is “the reduction in risk [in this century only] achieved by spending . C”.
    Millett and Snyder Be...
    14 min
  • EA - Apply to fall policy internships (we can help) by Elika
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply to fall policy internships (we can help), published by Elika on July 3, 2023 on The Effective Altruism Forum.
    Many U.S. congressional internship applications are closing in the next few weeks for Fall (Sep- Dec) internships. This is a relatively low-effort, high reward thing to do if you you’re interested in testing your fit for policy.
    I (Elika) interned in my congressional office for a semester just from off-the-cuff applying to test my fit and build my resume. This experience has been incredibly helpful (I now work for the US government and it gives me some more credibility in D.C). Many applications are closing within the next 1-2 weeks. We’re offering to support anyone considering applying.
    This is a particularly good fit if you’re:
    Interested in working in policy, politics, or governance solutions to problems
    An undergraduate student
    Able to work part-time (10+ hours per week)
    If you think this could be a good opportunity, we recommend:
    Reading this guide to internships which has information on which offices to choose from and how to apply and more (including this helpful link of all the Congressional office internships)
    Making a list of offices you think you’d be a good fit for
    Applying! When in doubt, apply - there’s no harm in applying if you’re serious about exploring this opportunity. We’re offering to support if you’re interested.
    Sign up to get support applying here
    Things we can help with:
    Whether or not you’d be a good fit for the positions
    Review your resume, cover letter & offices you’re interested in
    Accountability for submitting applications by the deadline
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • EA - Douglas Hoftstadter concerned about AI xrisk by Eli Rose
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Douglas Hoftstadter concerned about AI xrisk, published by Eli Rose on July 3, 2023 on The Effective Altruism Forum.
    Douglas Hofstadter is best known for authoring Godel, Escher, Bach, a book on artificial intelligence (among other things) which is sort of a cult classic. In a recent interview, he says he's terrified of recent AI progress and expresses beliefs similar to many people who focus on AI xrisk.
    Hoftstadter: The accelerating progress has been so unexpected that it has caught me off guard... not only myself, but many many people. There's a sense of terror akin to an oncoming tsunami that could catch all of humanity off guard. It's not clear whether this could mean the end of humanity in the sense of the systems we've created destroying us, it's not clear if that's the case but it's certainly conceivable. If not, it's also that it just renders humanity a small, almost insignificant phenomenon, compared to something that is far more intelligent and will become as incomprehensible to us as we are to cockroaches.
    Interviewer: That's an interesting thought.
    Hoftstadter: Well I don't think it's interesting. I think it's terrifying. I hate it.
    I think this is the first time he's publicly expressed this, and his views seem to have changed recently. Previously he published this which listed a bunch of silly questions GPT-3 gets wrong and concluded that
    There are no concepts behind the GPT-3 scenes; rather, there’s just an unimaginably huge amount of absorbed text upon which it draws to produce answers
    though it ended with a gesture to the fast pace of change and inability to predict the future. I randomly tried some of his stumpers on GPT-4 and it gets them right (and I remember being convinced when this came out that GPT-3 could get them right too with a bit of prompt engineering, though I don't remember specifics).
    I find this a bit emotional because of how much I loved Godel, Escher, Bach in early college. It was my introduction to "real" math and STEM, which I'd previously disliked and been bad at; because of this book, I majored in computer science. It presented a lot of philosophical puzzles for and problems with AI, and gave beautiful, eye-opening answers to them. I think Hofstadter expected us to understand AI much better before we got to this level of capabilities; expected more of the type of understanding his parables and thought experiments could sometimes create.
    Now I work professionally on situations along the lines of what he describes in the interview (and feel a similar way about them) — it's a weird way to meet Hofstadter again.
    See also Gwern's post on LessWrong.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - Meta-rationality and frames by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta-rationality and frames, published by Richard Ngo on July 3, 2023 on LessWrong.
    How should we think about thinking? In this sequence I outline an epistemology called meta-rationality, which in my opinion corrects a number of the mistakes of existing epistemologies. This first post will introduce some of the key concepts in meta-rationality (in particular the concept of frames), provide some of the intellectual context behind my conception of meta-rationality, and list the main claims I’ll be making in the next half-dozen posts. Those posts, which focus on introducing meta-rationality, constitute the first half of the sequence, and will be posted over the next month; the second half of the sequence focuses on more complex aspects of meta-rationality, and will be posted over the following month or two.
    The traditional approach to epistemology is to focus on our knowledge of propositions like “there is a red car in the garage” or “I’m feeling thirsty”, which can in principle be evaluated as true or false. At a high level, meta-rationality is about making epistemology less reductionist by focusing less on assigning credences to isolated propositions like these, and more on the larger-scale mental entities which we actually use when thinking about complex domains—entities including:
    Ideologies like environmentalism, neoliberalism, communism, longtermism, etc
    Scientific paradigms like darwinism, keynesianism, quantum physics, deep learning, etc
    Life philosophies like stoicism, conformism, careerism, etc
    Moral drives like egalitarianism, patriotism, compassion, etc
    Epistemologies like empiricism, scientism, various schools of rationalism, etc
    Persistent personality traits like openness to experience, ambition, narcissism, etc
    Wide-ranging heuristics like “follow common-sense advice” or “move as fast as possible”
    I’ll call these frames. I’ll very roughly define a frame as a cluster of mental entities and processes (such as concepts, beliefs, heuristics, instincts, habits, skills, mental models, desires, values, etc) which tend to be activated in conjunction with each other. Under this definition, the extent to which a group of traits qualifies as a frame is a matter of degree, and we might focus on frames at different levels of abstraction in different contexts (although for convenience I'll mostly talk about frames in binary terms). For example, the concept of the voltage across a component in an electrical circuit tends to be associated with concepts like electrical current and resistance; and at a higher level, with knowledge about how to design circuits, and knowledge of electrical engineering more generally; and at an even higher level, with the idea that the world can be understood mechanistically and scientifically.
    I’ll primarily be focusing on high-level frames which apply across a broad range of contexts, but many of my conclusions apply to low-level frames too. (For an in-depth exploration of low-level frames, such as the ones learned by young children, I recommend Minsky’s Society of Mind.)
    The key contention of this sequence is that we should think about thinking primarily in terms of frames and the interactions between them. I’ll be giving detailed arguments for this claim in later posts; for now, I want to start off by appealing to high-level intuitions. We’re all familiar with the feeling of having strong beliefs that we can’t easily argue for, which are derived not from a straightforward argument, but from our background perspective on the world, which itself has been learned from many different datapoints. We don’t always apply the same perspective, though—we often act very differently in different contexts, or with different people. A particularly striking example: even people who are very good at building up a coherent understanding of complex domains...
    10 min
  • LW - Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)? by gwern
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)?, published by gwern on July 3, 2023 on LessWrong.
    A podcast interview (posted 2023-06-29) with noted AI researcher Douglas Hofstadter discusses his career and current views on AI.
    Hofstadter has previously energetically criticized GPT-2/3 models (and deep learning and compute-heavy GOFAI). These criticisms were widely circulated & cited, and apparently many people found Hofstadter a convincing & trustworthy authority when he was negative on deep learning capabilities & prospects, and so I found his comments in this most recent discussion of considerable interest (via Edward Kmett).
    Below I excerpt from the second half where he discusses DL progress & AI risk:
    Q: ...Which ideas from
    GEB are most relevant today?
    Hofstadter: ...In my book, I Am a
    Strange
    Loop, I tried to set forth what it is that really makes a self or a soul. I like to use the word
    "soul", not in the religious sense, but as a synonym for "I", a human "I", capital letter "I." So, what is it that makes a human being able to validly say "I"? What justifies the use of that word? When can a computer say "I" and we feel that there is a genuine "I" behind the scenes?
    I don't mean like when you call up the drugstore and the chatbot, or whatever you want to call it, on the phone says, "Tell me what you want. I know you want to talk to a human being, but first, in a few words, tell me what you want. I can understand full sentences." And then you say something and it says, "Do you want to refill a prescription?" And then when I say yes, it says, "Gotcha", meaning "I got you." So it acts as if there is an "I" there, but I don't have any sense whatsoever that there is an "I" there. It doesn't feel like an "I" to me, it feels like a very mechanical process.
    But in the case of more advanced things like
    ChatGPT-3 or
    GPT-4, it feels like there is something more there that merits the word "I." The question is, when will we feel that those things actually deserve to be thought of as being full-fledged, or at least partly fledged, "I"s?
    I personally worry that this is happening right now. But it's not only happening right now. It's not just that certain things that are coming about are similar to human consciousness or human selves. They are also very different, and in one way, it is extremely frightening to me. They are extraordinarily much more knowledgeable and they are extraordinarily much faster. So that if I were to take an hour in doing something, the ChatGPT-4 might take one second, maybe not even a second, to do exactly the same thing.
    And that suggests that these entities, whatever you want to think of them, are going to be very soon, right now they still make so many mistakes that we can't call them more intelligent than us, but very soon they're going to be, they may very well be more intelligent than us and far more intelligent than us. And at that point, we will be receding into the background in some sense. We will have handed the baton over to our successors, for better or for worse.
    And I can understand that if this were to happen over a long period of time, like hundreds of years, that might be okay. But it's happening over a period of a few years. It's like a tidal wave that is washing over us at unprecedented and unimagined speeds. And to me, it's quite terrifying because it suggests that everything that I used to believe was the case is being overturned.
    Q: What are some things specifically that terrify you? What are some issues that you're really...
    D. Hofstadter: When I started out studying cognitive science and thinking about the mind and computation, you know, this was many years ago, around 1960, and I knew how computers worked and
    I knew how extraordinarily rigid they were. You made the slightest typing error and it comp...
    10 min