The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Why Balsa Research is Worthwhile by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Balsa Research is Worthwhile, published by Zvi on October 10, 2022 on LessWrong.
    Previously: Announcing Balsa Research
    It is easy to see why improving government policy would be impactful. The part where one has a chance of pulling it off requires explanation. Here are four fair questions.
    Why believe there is an opening in what would appear to be a well-covered, highly crowded space of trained professionals pushing their preferred policies?
    Why is this tractable or neglected?
    What is the theory of change?
    You?
    Tell the Story So Far
    My answer to all of these, as it is in most other places, has two central explanations.
    There are no adults. In most places, also no trained professionals. There are only a bunch of adaptation executors, rewarded when they are seen cutting the enemy rather than ensuring the enemy is cut, and for reinforcing the party line.
    Where adults do exist, they do not have the right incentives. This causes them to mostly do things that don’t help solve the underlying problems. Existing organizations are instead focused on pleasing their donor bases and otherwise looking good. This makes them bad at generating effective solutions and at getting anything implemented.
    There are several entire ecosystems of think tanks and political activists seeking to make their case, have their voices heard and enact change. We have no shortage of such projects, from all sides: Left wing and right wing, statist, liberal and libertarian. Some generalize while others narrow their focus.
    Almost none of them are taking actions designed to cause change. That is not what gets them or their employees more status or more funding. That is not what their organizational memory or culture says to do.
    This is why, from the perspective of someone looking for good policy as a result, such results seem to be few and far between, no matter one’s underlying preferences. When it comes time to choose policies on which to campaign or laws to enact, the choices and details reliably fail to seem to be anything approaching optimal and are often atrociously bad. They are not much informed by what preparatory work has been done.
    Recently I talked to a number of Democratic operatives, and all of them winced at most of what the party has been centrally attempting (often successfully) to pass, both on political and practical grounds.
    When Republicans have had the power to pass legislation recently, they seem to have little idea how to in practice implement either general good governance or their own core principles. They have instead mostly focused on either looking tough on culture war issues and point making or trying to mechanically win future elections.
    When the time came to ‘repeal and replace’ the ACA, with years to prepare, the proposal that was voted on was the null proposal. No alternatives seem to have been seriously considered.
    Corporate tax reform was a top Republican priority forever. Its implementation was technically botched.
    With a new opportunity to pass abortion laws, we see once again that the research simply was not done, by anyone, on any side.
    Leftist and statist organizations propose policy primarily on the basis of symbolism rather than whether the proposal would work in practice. Usually what they propose would sabotage the very cause they say they wish to promote if it were implemented. Often the motivations are various internal struggles.
    Libertarian groups are good at pointing out the flaws in new proposals and existing laws. Alas, they mostly do not present these arguments in the forms or with the credibility that causes the system to listen. Everyone is tired of hearing it, often including me. Nor do they pack a sufficient lobbying punch. Nor do they sculpt proposals in ways that allow them to be picked up by the media or would make them popular with voters, or ...
    13 min
  • EA - Lessons learned from talking to >100 academics about AI safety by mariushobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons learned from talking to >100 academics about AI safety, published by mariushobbhahn on October 10, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - Lessons learned from talking to >100 academics about AI safety by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons learned from talking to >100 academics about AI safety, published by Marius Hobbhahn on October 10, 2022 on The AI Alignment Forum.
    I’d like to thank MH, Jaime Sevilla and Tamay Besiroglu for their feedback.
    During my Master's and Ph.D. (still ongoing), I have spoken with many academics about AI safety. These conversations include chats with individual PhDs, poster presentations and talks about AI safety.
    I think I have learned a lot from these conversations and expect many other people concerned about AI safety to find themselves in similar situations. Therefore, I want to detail some of my lessons and make my thoughts explicit so that others can scrutinize them.
    TL;DR: People in academia seem more and more open to arguments about risks from advanced intelligence over time and I would genuinely recommend having lots of these chats. Furthermore, I underestimated how much work related to some aspects AI safety already exists in academia and that we sometimes reinvent the wheel. Messaging matters, e.g. technical discussions got more interest than alarmism and explaining the problem rather than trying to actively convince someone received better feedback.
    Executive summary
    I have talked to somewhere between 100 and 200 academics (depending on your definitions) ranging from bachelor students to senior faculty. I use a broad definition of “conversations”, i.e. they include small chats, long conversations, invited talks, group meetings, etc.
    Findings
    Most of the people I talked to were more open about the concerns regarding AI safety than I expected, e.g. they acknowledged that it is a problem and asked further questions to clarify the problem or asked how they could contribute.
    Often I learned something during these discussions. For example, the academic literature on interpretability and robustness is rich and I was pointed to resources I didn’t yet know. Even in cases where I didn’t learn new concepts, people scrutinized my reasoning such that my explanations got better and clearer over time.
    The higher up the career ladder the person was, the more likely they were to quickly dismiss the problem (this might not be true in general, I only talked with a handful of professors).
    Often people are much more concerned with intentional bad effects of AI, e.g. bad actors using AI tools for surveillance, than unintended side-effects from powerful AI. The intuition that “AI is just a tool and will just do what we want” seems very strong.
    There is a lot of misunderstanding about AI safety. Some people think AI safety is the same as fairness, self-driving cars or medical AI. I think this is an unfortunate failure of the AI safety community but is quickly getting better.
    Most people really dislike alarmist attitudes. If I motivated the problem with X-risk, I was less likely to be taken seriously.
    Most people are interested in the technical aspects, e.g. when I motivated the problem with uncontrollability or interpretability (rather than X-risk), people were more likely to find the topic interesting. Making the core arguments for “how deep learning could go wrong” as detailed, e.g. by Ajeya Cotra or Richard Ngo usually worked well.
    Many people were interested in how they could contribute. However, often they were more interested in reframing their specific topic to sound more like AI safety rather than making substantial changes to their research. I think this is understandable from a risk-reward perspective of the typical Ph.D. student.
    People are aware of the fact that AI safety is not an established field in academia and that working on it comes with risks, e.g. that you might not be able to publish or be taken seriously by other academics.
    In the end, even when people agree that AI safety is a really big problem and know that they could contribute, they rarely change their...
    20 min
  • EA - We can do better than argmax by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We can do better than argmax, published by Jan Kulveit on October 10, 2022 on The Effective Altruism Forum.
    Summary: A much-discussed normative model of prioritisation in EA is akin to argmax (putting all resources on your top option). But this model often prescribes foolish things, so we rightly deviate from it – but in ad hoc ways. We describe a more principled approach: a kind of softmax, in which it is best to allocate resources to several options by confidence. This is a better yardstick when a whole community collaborates on impact; when some opportunities are fleeting or initially unknown; or when large actors are in play.
    Epistemic status: Relatively well-grounded in theory, though the analogy to formal methods is inexact. You could mentally replace “argmax” with “all-in” and “softmax” with “smooth” and still get the gist.Gavin wrote almost all of this one, based on Jan’s idea.
    > many EAs’ writings and statements are much more one-dimensional and “maximizy” than their actions.
    – Karnofsky
    Cause prioritisation is often talked about like this:
    Evaluate a small number of options (e.g. 50 causes);
    Estimate their {importance, tractability, and neglectedness} from expert point estimates;
    Give massive resources to the top option.
    You can see this as taking the argmax: as figuring out which input (e.g. “trying out AI safety”; “going to grad school”) will get us the most output (expected impact). So call this argmax prioritisation (AP).
    AP beats the hell out of the standard procedure (“do what your teachers told you you were good at”; “do what polls well”). But it’s a poor way to run a portfolio or community, because it only works when you’re allocating marginal resources (e.g. one additional researcher); when your estimates of the effect or cost-effect are not changing fast; and when you already understand the whole action space.
    It serves pretty well in global health. But where these assumptions are severely violated, you want a different approach – and while alternatives are known in technical circles, they are less understood by the community at large.
    Problems with AP, construed naively:
    Monomania: the argmax function returns a single option; the winner takes all the resources. If people naively act under AP without coordinating, we get diminishing returns and decreased productivity (because of bottlenecks in the complements to adding people to a field, like ops and mentoring). Also, under plausible assumptions, the single cause it picks will be a poor fit for most people. To patch this, the community has responded with the genre "You should work on X instead of AI safety" or “Why X is actually the best way to help the long-term future”. We feel we need to justify not argmaxing, or to represent our thing as the true community argmax. And in practice justification often involves manipulating your own beliefs (to artificially lengthen your AI timelines, say), appealing to ad hoc principles like worldview diversification , or getting into arguments about the precise degree of crowdedness of alignment.
    Stuckness: Naive argmax gives no resources to exploration (because we assume at the outset that we know all the actions and have good enough estimates of their rank). As a result, decisions can get stuck at local maxima. The quest for "Cause X" is a meta patch for a lack of exploration in AP. Also, from experience, existing frameworks treat value-of-information as an afterthought, sometimes ignoring it entirely.
    Flickering: If the top two actions have similar utilities, small changes in the available information lead to endless costly jumps between options. (Maybe even cycles!) Given any realistic constraints about training costs or lags or bottlenecks, you really don't want to do this. This has actually happened in our experience, with some severe switching costs (yea...
    20 min
  • EA - Possible miracles by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Possible miracles, published by Akash on October 9, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Possible miracles by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Possible miracles, published by Akash on October 9, 2022 on LessWrong.
    Epistemic status: Speculative and exploratory.
    Contributions: Akash wrote the initial list; Thomas reviewed the list and provided additional points. Unless specified otherwise, writing in the first person is by Akash and so are the opinions. Thanks to Joshua Clymer, Tom Shlomi, and Eli Lifland for comments. Thanks to many others for relevant conversations.
    If we need a miracle, where might it come from? What would it look like?
    Many of the arguments presented in List of Lethalities are compelling to me. Some of my colleagues and I spend many hours thinking about how hard the alignment problem is, analyzing various threat models, and getting familiar with all the ways we could fail.
    I thought it would be a useful exercise to intentionally try to think in the opposite way. How might we win?
    I have found “miracles” to be a helpful frame, but it’s misleading in some ways. For example, I think the “miracles” frame implies an extremely low chance of success (e.g., <1%) and fosters a “wait and hope” mentality (as opposed to a “proactively make things happen” mentality). I was considering titling this something else (e.g., “Reasons for Hope” or “Possible Victory Conditions”), but these frames also didn’t feel right. With the phrase “miracles,” I’m trying to convey that (a) I don’t feel particularly confident in these ideas, (b) I am aware of many counterarguments to these claims [and indeed some are already presented in List of Lethalities], and (c) I don’t think “hope” or “victory” sets the right tone-- the right tone is “ah gosh, things seem really hard, but if we win, maybe we’ll win because of something like this.”
    I have found it helpful to backchain from these miracles to come up with new project ideas. If you think it’s plausible that we need a miracle, I encourage you to form your own list of possible miracles & think carefully about what kinds of projects might make each one more likely to occur (or make it more likely that we notice one in time).
    With this in mind, here’s my list of possible miracles.
    New Agendas
    1. New people are entering the field of alignment at a faster rate than ever before. Some large programs are being designed to attract more AI safety researchers. One of these new researchers could approach the problem from an entirely new perspective (one that has not yet been discovered by the small community of <300 researchers). Other scientific fields have had moments in which one particularly gifted individual finds a solution that others had missed.
    a. There are some programs that are optimizing for finding talented people, exposing them to alignment arguments, and supporting the ones who become interested in alignment.
    b. A small fraction of people in the US and UK have been exposed to AI x-risk.
    c. Even less effort has gone into outreach to people outside the US and UK.
    2. More resources are being poured into the mentorship and training of new alignment researchers. Perhaps the miracle will not come from a new “genius” but from a reasonably smart person (or set of people) who receive high-quality mentorship.
    a. The pool of mentors is also expanding. It seems plausible to me that the top 2-10% of junior alignment researchers will soon be ready to take on their own mentees. This not only increases the total pool of mentors but also increases the diversity of thought in the mentorship pool.
    b. Some training programs are explicitly trying to get people to think creatively and come up with new agendas (e.g., Refine).
    c. Some training programs are explicitly trying to get people from different [non-CS/math] disciplines (e.g., Philosophy Fellowship; PIBBSS).
    3. The bar for coming up with new agendas lowers each year. At the very beginning of the field, people needed to identify the pr...
    15 min
  • EA - A peek at pairwise preference estimation in economics, marketing, and statistics by Jonas Moss
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A peek at pairwise preference estimation in economics, marketing, and statistics, published by Jonas Moss on October 8, 2022 on The Effective Altruism Forum.
    Introduction
    In my earlier post Estimating value from pairwise comparisons I wrote about a reasonable statistical model for the pairwise comparison experiments that Nuño Sempere at QURI have been doing (see also his sequence on estimating value). While writing that post I started thinking about fields where utility extraction is important and decided to take a look at health economics and environmental economics. This post is a write-up of my attempt at a light survey of the literature on this topic, with particular attention paid on pairwise experiments.
    What do I mean by pairwise comparisons? Suppose I ask you "Do you prefer to lose your arm or your leg?" That's a binary pairwise comparison between the two outcomes A, B, where A=lose a leg and B=lose an arm. Such comparison studies are truly widespread, going back at least to McFadden (1973), which has 23000 Google Scholar citations! Models such as these are called discrete choice models, and I will refer to them as dichotomous (binary) comparisons as well, which is terminology I've seen in the economics literature. These models cannot measure the scale of the preferences properly though. There are many reasons why we care about the scale of preferences/utilities. For instance, we need scaling to compare preferences between different studies, and we need scales when we face uncertainty, as part of expected utility theory.
    To take scale into account we can ask questions such as "How many times worse would it be to lose an arm than losing a leg?". Then you might answer, say, 1/10, so you think losing a leg is ten times worse than losing an arm. Or 10, so you think losing an arm is ten times worse than losing a leg. These questions are harder than the corresponding binary questions though, and I can image respondents being flabbergasted by them. Questions of this kind are called graded (or ratio) comparisons in the literature. The idea is old -- it goes way back to Thurstone (1927)!
    I'm excited about the prospect of using pairwise comparisons on a large scale. Here are some applications:
    Estimate the value of research, both in the context of academia and effective altruism. This post presents a small-scale experiment in the EA context. It would be interesting to do a similar experiment inside of academia. Probably more rigorous and lengthy though. In my experience many academics do not feel that their or other people's work is important. They research whatever is publishable since it's their job. Attempting to quantify researchers understanding of the value of their and other people's research could at least potentially push some researchers into a more effective direction.
    Estimating the value of EA projects. This should be pretty obvious. One of the potentials of the pairwise value estimation method is crowd-sourcing -- since it's so easy to say "I prefer A to B", or perhaps "A is 10 times better than B" -- the bar for participation is likely to be lower than, say, participating in Metaculus, which is a real hassle. Possible applications would be crowd-sourcing of valuation of small projects, e.g. something like Relative Impact of the First 10 EA Forum Prize Winners.
    Descriptive ethics. You could estimate moral weights for various species. You could get an understanding about how people vary in the their moral valuations. You could run experiments akin to the experiments underlying moral foundations theory, but with a much more quantitative flavor. I haven't thought deeply about it, but I imagine studies of this sort would be important in the context of moral uncertainty.
    Summary of thoughts from the short literature review
    See the linked post for more informat...
    6 min
  • LW - The Balto/Togo theory of scientific development by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Balto/Togo theory of scientific development, published by Elizabeth on October 9, 2022 on LessWrong.
    Tragically I gave up on the Plate Tectonics study before answering my most important question: “Is Alfred Wegener the Balto of plate tectonics?”
    Let me back up.
    Balto
    Balto is a famous sled dog. He got a statue in NYC for leading a team of dogs through a blizzard to deliver antibody serum to Nome, Alaska in 1925, ending a diphtheria outbreak. Later Disney made a movie about how great he was.
    Except that run was a relay, and Balto only got famous because he did the last leg, which had the most press coverage but was also the easiest. The real hero was Togo, the dog who led the team through the hardest terrain and covered by far the most miles as well. Disney later made a movie about him that makes no mention of Balto for the first 90%, and then goes out of its way to talk about what a shit dog he was, that’s why he didn’t get included in any of the important teams, but Togo had had to do so many hard things they needed a backup team for the trivial last leg so Balto would have to do.
    Togo’s owner died mad about the US mainland believing Balto was a hero. But since all the breeders knew who did the hard part Togo enjoyed a post-Nome level of reproductive success that Ghengis Khan could only dream about, so I feel like he was happy with his choices.
    But it’s not like Togo did this alone either. He led one team in a relay, and there were 20 humans and 150 dogs that contributed to the overall run. Plus someone had to invent the serum, manufacture it, and get it to the start of the dog relay at Nenana, Alaska. So exactly how much credit should Togo get here?
    The part with Wegener
    I was pretty sure Alfred Wegener, popularly credited as the discoverer/inventor of continental drift and mentioned more prominently than any other scientist in discussions of plate tectonics, is a Balto.
    First of all, continental drift is not plate tectonics. Continental drift is an idea that maybe some stuff happened one time. Plate tectonics is a paradigm with a mechanism that makes predictions and explains a lot of data no one knew was related until that moment.
    Second, Wegener didn’t discover any of the evidence he cited, he wasn’t the first to have the idea, and it’s not even clear he did much of the synthesis of the evidence. His original paper refers to “Concerning South America and Africa, biologists and geologists are in close agreement that a Brazilian–African continent existed in the Mesozoic”
    So he didn’t invent the idea, gather the data, or even really synthesize the evidence. His guess at the mechanism was wrong. But despite spending hours digging into the specific discovers and synthesizers that contributed to plate tectonics, the only name I remember is Wegener’s. Classic Balto.
    On the other hand, some of the people who gathered the data used to discover plate tectonics were motivated by the concept of continental drift, and by Wegener specifically. That seems like it should count for something. My collaborator Jasen Murray thinks it counts for a lot
    Jasen would go so far as to argue that shining a beacon in unknown territory that inspires explorers to look for treasure in the right place makes you the Togo, racing through fractured ice rapids social ridicule and self-doubt to do the real work of getting an idea considered at all. Showing up at the finish line to formalize a theory after there’s enough work to know it’s true is Balto work to him. This makes me profoundly uncomfortable because strongly advocating for something unproven terrifies me, but as counterargument arguments go that’s pretty weak.
    One difficulty is it’s hard to distinguish “ahead of their time beacon shining” from “lucky idiot”, and even Jasen admits he doesn’t know enough to claim Wegener in particular is...
    4 min
  • EA - Why defensive writing is bad for community epistemics by Emrik
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why defensive writing is bad for community epistemics, published by Emrik on October 8, 2022 on The Effective Altruism Forum.
    I see this as a big problem that makes communication and learning really inefficient. In a culture where defensive writing is the norm, readers learn to expect that their reputation is greatly at stake if they were to publish anything themselves.
    I advocate writing primarily based on what you think will help the reader. I claim this is an act of introspection that's harder than what it seems at first. I'm not trying to judge anyone here. I find this hard myself, so I hope to help others notice what I've noticed in myself.
    TL;DR: A summary is hard, but: You may be doing readers a disservice by not being aware of when you're optimising your writing purely for helping yourself vs at-least-partially helping readers as well. Secondly, readers should learn to interpret charitably, lest they perpetuate an inefficient and harmfwl culture of communication.
    Definitions
    Self-centered writing is when you optimise your writing based on how it reflects on your character, instead of on your expectation of what will help your readers.
    Defensive writing is a subcategory of the above, where you're optimising for making sure no one will end up having a bad impression of you.
    Judgmental reading is when you optimise your reading for making inferences about the author rather than just trying to learn what you can learn from the content.
    Naturally, these aren't mutually exclusive, and you can optimise for more than one thing at once.
    Takeaways
    A culture of defensive writing and judgmental reading makes communication really inefficient, and makes it especially scary for newcomers to write anything. Actually, it makes it scary for everyone.
    There's a difference between trying to make your statements safe to defer to (minimsing false-positives), vs not directly optimising for that e.g. because you're just sharing tools that readers can evaluate for themselves (minimising false-negatives). Where appropriate, writers should be upfront about when they're doing what.
    As an example of this, I'm not optimising this post for being safe to defer to. I take no responsibility for whatever nonsense of mine you end up actually believing. :p
    You are not being harmed when someone, according to you, uses insufficiently humble language. Downvoting them for it is tantamount to bullying someone for harmless self-expression.
    What does a good epistemic community look like?
    In my opinion, an informed approach to this is multidisciplinary and should ideally draw on wisdom from e.g. social epistemology, game theory, metascience, economics of science, graph theory, rationality, several fields in psychology, and can be usefwly supplemented with insights from distributed & parallel computing, evolutionary biology, and more. There have also been relevant discussions on LessWrong over the years.
    I'm telling you this because the first principle of a good epistemic community is:
    Community members should judge practices based on whether the judgment, when universalised, will lead to better or worse incentives in the community.
    And if we're not aware that there even exists a depth of research on what norms a community can try to encourage in order to improve its epistemic health, then we might have insufficient humility and forget to question our judgments. I'm not saying we should be stifled by uncertainty, but I am advocating that we at least think twice about how to encourage positive norms, and not rely too much on cached sensibilities.
    I'll summarise some of what I think are some of the most basic problems.
    1) We judge people much too harshly for what they don't know
    Remember, this isn't an academic prestige contest, and the only thing that matters is whether we have the knowledge we need in ord...
    14 min
  • LW - The "you-can-just" alarm by Emrik
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The "you-can-just" alarm, published by Emrik on October 8, 2022 on LessWrong.
    Daniel Dennett argues for the idea of a "Surely alarm" which says that whenever you hear someone say "surely" in an argument, an alarm should sound in your head—ding!—and you should pay attention for weaknesses. When someone says "surely", it's often just meant to appeal to a sense of disbelief that anyone could possibly disagree with them.
    Readers of the Sequences will be familiar with similar alarms upon hearing "emergence", "by definition", "scientific", "rational", "merely", and more. And if you're really unlucky, some words just explode without warning.
    Anyway, when you notice that people aren't doing something to attain what they've explicitly expressed a preference for, you should be wondering what's going on. I've seen countless examples of people noticing the discrepancy only to brush it aside because "surely if they wanted to do it, they would have." People routinely underestimate the cost of some seemingly insignificant obstacles.
    The fact that A) it's obvious that people really want Y, and that B) it's also obvious that there's almost no cost to doing X in order to attain Y, should not just cause you notice-it-as-plain-fact. There's a deep mystery here, and if you also believe both A and B, there's real epistemic profit to be had by paying close attention. The potential profit is proportional to the total strength of the beliefs you've just noticed are in direct conflict.
    You might naively think that if the confusion is so profitable, surely you—with all your ability to reason and pay close attention to reality—would've already noticed and exploited it. But if acquired beliefs A and B from a pre-attentive stage of your life, this may also explain why you've skipped over this obvious opportunity until now.
    Besides, it's really cheap to do a second look-over, so why-don't-you-just investigate?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min