The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Concrete Steps to Get Started in Transformer Mechanistic Interpretability by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Steps to Get Started in Transformer Mechanistic Interpretability, published by Neel Nanda on December 25, 2022 on The AI Alignment Forum.
    Disclaimer: This post mostly links to resources I've made. I feel somewhat bad about this, sorry! Transformer MI is a pretty young and small field and there just aren't many people making educational resources tailored to it. Some links are to collations of other people's work, and I link to more in the appendix.
    Introduction
    Feel free to just skip the intro and read the concrete steps
    The point of this post is to give concrete steps for how to get a decent level of baseline knowledge for transformer mechanistic interpretability (MI). This is an experiment in trying to give concrete, actionable, goal-oriented advice that I think is enough to get decent outcomes. Naturally, this can be constraining and the best path will differ a lot between your background and precise goals! As an accompaniment, I’m writing a future post fleshing out my philosophy behind doing mechanistic interpretability research, what I think the sub-skills are, and fleshing out how I think people should learn them.
    A core belief I have about learning mechanistic interpretability is that you should spend at least a third of your time writing code and playing around with model internals, not just reading papers. MI has great feedback loops, and a large component of the skillset is the practical, empirical skill of being able to write and run experiments easily. Unlike normal machine learning, once you have the basics of MI down, you should be able to run simple experiments on small models within minutes, not hours or days. Playing around with models builds this empirical skill, but also helps enhance the reading and learning. It builds intuitions for how the model actually works internally, what techniques are easy vs hard, etc, which is key context when understanding the ideas in papers.
    Further, because the feedback loops are so tight, I don’t think there’s a sharp boundary between reading and doing research. If you want to deeply engage with a paper, you should be playing with the model studied, and testing the paper’s basic claims. And if you’re doing this, you can try to answer the questions that come up as you read the paper. And there’s a smooth continuum between this and doing real original research.
    The intended audience is for people new-ish to Mechanistic Interpretability but who know they want to learn about it - if you have no idea what MI is, check out my MI Explainer or Circuits: Zoom In.
    One of my projects for the past few months has been trying to make it much easier to get into the field of reverse engineering language models, and making better open source tooling and educational materials. I compile the most relevant ones in this post, but if you're interested you can see a full list of side projects here
    Defining “Decent Baseline”
    Scoping out my goals here, I want you to be able to take a behaviour in a transformer that you want to understand, and have some idea of how to get started and get traction. Breaking this down further:
    A good grounding in the key concepts of ML and MI
    An intuition for how a transformer actually works as a mathematical object - what the moving parts are, how it all fits together, and how to reason about the overall system
    Familiarity with tooling, such that you can easily spin up a model and run quick and dirty experiments.
    A rough map of the literature, what’s known in the field, and big categories of open problems - not necessarily a deep knowledge, but hopefully enough to get a sense for techniques used, and where you could go and read a relevant paper if you want to.
    A sense of basic techniques, what compelling evidence about model internals looks like, and how to get started when poking around at a model....
    20 min
  • EA - YCombinator fraud rates by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: YCombinator fraud rates, published by Ben West on December 25, 2022 on The Effective Altruism Forum.
    Summary
    I estimate that 1-2% of $100M+ YCombinator-backed companies have faced serious allegations of fraud.
    Details
    I’m interested in better understanding the base rates of fraud in high-growth companies. YCombinator is a convenience sample for “high-growth companies”, as they are relatively public about who they back, and act as a filter for e.g. “obviously fraudulent” companies.
    I was able to find 3 companies which closed due to (alleged) fraud, two more which had their business substantially impacted by fraud claims, and three which had mild business impacts from fraud claims.
    YCombinator has incubated 3,951 companies, which gives a naïve rate of ~0.1% of companies having major allegations of fraud.
    I estimate that there are around 400 YCombinator-incubated companies with a valuation over $100M. 3 of the 5 companies with major fraud had $100M+ valuations, implying that 3/400 = 1% of $100M+ companies have major fraud charges. I think that this 1% number is probably more useful than the previous 0.1% number, because I expect that many smaller companies were fraudulent but no one bothered to charge them or report on it.
    I also wouldn’t be surprised to learn if I missed some cases, though I’d be surprised if I missed more than half of the $100M+ cases.
    So overall, I estimate that maybe 1-2% of $100M+ YCombinator-backed companies have faced serious charges of fraud.
    Additional Comments
    It’s notable that almost half of the companies on this list are financial services, which I believe makes them overrepresented.
    I expect that YCombinator is better at filtering out fraudulent companies than other angel investors, but they are also more likely to generate highly valuable companies. I’m not sure how these factors balance out, but would guess that they are roughly equal, implying that 1-2% of all high-growth $100M+ companies face serious fraud charges.
    Data
    CompanyValuationSeverityNotesuBiome$298MFatal“the company shut down in 2019 following an investigation into possible insurance fraud. Since 2021, the FBI has considered [the founders] to be fugitives.” (Wikipedia)LendUp$500MFatal"We are shuttering the lending operations of this fintech for repeatedly lying and illegally cheating its customers," said CFPB Director Rohit Chopra. (Reuters)Stablegains$6-15MFatalInvested customer funds into a cryptocurrency which collapsed. Customers allege this was deceptive, though I can’t find any evidence of them actually being charged (yet). (The Defiant)Synapsica$8-20MMajorTwo cofounders charged with defrauding the third and other investors. 1/3 of employees were laid off, which the company claims is unrelated to the fraud charges.Flutterwave$3BMajorAssets frozen over claimed violations of anti-money laundering laws. The company denies misconduct and seems to still be operating at a large scale.
    I am unfamiliar with the Nigerian legal/financial system, and it’s unclear to me how serious these charges are.Bikayi$21-50MMildSalespeople forged signatures of customers. It looks to me like the company separately encountered financial trouble and this fraud in particular was maybe just used as a cover to fire people.Momentus$65MMildSEC settled charges that the CEO misled investors about his immigration status and the results of a test of their technology during an attempt to take the company public. The company has now successfully gone public; my impression is that the CEO was fired but otherwise the company is fine.Modern Health$1.2BMildOne cofounder sued the other for defrauding investors by not disclosing incentives provided to customers which may have boosted sales.
    It seems like few media venues have picked up on the story, and so far the impact on their business seems mild.Dreamworld$1....
    7 min
  • AF - Take 14: Corrigibility isn't that great. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 14: Corrigibility isn't that great., published by Charlie Steiner on December 25, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days.
    It's the end (I saved a tenuous one for ya')! Kind of disappointing that this ended up averaging out to one every 2 days, but this was also a lot of work and I'm happy with the quality level. Some of the drafts that didn't work as "hot takes" will get published later.
    I
    There are certainly arguments for why we want to build corrigible AI. For example, the problem of fully updated deference says that if you build an AI that wants things, even if it's uncertain about what it wants, it knows it can get more of what it wants if it doesn't let you turn it off.
    The metal image this conjures up is of an AI doing something that's obvious-to-humans bad, and us clamoring to stop, but it blocking us from turning it off because we didn't solve the problem of fully updated deference. It would be better if we built an AI that took things slow, and that would let us shut it off if we got to look at what it was doing and saw that it was obviously bad.
    Don't get me wrong, this could be a nice property to have. But I don't think it's all that likely to come up, because aiming at aligned AI means building AI that tries not to do obviously bad stuff.
    A key point is that corrigibility is only desirable if you actually expect to use it. Its primary sales pitch is that it might give us a mulligan on an AI that starts doing obviously bad stuff. If everything goes great and we wind up in a post-scarcity utopia, I'm not worried about whether the AI would let me turn it off if I counterfactually wanted to.
    A world where corrigibility is useful might look like us building an agenty AI with a value learning process that we're not confident in, letting it run and interacting with it to try to judge how the value learning is going, and then (with moderate probability) turning it off and trying again with another idea for value learning. What does corrigibility have to do in this world? The AI shouldn't deliberately try to get shut down by doing obviously-bad things, but it also shouldn't try to avoid being shut down by instrumentally hiding bad behavior, or by backing itself up on AWS.
    Such indifference to the outside world is the default for limited AI that doesn't model that part of the world, or doesn't make decisions in a very coherent way. But in an agent that's good at navigating the real world, a lot of corrigibility is made out of value learning. The AI probably has to actively notice when it's coming into conflict with humans (and specifically humans, rather than head lice) and defer to them, even if those humans want to shut down the AI or rewrite its value learning process.
    So the first issue: if you can already do things like noticing when you're coming into conflict with humans, I fully expect you can build an AI that tries not to do things the humans think are obviously bad. And even though this has dangers, notably making corrigibility less likely to be used by making AIs avoid doing obviously-bad things, what the hell are you trying to do value learning for if you're not going to use it to get the AI to do good things and not bad things?
    II
    Second issue: sometimes agenty properties are good. An incorrigible AI is one that endorses some value learning process or meta-process, and will defend that good process against random noise, and against humans who might try to modify the process selfishly or short-sightedly.
    The point of corrigibility is that it the AI should not trust its own judgement about what counts as "short-sighted" for the human, and should let itself be shut down or modified. But sometimes humans are like a toddler i...
    5 min
  • EA - May The Factory Farms Burn by Omnizoid
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: May The Factory Farms Burn, published by Omnizoid on December 25, 2022 on The Effective Altruism Forum.
    Cross-post of this.
    “King Lear, late at night on the cliffs asks the blind Earl of Gloucester
    “How do you see the world?”
    And the blind man Gloucester replies “I see it feelingly”.
    Shouldn’t we all?
    Animals must be off the menu – because tonight they are screaming in terror in the slaughterhouse, in crates, and cages. Vile ignoble gulags of Despair.
    I heard the screams of my dying father as his body was ravaged by the cancer that killed him. And I realised I had heard these screams before.
    In the slaughterhouse, eyes stabbed out and tendons slashed, on the cattle ships to the Middle East and the dying mother whale as a Japanese harpoon explodes in her brain as she calls out to her calf.
    Their cries were the cries of my father.
    I discovered when we suffer, we suffer as equals.
    And in their capacity to suffer, a dog is a pig is a bear. . . . . . is a boy.”
    Phillip Wollen in a speech I’d highly recommend
    When a quarter million birds are stuffed into a single shed, unable even to flap their wings, when more than a million pigs inhabit a single farm, never once stepping into the light of day, when every year tens of millions of creatures go to their death without knowing the least measure of human kindness, it is time to question old assumptions, to ask what we are doing and what spirit drives us on.
    Matthew Scully “Dominion”
    Some ethical questions are difficult. One can’t be particularly confident in their views about population ethics, most political issues, normative ethics, etc. Yet the morality of our current consumption of meat is not one of those difficult issues. It is as close to a no-brainer as you get in normative ethics. The question we face is whether we should torture billions of sentient beings before killing them because we like the taste of their flesh.
    Whether pigs should be roasted to death, hot steam choking and burning them to death. Whether pigs who are smarter than dogs should be forced to give birth in tiny crates where they can’t move around. Whether they should be castrated with no anesthetic, have their tails and teeth cut off with a sharp object and no anesthetic, whether parts of their ears should be cut off for identification purposes, cruelly cramped together during transport unable to stand or move around, and whether they should have a knife dragged across their throat. All of these are the price we pay for cheap pig flesh.
    Whether chickens should be hung upside down by one leg before being brought on a conveyor to a knife being dragged across their throat—the only saving grace being an error prone electric bath that knocks them unconscious; sometimes. Of course, the combination of blade and electric bath is sufficiently error-prone to boil to death half a million or so sentient beings every year. It becomes abundantly clear that we’re acting horrendously when animal advocates are hoping that we’ll gas animals to death—kill them the way the Nazi’s did, for the ways we do it now are far crueler. Whether chickens should be crammed in a space far smaller than a sheet of paper, living their whole lives without seeing the sun, except in the moments before they’re transferred to their grisly slaughter. We do this to about 80 billion land animals and trillions of sea creatures.
    Are cheap eggs worth forcing sentient beings to live in shit, with the smell of feces being the only thing detectable from inside the barn? Forcing sentient beings to get osteoporosis and heart disease, all in an attempt to reduce the cost of eggs. This barrage of shit is not limited to egg-laying hens—it’s why a full 80% of pigs have pneumonia upon slaughter. A life lived in so much shit that it causes pneumonia the vast majority of the time is not how we ought to treat...
    22 min
  • EA - On applause lights and costly counters by Will Aldred
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On applause lights and costly counters, published by Will Aldred on December 24, 2022 on The Effective Altruism Forum.
    Epistemic status: stream of consciousness.
    Edited to add: This was written as a personal blogpost (with the frontpage checkbox at the bottom unchecked), but it appears to be showing up on the frontpage. I'm not sure what's up with that (though it's probably my bad, I forgot to uncheck the box when I initially published, then unchecked later); if you do read this post, know that it's especially rough and off-the-cuff and to some extent the downstream effect of red wine.
    An applause light is an empty statement which evokes positive affect without providing new information.
    (LessWrong Wiki)
    I've heard a couple of applause lights in the recent past, spoken by EAs.
    The first: "I'm confused [about x]."
    EA culture tends to reward people who are open and transparent about what they don't know. This is a good thing... until it starts getting Goodharted. The statement, "I'm confused", is productive in so far as it gets others to:
    apply appropriate grains of salt to the claims you're making on the topic you say you're confused about
    help you become less confused
    think more critically and flag appropriate uncertainty on claims they themselves make
    However, several times now I've heard empty statements of "I'm confused" where there's really no intention to achieve any of the above, so far as I can tell. "I'm confused" is being used as an applause light.
    The second: "diversity".
    This one's particularly pernicious, because it's costly to argue against. A concrete example: Alice and Bob, plus a handful more people, run an internship. In a team meeting on the application process, Bob says, "We should promote diversity." Heads nod. Satisfied, Bob does not continue. He's made his point, he's scored some social points.
    Meanwhile, in Alice's head:
    Hmm, if we do want to promote diversity, what actions should we take to make that happen?
    And, which is more, do we actually want to promote diversity in the first place?...
    ... On the one hand, one could make the moral claim, "diversity is a terminal goal."...
    ... On the other hand, one could make an instrumental claim along the lines of, "Greater diversity entails a wider set of viewpoints. This leads to a more informed discourse, better conclusions, and, ultimately, better decisions. Better decisions are upstream of better achieving our terminal goals, such as raising the probability of a flourishing future."
    Now, I know Bob, he's an EA, he's pretty in on the whole utilitarian deal, I don't think he views diversity as a terminal goal. Moreover, diversity is definitely not a terminal goal of this internship: our raison d'être is to reduce x-risk.
    Okay, so, I'll assume Bob's statement was made in good faith, and that it's really about the extent to which diversity is instrumentally useful toward reducing x-risk. This is an empirical question. There's a trade-off at play here. I've mentioned some reasons for more diversity above (more viewpoints, and so on), but there are also reasons against more diversity. Top of this list is that most of our best applicants are male and white. So, if we want to include more females and more non-white folks, then, given our fixed quota of places, this will mean distorting the bar to being accepted. Now, I don't like that this is the reality of the situation. I would like it be easy to promote diversity (in order to get the instrumental benefits of diversity without giving anything up, like having lower-aptitude people as fellows), but that's not the reality we live in. We have to weight up costs and benefits and make a choice accordingly.
    Right, I'll say what I'm thinking...
    Alice's words were met with awkward silence, averted gazes, and negative social points.
    Inside Alice's head:
    Ah, I see....
    6 min
  • LW - A hundredth of a bit of extra entropy by Adam Scherlis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A hundredth of a bit of extra entropy, published by Adam Scherlis on December 24, 2022 on LessWrong.
    There are two ways to calculate the amount of information in one term of a continued fraction:
    The entropy of the Gauss-Kuzmin distribution is about 3.4325 bits.
    Twice the logarithm of the Khinchin-Lévy constant is about 3.4237 bits.
    These differ by about 0.0088 bits. It took me a while to figure out why they were different at all, and now I'm surprised by how close they are.
    Continued fractions
    π is about 3.
    Actually, 3 is a bit too small, by about 1/7.
    Actually, 7 is a bit too small, by about 1/15.
    Actually, 15 is a bit a lot too small, by almost 1/1.
    But the 1 in the denominator is a tiny bit too small, by about 1/292.
    We can keep doing this forever, and get an expression like:
    π=3+17+115+11+1292+⋱
    This is pretty useful for things like finding rational approximations. We can drop that 1/(292+...), since it's really small, and get the excellent approximation π≈355113. Or we can drop the 1/(15+...) and get the rougher approximation π≈227.
    In general, the continued fraction for a real number r is of the form
    r=a0+1a1+1a2+1a3+⋱
    where the terms an are positive integers (except a0 which is an arbitrary integer).
    This is infinite for any irrational number; for a rational number, it stops after some number of terms, e.g.
    355113=3+17+115+11
    Truncating a continued fraction is the optimal way to approximate a real number with a rational: it gives a better approximation than anything with a smaller denominator.
    Continued fractions show up every now and then in different ways, but the approximation property is usually how I end up bumping into them.
    The Gauss-Kuzmin distribution
    Suppose we choose a random real number (say, uniformly between 0 and 1). The probability that the nth term is equal to k converges to p(k)=−log2(1−1(1+k)2) as n∞. This is the Gauss-Kuzmin distribution.
    I think it's also true that the fraction of k's in the continued fraction converges to p(k).
    The entropy of this distribution is just ∑∞k=1−p(k)log2(p(k)), which is pretty quick to compute out to a few digits of accuracy in your favorite programming language. It works out to about 3.4325 bits, so that's how much information there is in the term an, for large n.
    Lochs's theorem
    Lochs's theorem: For almost all real numbers, asymptotically, the accuracy of the continued fraction improves by a factor of eπ2/6ln2≈10.731 per term. This is amusingly close to 10, which is how much the accuracy of decimals improves per digit.
    You get log210 bits of information from a digit of decimal, so this means that you get π26(ln2)2≈3.4237 bits of information per term of a continued fraction, asymptotically.
    The difference
    So what gives? Why aren't those the same??
    Hint:
    It's not about how you take the limits to get these two things, as I initially thought. Eventually, the error in both numbers above will drop so far that 0.0088 bits looks enormous by comparison, so we can just think about the behavior after a bajillion terms and not worry about the fact that both theorems are asymptotic.
    The answer, unless I'm mistaken:
    The Gauss-Kuzmin distribution is a marginal distribution, so its entropy is the entropy of the nth term if you don't know anything about previous terms.The Khinchin-Lévy constant is the entropy of the conditional distribution; it measures the entropy of the nth term if you know every previous term.These are different because -- evidently -- different terms of the continued fraction are correlated, even for a generic real number, even in the limit of large n!I implicitly assumed that the correlation would tend to zero, and barely noticed I was making this assumption.It seems that if you know the first bajillion terms of a continued fraction, you actually have a tiny amount of information about the bajillion-pl...
    7 min
  • EA - [Link-post] Politico: "Ex-Google boss helps fund dozens of jobs in Biden’s administration" by Pranay K
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link-post] Politico: "Ex-Google boss helps fund dozens of jobs in Biden’s administration", published by Pranay K on December 24, 2022 on The Effective Altruism Forum.
    Politico article from Thursday December 22, 2022: "Ex-Google boss helps fund dozens of jobs in Biden’s administration"
    1. Summary:
    In three sentences:
    "Eric Schmidt, the former CEO of Google who has long sought influence over White House science policy, is helping to fund the salaries of more than two dozen officials in the Biden administration under the auspices of an outside group, the Federation of American Scientists."
    It is worth noting that Schmidt Futures (Schmidt's philanthropic ventures) does not directly fund these officials' salaries: Schmidt Futures provides < 30% to the Federation of American Scientists' "Day One fund" which funds these officials' salaries. Eric Schmidt seems to me to have called for the US government to aggressively invest in AI development.
    Some more context:
    Eric Schmidt chaired the National Security Commission on Artificial Intelligence from 2018-2021, in which the commision called on the US government to spend $40 billion on AI development.
    Schmidt Futures (Schmidt's philanthropic ventures) funds < 30% of the contributions to the Day One Project, a project within the Federation of American Scientists (FAS), which (among other things) provides the salaries of "FAS fellows" who hold "more than two dozen officials in the Biden administration" (from the main Politico article being discussed in this post). This includes 2 staffers in the Office of Science and Technology Policy (a different Politico article).
    The FAS is a "nonprofit global policy think tank with the stated intent of using science and scientific analysis to attempt to make the world more secure" (Wikipedia). The Day One project was started to recruit people to fill "key science and technology positions in the executive branch" (from the main Politico article).
    2. My question: Are Schmidt's projects harmfully advancing AI capabilities research?
    I've seen discussion among the EA community about how OpenAI and Anthropic may be harmfully advancing AI capabilities research. (The best discussion that comes to mind is this recent Scott Alexander post about ChatGPT; if anyone knows any other resources discussing this hypothesis - for or against - please comment below).
    I have not seen much discussion about Eric Schmidt's harmful or beneficial contributions to AI development in the US government. What do people think about this? Is this something that should concern us?
    3. Some more excerpts from the article about AI
    “Schmidt is clearly trying to influence AI policy to a disproportionate degree of any person I can think of,” said Alex Engler, a fellow at the Brookings Institution who specializes in AI policy. “We’ve seen a dramatic increase in investment toward advancing AI capacity in government and not much in limiting its harmful use.”
    Schmidt’s collaboration with FAS [Federation of American Scientists] is only a part of his broader advocacy for the U.S. government to invest more in technology and particularly in AI, positions he advanced as chair of the federal National Security Commission on Artificial Intelligence from 2018 to 2021.
    The commission’s final report recommended that the government spend $40 billion to “expand and democratize federal AI research and development” and suggested more may be needed.
    “If anything, this report underplays the investments America will need to make,” the report stated.
    “Other countries have made AI a national project. The United States has not yet, as a nation, systematically explored its scope, studied its implications, or begun the process of reconciling with it,” they wrote. “If the United States and its allies recoil before the implications of these capabilities and halt...
    4 min
  • LW - Shared reality: a key driver of human behavior by kdbscott
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shared reality: a key driver of human behavior, published by kdbscott on December 24, 2022 on LessWrong.
    Or: how to have a nice time with your family during the holidays.
    Model status: Well refined and very useful personally. But I haven't taught it, not sure how well it maps for others.
    I once asked Robin Hanson if he really thought status-seeking was such a dominant driver of human behavior. I said humans had dozens of factors motivating their behavior, it was crazy to claim there was One Big Thing. He replied (something to the effect of) "well, even if each factor has a small effect – one percent, two percent – one of them has to be the biggest."
    There's a concept I refer to as 'shared reality' that I think is up there with 'status' as something humans seek, shaping a lot (maybe five percent?) of our behavior.
    Knowing and playing with the concept of shared reality has noticeably improved my relationships and given me more surface area on many social concepts (e.g. connection, attachment, idle banter, tribalism).
    What is shared reality?
    There is a Columbia research lab that studies shared reality among other topics. They define it as:
    the perceived commonality of inner states with others
    By example: If two people chuckle at a pun, and then see each other chuckling such that they both perceive that the other had a similar pun-chuckle-experience, they are in a state of shared reality (at least, for that pun-chuckle-experience in particular).
    Shared reality is at play when people
    go to sports events / concerts / movies together.
    That emotional rush when everyone cheers / sings / laughs at the same time.
    travel / eat / dance together.
    That moment when you both see / eat / intuit something cool, look at each other and see you've had a similar experience, and get a rush of connection.
    Or that moment when you have an experience and suddenly want to share it with others, so you offer 'try this!' or 'look at that!' or take a picture and share it with the people who you think will best 'get it'.
    Shared reality is nice
    The research literature mentions that people are 'motivated to create shared realities' but doesn't really discuss why. I think people want it because it's pleasant. And conversely, that the opposite of shared reality (disconnected reality?) is unpleasant, and something people avoid.
    This strikes me as an important part, because put together with the above definition it makes for a model that better predicts how people will behave, and what is sometimes causing people to feel better/worse.
    Here's a stab:
    people seek a perceived commonality of inner states with others (because it's pleasant). People try to maintain that perceived commonality and/or avoid perceived uncommonality (because failing to do so is unpleasant).
    Wow, that's pretty clunky. Oh well.
    Here be dragons reality masking puzzles
    As you might expect with something that involves 'people seeking a perceived X', attempts at shared reality often skip right over actually sharing an experience to merely convincing oneself/others that an experience was shared. I think this is often playing out in the various forms of conformity. Some quick examples:
    When spending the holidays with family, I often feel a gentle pressure/request to do the same activity (sit/eat/talk together), even if I’d rather be doing something else.
    When talking with my parents about AI, I notice myself bouncing between frustrated that they don’t think about it the same way I do, and subdued/confused, agreeing that this whole thing is probably overblown. I think this flailing is motivated by a desire to connect (and fear of disconnection / not being seen and known by them)
    Social drinking/smoking
    Bandwagoning / groupthink
    I've developed a kind of backing-away immune response to many of these conformity / shared reality pressures. I thin...
    12 min
  • LW - Three reasons to cooperate by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three reasons to cooperate, published by paulfchristiano on December 24, 2022 on LessWrong.
    In this post I’ll discuss three reasons to cooperate in a truly one-shot prisoner’s dilemma:
    Kindness: You care about the other player.
    Correlation. Your decision is correlated with the other player’s decision.
    Reciprocity. Your decision is correlated with the other player’s belief about your decision.
    Kindness makes common sense, but correlation and reciprocity are often lumped together under “weird acausal stuff.” I think they are worth distinguishing because they work quite differently.
    I’ll talk about details of correlation and reciprocity, and then argue that most incentives to cooperate in the true prisoner’s are likely to come from interactions of multiple mechanisms.
    Setup: the true prisoner’s dilemma
    Imagine that one day our civilization sees another through a wormhole—we’ve never met before, and we are far enough away from each other that after today we will never meet again. From each civilization, a single person is chosen to play a game.
    The two players then play a one-shot prisoner’s dilemma with survival as the stakes. Each player chooses to cooperate in which case they die with probability 1%, or defect in which case the other play dies with probability 2%. If both players cooperate they each die with 1% chance, if both defect they each die with 2% chance.
    No matter what the other player does, you have a lower probability of death if you defect. In this post we analyze reasons that you might cooperate anyway.
    I will not discuss the incentives to make commitments in advance based on the possibility that we may play a prisoner’s dilemma in the future (even though I think those incentives are important). I will analyze the question mostly for an agent who uses EDT or UDT.
    You could just as well replace the ratio 2%/1% with any X>1, and ask “how large does X need to be before I’d cooperate?” I think it is pretty robust that you should cooperate at some sufficiently large scale (unless you specifically want the other player to die) and so this is ultimately a quantitative question. For this post I’ll just keep X=2 and leave it up to the reader to generalize.
    Kindness
    The most boring reason to cooperate is that I don’t want the other player to die.
    That might be because I value their life and experiences for its own sake, in the same way that I value mine. Or it could be that I mostly care about other consequences of survival—the effect a death will have on the people around us, the things we will accomplish if we survive, and so on—and I think their survival will have nearly as many good consequences as mine.
    Let K be how much I value the other player’s survival. If K > 0.5 then I should cooperate. (Since then 2% K > 1%.)
    Correlation
    When my deliberation begins, I’m not sure what I’ll end up deciding. I’m also not sure what the other player will decide. Those two unknown quantities are slightly correlated because our decisions have some common structure: what happens when someone is whisked away from their everyday life and spends a day deciding how nice to be to someone from an alien civilization?
    A causal decision theorist doesn’t care about this correlation, but an evidential decision theorist considers it a reason to cooperate. We can measure the strength of the correlation as C = P(they cooperate | I cooperate) – P(they cooperate | I defect). If C > 0.5 then I should cooperate.
    Smaller correlations give us a partial reason to cooperate, and they combine linearly with kindness: if K + C > 0.5 I should cooperate.
    It’s really hard to estimate C. It’s probably not zero but it’s plausible that it’s extremely small. Much of the correlation is likely to be screened off by other stuff I’ve seen (e.g. observing other people playing the prisoner’s dilemma).
    The effectiv...
    16 min
  • LW - The case against AI alignment by andrew sauer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The case against AI alignment, published by andrew sauer on December 24, 2022 on LessWrong.
    Trigger warning: Discussion of seriously horrific shit. Honestly, everything is on the table here so if you're on the lookout for trigger warnings you should probably stay away from this conversation.
    Any community of people which gains notability will attract criticism. Those who advocate for the importance of AI alignment are no exception. It is undoubtable that you have all heard plenty of arguments against the worth of AI alignment by those who disagree with you on the nature and potential of AI technology. Many have said that AI will never outstrip humans in intellectual capability. Others have said that any sufficiently intelligent AI will “align” themselves automatically, because they will be able to better figure out what is right. Others say that strong AI is far enough in the future that the alignment problem will inevitably be solved by the time true strong AI becomes viable, and the only reason we can’t solve it now is because we don’t sufficiently understand AI.
    I am not here to level criticisms of this type at the AI alignment community. I accept most of the descriptive positions endorsed by this community: I believe that AGI is possible and will inevitably be achieved within the next few decades, I believe that the alignment problem is not trivial and that unaligned AGI will likely act against human interests to such an extent as to lead to the extinction of the human race and probably all life as well. My criticism is rather on a moral level: do these facts mean that we should attempt to develop AI alignment techniques?
    I say we should not, because although the risks and downsides of unaligned strong AI are great, I do not believe that they even remotely compare in scope to the risks from strong AI alignment techniques in the wrong hands. And I believe that the vast majority of hands this technology could end up in are the wrong hands.
    You may reasonably ask: How can I say this, when I have already said that unaligned strong AI will lead to the extinction of humanity? What can be worse than the extinction of humanity? The answer to that question can be found very quickly by examining many possible nightmare scenarios that AI could bring about. And the common thread running through all of these nightmare scenarios is that the AI in question is almost certainly aligned, or partially aligned, to some interest of human origin.
    Unaligned AI will kill you, because you are made of atoms which can be used for paper clips instead. It will kill you because it is completely uninterested in you. Aligned, or partially aligned AI, by contrast, may well take a considerable interest in you and your well-being or lack thereof. It does not take a very creative mind to imagine how this can be significantly worse, and a superintelligent AI is more creative than even the most deranged of us.
    I will stop with the euphemisms, because this point really needs to be driven home for people to understand exactly why I am so insistent on it. The world as it exists today, at least sometimes, is unimaginably horrible. People have endured things that would make any one of us go insane, more times than one can count. Anything you can think of which is at all realistic has happened to somebody at some point in history. People have been skinned alive, burned and boiled alive, wasted away from agonizing disease, crushed to death, impaled, eaten alive, succumbed to thousands of minor cuts, been raped, been forced to rape others, drowned in shit, trampled by desperate crowds fleeing a fire, and really anything else you can think of. People like Junko Furuta have suffered torture and death so bad you will feel physical pain just from reading the Wikipedia article. Of course, if you care about animal...
    8 min