The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Residual stream norms grow exponentially over the forward pass by Stefan Heimersheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Residual stream norms grow exponentially over the forward pass, published by Stefan Heimersheim on May 7, 2023 on The AI Alignment Forum.
    Summary: For a range of language models and a range of input prompts, the norm of each residual stream grows exponentially over the forward pass, with average per-layer growth rate of about 1.045 in GPT2-XL. We show a bunch of evidence for this. We discuss to what extent different weights and parts of the network are responsible.
    We find that some model weights increase exponentially as a function of layer number. We finally note our current favored explanation: Due to LayerNorm, it's hard to cancel out existing residual stream features, but easy to overshadow existing features by just making new features 4.5% larger.
    Thanks to Aryan Bhatt, Marius Hobbhahn, Neel Nanda, and Nicky Pochinkov for discussion.
    Plots showing exponential norm and variance growth
    Our results are reproducible in this Colab.
    Alex noticed exponential growth in the contents of GPT-2-XL's residual streams. He ran dozens of prompts through the model, plotted for each layer the distribution of residual stream norms in a histogram, and found exponential growth in the L2 norm of the residual streams:
    Here's the norm of each residual stream for a specific prompt:
    Stefan had previously noticed this phenomenon in GPT2-small, back in MATS 3.0:
    Basic Facts about Language Model Internals also finds a growth in the norms of the attention-out matrices WO and the norms of MLP out matrices Wout ("writing weights"), while they find stable norms for WQ, WK, and Win ("reading weights"):
    Comparison of various transformer models
    We started our investigation by computing these residual stream norms for a variety of models, recovering Stefan's results (rescaled by √dmodel=√768) and Alex's earlier numbers. We see a number of straight lines in these logarithmic plots, which shows phases of exponential growth.
    We are surprised by the decrease in Residual Stream norm in some of the EleutherAI models. We would have expected that, because the transformer blocks can only access the normalized activations, it's hard for the model to "cancel out" a direction in the residual stream. Therefore, the norm always grows. However, this isn't what we see above. One explanation is that the model is able to memorize or predict the LayerNorm scale. If the model does this well enough it can (partially) delete activations and reduce the norm by writing vectors that cancel out previous activations.
    The very small models (distillgpt2, gpt2-small) have superexponential norm growth, but most models show exponential growth throughout extended periods. For example, from layer 5 to 41 in GPT2-XL, we see an exponential increase in residual stream norm at a rate of ~1.045 per layer. We showed this trend as an orange line in the above plot, and below we demonstrate the growth for a specific example:
    BOS and padding tokens
    In our initial tests, we noticed some residual streams showed a irregular and surprising growth curve:
    As for the reason behind this shape, we expect that the residual stream (norm) is very predictable at BOS and padding positions. This is because these positions cannot attend to other positions and thus always have the same values (up to positional embedding). Thus it would be no problem for the model to cancel out activations, and our arguments about this being hard do not hold for BOS and padding positions. We don't know whether there is a particular meaning behind this shape.
    We suspect that is the source of the U-shape shown in Basic facts about language models during training:
    Theories for the source of the growth
    From now on we focus on the GPT2-XL case. Here is the residual stream growth curve again (orange dots), but also including the resid_mid hook between the two Attention and MLP sub...
    21 min
  • LW - Long Covid Risks: 2023 Update by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long Covid Risks: 2023 Update, published by Elizabeth on May 6, 2023 on LessWrong.
    Back in 2021 I wrote a post estimating the risk of long covid. Recently a client hired me to do an update, focusing on changes induced by Paxlovid and vaccination. This was a <5h project and the literature wasn’t very rich so nothing I say here is conclusive, but nothing I said last time was conclusive either so let’s enjoy this together.
    Some caveats:
    I spent 5 hours on this, and that includes client-specific work I’m not including here.
    Research that met my standard was really scarce; ultimately each conclusion is based on a single study. My goal was data that includes a large population not selected for having long covid, where reporting was automated so you don’t need to worry about response bias. In practice, this means I used data from large medical systems with integrated reporting, like the American Department of Veteran Affairs, national medical systems, and HMOs. Surveys from long covid support groups were ignored with prejudice.
    Summary
    Vaccination helps, a bit: Given a medically diagnosed infection (which means it was serious enough to actually get you to the doctor), up-to-date vaccination lowers the risk of long covid by about 20% (this does not include the reduction in risk of having diagnosable covid in the first place, which is substantial).
    Paxlovid helps, more: Nirmatrelvir, which is one of two drugs that make up Paxlovid, reduces long covid risk by about 30% for medically diagnosed infections (which means it was serious enough to actually get you to the doctor). An optimist might hope the other drug (which is in the same class, although most commonly used as an adjuvant) is also useful and round this to 50%.
    Most symptoms are temporary: Long covid does tend to get better over time, but how quickly depends on the symptom. At one year post-infection, the rate of heart issues is nearly indistinguishable from controls, but cognitive issues have a 50% chance of persisting.
    Calculate your absolute risk: Your absolute risk depends on your age and comorbidities. The measured risk for 70-year-old men (not controlling for comorbidities) of developing at least one serious sequelae of medically diagnosed covid n (which means it was serious enough to actually get the patient the doctor) is ~12%. If you want to norm this for your own demographic, you can get a very crude estimate by entering your demographic information in this calculator, dividing your risk of hospitalization by 3 and multiplying the total by 0.4 (which includes the 20% reduction from vaccination and the 50% reduction from Paxlovid). If you are a cis woman, multiply by 2 to account for increased risk (trans people: I have no idea, if you find good data please let me know).
    I cannot emphasize enough how crude this is. I got that 3 by making up a 70 year old man with some common comorbidities, which has a risk of hospitalization of ~36%, and noticed 36/12=3. I don’t think The Economist has been keeping up to date with the latest strains of covid or even the impact of vaccination; these proposed calculations are strictly for order-of-magnitude estimates.
    Sample calculation: a 35 year old woman with no comorbidities shows a 3.8% risk of hospitalization (with their data, which I believe is very old). 3.8%/3= 1.3%. 1.3%0.4= 0.5%. Times 2 for being female = 1.0%. So a covid infection bad enough to require medical attention has a 1 chance in 100 of a serious persistent issue post-covid.
    Studies
    Nirmatrelvir and the Risk of Post-Acute Sequelae of COVID-19 (pre-print)
    This study compared people who got covid and received Nirmatrelvir (half of Paxlovid). It used data from the American Department of Veteran Affairs, which means the participants are older (average age 65), overwhelmingly male (~90%), and very white (75%). La...
    7 min
  • LW - How much do you believe your results? by Eric Neyman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How much do you believe your results?, published by Eric Neyman on May 6, 2023 on LessWrong.
    Thanks to Drake Thomas for feedback.
    I.
    Here’s a fun scatter plot. It has two thousand points, which I generated as follows: first, I drew two thousand x-values from a normal distribution with mean 0 and standard deviation 1. Then, I chose the y-value of each point by taking the x-value and then adding noise to it. The noise is also normally distributed, with mean 0 and standard deviation 1.
    Notice that there’s more spread along the y-axis than along the x-axis. That’s because each y-coordinate is a sum of two independently drawn numbers from the standard normal distribution. Because variances add, the y-values have variance 2 (standard deviation 1.41), not 1.
    Statisticians often talk about data forming an “elliptical cloud”. You can see how the data forms into an elliptical shape. To put a finer point on it:
    Why an ellipse — what’s the mathematical significance of this shape? The answer pops out if you look at a plot of how likely different points on the plane are to be selected by the random generation procedure that I used.
    The highest density of points is near (0, 0), and as you get farther from the origin the density decreases. The green ellipse on the scatter plot is a level set of equal probability: if you were to select a datapoint using my procedure, you’d be more likely to land in any square millimeter inside the ellipse than in any square millimeter outside the ellipse — and you’d be equally likely to land in any location on the ellipse as on any other location on the ellipse.
    The line of best fit is a statistical tool for answering the following question: given an x-value, what is your best guess about the y-value?
    What is the line of best fit for this data? Here’s one line of reasoning: since the y-values were generated by taking the x-values and adding random noise, our best guess for y should just be x. So the line of best fit is y = x.
    Huh, weird. this line is weirdly “askew” of the ellipse, and it doesn’t reflect the fact that the y-values are more dispersed than the x-values. Maybe the line of best fit instead passes from the bottom-left to the top-right of the ellipse, along its major axis. It sure looks like the points are on average closer to this line than to the previous one.
    Which line is the line of best fit, and what’s wrong with the other line? I recommend pondering this for a bit before reading on.
    The answer is that the first line, y = x, is the line of best fit. The problem with the second line is that it doesn’t try to predict y given x. I mean, scroll back up and take a look at how low the line is at x = -2: it’s way below almost all of the points whose x-value is near -2! This line is instead doing a different, important thing: it indicates the axis of maximum variation of the data. It’s the line with the property that, if you project the data onto the line, the data will be maximally dispersed. This line is called the first principal component of the data, but it is not the line of best fit.
    Instead of going from the bottom-left to the top-right of the ellipse, the line of best fit goes from the left of the ellipse to the right. This is the line that has as much of the ellipse above it as below it, at every x-coordinate. This is what you want, because you want it the true y-value to be below your prediction as often as it is above your prediction.
    (Huh, what a weird asymmetry! I wonder why the line doesn’t instead go from the bottom of the ellipse to the top.)
    II.
    You are the director of a giant government research program that’s conducting randomized controlled trials (RCTs) on two thousand health interventions, so that you can pick out the most cost-effective ones and promote them among the general population.
    The quality of the two thousan...
    24 min
  • EA - Don't Interpret Prediction Market Prices as Probabilities by bob
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't Interpret Prediction Market Prices as Probabilities, published by bob on May 5, 2023 on The Effective Altruism Forum.
    Epistemic status: most of it is right, probably
    Prediction markets sell shares for future events. Here's an example for the 2024 US presidential election:
    This market allows any US person to bet on the gender of the 2024 president. Male shares and female shares are issued in equal amounts. If the demand for shares of one gender is higher than shares of the other, the price is adjusted.
    At the time of writing, female shares cost 17 cents, and male shares cost 83 cents. If a female president is elected in 2024, owners of female shares will be able to cash them out for $1 each. If not, male shares can be cashed out for $1. The prices of male and female shares sum up to $1, which makes sense given that only one of them will be worth $1 in the future.
    Because bettors think a female president is relatively unlikely, the price for male shares is higher. The bettors may be wrong here, but the beauty of prediction markets it that anyone can put their money where their mouth is. If you believe that a female president is more likely than a male president, you can buy female shares for 17 cents a piece. If you're right, each of these shares will likely appreciate to $1 by 2024, almost sextupling your investment. If enough people predict a female president to be more likely, the demand for female shares will grow until they are more expensive than male shares. As such, the price of the shares reflects the predictions of everyone involved in the market.
    Even if you believe a female president is, say, 25% likely, you'd still be inclined to buy a female share for 17 cents. (That is, if you'd take a 1 in 4 chance of a 500% return on investment.) The interesting thing is that whenever you buy shares, the price will move closer to the probability you perceive be true. Only when the price matches your perceived probability, the market is no longer interesting for you. Because of this, the price of a share reflects the crowd's perceived probability of the corresponding outcome. If the market believes the probability to be 17%, the price will be 17 cents.
    Or so the story goes.
    In reality, it's more complicated.
    You're betting in a currency and, as such, you're betting on a currency.
    Let's say you believe a male president is about 90% likely, so you're considering buying male shares at 83 cents. Every 83 cents you put in can only become $1, so your maximum return on investment (ROI) is about 20%. Your expected ROI is closer to about 8% because you believe there's only a 90% chance the president will be male. Still, that's a positive ROI, so why not make the bet?
    This bet is denominated in US dollars, and it will be only resolved in 20 months or so. The problem is that US dollars are subject to inflation.
    Instead of locking up our investing money in a long-term bet for nearly two years, we could instead put it in an index fund, like the S&P 500, or invest in a large number of random stocks. Both methods have historically had a 10% annualized return. That's much better than an 8% two-year return!
    Because everyone thinks this way, there will be an artificially low demand for boring long-term positions, like predicting that the next US president will be male. This will drive the price of these shares down, while driving the price of shares for low-probability events up. A share that pays out USD will never have a price that reflects the market's perceived probability, because most people believe there are better things to invest in than USD.
    There's a solution for this, although regulators might not like it: allow people to bet bonds or shares. The famous 1 million USD bet between Warren Buffett and Protege Partners was actually not denominated in USD, but in bonds and sh...
    8 min
  • LW - Explaining “Hell is Game Theory Folk Theorems” by electroswing
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Explaining “Hell is Game Theory Folk Theorems”, published by electroswing on May 5, 2023 on LessWrong.
    I, along with many commenters, found the explanation in Hell is Game Theory Folk Theorems somewhat unclear. I am re-explaining some of the ideas from that post here. Thanks to jessicata for writing a post on such an interesting topic.
    1-shot prisoner’s dilemma.
    In a 1-shot prisoner’s dilemma, defecting is a dominant strategy. Because of this, (defect, defect) is the unique Nash equilibrium of this game. Which kind of sucks, since (cooperate, cooperate) would be better for both players.
    Nash equilibrium.
    Nash equilibrium is just a mathematical formalism. Consider a strategy profile, which is a list of which strategy each player chooses. A strategy profile is a Nash equilibrium if no player is strictly better off switching their strategy, assuming everyone else continues to play their strategy listed in the strategy profile. Notably, Nash equilibrium says nothing about:
    What if two or more people team up and deviate from the Nash equilibrium strategy profile?
    What if people aren’t behaving fully rationally? (see bounded rationality)
    Nash equilibria may or may not have predictive power. It depends on the game. Much work in game theory involves refining equilibrium concepts to have more predictive power in different situations (e.g. subgame perfect equilibrium to handle credible threats, trembling hand equilibrium to handle human error).
    n-shot prisoner’s dilemma.
    OK, now what if people agree to repeat a prisoner’s dilemma n=10 times? Maybe the repeated rounds can build trust among players, causing cooperation to happen?
    Unfortunately, the theory says that (defect, defect) is still the unique Nash equilibrium. Why? Because in the 10th game, players don’t care about their reputation anymore. They just want to maximize payoff, so they may as well defect. So, it is common knowledge that each player will defect in the 10th game. Now moving to the 9th game, players know their reputation doesn’t matter in this game, because everyone is going to defect in the 10th game anyway. So, it is common knowledge that each player will defect in the 9th game. And so on. This thought process is called backwards induction.
    This shows that the unique Nash equilibrium is still (defect, defect), even if the number of repetitions is large. Why might this lack predictive power?
    In the real world there might be uncertainty about the number of repetitions.
    (again) people might not behave fully rationally—backwards induction is kind of complicated!
    Probably the simplest way to model “uncertainty about the number of repetitions” is by assuming an infinite number of repetitions.
    infinitely repeated prisoner’s dilemma. OK, now assume the prisoner’s dilemma will be repeated forever.
    Turns out, now, there exists a Nash equilibrium which involves cooperation! Here is how it goes. Each player agrees to play cooperate, indefinitely. Whenever any player defects, the other player responds by defecting for the rest of all eternity (“punishment”).
    technical Q: Hold on a second. I thought Nash equilibrium was “static” in the sense that it just says: given that everybody is playing a Nash equilibrium strategy profile, if a single person deviates (while everyone else keeps playing according to the Nash equilibrium strategy profile), then they will not be better off from deviating. This stuff where players choose to punish other players in response to bad behavior seems like a stronger equilibrium concept not covered by Nash.
    A: Nope! A (pure) strategy profile is a list of strategies for each player. In a single prisoner’s dilemma, this is just a choice of “cooperate” or “defect”. In a repeated prisoner’s dilemma, this is much more complicated. A strategy is a complete contingency plan of what the player plans to...
    9 min
  • EA - Maybe Family Planning Charities Are Better For Farmed Animals Than Animal Welfare Ones by Hank B
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maybe Family Planning Charities Are Better For Farmed Animals Than Animal Welfare Ones, published by Hank B on May 6, 2023 on The Effective Altruism Forum.
    This piece estimates that a donation to the Humane League, an animal welfare organization considered highly cost-effective, and which mainly engages in corporate lobbying for higher welfare standards, saved around 4 animals per dollar donated, mostly chickens. “Saving a farmed animal” here means “preventing a farmed animal from existing” or “improving the welfare of enough farmed animals by enough to count as preventing one farmed animal from existing.” That second definition is a little weird, sorry.
    If you’re trying to help as many farmed animals as possible this seems like a pretty good deal. Can we do better? Maybe.
    Enter MSI Reproductive Choices, an international family planning organization, which mainly distributes contraception and performs abortions. They reported in 2021 that they prevented around 14 million unintended pregnancies on a total income of 290 million pounds, or 360 million dollars at time of writing. This is roughly 25 dollars per unintended pregnancy prevented. Let’s pretend that for every unintended pregnancy prevented, a child who would have been born otherwise is not born. This is plausibly true for some of these unintended pregnancies. But not all. On the other hand, MSI also provided abortions which plausibly prevent child lives as well. Maybe that means MSI prevented 14 million child lives from starting in 2021 (if we think the undercounting from not including abortions counter perfectly the overcounting of unintended pregnancy). I have no reason to think that’s particularly plausible, but let’s just keep pretending that’s right.
    Let’s further pretend that all of MSI’s work happened in Zambia. MSI does work in Zambia, but they also do work in lots of other countries. I choose Zambia mostly because trying to do this math with all the countries that MSI works with would be hard. Zambia had a life expectancy at birth of 62 years in 2020 according to this. According to this, Zambians consumed an average of 28kg of meat per person per year. The important subfigures here are the 2.6kg of poultry and 13kg of seafood per person per year, since chickens and fish are much lighter than other animals killed for meat. One chicken provides say 1kg of meat (I’m sort of making this number up, but similar numbers come up on google). One fish provides say 0.5kg. This means that the average Zambian would eat 2.6 chickens and 26 fish per person per year. Over a lifetime, that’d be 62 years of consumption.
    If a human who would have otherwise existed no longer exists because of your efforts, they also no longer eat the meat they would have eaten otherwise. Thus, if MSI prevents one human lifetime for every $25 you donate, then you’d be saving 62(2.6+26) farmed animals which is around 1,750. That’s 70 animals saved per dollar donated.
    This analysis is so bad in so many ways. I took the number for animals saved per dollar donated to The Humane League on total faith. I also just assumed that MSI is correct in saying that they prevented 14 million unintended pregnancies and I made clearly bad assumptions to get from that number to number of human lifetimes prevented. At least we can have some confidence in the total weight of meat consumed on average by a Zambian per year and the life expectancy at birth in Zambia. However, my way of getting from total weight to animals slaughtered is pretty hokey and doesn’t even include cows, sheep, pigs, etc. There are many other problems too. For example, I took the average cost per unintended pregnancy prevented by MSI. However, the average is not the relevant figure here. We’d like the marginal cost of preventing an additional unintended pregnancy. This is a figure I don...
    5 min
  • LW - Naturalist Collection by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Naturalist Collection, published by LoganStrohl on May 6, 2023 on LessWrong.
    There’s a kind of rhythm to the four phases of naturalism. To me, they feel a bit like oscillation between zooming out and zooming in, or like inhaling and exhaling. The first phase (Locating Fulcrum Experiences) has a broad exploratory feel, like zooming out; it’s sort of meandering, touching on this and that until I’ve found the right thing to hone in on. The second phase (Getting Your Eyes On) emphasizes focused attention, and involves a lot of precision and detail. It’s more like zooming in.
    This third phase, “Collection”, zooms out again. After mining a small number of experiences for all the detail you can perceive, the next step is to train yourself to notice every single instance of your fulcrum experience, no matter when, where, or how it happens.
    You may find that the experiences you previously studied in detail represented a narrow subset of all the instances that happen; the modal instance might be far more subtle, might happen in a context you didn’t anticipate, or might be quite different in some other way. By the end of this phase, you’ll likely perceive some large patterns in the way this experience shows up in your life, and you’ll be very familiar with the circumstances that give rise to it.
    How To Collect Experiences: Gamified Noticing
    In the most basic version of collection, you’ll simply tap your leg (or make some other planned gesture) when you notice the experience you’re watching for. As you may recall from a previous essay, I call this “marking”. You don’t necessarily have to do anything special with your mind when you mark an experience; you can just endeavor to explicitly notice the experience every time (that is, to notice that you’ve noticed it).
    But the standard version of collection, the one I do when I’m serious about a study, is a little bit gamified. Rather than just making a marking gesture, you’ll also keep a tally of how many times you noticed over the course of a day. Your goal each day is to beat your high score.
    There are a few good ways to keep a noticing tally. One way to do it is just to count in your head, which apparently works fine for some people. Or you could get a little tally device such as a knitting counter, a golf counter, or a counter ring, and keep that with you. There are also phone apps that mimic golf and knitting counters. Personally, I typically use a tiny notebook these days and make literal tally marks with a pen, since I often like to jot down a few words as field notes anyway.
    What I like most about the gamified version of collection is that the people who really get into it tend to find themselves steering toward situations where they’ll encounter their fulcrum experience. You might worry that steering toward fulcrum experiences would skew the data on how the experience shows up in daily life; and that’s true, it does. But I find that the more I notice something, the easier it becomes to notice in the future; and this effect has proven robust among people I’ve worked with. A person who gets a much higher tally in the first week or two of collection will be far better at catching all the uncontrived instances of their fulcrum experience in the third and fourth weeks. You can always stop using the tally once you’re very good at noticing the experience.
    I think it’s well worth the tradeoff, so in addition to suggesting this incentive structure, I actively encourage people to design artificial situations where they’ll encounter their fulcrum a lot. (More on that, as well as other concerns with gamification, in “Troubleshooting” below.)
    The Noticing Timeline
    Tapping your leg when you notice an experience may sound simple enough; but in fact it usually takes people some time before they can consistently mark right as the experience i...
    24 min
  • LW - Orthogonal's Formal-Goal Alignment theory of change by carado
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Orthogonal's Formal-Goal Alignment theory of change, published by carado on May 5, 2023 on LessWrong.
    we recently announced Orthogonal, an agent foundations alignment research organization. in this post, i give a thorough explanation of the formal-goal alignment framework, the motivation behind it, and the theory of change it fits in.
    the overall shape of what we're doing is:
    building a formal goal which would lead to good worlds when pursued — our best candidate for this is QACI
    designing an AI which takes as input a formal goal, and returns actions which pursue that goal in the distribution of worlds we likely inhabit
    backchaining: aiming at solutions
    one core aspect of our theory of change is backchaining: come up with a at least remotely plausibly story for how the world is saved from AI doom, and try to think about how to get there. this avoids spending lots of time getting confused about concepts that are confusing because they were the wrong thing to think about all along, such as "what is the shape of human values?" or "what does GPT4 want?" — our intent is to study things that fit together to form a full plan for saving the world.
    alignment engineering and agent foundations
    alignment is not just not the default, it's a very narrow target. as a result, there are many bits of non-obvious work which need to be done. alignment isn't just finding the right weight to sign-flip to get the AI to switch from evil to good; it is the hard work of putting together something which coherently and robustly points in a direction we like.
    as yudkowsky puts it:
    The idea with agent foundations, which I guess hasn't successfully been communicated to this day, was finding a coherent target to try to get into the system by any means (potentially including DL ones).
    agent foundations/formal-goal alignment is not fundamentally about doing math or being theoretical or thinking abstractly or proving things. agent foundations/formal-goal alignment is about building a coherent target which is fully made of math — not of human words with unspecified meaning — and figuring out a way to make that target maximized by AI. formal-goal alignment is about building a fully formalized goal, not about going about things in a "formal" manner.
    current AI technologies are not strong agents pursuing a coherent goal (SGCA). the reason for this is not because this kind of technology is impossible or too confusing to build, but because in worlds in which SGCA was built (and wasn't aligned), we die. alignment ultimately is about making sure that the first SGCA pursues desirable goal; the default is that its goal will be undesirable.
    this does not mean that i think that someone needs to figure out how to build SGCA for the world to end of AI; what i expect is that there are ways in which SGCA can emerge out of the current AI paradigm, in ways that don't let particularly us choose what goal it pursues.
    you do not align AI; you build aligned AI.
    because this emergence does not let us pick the SGCA's goal, we need to design an SGCA whose goal we do get to choose; and separately, we need to design such a goal. i expect that pursuing straightforward progress on current AI technology leads to an SGCA whose goal we do not get to choose and which leads to extinction.
    i do not expect that current AI technology is of a kind that makes it easy to "align"; i believe that the whole idea of building a strange non-agentic AI about which the notion of goal barely applies, and then to try and make it "be aligned", was fraught from the start. if current AI was powerful enough to save the world once "aligned", it would have already killed us before we "aligned" it. to save the world, we have to design something new which pursues a goal we get to choose; and that design needs to have this in mind from the start, rather than ...
    7 min
  • AF - Orthogonal's Formal-Goal Alignment theory of change by Tamsin Leake
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Orthogonal's Formal-Goal Alignment theory of change, published by Tamsin Leake on May 5, 2023 on The AI Alignment Forum.
    we recently announced Orthogonal, an agent foundations alignment research organization. in this post, i give a thorough explanation of the formal-goal alignment framework, the motivation behind it, and the theory of change it fits in.
    the overall shape of what we're doing is:
    building a formal goal which would lead to good worlds when pursued — our best candidate for this is QACI
    designing an AI which takes as input a formal goal, and returns actions which pursue that goal in the distribution of worlds we likely inhabit
    backchaining: aiming at solutions
    one core aspect of our theory of change is backchaining: come up with a at least remotely plausibly story for how the world is saved from AI doom, and try to think about how to get there. this avoids spending lots of time getting confused about concepts that are confusing because they were the wrong thing to think about all along, such as "what is the shape of human values?" or "what does GPT4 want?" — our intent is to study things that fit together to form a full plan for saving the world.
    alignment engineering and agent foundations
    alignment is not just not the default, it's a very narrow target. as a result, there are many bits of non-obvious work which need to be done. alignment isn't just finding the right weight to sign-flip to get the AI to switch from evil to good; it is the hard work of putting together something which coherently and robustly points in a direction we like.
    as yudkowsky puts it:
    The idea with agent foundations, which I guess hasn't successfully been communicated to this day, was finding a coherent target to try to get into the system by any means (potentially including DL ones).
    agent foundations/formal-goal alignment is not fundamentally about doing math or being theoretical or thinking abstractly or proving things. agent foundations/formal-goal alignment is about building a coherent target which is fully made of math — not of human words with unspecified meaning — and figuring out a way to make that target maximized by AI. formal-goal alignment is about building a fully formalized goal, not about going about things in a "formal" manner.
    current AI technologies are not strong agents pursuing a coherent goal (SGCA). the reason for this is not because this kind of technology is impossible or too confusing to build, but because in worlds in which SGCA was built (and wasn't aligned), we die. alignment ultimately is about making sure that the first SGCA pursues desirable goal; the default is that its goal will be undesirable.
    this does not mean that i think that someone needs to figure out how to build SGCA for the world to end of AI; what i expect is that there are ways in which SGCA can emerge out of the current AI paradigm, in ways that don't let particularly us choose what goal it pursues.
    you do not align AI; you build aligned AI.
    because this emergence does not let us pick the SGCA's goal, we need to design an SGCA whose goal we do get to choose; and separately, we need to design such a goal. i expect that pursuing straightforward progress on current AI technology leads to an SGCA whose goal we do not get to choose and which leads to extinction.
    i do not expect that current AI technology is of a kind that makes it easy to "align"; i believe that the whole idea of building a strange non-agentic AI about which the notion of goal barely applies, and then to try and make it "be aligned", was fraught from the start. if current AI was powerful enough to save the world once "aligned", it would have already killed us before we "aligned" it. to save the world, we have to design something new which pursues a goal we get to choose; and that design needs to have this in mind from the ...
    7 min
  • EA - What is effective altruism? How could it be improved? by MichaelPlant
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is effective altruism? How could it be improved?, published by MichaelPlant on May 5, 2023 on The Effective Altruism Forum.
    The EA community has been convulsing since FTX. There's been lots of discontent, but almost no public discussion between community leaders, and little in the way of a constructive suggestions for what could change. In this post, I offer a reconceptualisation of what the EA community is and then use that to sketch some ideas for how to do good better together.
    I’m writing this purely in my personal capacity as a long-term member of the effective altruism community. I drafted this at the start of 2023, in large part to help me process my own thoughts. The ideas here are still, by my lights, dissatisfyingly underdeveloped. But I’m posting it now, in its current state and with minimal changes, because it's suddenly relevant to topical discussions about how to run the Effective Ventures Foundation and the Centre for Effective Altruism and I don't know if I would ever make time to polish it.
    [I'm grateful to Ben West, Chana Messinger, Luke Freeman, Jack Lewars, Nathan Young, Peter Brietbart, Sam Bernecker, and Will Troy for their comments on this. All errors are theirs mine]
    Summary
    We can think of effective altruists as participants in a market for maximum impact activities. It’s much like a local farmers’ market, except people are buying and selling goods and services for how best to help others.
    Just like people in a market, EAs don’t all share the same goal - a marketplace isn’t an army. Rather, people have different goals, based on their different accounts of what matters. The participants can agree, however, that they all want there to be a marketplace to allow them to meet and trade; this market is useful because people want different things.
    Presumably, the EA market should function as a free, competitive market. This means lots of choice and debate among the participants. It requires the market administrators to operate a level playing-field.
    Currently, the EA community doesn’t quite operate like this. The market administrators - CEA, its staff and trustees - are also major market participants, i.e. promoting particular ideas and running key organisations. And the market is dominated by one big buyer (i.e. it’s a ‘monopsony’).
    I suggest some possible reforms: CEA to have its trustees elected by the community; it should strive to be impartial rather than take a stand on the priorities. I don’t claim this will solve all the issues, but it should help. I'm sure there are other implications of the market model I've not thought of.
    These reforms seem sensible even without any of EA’s recent scandals. I do, however, explain how they would likely have helped lessened these scandals too.
    I’ve tried to resist getting into the minutiae of “how would EA be run if modelled on a free market?” and I would encourage readers also to resist this. I want people to focus on the basic idea and the most obvious implications, not get stuck on the details.
    I’m not very confident in the below. It’s an odd mix of ideas from philosophy, politics, and economics. I wrote it up in the hope others can develop the ideas and I can stop ruminating on the “what should FTX mean for EA?” question.
    What is EA? A market for maximum-impact altruistic activities
    What is effective altruism? It's described by the website effectivealtruism.org as a "research field and practical community that aims to find the best ways to help others, and put them into practice". That's all well and good, but it's not very informative if we want to understand the behaviour of individuals in the community and the functioning of the community as a whole.
    An alternative approach is to think of effective altruists, the people themselves, in economic terms. In this case, we might characterise the effe...
    27 min