The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - More findings on maximal data dimension by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More findings on maximal data dimension, published by Marius Hobbhahn on February 2, 2023 on The AI Alignment Forum.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
    I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.
    The post contains a lot of figures, so the suggested length is deceiving. Code can be found in this colab notebook.
    This is the second in a series of N posts on trying to understand memorization in NNs.
    Executive summary
    I look at a variety of settings and experiments to better understand memorization in toy models. My primary motivation is to increase our general understanding of NNs but I also suspect that understanding memorization better might increase our ability to detect backdoors/trojans. This post specifically focuses on measuring memorization with the maximal data dimensionality metric.
    In a comment to the “Superposition, Memorization and double descent” paper, Chris Olah introduces maximal data dimensionality D, a metric that supposedly tells to which degree a network memorized a datapoint compared to using features that are shared between datapoints. I extend the research on this metric with the following findings
    In the double descent setting, the metric describes exactly what we would predict, i.e. with few inputs the network memorizes all datapoints and with a lot of input it learns some features.
    On MNIST, I can reproduce the shape of the D curve and also the findings that memorized datapoints have high D, datapoints that share many features are in the middle and datapoints that the network is confused about have low D. However, I was surprised to find that the datapoints the network misclassified on the training data are evenly distributed across the D spectrum. I would have expected them to all have low D didn’t learn them.
    When we train the network to different levels of accuracy, we find that the distribution of errors is actually slightly left-heavy instead of right-heavy. I have not yet understood why it is the case but I’d be interest in follow-up research to see whether it tells us something interesting.
    Different classes are not evenly distributed across the spectrum, e.g. “8” is far more regular than “5” according to D. This is what we would expect.
    Across different hidden sizes, the shape of the D curve stays nearly the same but the spearman rank correlation between the datapoints increases the larger the difference in hidden size. This means the more similar the number of neurons, the more similar is the in which D sorts the datapoints.
    Networks of the same size trained on the same data with different seeds show nearly identical D curves and have high spearman rank correlation. This is what we would expect.
    Different dataset sizes produce different shapes of D, e.g. larger datasets have more shared features (they are flatter in the middle). This seems plausible.
    Different levels of weight decay have nearly no effect on the shape of D. The minor effect they have is the opposite of what I would have expected.
    The shape of D changes very little between initialization and the final training run. This was unexpected and I have no good explanation for this phenomenon yet. When we measure D over different batches we find the same phenomenon.
    Working with D can be a bit tricky (see Appendix for practical tips). The more I played around with D, the more I’m convinced that it tells us something interesting. Particularly the question about misclassifications and error rates and the unexpectedly small change during initialization and final training run seem like they could tell us something about NNs that we don’t yet know.
    Maximal data dimensionality
    There are two models u...
    18 min
  • EA - CE Incubation Programs - applications now open + updates to the programs by KarolinaSarek
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CE Incubation Programs - applications now open + updates to the programs, published by KarolinaSarek on February 2, 2023 on The Effective Altruism Forum.
    Applications are now open for our two upcoming Incubation Programs:
    July-August 2023 with a focus on biosecurity interventions and large-scale global health interventions
    February-March 2024 with a focus on farmed animals and global health and development mass-media interventions
    In this post we set out some of the key updates we’ve made to the program, namely:
    Increased funding
    More time for participants in person in London
    Extended stipends and support to provide an even bigger safety net for participants
    Even more ongoing support after the program
    More time for applications
    Context:
    In four years we’ve launched 23 new effective charities that have reached 120 million animals and over 10 million people. The Incubation Program provides you with two months of intensive training, well-researched charity ideas, and access to the funding you need to launch. All we care about is impact, and the most pressing determinant of success is finding potential founders.
    APPLY HERE
    Updates to the Incubation Program
    All the details are here on our website, but below we summarize the latest changes/improvements.
    Increased quantity and probability of funding
    In recent years, in part due to our portfolio’s track record, we’re seeing a significant uptick in the seed funding being achieved by our incubatees. In the most recent round, for example, eight out of nine participants started organizations and received $732,000, with grants ranging from $100,000 to $220,000. The ninth participant joined the CE team as a research analyst.
    A Bigger Safety Net
    In the past two years, we’ve trained 34 people. After the program:
    20 launched new charities and raised over $1.2 million in seed funding
    6 got jobs in EA orgs (including CE)
    1 worked on mental health research with funding in Asia (and 1 year later become a co-founder of a newly incubated by CE mental health charity)
    1 worked as a senior EA community manager
    1 got funded to do their own specialist research project and has since hired 3 people
    2 launched their own grantmaking foundation
    1 works for that grantmaking foundation
    1 is running for office in America and was elected to the district parliament
    1 kept on working on the project they co-founded in the alternative protein space
    1 runs a charity evaluator in China
    1 was hired by one of the previously incubated charities
    So in summary: 100% of participants, within weeks of finishing the program, landed relevant roles with high personal fit and excellent impact potential.
    During the program we will provide you with:
    Stipends to cover your living costs during the Incubation Program (e.g., rent, wifi, food, childcare). The stipends are around $2,000 per month and are based on participants' needs and adjusted accordingly.
    Travel and board costs for the 2 weeks in person in London.
    If, for any reason, you do not start a charity after the program, we provide:
    Career mentorship (our track record for connecting non-founder participants to research grants, related jobs, and other pathways to impact is near 100%).
    Two-month stipends to provide a safety net during the period of looking for alternative opportunities.
    More time in-person in London
    The Incubation Program lasts 8 weeks, followed by a 2 week seed-funding process.
    The 8 week program runs online, now with 2 weeks in person in CE’s London office
    During the 2 week seed-funding process you make final improvements to your proposal, which is submitted to the CE seed network that makes the final decision on your grant.
    Even more support after the program
    You will graduate the program with a co-founder, a high-quality charity idea, a plan for implementation, and a robust funding proposal. On top ...
    6 min
  • EA - Epoch Impact Report 2022 by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Epoch Impact Report 2022, published by Jaime Sevilla on February 2, 2023 on The Effective Altruism Forum.
    Epoch is a research group forecasting the development of transformative Artificial Intelligence. We try to understand how progress in AI happens and what economic impacts we might see from advanced AI.
    We want to enable better governance during this economic transition by gathering information about the timing of new developments, studying which levers can be used to influence AI progress and making current and past trends in ML more understandable.
    Founded in April of 2022, Epoch currently has a staff of 13 people, corresponding to 9 FTEs. We have received $1.96M in funding through a grant from Open Philanthropy. We are fiscally sponsored and operationally supported by Rethink Priorities, whose Special Projects team has been a core part of our success as an organisation.
    Epoch is fundraising a total of $7.07M over 2 years, or approximately $2.91M for September 2023 to September 2024, and $4.16M for September 2024 to September 2025. A detailed budget can be found in the full report.
    With this funding, we expect to continue and expand our research capacity in understanding the future of AI. Through our research, we hope to continue informing about key drivers of AI progress and when key AI capabilities will be developed.
    You can now donate to Epoch and subscribe to our newsletter.
    Read the full report to learn about our key insights this year, an overview of our research and non-research activities, and our plans for the future.
    Epoch is one of the coolest (and in my opinion underrated) research orgs for understanding trends in ML. Rather than speculating, they meticulously analyze empirical trends and make projections for the future. Lots of interesting findings in their data! – Jacob Steinhardt
    Read now the full report
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Research agenda: Formalizing abstractions of computations by Erik Jenner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research agenda: Formalizing abstractions of computations, published by Erik Jenner on February 2, 2023 on LessWrong.
    Big thanks to Leon Lang, Jérémy Scheurer, Adam Gleave, and Shoshannah Tekofsky for their feedback on a draft of this post, to Euan McLean (via FAR AI) for his feedback and a lot of help with editing, and to everyone else who discussed this agenda with me, in particular Johannes Treutlein for frequent research check-ins!
    Summary
    My current agenda is to develop a formal framework for thinking about abstractions of computations. These abstractions are ways to partially describe the “algorithm” a neural network or other computation is using, while throwing away irrelevant details.
    Ideally, this framework would tell us 1) all possible abstractions of a given computation, and 2) which of these are most useful (for a specific purpose, such as detecting deception). “Useful” doesn’t necessarily mean “easily human-understandable”—I see that as an open question.
    I anticipate the main applications to alignment to be automating interpretability or mechanistic anomaly detection. There are also potential connections to other alignment topics, such as natural abstractions or defining terms like “search process”.
    This agenda is at an early stage (I have been thinking about it for ~2 months, and about related topics for another ~2 months before that). So feedback now could change my future direction.
    I also list a few potential projects that seem self-contained. If you’re interested in working on any of those, or collaborating in some other way, please get in touch!
    I encourage you to skip around and/or only read parts of the post. Here’s an overview:
    Introduction and Q&A mostly talk about motivation and connections to alignment..
    What are abstractions of computations? discusses my current guess as to what the framework should look like.
    There’s a list of Some potential projects.
    Appendix: Related work gives a quick overview of relevant work in academia, and the relation between this agenda and other alignment research.
    This post doesn’t contain any actual theorems or experiments, so if you’re only interested in that, you can stop reading.
    Introduction
    Humans can’t just look at the weights of a neural network and tell what it’s doing. There are at least two reasons for this:
    Neural network weights aren’t a format we’re great at thinking about.
    Neural networks are often huge.
    The second point would likely apply to any system that does well on complicated tasks. For example, neural networks are decision trees, but this doesn’t mean we can look at the decision tree corresponding to a network and understand how it works. To reason about these systems, we will likely have to simplify them, i.e. throw away details that are irrelevant for whatever we want to find out. In other words, we are looking for abstractions of computations (such as neural networks).
    Abstractions are already how we successfully reason about many other complicated systems. For example, if you want to understand the Linux kernel, you wouldn’t start by reading the entire source code top to bottom. Instead, you’d try to get a high-level understanding—what are the different modules, how do they interact? Similarly, we use pseudocode to more easily communicate how an algorithm works, abstracting away low-level details.
    If we could figure out a general way to find useful abstractions of computations, or at least of neural networks, perhaps we could apply this to understand them in a similar way. We could even automate this process and mechanically search for human-understandable abstractions.
    Making things easier to understand for humans isn’t the only application of abstractions. For example, abstractions have been used for more efficient theorem proving and in model checking (e.g. abstract interpretation). ...
    53 min
  • LW - Can we “cure” cancer? by jasoncrawford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can we “cure” cancer?, published by jasoncrawford on February 1, 2023 on LessWrong.
    In an excellent recent essay on “big visions for biology,” Sam Rodriques writes:
    Ask most biologists about the cure for cancer, and they will tell you it doesn’t exist: cancer is many diseases that are mostly unrelated to each other, and that all have to be cured one at a time.
    Are “most biologists” right about this?
    We can get perspective on this from the history of infectious disease. After all, infection was also “many diseases,” with disparate causes (viruses, bacteria, protozoa) and disparate pathways (air, water, food, insects). And yet, while we did not exactly “cure infectious disease” (just ask the 248,544 people who got covid last week in the US alone), we did reduce every major metric of infectious disease by 90% to 99% in wealthy/developed countries:
    Overall mortality from infectious disease reduced by 90% in the 20th century alone
    Youth mortality (death before age 15) reduced from almost half to well under 1%
    Maternal mortality reduced from almost 1% to under 0.01%
    Surgical mortality reduced from half or more (for some major surgeries at least) to around 1% to 2%
    Infectious disease was not the whole story in all of the above metrics, but it was the biggest factor in all of them.
    We also eradicated smallpox worldwide and eliminated diseases such as cholera, malaria, and polio from many countries:
    Did we have to cure all these diseases “one at a time”? No, not exactly:
    Sanitation and hygiene efforts were effective against broad classes of germs: e.g., water sanitation was effective against all water-borne diseases.
    Disinfectants and antiseptics, such as bleach, are effective against most if not all germs.
    Antibiotic drugs are effective against broad classes of bacteria.
    Vaccines do have to be developed one disease at a time—but even here there are general techniques for creating them. In fact, with mRNA technology, we now have a very general technique that can create new vaccines with relatively little adaptation.
    Underlying all of this is a theory that explains the basic causal mechanism of infection: the germ theory. The theory alone doesn’t give us everything we need to cure each specific disease: new research is needed to understand the etiology and epidemiology of each one. But the theory does give us a conceptual framework and a set of tools to guide that research. Before the theory, we were able to make limited progress against some diseases; after it, we made much more rapid progress, and the diseases we weren’t able to solve became the exception rather than the rule.
    I think we are in the pre-theory stage for cancer. We are able to make progress against some forms of cancer, as we reduced lung cancer by public health efforts against smoking. But we don’t, to my knowledge, have the fundamental theory that we need, and so overall progress is slow. [UPDATE: @DirectedEvolution has convinced me that “pre-theory” is wrong, see their excellent comment below.]
    Infection and cancer are both “many diseases”—but those diseases have something in common. A deeper understanding of cancer will allow us to make much more rapid progress, on more fronts. There won’t be a silver bullet—no single treatment that will cure all cancers. More likely there will be a few major techniques, as with infection we had to develop sanitation, vaccination, and antibiotics; and even within each of those categories, we needed many specific efforts to develop each individual vaccine, antibiotic drug, and sanitation method.
    But there is no reason why we can’t do for cancer what we did for infection: reduce it by orders of magnitude and knock it down from a number one cause of death to a much more minor and more manageable threat. And eventually, with more advanced science and technology, perhaps we wil...
    4 min
  • EA - What advice would you give someone who wants to avoid doxing themselves here? by Throwaway012723
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What advice would you give someone who wants to avoid doxing themselves here?, published by Throwaway012723 on February 2, 2023 on The Effective Altruism Forum.
    From my recent comment, I gathered from the upvotes that there was some interest in people wanting me to share my negative experience with the community health team.
    My biggest concern in writing that post is accidentally doxing myself.
    The advice I'm seeking is, how do I include details to show the complexity of the situation the community health team dealt with (as I think this would be of public interest and only fair to them) without me accidentally doxing myself?
    Including details is hard as I have many friends in the EA Community, and I don't want them to identify me.
    Including simple details reduces the option space in people's minds of who could be writing this. I think with 2 or 3 simple details (e.g., in city X then in city Y), someone I'm friends with can be 95% sure that Y is writing.
    To illustrate my point, I intentionally chose "many friends" in the bullet point above as revealing an order of magnitude also reduces the option space in people's minds.
    But I also appreciate that people can read between the lines and still conclude the details I wanted to avoid. For example, intentionally saying "many friends" still suggests that I have a large number, so it was irrelevant whether I said an order of magnitude.
    As has been done in EA Forum posts that I've seen from others, I could include false details to throw people off, but I believe this is dishonest.
    The community health team will almost certainly know who I am from the post, as I will be sharing excerpts from my emails with them. I'm not worried about them doxing me as they would know if they do, they'll reduce the trust that the EA community has in them, and this is not what they want.
    However, as they will almost certainly reply to the forum post, I'm concerned they will reveal details I wanted to avoid revealing as I knew that those details would make it easier for others to identify me.
    Currently, my solution is to not worry too much about including and set the moderation guidelines on the post to "Reign of Terror - I delete anything I judge to be annoying or counterproductive". I set those guidelines here to see what it's like to have them on.
    I'm bullet-pointing as I have a distinctive writing style amongst my friends that I'm trying to avoid in these posts.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Exercise is Good, Actually by Gordon Seidoh Worley
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exercise is Good, Actually, published by Gordon Seidoh Worley on February 2, 2023 on LessWrong.
    Yesterday I wrote about how dressing nice is a good idea. This post is written similarly, but the topic is exercise. This is again somewhat casual writing that's in the spirit of what I wish someone had told me when I was 15.
    Until about 10 years ago I didn't care about exercise. I figured I could get on just fine in life without really being in shape. Sure, I knew that exercise was supposed to be good for me and I'd make half hearted attempts to include some activity in my life, but I never went to the gym or intentionally tried to get stronger or fitter.
    I was carrying around a story in my head like this one:
    I care about doing important intellectual and professional work that depends on my mind.
    Physical exercise doesn't much impact my ability to do that type of work.
    Additionally, only muscle-heads go to the gym all the time. Regular people just get enough exercise from living their lives.
    And besides, smart people in movies are usually weak anyway. Their power comes from being smart, not strong.
    I'm smarter than average, so why do I need to be strong?
    Conclusion: I don't need to work out or go to the gym or even care about physical fitness.
    I think a lot of this story can be traced back to how characters in stories work. They're created like D&D characters: they get a certain number of points and the author distributes those points around to different attributes. Some characters have more points than others to work with, but at the end of the day you have to make tradeoffs. Want to be really smart? Gonna have to take that out of something else. What to be really, really, really smart? You're going to have to have to put basically all of your points into intelligence and give up everything else.
    Authors do this because it helps make characters more relatable. When someone is too good at too many things the story gets boring because they don't face any real challenges, and we might even envy and hate a character that's too capable because they make us feel inadequate. But this isn't how real people work. In fact, positive traits tend to correlate in people. Yes, you find some jocks with tiny brains, politicians with terrible looks, and geniuses with major health problems, but most positive traits correlate in real humans so that it's not unreasonable for someone to be strong, smart, and good looking.
    Dismissing the need for exercise is a kind of coping mechanism for not being above average in physical ability. This is certainly how I coped. I have asthma. My peak flow is like 30% below normal (this means I can exchange less air with each breath). Even with significant training I can't do things lots of people do, like go jogging or ride a bike and hold a conversation; instead, I'm breathing heavily after 30 seconds. My best performance at physical activities, even after significant training, is just up to the level of what the average fit person can do with zero conditioning. So to protect my self-esteem it was important to think that exercise was for dumb people who weren't smart enough to do smart things like me.
    But this is just nonsense. I actually like doing things with my body. I like hiking and kayaking and mountain climbing and dancing. I'm just not very good at them, even with significant work (e.g. I've never managed to send a bouldering problem past V3 despite climbing 1-3 times a week for 7 years and getting training). That doesn't mean I can't still enjoy them. I just have limits to how much I can achieve.
    Knowing those limits, it's okay to accept the conventional wisdom that exercise is good. We live in our bodies. Even if you mostly care about your mind, your mind lives in your brain, and your brain lives in your body, so you do well to take care of your ...
    5 min
  • LW - You Don't Exist, Duncan by Duncan Sabien
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You Don't Exist, Duncan, published by Duncan Sabien on February 2, 2023 on LessWrong.
    This is an experimental essay, not in the typical LessWrong or Duncan Sabien style.
    Depending on how this goes, I might try writing a companion piece in the typical style, laying out the model clearly and explicitly and deriving concrete and specific recommendations from it.
    But it seemed worth it to try communicating at a lower and more emotional/visceral level, not least because that is the level at which I actually experience The Problem. Any clear, analytical essay would be the result of me trying to make sense of the thing that I'm going to try to directly convey, below.
    It is the year 1995. I am nine years old. In front of me there is a sheet of paper, upon which are written a dozen or so lines of math. The first is:
    I stare at it. I know that I can divide both sides of the equation by x, leaving me with:
    ...but this does not seem to do any good.
    I raise my hand. The afterschool volunteer comes over.
    "No," he says. "That's not right. X isn't a term on the left side. F is a function."
    He has explained nothing.
    "F is a function, so what this is saying is to take X, and square it, and add seven."
    I look up at him, confused. I am nine. I have never heard the word "function" used in this way before. No one has grounded me in the activity of the day; no one has oriented me; no one has told me today you are learning what a function is, and you will learn by looking at a bunch of examples. No one has said today, parentheses don't mean the thing you're expecting them to mean. No one has said f is a thing that eats xs, and what the right side is showing you is how it eats them—what it does to them.
    "So, like, if X is three, right?" he continues. "X is three? So F of X is three squared plus seven, which is sixteen."
    I say the words again in my mind, more slowly. F ... of ... (of? What?) ... X. ""F of X"" (okay, whatever, that's nonsense, but whatever) is sixteen.
    I look back down at the paper. If the right side of the equation is sixteen, and X is three...
    "F is five-point-three-repeating," I say, trying to inject a measure of confidence I do not feel into my tone.
    "What? No. F isn't anything. F is a function. It's not part of the equation."
    Not part of the equation, he says. Looking back from a distance of twenty-five years, I see (one of) his mistake(s). He doesn't tell me this isn't really an equation at all, not the way you're thinking of it. He doesn't tell me the equals sign here is more like telling you the definition of this thing, F of X—what F of X is is the thing on the other side of the equals sign. He doesn't say a function is when you set up a rule for dealing with numbers, and this rule is, whatever number you put in, you're going to square it, and add seven.
    Instead, he looks at me, and says more words, and the message lurking behind the words—the message implicit in his tone and posture and air of tolerant patience—is:
    I have given you an adequate explanation. If you were the kind of person who was good at math, my explanation would have been sufficient, and you would now understand. You still do not understand. Therefore...?
    My heart rate quickens.
    It is 1993. I am seven years old, roughhousing with my older brother and my father on the living room carpet. We clamber over top of him, laughing, pummeling him with tiny fists. He throws us both onto the couch, where we recover and launch ourselves back at him like pouncing tigers.
    My father tosses my brother back into the cushions a second time, grabs me in a gentle headlock, digs his knuckles into my scalp in a painful noogie.
    "Ow!" I shout, rolling away from him and clutching my head. "Ow. Ow."
    The pain is bright and hot, feeling halfway between a cut and a burn. Five seconds pass, and it has not yet begun to fade.
    "That...
    14 min
  • LW - Product safety is a poor model for AI governance by Richard Korzekwa
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Product safety is a poor model for AI governance, published by Richard Korzekwa on February 1, 2023 on LessWrong.
    Note: This post is intended to be accessible to readers with relatively little background in AI safety. People with a firm understanding of AI safety may find it too basic, though they may be interested in knowing which kinds of policies I have been encountering or telling me if I’ve gotten something wrong.
    I have recently encountered many proposals for policies and regulations intended to reduce risk from advanced AI. These proposals are highly varied and most of them seem to be well-thought-out, at least on some axes. But many of them fail to confront the technical and strategic realities of safely creating powerful AI, and they often fail in similar ways. In this post, I will describe a common type of proposal1 and give a basic overview of the reasons why it is inadequate. I will not address any issues related to the feasibility of implementing such a policy.
    Caveats
    It is likely that I have misunderstood some proposals and that they are already addressing my concerns. Moreover, I do not recommend dismissing a proposal because it pattern-matches to product safety on the surface level.
    These approaches may be good when combined with others. It is plausible to me that an effective and comprehensive approach to governing AI will include product-safety-like regulations, especially early in the process when we’re still learning and setting the groundwork for more mature policies.
    The product safety model of AI governance
    A common AI policy structure, which I will call the ‘product safety model of AI governance’2, seems to be built on the assumption that, while the processes involved in creating powerful AI may need to be regulated, the harm from a failure to ensure safety occurs predominantly when the AI has been deployed into the world. Under this model, the primary feedback loop for ensuring safety is based on the behavior of the model after it has been built. The product is developed, evaluated for safety, and either sent back for more development or allowed to be deployed, depending on the evaluation. For a typical example, here is a diagram from a US Department of Defense report on responsible AI:3
    The system in this diagram is not formally evaluated for safety or performance until after “Acquisition/Development”.4
    I do not find it surprising that this model is so common. Most of the time when we are concerned about risk from technology, we are worried about what happens when the technology has been released into the world. A faulty brake line on a car is not much of a concern to the public until the car is on public roads, and the facebook feed algorithm cannot be a threat to society until it is used to control what large numbers of people see on their screens. I also think it is reasonable to start with regulations that are already well-understood, and work from there. But this model, on its own, is inadequate for AI, for reasons I will explain in the next section.
    This model fails when applied to advanced AI
    This approach cannot be relied on to prevent the major risks associated with advanced AI. I will give two reasons for this, though there may be others.
    Safety cannot (yet) be evaluated reliably in a fully-developed AI
    A fundamental difficulty of ensuring that AI will behave as intended is that examining an AI or observing its behavior during development and testing is not a reliable indicator of how it will behave once deployed. This is a difficult, unsolved problem that resists simple solutions.5 I will not explain this problem in detail, but I will list some aspects of it here, along with real-world examples, and you can read more about it elsewhere.67
    AI may learn approaches to problems or, if sufficiently advanced, acquire goals that result ...
    9 min
  • EA - Questions about OP grant to Helena by DizzyMarmot
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Questions about OP grant to Helena, published by DizzyMarmot on February 2, 2023 on The Effective Altruism Forum.
    TL;DR
    A biosecurity & pandemic preparedness grant was made by Open Philanthropy to Helena for $500,000 in Nov 2022
    The grant profile raises questions about how it was justified and approved
    There is scant public information that could justify it as the best-placed and most appropriate recipient, a clear risk of nepotism inherent in the recipient organization, and what appear to be previous false representations made by the recipient organization
    This post asks OP to, in an effort to support accountability and transparency, publish further grant details and the investigation documents for the grant
    Notes: I apologize in advance as I am not experienced in writing using the EA syntax and structure, all errors are my own. Full disclosure, I previously applied to and was rejected for a role at Open Philanthropy. I work for an organization that in the past has been a recipient of funding from OP, as a person who has an affinity with but does not identify strongly as an EA. I think it is a good thing for accountability, transparency, and the broader governance of EA that OP listed the grant online, that I am able to question the grant in this forum and expect that OP will respond in good faith.
    Content
    I have watched with interest as EA’s influence has grown within global health and development, in particular Open Philanthropy’s work in health security and biosecurity. Recent calls for transparency and accountability within effective altruism have thankfully been met by new initiatives like openbook.fyi. In light of this, a particular grant seems like it warrants further inspection and from OP’s perspective raises enough questions that it should have been accompanied by more information.
    I occasionally trawl the grants databases for philanthropic foundations working in global health & health security and found a $500,000 grant to ‘Helena’ awarded in November 2022. I didn’t think anything of it until I saw it again in the openbook log, and upon seeing it looked into the organization.
    I hope that my initial perception of the grant and the organization is entirely wrong, but the grant is striking in that it has scant details on what is a sizable grant to an organization with no track record in the subject matter that appears to have pretty clearly misrepresented itself in the recent past.
    The grant commitment and recipient organization as they stand are emblematic of the challenges in effective and equitable grantmaking for philanthropies like Open Philanthropy. Henry Elkus and Helena’s path can be summed up as:
    Henry’s VC dad uses his resources to support Henry’s interests
    Henry drops out of school because he thinks he is exceptionally smarter and better equipped to solve 'our problems’
    Henry starts an obtuse org with for-profit venture capital and non-profit 501(c)3 sides
    The organization has a confusing and meandering scope of work, that appears to be a collection of personal interests at each point in time
    Has no proven tangible value-add beyond ‘networking’, with previous programmatic activities summed up as funding a semi-academic sociology event and acting as a rogue PPE procurement agency during COVID
    Helena then gets half a million from OP to explore ‘policy related to health security’ work that they have no track record of doing or apparent expertise in
    It feels wrong when Helena (subjectively) seems like a self-aggrandizing nepotism project at best, and (objectively) gets a large grant with no details and no track record in the area of work. It is also notable that Henry and Helena have no track record of participating in EA settings. Participating in transparency and holding your grantmaking to account meaningfully are two different things. This all lea...
    9 min