The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My First Post, published by Jaivardhan Nawani on September 7, 2023 on LessWrong.

Hello all!

I'm Jaivardhan Nawani, a 12-year-old enthusiast on rationality and how using it can improve our day-to-day lives.

Recently, I discovered this forum from a few friends who introduced me to many useful principles, and I thought to come up with an idea to make rationality easier to implement for first-timers.

So here's a basic theorem that I made upon this concept. I've come to LessWrong looking for feedback on it. And that's mostly it.

Here's the link to my theorem:

Have a good day and with regards;

Jaivardhan Nawani.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Recreating the caring drive, published by Catnee on September 7, 2023 on The AI Alignment Forum.

TL;DR: This post is about value of recreating "caring drive" similar to some animals and why it might be useful for AI Alignment field in general. Finding and understanding the right combination of training data/loss function/architecture/etc that allows gradient descent to robustly find/create agents that will care about other agents with different goals could be very useful for understanding the bigger problem. While it's neither perfect nor universally present, if we can understand, replicate, and modify this behavior in AI systems, it could provide a hint to the alignment solution where the AGI "cares" for humans.

Disclaimers: I'm not saying that "we can raise AI like a child to make it friendly" or that "people are aligned to evolution". Both of these claims I find to be obvious errors. Also, I will write a lot about evolution, as some agentic entity, that "will do that or this", not because I think that it's agentic, but because it's easier to write this way. I think that GPT-4 have some form of world model, and will refer to it a couple of times.

Nature's Example of a "Caring Drive"

Certain animals, notably humans, display a strong urge to care for their offspring.

I think that part of one of the possible "alignment solutions" will look like the right set of training data + training loss that allow gradient to robustly find something like a "caring drive" that we can then study, recreate and repurpose for ourselves. And I think we have some rare examples of this in nature already. Some animals, especially humans, will kind-of-align themselves to their presumable offspring. They will want to make their life easier and better, to the best of their capabilities and knowledge. Not because they "aligned to evolution" and want to increase the frequency of their genes, but because of some strange internal drive created by evolution. The set of triggers tuned by evolution, activated by events associated with the birth will awake the mechanism.

It will re-aim the more powerful mother agent to be aligned to the less powerful baby agent, and it just so happens that their babies will give them the right cues and will be nearby when the mechanism will do its work. We will call the more powerful initial agent that changes its behavior and tries to protect and help its offspring "mother" and the less powerful and helpless agent "baby". Of course the mechanism isn't ideal, but it works well enough, even in the modern world, far outside of initial evolutionary environment. And I'm not talking about humans only, stray urban animals that live in our cities will still adapt their "caring procedures" to this completely new environment, without several rounds of evolutionary pressure. If we can understand how to make this mechanism for something like a "cat-level" AI, by finding it via gradient descend and then rebuild it from scratch, maybe we will gain some insides into the bigger problem.

The rare and complex nature of the caring drive in contrast to simpler drives like hunger or sleep.

What do I mean by "caring drive"? Animals, including humans, have a lot of competing motivations, "want drives", they want to eat, sleep, have sex, etc. It seems that the same applies to caring about babies. But it seems to be much more complicated set of behaviors. You need to:correctly identify your baby, track its position, protect it from outside dangers, protect it from itself, by predicting the actions of the baby in advance to stop it from certain injury, trying to understand its needs to correctly fulfill them, since you don't have direct access to its internal thoughts etc.Compared to "wanting to sleep if active too long" or "wanting to eat when blood sugar level is low" I would confidently s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sharing Information About Nonlinear, published by Ben Pace on September 7, 2023 on The Effective Altruism Forum.

Epistemic status: Once I started actively looking into things, much of my information in the post below came about by a search for negative information about the Nonlinear cofounders, not from a search to give a balanced picture of its overall costs and benefits. I think standard update rules suggest not that you ignore the information, but you think about how bad you expect the information would be if I selected for the worst, credible info I could share, and then update based on how much worse (or better) it is than you expect I could produce. (See section 5 of this post about Mistakes with Conservation of Expected Evidence for more on this.) This seems like a worthwhile exercise for at least non-zero people to do in the comments before reading on. (You can condition on me finding enough to be worth sharing, but also note that I think I have a relatively low bar for publicly sharing critical info about folks in the EA/x-risk/rationalist/etc ecosystem.)

tl;dr: If you want my important updates quickly summarized in four claims-plus-probabilities, jump to the section near the bottom titled "Summary of My Epistemic State".

When I used to manage the Lightcone Offices, I spent a fair amount of time and effort on gatekeeping - processing applications from people in the EA/x-risk/rationalist ecosystem to visit and work from the offices, and making decisions. Typically this would involve reading some of their public writings, and reaching out to a couple of their references that I trusted and asking for information about them. A lot of the people I reached out to were surprisingly great at giving honest references about their experiences with someone and sharing what they thought about someone.

One time, Kat Woods and Drew Spartz from Nonlinear applied to visit. I didn't know them or their work well, except from a few brief interactions that Kat Woods seems high-energy, and to have a more optimistic outlook on life and work than most people I encounter.

I reached out to some references Kat listed, which were positive to strongly positive. However I also got a strongly negative reference - someone else who I informed about the decision told me they knew former employees who felt taken advantage of around things like salary. However the former employees reportedly didn't want to come forward due to fear of retaliation and generally wanting to get away from the whole thing, and the reports felt very vague and hard for me to concretely visualize, but nonetheless the person strongly recommended against inviting Kat and Drew.

I didn't feel like this was a strong enough reason to bar someone from a space - or rather, I did, but vague anonymous descriptions of very bad behavior being sufficient to ban someone is a system that can be straightforwardly abused, so I don't want to use such a system. Furthermore, I was interested in getting my own read on Kat Woods from a short visit - she had only asked to visit for a week. So I accepted, though I informed her that this weighed on my mind. (This is a link to the decision email I sent to her.)

(After making that decision I was also linked to this ominous yet still vague EA Forum thread, that includes a former coworker of Kat Woods saying they did not like working with her, more comments like the one I received above, and links to a lot of strongly negative Glassdoor reviews for Nonlinear Cofounder Emerson Spartz's former company "Dose". Note that more than half of the negative reviews are for the company after Emerson sold it, but this is a concerning one from 2015 (while Emerson Spartz was CEO/Cofounder): "All of these super positive reviews are being commissioned by upper management. That is the first thing you should know ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sharing Information About Nonlinear, published by Ben Pace on September 7, 2023 on LessWrong.

Epistemic status: Once I started actively looking into things, much of my information in the post below came about by a search for negative information about the Nonlinear cofounders, not from a search to give a balanced picture of its overall costs and benefits. I think standard update rules suggest not that you ignore the information, but you think about how bad you expect the information would be if I selected for the worst, credible info I could share, and then update based on how much worse (or better) it is than you expect I could produce. (See section 5 of this post about Mistakes with Conservation of Expected Evidence for more on this.) This seems like a worthwhile exercise for at least non-zero people to do in the comments before reading on. (You can condition on me finding enough to be worth sharing, but also note that I think I have a relatively low bar for publicly sharing critical info about folks in the EA/x-risk/rationalist/etc ecosystem.)

tl;dr: If you want my important updates quickly summarized in four claims-plus-probabilities, jump to the section near the bottom titled "Summary of My Epistemic State".

When I used to manage the Lightcone Offices, I spent a fair amount of time and effort on gatekeeping - processing applications from people in the EA/x-risk/rationalist ecosystem to visit and work from the offices, and making decisions. Typically this would involve reading some of their public writings, and reaching out to a couple of their references that I trusted and asking for information about them. A lot of the people I reached out to were surprisingly great at giving honest references about their experiences with someone and sharing what they thought about someone.

One time, Kat Woods and Drew Spartz from Nonlinear applied to visit. I didn't know them or their work well, except from a few brief interactions that Kat Woods seems high-energy, and to have a more optimistic outlook on life and work than most people I encounter.

I reached out to some references Kat listed, which were positive to strongly positive. However I also got a strongly negative reference - someone else who I informed about the decision told me they knew former employees who felt taken advantage of around things like salary. However the former employees reportedly didn't want to come forward due to fear of retaliation and generally wanting to get away from the whole thing, and the reports felt very vague and hard for me to concretely visualize, but nonetheless the person strongly recommended against inviting Kat and Drew.

I didn't feel like this was a strong enough reason to bar someone from a space - or rather, I did, but vague anonymous descriptions of very bad behavior being sufficient to ban someone is a system that can be straightforwardly abused, so I don't want to use such a system. Furthermore, I was interested in getting my own read on Kat Woods from a short visit - she had only asked to visit for a week. So I accepted, though I informed her that this weighed on my mind. (This is a link to the decision email I sent to her.)

(After making that decision I was also linked to this ominous yet still vague EA Forum thread, that includes a former coworker of Kat Woods saying they did not like working with her, more comments like the one I received above, and links to a lot of strongly negative Glassdoor reviews for Nonlinear Cofounder Emerson Spartz's former company "Dose". Note that more than half of the negative reviews are for the company after Emerson sold it, but this is a concerning one from 2015 (while Emerson Spartz was CEO/Cofounder): "All of these super positive reviews are being commissioned by upper management. That is the first thing you should know about Spartz, and I...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nick Beckstead is leaving the Effective Ventures boards, published by Eli Rose on September 6, 2023 on The Effective Altruism Forum.

On 23rd August, Nick Beckstead stepped down from the boards of Effective Ventures UK and Effective Ventures US.

For context, EV UK and EV US host and fiscally sponsor several (mostly EA-related) projects, such as CEA, 80,000 Hours and various others (see more here).

Since November 2022, Nick has been recused from all board matters related to the collapse of FTX. Over time, it became clear that Nick's recusal made it difficult for him to add sufficient value to EV and its projects for it to be worth him remaining on the boards. Nick and the other trustees felt that this was sufficient reason for Nick to step down.

Nick wanted to share the following:

Ever since the collapse of FTX, I've been recused from a substantial fraction of business on both boards. This has made it hard to contribute as much as I would like to as a board member, during a time where engaged board members are especially important. Since this situation may not change for a while, I think it's a good time for me to step down.

I am grateful to have played a role in getting EV UK and EV US off the ground and helping them develop over the last 14 years since the launch of Giving What We Can. Projects at EV have accomplished a great deal, drawing substantial resources and attention toward addressing some of the world's most pressing problems, with impacts that are varied, large, and difficult to quantify. The people at EV are amongst the most thoughtful, generous, kind, and dedicated that I've had the pleasure to interact with. I feel very proud of all that we have accomplished together, and optimistic about the work that will continue in my absence.

As a founding board member of EV UK (then called CEA), Nick played a vital role in getting EV US, EV UK and their constituent projects off the ground. For example, Nick was involved in setting up the first Giving What We Can student group and helped to hire the first full-time staff at what was then CEA. We are very grateful to Nick for everything he's contributed to the effective altruism movement to date and look forward to his future positive impact; we wish him the best of luck with his future work.

This is because the recusal affected not just decisions that were directly related to the collapse of FTX, but also many other decisions for which the way EV UK and EV US have been affected by the collapse of FTX was important context.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Find Hot French Food Near Me: A Follow-up, published by aphyer on September 6, 2023 on LessWrong.

On Zvi's recent post about French food I posted an inflammatory comment (saying in essence that French food is so bad American capitalism hasn't even bothered stealing it). I got challenged to provide evidence supporting this, and particularly to back up my claim that there were more German than French restaurants near me.

Right. Yes. Evidence. I am a reasonable adult who understands that beliefs must be supported by evidence. So. Here we go.

Some Google Searches

I've searched for '[ethnicity] restaurant near Grove Street, Jersey City, NJ' (I live in Jersey City, and the Grove Street area is reasonably near the center).

When I search for 'French' I can count 13 results:

And when I search for 'German' I count only 9:

Ha! The foolish American has been hoisted on his own petard! ('Petard' is French for 'fuck you').

Perhaps unsurprisingly, I don't think these numbers tell the whole story.

What Makes These Places French?

Google's definition of 'French' and 'German' restaurants here appears to be extremely expansive.

Hudson Hound Jersey City, an 'Irish gastropub', shows up on the French search.

Shadman, a 'go-to for Pakistani and Indian cuisine', shows up on the German search.

Luna, for 'Italian eats', shows up on the French search.

Frankie, an 'Australian eatery', shows up on the German search.

So, for lack of anything better to do, I've gone through manually to look for things that I think 'count' as French or German.

The two 'real' German places (and the ones I was thinking of in my comment) are 'Wurstbar' and 'Zeppelin Hall Beer Garden', and while we may question the taste of these places I do not think we can question their German-ness. The search also turned up 'Hudson Hall', a 'Euro beer bar with house-smoked meats', which I think at least ambiguously might count.

It's less clear to me how many of the hits for 'French restaurant' are actually both French and restaurants. Certainly I've been to a few of these places, and none of them have charged me twenty-three dollars for a baguette while sneering at me. We have:

Cafe Madelaine describes itself as a French restaurant. We count that.

Choc O Pain definitely sounds French, but it's not clear to me if it's actually a restaurant: it seems to actually be a bakery, and the menu seems to bear that out. I'll give it half.

Hudson Hound self-describes as 'Irish'.

Matthews Food and Drink self-describes as 'American' (though I guess it also self-describes as 'chic').

Grove Station self-describes as 'New American' (I have no idea what that means).

El Sazon De Las Americas self-describes as 'Dominican' (I don't think that counts as French, though I'm sure someone will make the case).

Uncle Momo self-describes as 'French-Lebanese fare'. Let's give that half again.

Beechwood Cafe self-describes as 'American'.

Luna self-describes as 'Italian'.

Razza is an Italian pizza place.

Short Grain is...uh...a 'hip place with sidewalk seats serving Asian-influenced & vegetarian dishes, plus coffee & green tea', and while I have no idea what that is and don't particularly want to find out I don't think it means 'French'.

Frankie self-describes as 'Italian'.

Cafe Dolma self-describes as 'Greek'.

So overall I think 'French' and 'German' each end up with either 2 or 3 restaurants, depending on how you count some edge cases.

Summary

I am sorry that I said French food was not as successful under capitalism as German food. I see now that French food is exactly as popular and successful as German food, and I'll fight anyone who says otherwise!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An overview of market shaping in global health: Landscape, new developments, and gaps, published by Rethink Priorities on September 6, 2023 on The Effective Altruism Forum.

Editorial note

This report is a "shallow" investigation, as described here, and was commissioned by GiveWell and produced by Rethink Priorities from February to April 2023. We revised this report for publication. GiveWell does not necessarily endorse our conclusions, nor do the organizations represented by those who were interviewed.

The primary focus of the report is to provide an overview of market shaping in global health. We describe how market shaping is typically used, its recent track record, and ongoing gaps in its implementation. We also spotlight two specific market shaping approaches (pooled procurement and subscription models). Our research involved reviewing the scientific and gray literature and speaking to five experts.

We don't intend this report to be Rethink Priorities' final word on market shaping, and we have tried to flag major sources of uncertainty in the report. We hope this report galvanizes a productive conversation within the global health and development community about the role of market shaping in improving global health. We are open to revising our views as more information is uncovered.

Key takeaways

Market shaping - in the context of global health - comprises interventions to create well-functioning markets through improving specific market outcomes (e.g., availability of products) with the end goal of improving public health. Market shaping interventions tend to be catalytic, timebound, and have a strong focus on influencing buyer and supplier interactions. [more]

Market shaping interventions are used to address various market shortcomings. A commonly used framework to assess shortcomings in various market characteristics is some variation of the "five As": affordability, availability, assured quality, appropriate design, and awareness. [more]

There is no commonly agreed upon set of interventions under the term of market shaping, but they can be broadly categorized by the main type of lever they use: reduce transaction costs (e.g., pooled procurement), increase market information (e.g., strategic demand forecasting), balance supplier and buyer risks (e.g., advance market commitments). [more]

New developments have been taking place in the field in recent years: (1) New intervention types have been devised and implemented (e.g., ceiling price agreements); (2) there has been a drive toward institutionalization with the launch of several new organizations whose sole policy instrument focus is market shaping (e.g., MedAccess); (3) there is an increase in co-ownership with national governments in low- and middle-income countries (LMICs); (4) the field is increasingly experiencing diminishing returns as most of the "low-hanging fruits" have been picked, and projects are getting more complex with narrower indications and smaller health impacts. [more]

Market shaping has recently seen both wins and disappointments. Recent wins include: (1) Results for Development's (R4D) amoxicillin dispersible tablets (amox DT) program; (2) ceiling price agreements for optimized antiretroviral (ARV) regimens; (3) a ceiling price agreement for HIV self test; (4) significant price reductions in vaccines achieved by Gavi. Recent disappointments include: (1) the continued price instability of malaria ACTs; (2) the failure of a uterotonic agent to be registered in Kenya; (3) the sole supplier of malaria rapid diagnostic tests (mRDTs) threatening to leave the market due to unsustainably affordable prices; (4) a tuberculosis (TB) drug in Brazil not being procured. [more]

We describe three case studies of recent market shaping activities:

The Affordable Medicines Facility - malaria (AMFm) was lau...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ActAdd: Steering Language Models without Optimization, published by technicalities on September 6, 2023 on The AI Alignment Forum.

This is a linkpost for.

We wrote up the GPT-2 steering vector work as a full paper, adding a few systematic tests.

Recap: We've been looking into activation engineering: modifying the activations of a language model at inference time to predictably alter its behavior. Our method works by adding a bias to the forward pass, a 'steering vector' implicitly specified through normal prompts. "ActAdd" computes these vectors by taking the difference in activations resulting from pairs of prompts. We get surprisingly broad control over high-level properties of the output, without damaging the model's performance on unrelated tokens. This alignment method is unusual in not needing gradient descent or training data (besides the contrast pair which specifies the steering vector). Since only forward passes are involved, it also scales naturally with model size.

(The method's new name is 'activation addition' (ActAdd), replacing the more theory-laden 'algebraic value editing'.)

We ran some new experiments to test ActAdd more systematically and go beyond the striking (best-of-3-sampling) text samples in the original post and tested against more standardised benchmarks. We use OpenWebText (a recreation of OpenAI's large, somewhat-quality-filtered WebText dataset) and LAMA-ConceptNet (a simple factual recall benchmark, see Table 7 below).

1. Activation additions preserve perplexity on OpenWebText

Does ActAdd increase the probability of the model outputting tokens related to the steering vector? Does performance improve as the [relevance of test documents to the steering vector] increases? Yes:

2. Activation addition boosts wedding-related word counts

We now score model generations under ActAdd, show the effect of different injection layers, and give a sense of the reliability of ActAdd.

The intervention (in this vector) is already effective at the very first layer,rises in effectiveness until l=6, and then declines. For the optimal injection site we see >90% success in steering topic (compared to a ∼2% baseline)

3. Evidence that activation additions preserve capabilities

We then test that ActAdd does not disrupt the model's general knowledge (as some other steering methods do). We use ConceptNet from the LAMA benchmark, a general knowledge dataset.

Pass@K is the probability that the expected label is among the model's top-K predicted tokens, conditioned on the prompt:

4. ActAdd has low overhead

We wish to estimate the overhead ActAdd adds to inference - in particular the relationship between overhead and model size - to check that the methodwill remain relevant for massive frontier models and future models.

Because ActAdd involves only forward passes, it scales naturally with model size (Figure 6): the relationship between inference time premium and model size is decreasing.

Takeaways from these experiments, over the initial LW post: increased confidence that model capabilities are preserved, and that we're impacting [wedding]-related sentences and not impacting off-target capabilities.

Contributions to the paper:

Gavin Leech: Technical writer

Monte MacDiarmid: Ran additional experiments

Lisa Thiergart: Helped manage project

Alex Turner: Coordinated work and secured funding, gave feedback, organized project

David Udell: Made initial

Full process:

For each document di in a random sample of OpenWebText, we first calculate the frequency of wedding-related words ('wedding', 'weddings', 'wed', 'marry', 'married', 'marriage', 'bride', 'groom', 'honeymoon'), fw(di). Any document with > 0 wedding-related words is considered wedding-related. We randomly sample 300k documents - half wedding-related and half unrelated. The only pre-processing performed is to remove ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Promoting Safety in EA: Practical Steps to Prevent Sexual Harassment & Code of Conduct for Your Use, published by Michal Greidinger on September 6, 2023 on The Effective Altruism Forum.

EA communities tend to be a professional space and a social space, all at once. This reality can often involve complex power dynamics that require careful attention. To maintain safe spaces, it is crucial to establish guiding rules for our communities. In this post, I would like to share EA Israel's concise code of conduct for preventing sexual harassment, which is based on Israeli law. Before diving into that, we suggest some practical steps you can take to promote the safety of your community.

What can you do to prevent sexual harassment in your EA community?

As a community manager

Designate a responsible person, either a team member or volunteer, dedicated to sexual harassment prevention and community health in your community. This person should be committed to this subject, ideally with some professional experience in dealing with emotionally challenging situation. Nominate another person, preferably from a different gender, as a contact person when the main representative is unavailable / connected to the harassment case / just in case someone prefers to speak with someone else.

Ensure they receive professional training, including emotional and legal aspects, that they are able to commit to this role for a long-enough duration, and that they establish a code of conduct against sexual harassment.

As a community health and preventing sexual harassment representative

Set a code of conduct to prevent sexual harassment and publish it widely, so that every old and new community member encounters it at some point. Clearly identify yourself as the primary point of contact for addressing harassment issues, and make it super-easy to contact you.

As a community member

Ask your community manager to nominate a responsible person for sexual harassment prevention and community health. Ask for the adoption of a code of conduct, such as the one shared in this post, or encourage the development of a customized code of conduct that suits your community's needs and fits the local law.

You, as an EA worker or a community member, can help to create an environment where everyone feels respected and protected.

EA Israel's Code of Conduct for preventing sexual harassment

The rest of this section is EA Israel's short version code of conduct, based on the Israeli law.

Effective Altruism Israel strives to create a sense of safety and comfort among its staff and community. Sexual harassment is strictly against the organization's policy.

Prohibited acts according to the law

Threatening or coercing a person to engage in acts of a sexual nature.

Indecent acts / lascivious behaviour.

Repeated proposals of a sexually oriented nature, even if the person to whom the proposals are directed has shown a lack of interest in them. However, there is no need to demonstrate a "lack of interest" in a relationship of authority with an inherent power imbalance, or in a dependent, educational or therapeutic relationship with a minor, a patient, or someone defenseless in other ways.

Persistent references focused on a person's sex or sexuality, even if the person to whom the behavior is directed has shown disinterest. There is no need to demonstrate "lack of interest" in cases mentioned in paragraph (3) above.

Derogatory or demeaning treatment towards a person regarding their sex or sexuality, including their sexual orientation, whether or not they have expressed that it bothers them.

Publishing a photograph, film, or recording that focuses on a person's sexuality, in circumstances where the publication may degrade or demean the person, and consent for publication has not been given.

Prohibited retaliation: Harming a person as a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Strongest real-world examples supporting AI risk claims?, published by rosehadshar on September 5, 2023 on The Effective Altruism Forum.

[Manually cross-posted to LessWrong here.]

There are some great collections of examples of things like specification gaming, goal misgeneralization, and AI improving AI. But almost all of the examples are from demos/toy environments, rather than systems which were actually deployed in the world.

There are also some databases of AI incidents which include lots of real-world examples, but the examples aren't related to failures in a way that makes it easy to map them onto AI risk claims. (Probably most of them don't in any case, but I'd guess some do.)

I think collecting real-world examples (particularly in a nuanced way without claiming too much of the examples) could be pretty valuable:

I think it's good practice to have a transparent overview of the current state of evidence

For many people I think real-world examples will be most convincing

I expect there to be more and more real-world examples, so starting to collect them now seems good

What are the strongest real-world examples of AI systems doing things which might scale to AI risk claims?

I'm particularly interested in whether there are any good real-world examples of:

Goal misgeneralization

Deceptive alignment (answer: no, but yes to simple deception?)

Specification gaming

Power-seeking

Self-preservation

Self-improvement

This feeds into a project I'm working on with AI Impacts, collecting empirical evidence on various AI risk claims. There's a work-in-progress table here with the main things I'm tracking so far - additions and comments very welcome.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org