The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In favour of exploring nagging doubts about x-risk, published by Owen Cotton-Barratt on June 26, 2024 on The Effective Altruism Forum.

I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.

I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I'd encourage them to spend time trying to name their fears, and see what they think then.

And I'd encourage them to talk about these things with other people, or write about the complexities of their thinking.

Partly this is because I don't expect people who are using intellectual arguments to override their gut to do a good job of consistently tracking what the most important things to do are on a micro-scale. So it would be good to get the different parts of them to sync more.

And partly because it seems like it would be a public good to explore and write about these things. Either their gut is onto something with parts of its scepticism, in which case it would be great to have that articulated; or their gut is wrong, but if other people have similar gut reactions then playing out that internal dialogue in public could be pretty helpful.

It's a bit funny to make this point about x-risk in particular because of course the above all applies to whatever topic. But I think people normally grasp it intuitively, and somehow that's less universal around x-risk.

I guess maybe this is because people don't have any first-hand experience with x-risk, so their introductions to it are all via explicit arguments … and it's true that it's a domain where we should be unusually unwilling to trust our gut takes without hearing the arguments, but it seems to me like people are unusually likely to forget that they can know anything which has bearing on the questions without already being explicit (and also that perhaps the social environment, in encouraging people

to take explicit arguments seriously, can accidentally overstep and end up discouraging people from taking anything else seriously).

These dynamics seem especially strong in the case of AI risk - which I regard as the most serious source of x-risk, but also the one where I most wish people spent more time exploring their nagging doubts.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is a Tool?, published by johnswentworth on June 25, 2024 on The AI Alignment Forum.

Throughout this post, we're going to follow the Cognition -> Convergence -> Consequences methodology[1]. That means we'll tackle tool-ness in three main stages, each building on the previous:

Cognition: What does it mean, cognitively, to view or model something as a tool?

Convergence: Insofar as different minds (e.g. different humans) tend to convergently model the same things as tools, what are the "real patterns" in the environment which give rise to that convergence?

Consequences: Having characterized the real patterns convergently recognized as tool-ness, what other properties or consequences of tool-ness can we derive? What further predictions does our characterization make?

We're not going to do any math in this post, though we will gesture at the spots where proofs or quantitative checks would ideally slot in.

Cognition: What does it mean, cognitively, to view or model something as a tool?

Let's start with a mental model of (the cognition of) problem solving, then we'll see how "tools" naturally fit into that mental model.

When problem-solving, humans often come up with partial plans - i.e. plans which have "gaps" in them, which the human hasn't thought through how to solve, but expects to be tractable. For instance, if I'm planning a roadtrip from San Francisco to Las Vegas, a partial plan might look like "I'll take I-5 down the central valley, split off around Bakersfield through the Mojave, then get on the highway between LA and Vegas".

That plan has a bunch of gaps in it: I'm not sure exactly what route I'll take out of San Francisco onto I-5 (including whether to go across or around the Bay), I don't know which specific exits to take in Bakersfield, I don't know where I'll stop for gas, I haven't decided whether I'll stop at the town museum in Boron, I might try to get pictures of the airplane storage or the solar thermal power plant, etc.

But I expect those to be tractable problems which I can solve later, so it's totally fine for my plan to have such gaps in it.

How do tools fit into that sort of problem-solving cognition?

Well, sometimes similar gaps show up in many different plans (or many times in one plan). And if those gaps are similar enough, then it might be possible to solve them all "in the same way". Sometimes we can even build a physical object which makes it easy to solve a whole cluster of similar gaps.

Consider a screwdriver, for instance. There's a whole broad class of problems for which my partial plans involve unscrewing screws. Those partial plans involve a bunch of similar "unscrew the screw" gaps, for which I usually don't think in advance about how I'll unscrew the screw, because I expect it to be tractable to solve that subproblem when the time comes. A screwdriver is a tool for that class of gaps/subproblems[2].

So here's our rough cognitive characterization:

Humans naturally solve problems using partial plans which contain "gaps", i.e. subproblems which we put off solving until later

Sometimes there are clusters of similar gaps

A tool makes some such cluster relatively easy to solve.

Convergence: Insofar as different minds (e.g. different humans) tend to convergently model the same things as tools, what are the "real patterns" in the environment which give rise to that convergence?

First things first: there are limits to how much different minds do, in fact, convergently model the same things as tools.

You know that thing where there's some weird object or class of objects, and you're not sure what it is or what it's for, but then one day you see somebody using it for its intended purpose and you're like "oh, that's what it's for"? ()

From this, we learn several things about tools:

Insofar as different humans convergently model the same thing...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Sourcing Metaculus, published by christian on June 25, 2024 on The Effective Altruism Forum.

This is a linkpost for an announcement by Metaculus CEO, Deger Turan, originally published June 5th, 2024. In addition to spreading the word here, we are interested to gather feedback and ideas from the EA community.

We make reasoning and coordination tools for the public benefit, and that requires the trust and participation of the wider forecasting community. That's why we're going open source. GitHub issues, pull requests, and community-sourced forecasts will dictate much of what we build and how. Ultimately, forecasters will benefit from better infrastructure, the Metaculus team will benefit from more idea diversity and greater capacity, and readers will get more accurate aggregates.

What's next

Metaculus will go open source in Q3. There will be opportunities to audit, critique, and improve on our efforts. Creating a better epistemic environment is a collective endeavor.

As part of open-sourcing Metaculus, we'll help implement our best features into other open-source forecasting codebases. We'll make our roadmap and building process public, and we'll give more visibility into our decision-making criteria and discussions. Our codebase is far from perfect, but we can improve it better together out in the open.

In a future post I'll write more about how others can contribute to this project, and specifications for how code can be modified, used, and distributed - as well as guidelines for coding standards and our submission process. We'll provide processes for pull requests, and how we'll implement submissions from the community.

We want to be transparent that we may still choose to offer proprietary offerings targeting particular enterprise and commercial use cases: We believe that part of making forecasting more useful and more used is by productizing it and generating revenue.

Why isn't Metaculus already open source?

Before I joined as CEO I wondered why Metaculus wasn't already open source. I discovered the idea was being discussed, but that there were some concerns about platform security, about accessibility of the codebase and about the potential for exploits that would allow some forecasters to game the system and create an uneven playing field. Now that we've been able to observe the new scoring system at work for some time with no issues, we're confident the benefits outweigh the risks.

While the codebase is nowhere near perfect, we believe making improvements on the platform is better done in the open, with feedback from the larger ecosystem guiding us explicitly.

We're looking for

Researchers who want to backtest their aggregation algorithms on our data

UX specialists who see better ways to translate beliefs into forecasts

Nonprofits who want to spin up their own internal forecasting engines built on our architecture

Developers who can help add new features and enhance existing ones

As our plan evolves I'll provide more details. Until then, let me know what you think: What resources and documentation will you want to see? What would make contributing easier?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Extraordinary cost-effectiveness analyses call for theoretical buttressing, published by Seth Ariel Green on June 25, 2024 on The Effective Altruism Forum.

What needs to be true for an estimate to be reasonable?

Pure Earth is a GiveWell grantee that works to reduce lead and mercury exposure. In an August 2023 post, they provided a "preliminary analysis" suggesting that their lead reduction program in Bangladesh "can avert an equivalent DALY for just under $1." By contrast, they estimate that GiveDirectly "has a cost-effectiveness of approximately $836 per DALY-equivalent averted."

About 86% of the money GiveDirectly spends goes directly to recipients, so an $836 donation to GiveDirectly results in about~$719 going directly to a very poor person. In effect, Pure Earth is claiming that either GiveDirectly can give a person in poverty $719, or Pure Earth can spend $1 helping people in Bangladesh, and these would be about equally good for human welfare.

Pure Earth calls this an "extraordinary result" and forthrightly identifies places where their analysis might go wrong. But what I'm missing is a sense of why their analysis might be right -- a story about why this extraordinary opportunity exists. In other words, theory.

I find this to be a pretty common lacuna in effective altruist CEAs.[1] Our explanations tend to be technical but not contextual. They tend to focus on our assumptions about the world at large but lack granularity about social and political conditions. But for my tastes, any claim that that boils down to 'we can help poor people more effectively than they can help themselves' requires theoretical buttressing in the form of a plausible story.

What kinds of theories might suffice?

In the case of lead reduction in spices, six come to mind.

1. Poor people are generally bad at managing their own affairs and need external guidance. (I do not find this very plausible.)

2. There is extraordinary political gridlock that an external organization is especially well-suited to solving. (Pure Earth says that the "project's impact lies not in identifying and enforcing food safety regulations, but rather in expediting its implementation by several years.")

3. There is a narrow opportunity for positive impact stemming from deep insight about a particular context.

4. There is a market failure, e.g. a collective action problem, where a push from an NGO can create a self-sustaining equilibrium.

5. There is a cognitive bias and/or cultural failure that leads people to undervalue something that is good, or overvalue something that is bad, and they need a push in the right direction.

6. Unconditional cash transfers engender negative spillovers, whereas public health interventions typically have positive spillovers.

The story will vary from case to case. For New Incentives, I assume it's some version of theories 4, 5, and 6. For anti-malarial interventions, I am not sure (and am generally a skeptic). For Pure Earth, I am also not sure, because I am not an expert. As a potential donor, I am looking for explanation about aspects of the world that I don't understand. I can potentially back that story out of a spreadsheet. But I'd prefer to hear it directly.

1. ^

I use Pure Earth to illustrate because their CEA was on the far tail of the distribution, so it stuck in my memory, but this point could be made about many cost-effective analyses posted the forum. As far as I can tell, Pure Earth is doing great work on an important, neglected, and tractable issue.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distancing EA from rationality is foolish, published by Jan Kulveit on June 25, 2024 on The Effective Altruism Forum.

Recently, I've noticed a growing tendency within EA to dissociate from Rationality. Good Ventures have stopped funding efforts connected with the rationality community and rationality, and there are increasing calls for EAs to distance themselves.

This trend concerns me, and I believe it's good to make a distinction when considering this split.

We need to differentiate between 'capital R' Rationality and 'small r' rationality. By 'capital R' Rationality, I mean the actual Rationalist community, centered around Berkeley: A package deal that includes ideas about self-correcting lenses and systematized winning, but also extensive jargon, cultural norms like polyamory, a high-decoupling culture, and familiarity with specific memes (ranging from 'Death with Dignity' to 'came in fluffer').

On the other hand, 'small r' rationality is a more general concept. It encompasses the idea of using reason and evidence to form conclusions, scout mindset, and empiricism. It also includes a quest to avoid getting stuck with beliefs resistant to evidence, techniques for reflecting on and improving mental processes, and, yes, some of the core ideas of Rationality, like understanding Bayesian reasoning.

If people want to distance themselves, it's crucial to be clear about what they're distancing from. I understand why some might want to separate from aspects of the Rationalist community - perhaps they dislike the discourse norms, worry about negative media coverage, or disagree with prevalent community views.

However, distancing yourself from 'small r' rationality is far more radical and likely less considered. It's similar to rejecting core EA ideas like scope sensitivity or cause prioritization just because one dislikes certain manifestations of the EA community (e.g., SBF, jargon, hero worship).

Effective altruism is fundamentally based on pursuing good deeds through evidence, reason, and clear thinking - in fact when early effective altruists were looking for a name, one of the top contenders was rational altruism. Dissecting the aspiration to think clearly would in my view remove something crucial.

Historically, the EA community inherited a lot of epistemic aspects from Rationality[1] - including discourse norms, emphasis on updating on evidence, and a spectrum of thinkers who don't hold either identity closely, but can be associated with both EA and rationality. [2]

Here is the crux: if the zeitgeist pulls effective altruists away from Rationality, they should invest more into rationality, not less. As it is critical for effective altruism to cultivate reason, someone will need to work on it. If people in some way connected to Rationality are not who EAs will mostly talk to, someone else will need to pick up the baton.

1. ^

Clare Zabel in 2022 expressed similar worry:

Right now, I think the EA community is growing much faster than the rationalist community, even though a lot of the people I think are most impactful report being really helped by some rationalist-sphere materials and projects. Also, it seems like there are a lot of projects aimed at sharing EA-related content with newer EAs, but much less in the way of support and encouragement for practicing the thinking tools I believe are useful for maximizing one's impact (e.g.

making good expected-value and back-of-the-envelope calculations, gaining facility for probabilistic reasoning and fast Bayesian updating, identifying and mitigating one's personal tendencies towards motivated or biased reasoning). I'm worried about a glut of newer EAs adopting EA beliefs but not being able to effectively evaluate and critique them, nor to push the boundaries of EA thinking in truth-tracking directions.

2. ^

EA community act...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Responses to Manifest are not about Manifest, published by timunderwood on June 25, 2024 on The Effective Altruism Forum.

This is obvious in one way, but I think forgotten in a lot of the details about these arguments: People do not actually care very much about whether Manifest invited Hanania, they care about the broader trend.

And what I mean by that is specifically that the group that argues that people like Hanania should not be invited to events like Manifest are scared of things like:

They care about whether minorities are being excluded and made unwelcome in EA spaces.

They care about an identity they view as very important being connected to racists

More broadly, they are ultimately scared about the world returning to the sort of racism that led to the Holocaust, to segregation, and they are scared that if they do not act now, to stop this they will be part of maintaining the current system of discrimination and racial injustice.

They feel like they don't belong in a place where people like Hanania are accepted

I apologize if I did not characterize the fears correctly, I am part of the other group, and my model of what motivates the people I disagree with is almost always going to be worse than my model of what motivates me. I am scared of things like:

Making a policy that people like Hanania should never be invited to speak is pushing society in a direction that leads to things like Maoist struggle sessions, McCarthyism (I think we are currently at the level of badness that McCarthyism represented), and at an actual extreme, the thought police from 1984.

The norms cancel culture embraces functionally involve powerful groups being allowed to silence those they dislike. This is still the case no matter what the details of the arguments for the positions are.

Assuming a priori that we know that a certain person's policy arguments or causal model is false leads us to have stupider opinions on average.

I don't belong in a place where adults are not be allowed to read whichever arguments they are interested in about controversial topics, and then form their own opinions, even if those opinions disagree with social orthodoxy.

The biggest point I want to make is that none of these things are arguments against each other.

Cancel culture norms might be creating a tool for power, and make minorities more welcome.

This might push society to be more like a McCarthyist or Maoist place where people are punished for thinking about the wrong questions and having the wrong friends, and at the same time it might prevent backsliding on racial justice, and lead to improvements in equality between racial groups.

Perhaps McCarthy actually made the US meaningfully safer from communist takeover. Most of the arguments that McCarthy was terrible that I recall from university seemed to just take as a given that there was no real risk of a communist takeover, but even if the odds of that were low, making those odds even lower was worth doing things that had costs elsewhere (unless, of course, you think that a communist revolution would have been a good thing).

If we are facing a situation where the policy favored by side A leads to costs that side B is very conscious of, and vice versa, it is likely that if instead of arguing with each other, we attempted to build ideas that addressed each others core concerns, we might come up with ideas that let each side get more of what they want at a smaller cost to what the other side wants.

The second point I'd like to make, is that arguing passionately, with better and better thought experiments that try to trigger the intuitions underlying your position, while completely ignoring the things that actually led the people you are arguing with to the positions they hold, is unlikely to be productive.

Engage with their actual fears if you want to c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mistakes people make when thinking about units, published by Isaac King on June 25, 2024 on LessWrong.

This is a linkpost for Parker Dimensional Analysis. Probably a little elementary for LessWrong, but I think it may still contain a few novel insights, particularly in the last section about Verison's error.

A couple years ago, there was an interesting clip on MSNBC.

A few weeks later, Matt Parker came out with a video analyzing why people tend to make mistakes like this. Now I'm normally a huge fan of Matt Parker. But in this case, I think he kinda dropped the ball.

He does have a very good insight. He realizes that people are treating the "million" as a unit, removing it from the numbers before performing the calculation, then putting it back on. This is indeed the proximate cause of the error. But Matt goes on to claim that the mistake is the treating of "million" as a unit; the implication being that, as a number suffix or a multiplier or however you want to think of it, it's not a unit, and therefore cannot be treated like one. This is false.

So what is a unit, really? When we think of the term, we probably think of things like "meters", "degrees Celcius", "watts", etc.; sciency stuff. But I think the main reason we think of those is due to unit conversion; when you have to convert from meters to feet, or derive a force from mass and acceleration, this makes us very aware of the units being used, and we associate the concept of "unit" with this sort of physics conversion.

In reality, a unit is just "what kind of thing you're counting". Matt uses two other examples in his video: "dollars" and "sheep". Both of these are perfectly valid units! If I say "50 meters", that's just applying the number "50" to the thing "meters", saying that you have 50 of that thing. "50 sheep" works exactly the same way.

So what about "millions"? Well, we can definitely count millions! 1 million, 2 million, etc. You could imagine making physical groupings of a million sheep at a time, perhaps using some very large rubber bands, and then counting up individual clusters. "Millions" is a unit![1]

So if millions is a perfectly valid unit, why do we get an incorrect result if we take it off and then put it back on again after the calculation? Well, because you can't do that with other units either! 100 watts divided by 20 watts does not equal 5 watts. It equals the number 5, with no unit.

This is a somewhat subtle distinction, and easy to miss in a casual conversation. But it makes sense when you think about the actual things you're counting. 50 sheep is certainly not the same thing as 50 horses. And 50 sheep is also not the same thing as the abstract number 50; one is a group of animals, the other a mathematical concept. If someone were to say something to you involving the number 50, you would not simply assume that they're talking about sheep.

This perfectly solves the problem. If 100 watts / 20 watts equals only the number 5, with no "watts", then 100 million / 20 million also equals only the number 5, with no "million".

But what about Matt's example? 80 million sheep - 50 million sheep = 30 million sheep; not just 30. That's because this is subtraction, not division. Units work differently depending on what operation you're performing! If you're doing addition or subtraction, the units are preserved; you can take them off at the beginning and then put them back on at the end. But for multiplication and division, this is not the case.

Division cancels out the units, removing them entirely, and multiplication gives you a new unit, equal to the previous unit squared.

This seems kind of arbitrary, right? Why do they work differently depending on the operation? To understand this, let's go back to a different example that Matt used in his video. Near the beginning, when he's performing the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Formal verification, heuristic explanations and surprise accounting, published by Jacob Hilton on June 25, 2024 on The AI Alignment Forum.

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on

Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives.

In this post, we will:

Summarize ARC's takeaways from this work and the problems we see with provable guarantees

Explain ARC's notion of a heuristic explanation and how it is intended to overcome these problems

Describe with the help of a worked example how the quality of a heuristic explanation can be quantified, using a process we have been calling surprise accounting

We are also sharing a draft by Gabriel Wu (currently visiting ARC) describing a heuristic explanation for the same model that appears in the above paper:

max_of_k Heuristic Estimator

Thanks to Stephanie He for help with the diagrams in this post. Thanks to Eric Neyman, Erik Jenner, Gabriel Wu, Holly Mandel, Jason Gross, Mark Xu, and Mike Winer for comments.

Formal verification for neural networks

In

Compact Proofs of Model Performance via Mechanistic Interpretability, the authors train a small transformer on an algorithmic task to high accuracy, and then construct several different formal proofs of lower bounds on the network's accuracy. Without foraying into the details, the most interesting takeaway from ARC's perspective is the following picture:

In the top right of the plot is the brute-force proof, which simply checks every possible input to the network. This gives the tightest possible bound, but is very long. Meanwhile, in the bottom left is the trivial proof, which simply states that the network is at least 0% accurate. This is very short, but gives the loosest possible bound.

In between these two extremes, along the orange Pareto frontier, there are proofs that exploit more structure in the network, leading to tighter bounds for a given proof length, or put another way, shorter proofs for a given bound tightness.

It is exciting to see a clear demonstration that shorter proofs better explain why the neural network has high accuracy, paralleling a common mathematical intuition that shorter proofs offer more insight. One might therefore hope that if we understood the internals of a neural network well enough, then we would be able to provide provable guarantees for very complex behaviors, even when brute-force approaches are infeasible.

However, we think that such a hope is not realistic for large neural networks, because the notion of proof is too strict.

The basic problem with provable guarantees is that they must account for every possible way in which different parts of the network interact with one another, even when those interactions are incidental to the network's behavior. These interactions manifest as error terms, which the proof must provide a worst-case bound for, leading to a looser bound overall. The above picture provides a good demonstration of this: moving towards the left of the plot, the best bound gets looser and looser.

Mo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm a bit skeptical of AlphaFold 3, published by Oleg Trott on June 25, 2024 on LessWrong.

(also on https://olegtrott.substack.com)

So this happened: DeepMind (with 48 authors, including a new member of the British nobility) decided to compete with me. Or rather, with some of my work from 10+ years ago.

Apparently, AlphaFold 3 can now

predict how a given drug-like molecule will bind to its target protein. And it does so better than AutoDock Vina (the most

cited molecular docking program, which I

built at Scripps Research):

On top of this, it doesn't even need a 3D structure of the target. It predicts it too!

But I'm a bit skeptical. I'll try to explain why.

Consider a hypothetical scientific dataset where all data is duplicated: Perhaps the scientists had trust issues and tried to check each others' work. Suppose you split this data randomly into training and test subsets at a ratio of 3-to-1, as is often done:

Now, if all your "learning" algorithm does is memorize the training data, it will be very easy for it to do well on 75% of the test data, because 75% of the test data will have copies in the training data.

Scientists mistrusting each other are only one source of data redundancy, by the way. Different proteins can also be related to each other. Even when the sequence similarity between two proteins is low, because of evolutionary pressures, this similarity tends to be concentrated where it matters, which is the binding site.

Lastly, scientists typically don't just take random proteins and random drug-like molecules, and try to determine their combined structures. Oftentimes, they take baby steps, choosing to study drug-like molecules similar to the ones already discovered for the same or related targets.

So there can be lots of redundancy and near-redundancy in the public 3D data of drug-like molecules and proteins bound together.

Long ago, when I was a PhD student at Columbia, I

trained a neural network to predict protein flexibility. The dataset I had was tiny, but it had interrelated proteins already:

With a larger dataset, due to the Birthday Paradox, the interrelatedness would have probably been a much bigger concern.

Back then, I decided that using a random train-test split would have been wrong. So I made sure that related proteins were never in both "train" and "test" subsets at the same time. With my model, I was essentially saying "Give me a protein, and (even) if it's unrelated to the ones in my training data, I can predict …"

The authors don't seem to do that. Their analysis reports that most of the proteins in the test dataset had kin in the training dataset with sequence identity in the 95-100 range. Some had sequence identity below 30, but I wonder if this should really be called "low":

This makes it hard to interpret. Maybe the results tell us something about the model's ability to learn how molecules interact. Or maybe they tell us something about the redundancy of 3D data that people tend to deposit? Or some combination?

Docking software is used to scan millions and billions of drug-like molecules looking for new potential binders. So it needs to be able to generalize, rather than just memorize.

But the following bit makes me really uneasy. The authors say:

The second class of stereochemical violations is a tendency of the model to occasionally produce overlapping (clashing) atoms in the predictions. This sometimes manifests as extreme violations in homomers in which entire chains have been observed to overlap (Fig. 5e).

If AlphaFold 3 is actually learning any non-obvious insights from data, about how molecules interact, why is it missing possibly the most obvious one of them all, which is that interpenetrating atoms are bad?

On the other hand, if most of what it does is memorize and regurgitate data (when it can), this would explain such fail...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Review: Righteous Victims - A History of the Zionist-Arab Conflict, published by Yair Halberstadt on June 25, 2024 on LessWrong.

I originally entered this to the ACX Book Review competition. Since it has not been selected as a finalist I'm now free to post it here. In truth it's a followup to my review of Morris's history of Israel's War of Independence.

In the wake of the October 7th attack on Israel and Israel's response, everyone seemed to agree that one side of the conflict was the epitome of evil, the reincarnation of the Nazis, with warfare in their blood and a pure unfiltered hatred of the enemy in their minds. The other side was a force for good, who just wanted peace and was doing the best they could in a difficult situation. The only problem is no one could agree which side was which.

This is unfair. While the loudest voices may paint the world in black and white, as soon as you ignore them, you begin to encounter a whole range of more nuanced views - yet still find yourself no less confused.

Now for the most part my view is that unless you're willing to put in the effort to deeply understand conflicts in far off lands, you're best off not having an opinion on them, and definitely not one fed to you by the twitter or tiktok feed. Expressing loud, confident opinions on unfamiliar conflicts often does more harm than good.

Alas this conflict is not in a far away land. I live 20km from the border with Gaza. Most of my friends were called up to do reserve duty in the IDF. My children almost certainly will have to do the same once they grow up. Far too much of my income goes towards military spending rather than my bank account. I can't take the easy way out, so I have to do things the hard way. So I bought a copy of Benny Morris's Righteous Victims at exorbitant cost[1], and plowed through it.

And I thought I'd share with you what I learned, so that if you do decide to opine on the Israel Palestine conflict, your opinion will hopefully be more educated.

Righteous Victims is a history of the Arab Zionist conflict from 1881 till 2001, written by one of the most respected historians of this conflict.

Bias

Morris is a liberal Zionist, but one whose aim in studying history was to strip back the comforting lies he'd been taught as a child, and find out the actual truth.

None of his (serious) critics accuse him of lying, and his mastery of the primary sources is undisputed. Instead there are two main accusations leveled against him.

The first he readily admits himself in the introduction. Almost all sources about this conflict come from British or Israeli archives. Arab literacy was far lower, Arab historiography of this conflict is a relatively new and small field, and Arab documents have for the most part not been made publicly available even when they exist. Meanwhile a wealth of Zionist material has been released to the public, and we have plenty of contemporary documents to rely on.

While he tries to decipher the Arab perspective from the Zionist one, and relies on Arab documents when they are available, this is naturally going to be both a blindspot and a source of systematic bias.

The second is in choosing which events to highlight and which to ignore. This is an impossible task - over 120 years the amount of relevant information is going to outweigh by many orders of magnitude the amount of space you have in your book, and by carefully selecting which facts to tell you can paint any story you like without ever actually lying.

In practice you deal with this by covering the most important[2] events in plenty of detail, picking representative examples of other events, and giving aggregate statistics[3] to place the representative sample in context. However hard one tries here, it's always possible to accuse the author of favoring facts which paint one side or...