Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A note on 'semiotic physics', published by metasemi on February 11, 2023 on The AI Alignment Forum.

Introduction

This is an attempt to explain to myself the concept of semiotic physics that appears in the original Simulators post by janus and in a later post by Jan Hendrik Kirchner. Everything here comes from janus and Jan's work, but any inaccuracies or misinterpretations are all mine.

TL;DR

The prototypical simulator, GPT, is sometimes said to "predict the next token" in a text sequence. This is accurate, but incomplete.

It's more illuminating to consider what happens when GPT, or any simulator, is run repeatedly to produce a multi-token forward trajectory, as in the familiar scenario of generating a text completion in response to a prompt.

The token-by-token production of output is stochastic, with a branch point at every step, making the simulator a multiverse generator analogous to the time evolution operator of quantum mechanics.

In this analogical sense, a simulator such as GPT implements a "physics" whose "elementary particles" are linguistic tokens. When we experience the generated output text as meaningful, the tokens it's composed of are serving as semiotic signs. Thus we can refer to the simulator's physics-analogue as semiotic physics.

We can explore the simulator's semiotic physics through experimentation and careful observation of the outputs it actually produces. This naturalistic approach is complementary to analysis of the model's architecture and training.

Though GPT's outputs often contain remarkable renditions of the real world, the relationship between semiotic physics and quantum mechanics remains analogical. It's a misconception to think of semiotic physics as a claim that the simulator's semantic world approximates or converges on the real world.

Trajectories

GPT, the prototypical simulator, is often said to "predict the next token" in a sequence of text. This is true as far as it goes, but it only partially describes typical usage, and it misses a dynamic that's essential to GPT's most impressive performances. Usually, we don't simply have GPT predict a single token to follow a given prompt; we have it roll out a continuous passage of text by predicting a token, appending that token to the prompt, predicting another token, appending that, and so on.

Thinking about the operation of the simulator within this autoregressive loop better matches typical scenarios than thinking about single token prediction, and is thus a better fit to what we typically mean when we talk about GPT. But there's more to this distinction than descriptive point of view. Crucially, the growing sequence of prompt+output text, repeatedly fed back into the loop, preserves information and therefore constitutes state, like the tape of a Turing machine.

In the Simulators post, janus writes:

I think that implicit type-confusion is common in discourse about GPT. “GPT”, the neural network, the policy that was optimized, is the easier object to point to and say definite things about. But when we talk about “GPT’s” capabilities, impacts, or alignment, we’re usually actually concerned about the behaviors of an algorithm which calls GPT in an autoregressive loop repeatedly writing to some prompt-state...

The Semiotic physics post defines the term trajectory to mean the sequence of tokens—prompt plus generated-output-so-far—after each iteration of the autoregressive loop. In semiotic physics, as is common in both popular and technical discourse, by default we talk about GPT as a generator of (linguistic) trajectories, not context-free individual tokens.

Simulators are multiverse generators

GPT's token-by-token production of a trajectory is stochastic: at each autoregressive step, the trained model generates an output probability distribution over the token vocabulary, samples from t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I’m not working on {debate, RRM, ELK, natural abstractions}, published by Steven Byrnes on February 10, 2023 on LessWrong.

[For background & spelling out the acronyms in the title, see: Debate (AI safety technique), Recursive Reward Modeling, Eliciting Latent Knowledge, Natural Abstractions.]

When I say “Why I’m not working on X”, I am NOT trying to find a polite & diplomatic way to say “Nobody should work on X because X is unhelpful for AGI safety”. Hmm, OK, well, maybe it’s just a little bit that. But really, I don’t feel strongly. Instead, I think:

A lot of disagreement about what a solution to technical AGI safety looks like is really downstream of disagreements about questions like “How will AGI be built? What will it look like? How will it work?”

Nobody really knows the answers to those questions.

So we should probably be contingency-planning, by going through any possible answers to those questions that at least some reasonable person finds plausible, and doing AGI safety research conditional on those answers being correct.

But still, I have my own opinions about the answers to those questions, and obviously I think my opinions are right, and I am not going to work on something unless it makes sense on my own models. And since people ask me from time to time, it seems worth explaining why the various research programs in the post title do not seem to be a good use of time, on my own models of how AGI will be developed and what AGI will look like.

I wrote this post quickly and did not run it by the people I’m (sorta) criticizing. Do not assume that I described anything fairly and correctly. Please leave comments, and I’ll endeavor to update this post or write a follow-up in the case of major errors / misunderstandings / mind-changes. (But maybe not until after the weekend.)

(By the way: If I’m not working on any of those research programs, then what am I working on? See here. I listed six other projects that seem particularly great to me here, and there are many others besides.)

1. Background

1.1 “Trying” to figure something out seems both necessary & dangerous

(Partly self-plagiarized from here.)

Let’s compare two things: “trying to get a good understanding of some domain by building up a vocabulary of new concepts and their relations” versus “trying to win a video game”. At a high level, I claim they have a lot in common!

In both cases, there are a bunch of possible “moves” you can make (you could think the thought “what if there’s some analogy between this and that?”, or you could think the thought “that’s a bit of a pattern; does it generalize?”, etc. etc.), and each move affects subsequent moves, in an exponentially-growing tree of possibilities.

In both cases, you’ll often get some early hints about whether moves were wise, but you won’t really know that you’re on the right track except in hindsight.

And in both cases, I think the only reliable way to succeed is to have the capability to repeatedly try different things, and learn from experience what paths and strategies are fruitful.

Therefore (I would argue), a human-level concept-inventing AI needs “RL-on-thoughts”—i.e., a reinforcement learning system, in which “thoughts” (edits to the hypothesis space / priors / world-model) are the thing that gets rewarded.

Next, consider some of the features that we plausibly need to put into this RL-on-thoughts system, for it to succeed at a superhuman level:

Developing and pursuing instrumental subgoals—for example, suppose the AI is “trying” to develop concepts that will make it superhumanly competent at assisting a human microscope inventor. We want it to be able to “notice” that there might be a relation between lenses and symplectic transformations, and then go spend some compute cycles developing a better understanding of symplectic transformations. For ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FLI Podcast: Connor Leahy on AI Progress, Chimps, Memes, and Markets (Part 1/3), published by remember on February 10, 2023 on LessWrong.

We often prefer reading over listening to audio content, and have been testing transcribing podcasts using our new tool at Conjecture, Verbalize, with some light editing and formatting. We're posting highlights and transcripts of podcasts in case others share our preferences, and because there is a lot of important alignment-relevant information in podcasts that never made it to LessWrong.If anyone is creating alignment-relevant audio content and wants to transcribe it, get in touch with us and we can give you free credits!

The podcast episode transcribed in this post is available here.

Topics covered include:

Defining artificial general intelligence

What makes humans more powerful than chimps?

Would AIs have to be social to be intelligent?

Importing humanity's memes into AIs

How do we measure progress in AI?

Gut feelings about AI progress

Connor's predictions about AGI

Is predicting AGI soon betting against the market?

How accurate are prediction markets about AGI?

Books cited in the episode include:

The Incerto Series by Nassim Nicholas Taleb

The Selfish Gene, Richard Dawkins

Various books on primates and animal intelligence by Frans De Wall

Inadequate Equilibria by Eliezer Yudkowsky

Highlights

On intelligence in humans and chimps:

We are more social because we're more intelligent and we're more intelligent because we are more social. These things are not independent variables. So at first glance, if you look at a human brain versus a chimp brain, it's basically the same thing. You see like all the same kind of structures, same kind of neurons, though a bunch of parameters are different. You see some more spindle cells, it's bigger. Human brain just has more parameters, it's just GPT-3 versus GPT-4...

But really, the difference is, is that humans have memes. And I mean this in the Richard Dawkins sense of evolved, informational, programmatic virtual concepts that can be passed around between groups. If I had to pick one niche, what is the niche that humans are evolved for?

I think the niche we're evolved for is memetic hosts.

On benchmarks and scaling laws:

Benchmarks are actually coordination technologies. They're actually social technologies. What benchmarks are fundamentally for is coordination mechanisms. The kind of mechanisms you need to use when you're trying to coordinate groups of people around certain things....

So we have these scaling laws, which I think a lot of people misunderstand. So scaling laws give you these nice curves which show how the loss of performance on the model smoothly decreases as they get larger. These are actually terrible, and these actually tell you nothing about the model. They tell you what one specific number will do. And this number doesn't mean anything. There is some value in knowing the loss. But what we actually care about is can this model do various work? Can it do various tasks? Can it reason about its environment? Can it reason about its user?...

So currently there are no predictive theories of intelligence gain or task. There is no theory that says once it reaches 74.3 billion parameters, then it will learn this task. There's no such theory. It's all empirical. And we still don't understand these things at all. I think there's, so another reason I'm kind of against benchmarks, and I'm kind of being a bit pedantic about this question is because I think they're actively misleading in the sense that people present them as if they mean something, but they just truly, truly don't. A benchmark in a vacuum means nothing.

On the dangerous of having a good metric of progress towards AGI:

So this is an interesting question. And not just from a scientific perspective, but it's also interesting...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Select Challenges with Criticism & Evaluation Around EA, published by Ozzie Gooen on February 10, 2023 on The Effective Altruism Forum.

I think that critique (or sharing uncomfortable information) is very important, difficult, and misunderstood. I've started to notice many specific challenges when dealing with criticism - here's my quick take on some of the important ones. This comes mostly from a mix of books and personal experiences.

This post is a bit rambly and rough. I expect much of the content will be self-evident to many readers.

A whole lot has been written about criticism, but I've found the literature scattered and not very rationalist-coded (see point # 6). I wouldn't be surprised if there were better related lists out there that would apply to EA, please do leave comments if you know of them.

1. Many People Hate Being Criticized

Many people and organizations actively detest being open or being criticized. Even if those being evaluated wouldn’t push back against evaluation, evaluators really don’t want to harm people. The fact that antagonistic actors online are using public information as ammunition makes things worse.

There’s a broad spectrum regarding how much candidness different people/agents can handle. Even the most critical people can be really bad at taking criticism. One extreme is Scandal Markets. These could be really pragmatically useful, but many people would absolutely hate showing up on one. If any large effective altruist community enforced high-candidness norms, that would exclude some subset of potential collaborators.

In business, there are innumerable books about how to give criticism and feedback. Similar to romantic relationships. It’s a major social issue!

2. Not Criticizing Leads to Distrust

What’s scarier than getting negative feedback? Knowing that people who matter to you dislike you, but not knowing why.

When there are barriers in communication, people often assume the worst. They make up bizarre stories.

The number one piece of advice I’ve seen for resolving tense situations in business and romance is to just talk to each other. (Sometimes bringing in a moderator can help with the early steps!)

If you have critical takes on someone, you could:

Make it clear you have issues with them, be silent on the matter, or pretend you don’t have issues with them.

Reveal these issues, or hide them.

Lying in Choice 1 can be locally convenient, but it often comes with many problems.

You might have to create more lies to justify these lies.

You might lie to yourself to make yourself more believable, but this messes with your epistemics in subtle ways that you won’t be able to notice.

If others catch on that you do this, they won’t be able to trust many important things you say.

If you are honest in Choice 1 (this can include social hints) but choose to conceal in Choice 2, then the other party will make up reasons why you dislike them. These are heated topics (people hate being disliked). I suspect that’s why guesses here are often particularly bad.

Some things I’ve (loosely) heard around the EA community include:

That’s why [person X] rejected my application. It’s probably because I’m just a bad researcher, and I will never be useful.

Funders don’t like me because I don’t attend the right parties.

Funders don’t like me because I criticized them, and they hate criticism.

EAs hate my posts because I use emotion, and EAs hate emotion.

All the EA critics just care about Woke buzzwords.

When I post to the EA Forum about my reasons for (important actions), I get downvoted. That’s because they just care about woke stuff and no longer care about epistemics.

3. People have wildly different abilities to convey candid information gracefully

First, there seems to be a large subset of people that assumes that social grace or politeness is all BS. You ca...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Need For Work On Technical AI Alignment (Intro Explainer), published by Daniel Eth on February 11, 2023 on The AI Alignment Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Engineer’s Interpretability Sequence (EIS) I: Intro, published by scasper on February 9, 2023 on LessWrong.

Part 1 of 12 in the Engineer’s Interpretability Sequence.

If we want to reduce near and long term risks from AI, we should care a lot about interpretability tools. This is a very uncontroversial claim to make inside the AI safety community. Almost every agenda for safe advanced AI incorporates interpretability in some way. The key value of interpretability tools is that they aid in human oversight by enabling open-ended evaluation.

Short of actually deploying a system, any method of evaluating it can only be a proxy for its actual performance. The most common way to evaluate a model is by its performance in some test set or environment. But test sets alone can fail to reveal – and often incentivize – undesirable solutions involving overfitting, biases, deception, etc. This highlights the need for other ways to evaluate models, and an interpretability toolbox full of effective tools may go a long way.

Some of the seeds of the AI safety community’s interest in interpretability were planted by Distill in 2017. But 2022 was an inflection point with a massive new surge in interest and work on interpretability tools. Anthropic was founded a little over a year ago. ARC started less than a year ago. Redwood has begun to push for much more interpretability work, including with the REMIX program. We are seeing a number of pushes to get many more people involved in interpretability work. And as someone on the ground, I have subjectively observed a surge in interest over 2022. And the popularity of interpretability hasn’t been limited to the AI safety community. There is now so much work in interpretability that we now have a dataset of 5199 interpretability papers (Jacovi, 2023). See also a survey of 300+ of them from some coauthors and me (Räuker et al., 2022).

Growth in the interpretability literature by year from Jacovi (2023).

But despite all this work, interpretability research has limitations. One of the goals of this sequence is to argue that:

Interpretability research both within the AI safety space and at large is not very productive and may be on course to stay this way.

This is intentionally baitey, and I mean to make this point with a large amount of detail and nuance over the course of this sequence. But one striking thing about interpretability research is that:

For all the interpretability work that exists, there is a significant gap between this research and engineering applications.

This is not to say that purely exploratory work is not good and necessary. But the problem of AI safety is an engineering problem at its core. If one of our main goals for interpretability research is to help us with aligning highly intelligent AI systems in high stakes settings, shouldn’t we be seeing tools that are more helpful in the real world? Hence the name of this sequence: The Engineer’s Interpretability Sequence (EIS).

This sequence will have twelve parts.

EIS I: Intro

EIS II: What is “Interpretability”?

EIS III Broad critiques of Interpretability Research

EIS IV: A Spotlight on Feature Attribution/Saliency

EIS V: Blind Spots In AI Safety Interpretability Research

EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety

EIS VII: A Challenge for Mechanists

EIS VIII: An Engineer’s Understanding of Deception

EIS IX: Interpretability and Adversaries

EIS X: Continual Learning, Modularity, Compression, and Biological Brains

EIS XI: Moving Forward

EIS XII: Summary

In the coming days, I plan to post a new installment every day or so. Thanks to my labmates, advisor, friends, and many others in the interpretability community for lots of good conversations and inspiration in the past year. Thanks to Rio Popper for feedback on this intro post. I'll be thanking othe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prizes for the 2021 Review, published by Raemon on February 10, 2023 on LessWrong.

If you received a prize, please fill out your payment contact email and PayPal.

A'ight, one final 2021 Review Roundup post – awarding prizes. I had a week to look over the results. The primary way I ranked posts was by a weighted score, which gave 1000+ karma users 3x the voting weight. Here was the distribution of votes:

I basically see two strong outlier posts at the top of the ranking, followed by a cluster of 6-7 posts, followed by a smooth tail of posts that were pretty good without any clear cutoff.

Post Prizes

Gold Prize Posts

Two posts stood noticeably out above all the others, which I'm awarding $800 to.

Strong Evidence is Common by Mark Xu

“PR” is corrosive; “reputation” is not, by Anna Salamon.

I also particularly liked Akash's review.

Silver Prize Posts

And the second (eyeballed) cluster of posts, each getting $600, is:

Your Cheerful Price, by Eliezer Yudkowsky.

This notably had the most reviews – a lot of people wanted to weigh in and say "this personally helped me", often with some notes or nuance.

ARC's first technical report: Eliciting Latent Knowledge by Paul Christiano, Ajeya Cotra and Mark Xu.

This Can't Go On by Holden Karnofsky

Rationalism before the Sequences, by Eric S Raymond.

I liked this review by A Ray who noted one source of value here is the extensive bibliography.

Lies, Damn Lies, and Fabricated Options, by Duncan Sabien

Fun with +12 OOMs of Compute, by Daniel Kokotajlo.

Nostalgebraist's review was particularly interesting.

What 2026 looks like by Daniel Kokotajlo

Ngo and Yudkowsky on alignment difficulty. This didn't naturally cluster into the same group of vote-totals as the other silver-prizes, but it was in the top 10. I think the post was fairly hard to read, and didn't have easily digestible takeaways, but nonetheless I think this kicked off some of the most important conversations in the AI Alignment space and warrants inclusion in this tier.

Bronze Prize Posts

Although there's not a clear clustering after this point, when I eyeball how important the next several posts were, it seems to me appropriate to give $400 to each of:

How To Write Quickly While Maintaining Epistemic Rigor, by John Wentworth

Science in a High-Dimensional World by John Wentworth

How factories were made safe by Jason Crawford

Cryonics signup guide #1: Overview by Mingyuan

Making Vaccine by John Wentworth

Taboo "Outside View" by Daniel Kokotaljo

All Possible Views About Humanity's Future Are Wild by Holden Karnofsky

Another (outer) alignment failure story by Paul Christiano

Split and Commit by Duncan Sabien

What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) by Andrew Critch

There’s no such thing as a tree (phylogenetically), by eukaryote

The Plan by John Wentworth

Trapped Priors As A Basic Problem Of Rationality by Scott Alexander

Finite Factored Sets by Scott Garrabrant

Selection Theorems: A Program For Understanding Agents by John Wentworth

Slack Has Positive Externalities For Groups by John Wentworth

My research methodology by Paul Christiano

Honorable Mentions

This final group has the most arbitrary cutoff at all, and includes some judgment calls about how many medium or strong votes it had, among 1000+ karma users, and in some edge cases my own subjective guess of how important it was.

These authors each get $100 per post.

The Rationalists of the 1950s (and before) also called themselves “Rationalists” by Owain Evans

Ruling Out Everything Else by Duncan Sabien

Leaky Delegation: You are not a Commodity by Darmani

Feature Selection by Zack Davis

Cup-Stacking Skills (or, Reflexive Involuntary Mental Motions) by Duncan Sabien

larger language models may disappoint you [or, an eternally unfinished draft] by Nostalgebraist

Self-Integrity and the Drowni...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There can be highly neglected solutions to less-neglected problems, published by Linda Linsefors on February 10, 2023 on The Effective Altruism Forum.

This post is based on an old draft that I wrote years ago, but never quite finished. The specific references might be a bit outdated, but think the main point is still relevant.

Thanks to Amber Dawn Ace for turning my draft into something worth posting.

EAs are often interested in how ‘neglected’ problems are: that is, how much money is being spent on solving them, and/or how many people are working on them. 80,000 Hours, for example, prioritizes pressing problems using a version of the importance, tractability, neglectedness (ITN) framework, where a problem’s ‘neglectedness’ is one of three factors that determine how much we should prioritize it relative to other problems.

80,000 Hours define neglectedness as:

“How many people, or dollars, are currently being dedicated to solving the problem?

I suggest that a better definition would be:

“How many people, or dollars, are currently being dedicated to this particular solution?”

I think it makes more sense to assess solutions for neglectedness, rather than problems. Sometimes, a problem is not neglected, but effective solutions to that problem are neglected. A lot of people are working on the problem and a lot of money is being spent on it, but in ineffective (or less effective) ways.

Here’s the example that 80,000 Hours uses to illustrate neglectedness:“[M]ass immunisation of children is an extremely effective intervention to improve global health, but it is already being vigorously pursued by governments and several major foundations, including the Gates Foundation. This makes it less likely to be a top opportunity for future donors.”

Note that mass immunisation of children is a solution, not a problem. But it makes sense for 80K to think about this: we can imagine a world in which charities spent just as much money on preventing or curing diseases, but they spent it on less effective solutions. In that world, even though global disease would not be a neglected problem, mass vaccination would be a neglected (effective) solution, and it would make sense for donors to prioritize it.

There’s something fractal about solutions and problems. Every solution to a problem presents its own problem: how best to implement it. Mass vaccination is an effective solution to the problem of disease. But then there’s the problem of ‘how can we best achieve mass vaccination?’. And solutions to that problem - for example, different methods of vaccine distribution, or interventions to address vaccine skepticism - pose their own, more granular problems.

Here’s another example: hundreds of millions of dollars is spent each year on preventing nuclear war and nuclear winter. However, only a small fraction of this is spent on interventions intended to mitigate the negative impacts of nuclear winter - for example, ALLFED’s work researching food sources that are not dependent on sunlight. But in the event that there is a nuclear winter, these sorts of interventions will be extremely effective: they’ll enable us to produce much more food, so far fewer people will starve. Mitigation interventions are thus a highly neglected class of solutions for a problem - nuclear war - that is comparatively less neglected.

As another example: climate change is not a neglected problem. But different solutions get very different amounts of attention. One of the most effective interventions seems to be preserving the rainforests (and other bodies of biomass), yet only a lonely few organizations (e.g. Cool Earth and the Carbon Fund) are working on this.

Evaluating the neglectedness of solutions has its own problems. It means that neglectedness no longer lines up so neatly with solvability/tractability and scale/import...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A proposed method for forecasting TAI, published by Matthew Barnett on February 10, 2023 on LessWrong.

In 2021, I proposed measuring progress in the perplexity of language models and extrapolating past results to determine when language models were expected to reach roughly "human-level" performance. Here, I build on that approach by introducing a more systematic and precise method of forecasting progress in language modeling that employs scaling laws to make predictions.

The full report for this forecasting method can be found in this document. In this blog post I'll try to explain all the essential elements of the approach without providing excessive detail regarding the technical derivations.

As a bonus, this approach can be contrasted with Ajeya Cotra's Bio Anchors model, providing a new method for forecasting the arrival of transformative AI (TAI). I will tentatively call it the "Direct Approach", since it makes use of scaling laws directly to make predictions about compute requirements for AI.

Naturally, the Direct Approach is a very speculative framework and might end up being useless for forecasting TAI (in fact, I consider this the most likely outcome). Nonetheless, I'm hopeful that something like it can serve as a better foundation for current TAI timelines models, which I currently view as likely even worse. Note that there may be errors in the report and Colab notebook, as they were not extensively fact-checked.

Some background

In a nutshell, this approach is simply about taking the cross-entropy loss of an autoregressive model and trying to find a way of interpreting that quantity qualitatively: that is, something we can put on a chart and extrapolate until the quantity reaches a natural threshold that we identify with something important.

In my 2021 post about predicting language model performance, I drew a trendline through a plot of language model perplexities on various benchmarks and noted when the trendline went through estimates of "human-level" perplexity. This approach felt reasonable to me at the time, but I now think it too easily hand-waved away some important details.

The error of omission I committed in my old approach becomes more apparent when you think about language model performance from the perspective of scaling laws, for example the parametric scaling law from Hoffmann et al. 2022:

Here, we see cross-entropy loss as a function of parameters N and training tokens D seen during training. Notably, if we take the limit as the number of parameters and training tokens goes to infinity, then we're left with E. Theoretically, E corresponds to the "entropy of natural text", which is precisely the thing I identified with "roughly human-level" performance in my previous post. In other words, if we take this scaling law naively, it seems as though it will take infinite compute to reach human-level performance.

I believe the resolution to this apparent issue is to say that "human-level" performance will not be obtained when loss hits E, but rather some small level above E. How close to E is enough? Well, that's the question we tried to answer with this report.

Summary of the Direct Approach

We begin by considering a language task, which in this post will be scientific research for illustration. For simplicity, let's imagine that this task consists of writing high-quality research papers or reports, although more nuanced specifications are possible.

Of course, real scientific research involves more than merely writing research papers. It involves proposing hypotheses, devising experiments, and collecting data, but for now, let's imagine that we can simplify all these steps into one step that involves writing high quality research papers. This simplification may not be entirely unrealistic, since if the papers are genuinely judged to be high quali...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Pervasive Illusion of Seeing the Complete World, published by shminux on February 9, 2023 on LessWrong.

It is a tautology that we do not notice our blind spots.

It is not a tautology that we forget they exist, shortly after learning that they do.

Michael Crichton's Gell-Mann Amnesia effect as quoted by gwern is one of many examples: we know that we cannot model the news veracity with any accuracy, yet we forget it the moment this observation stops hitting us in face.

Scott Alexander's classic What Human Experiences Are You Missing Without Realizing It is even more egregious: the data about our blind spots keeps coming and we intuitively rationalize it away.

The ironic part is that everyone's favorite LLM keeps forgetting about its own blind spot, the way a human would.

I guess there is something about blind spots that is antimemetic, not very surprisingly.

Speculation: These meta-blind spots tend to develop around actual blind spots naturally, because of the way the brain works. We notice stuff that changes, because the brain is akin to a multi-level prediction error minimization machine. If you wear cracked or dirty glasses, you stop noticing them after a short time, unless the cracks or dirt actively interfere with something you have to see, reminding you of the cracks. Worse than that, you forget that the cracks exist, unless reminded. This meta-blind spot, or a tower of blind spots can probably go several levels up, if there is no prediction error detected at that level.

Another speculation: the tower of blind spots creates an illusion of seeing the complete world, with nothing else existing. After all, to notice the existence of something the brain needs to be able to compare predictions with inputs, and if there are no inputs at any level, there is nothing to activate the prediction error minimization machine.

This was the descriptive part. The prescriptive part is, as usual, much more speculative.

An aside: It is worth explicitly paying attention when a write-up switches from descriptive to prescriptive, from analysis to offering solutions. For example, Marx gave a fantastically good analysis of problems with 19th century capitalism, but then offered a fantastically bad prescription for fixing it, with disastrous consequences. My wild guess is that the difference is because our prediction abilities are a blind spot in itself, and self-calibration is a comparatively new, rare and hard rationality skill.

So, the prescriptive part is to identify (hard) and topple (easier) the blind spot towers. For example, once you conceive of God not being the ultimate source of everything, you can start questioning the unstated assumptions, jump-starting the predictive error minimization machine, with the data coming from outside and from "inside". The source of data can, of course, be corrupted by emotions, and is a tower of blind spots in itself. Thus one can reason oneself into atheism or Pascal's wager equally easily.

Oh, and just to undermine everything I said so far, here is a completely personal view that I reasoned myself into some years ago, that clashes severely with this site's consensus. The consensus is that there is an external reality that we, as embedded agents, build maps of, the map/territory dichotomy. I believe it is one of those blind spots, and a more accurate model is that it is maps all the way down. (And that the terms like "exist", "reality", "truth" and "fact" have a limited domain of applicability that is constantly and subconsciously exceeded by nearly everyone.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.