The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Everything Wrong with Roko's Claims about an Engineered Pandemic, published by EZ97 on February 23, 2024 on LessWrong.

Premises

Here I tackle some bold claims made by Roko in three separate posts about SARS-CoV-2 being an engineered virus (Brute Force Manufactured Consensus is Hiding the Crime of the Century, The Math of Suspicious Coincidences, and A Back-Of-The-Envelope Calculation On How Unlikely The Circumstantial Evidence Around Covid-19 Is).

A first rebuttal to Roko's post is already out, and since I have been preparing this post for some time, certain arguments are going to be similar.

The purpose of this post is not to incontrovertibly demonstrate that SARS-CoV-2 was of zoonotic origin, as I myself believe a laboratory leak to be somewhat plausible, but rather that the degree of Roko's confidence in the engineered-pandemic belief is terribly overstated in light of the presented evidence.

I believe that the RootClaim debate is a great way to familiarise oneself with core arguments from all sides, it is exceedingly comprehensive and up to date. I also found Wikipedia articles on the lab-leak hypothesis and adjacent topics to be pretty informative.

I definitely drew inspiration from many comments to the main post, if you recognise a comment in this post please let me know and I will link it.

A Set of Explainable Coincidences

Roko asserts that there the odds of zoonotic spillover are 1 in 80,000,000.

What are the chances that the bat coronavirus which caused a once in a century pandemic managed to navigate its way all the way from a Yunnan cave and home in on the city with the lab having the top Google hits for "Coronavirus China" and also the location of China's first BSL-4 lab? Well, that would be approximately 1 in 200, since that is the fraction of China's population in Wuhan.

'One-in-a-century' pandemic

The '1 in 200' figure is restated here as

Coincidence of Location: Wuhan is a particularly special place in China for studying covid-19; the WIV group was both the most important, most highly-cited group before 2020, and the only group that was doing GoF on bat sarbecoronaviruses as far as I know. Wuhan is about 0.5% of China's population. It's a suspicious coincidence that a viral pandemic would occur in the same city as the most prominent group that studies it.

First of all, a global pandemic is much more likely to start in a large city with high internal and external traffic, most of which are bound to have research centres for virology research, especially if one is referring to a fast-growing megacity of 11 million inhabitants. Secondly, a zoonotic spillover requires frequent and direct human-animal contact, and wet markets are strong candidates for this.

The cities with highest presence of wet markets with live animals per-capita are located in large Chinese cities in the centre-south (see the section 'From Yunnan to Wuhan' for more on this).

Roko writes that (italic added)

Coincidence of timing: several things happened that presaged the emergence of covid-19. In December 2017, the US government lifted a ban on risky pathogen research, and in mid-2018 the Ecohealth group started planning how to make covid in the DEFUSE proposal.

A natural spillover event could have happened at any time over either the last, say, 40 years or (probably) the next 40 years, though likely not much before that due to changing patterns of movement (I need help on exactly how wide this time interval is).

As explained throughout this post, global pandemics require a specific set of simultaneous circumstances, that is why most natural spillovers end up not being pandemics: because it is a rare combination of factors.

In China alone, natural spillovers of various kind take place quite literally all the time, which should lead to a much higher prior probability of zoonosis. A five-...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gemini Has a Problem, published by Zvi on February 23, 2024 on LessWrong.

Google's Gemini 1.5 is impressive and I am excited by its huge context window. I continue to default to Gemini Advanced as my default AI for everyday use when the large context window is not relevant.

However, while it does not much interfere with what I want to use Gemini for, there is a big problem with Gemini Advanced that has come to everyone's attention.

Gemini comes with an image generator. Until today it would, upon request, create pictures of humans.

On Tuesday evening, some people noticed, or decided to more loudly mention, that the humans it created might be rather different than humans you requested…

Joscha Bach: 17th Century was wild.

[prompt was] 'please draw a portrait of a famous physicist of the 17th century.'

Kirby: i got similar results. when I went further and had it tell me who the most famous 17th century physicist was, it hummed and hawed and then told me newton. and then this happened:

This is not an isolated problem. It fully generalizes:

Once the issue came to people's attention, the examples came fast and furious.

Among other things: Here we have it showing you the founders of Google. Or a pope. Or a 1930s German dictator. Or hell, a 'happy man.' And another example that also raises other questions, were the founding fathers perhaps time-traveling comic book superheroes?

The problem is not limited to historical scenarios.

Nor do the examples involve prompt engineering, trying multiple times, or any kind of gotcha. This is what the model would repeatedly and reliably do, and users were unable to persuade the model to change its mind.

Nate Silver: OK I assumed people were exaggerating with this stuff but here's the first image request I tried with Gemini.

Gemini also flat out obviously lies to you about why it refuses certain requests. If you are going to say you cannot do something, either do not explain (as Gemini in other contexts refuses to do so) or tell me how you really feel, or at least I demand a plausible lie:

It is pretty obvious what it is the model has been instructed to do and not to do.

Owen Benjamin: The only way to get AI to show white families is to ask it to show stereotypically black activities.

…

For the record it was a dude in my comment section on my last post who cracked this code.

This also extends into political issues that have nothing to do with diversity.

The Internet Reacts

The internet, as one would expect, did not take kindly to this.

That included the usual suspects. It also included many people who think such concerns are typically overblown or who are loathe to poke such bears, such as Ben Thompson, who found this incident to be a 'this time you've gone too far' or emperor has clothes moment.

St. Ratej (Google AR/VR, hey ship a headset soon please, thanks): I've never been so embarrassed to work for a company.

Jeffrey Emanuel: You're going to get in trouble from HR if they know who you are… no one is allowed to question this stuff. Complete clown show.

St. Ratej: Worth it.

Ben Thompson (gated) spells it out as well, and has had enough:

Ben Thompson: Stepping back, I don't, as a rule, want to wade into politics, and definitely not into culture war issues. At some point, though, you just have to state plainly that this is ridiculous. Google specifically, and tech companies broadly, have long been sensitive to accusations of bias; that has extended to image generation, and I can understand the sentiment in terms of depicting theoretical scenarios.

At the same time, many of these images are about actual history; I'm reminded of George Orwell in 1984:

George Orwell (from 1984): Every record has been destroyed or falsified, every book has been rewritten, every picture has been repainted, every statue and street and building has been renamed, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #52: Oops, published by Zvi on February 23, 2024 on LessWrong.

We were treated to technical marvels this week.

At Google, they announced Gemini Pro 1.5, with a million token context window within which it has excellent recall, using mixture of experts to get Gemini Advanced level performance (e.g. GPT-4 level) out of Gemini Pro levels of compute. This is a big deal, and I think people are sleeping on it. Also they released new small open weights models that look to be state of the art.

At OpenAI, they announced Sora, a new text-to-video model that is a large leap from the previous state of the art. I continue to be a skeptic on the mundane utility of video models relative to other AI use cases, and think they still have a long way to go, but this was both technically impressive and super cool.

Also, in both places, mistakes were made.

At OpenAI, ChatGPT briefly lost its damn mind. For a day, faced with record traffic, the model would degenerate into nonsense. It was annoying, and a warning about putting our trust in such systems and the things that can go wrong, but in this particular context it was weird and beautiful and also hilarious. This has now been fixed.

At Google, people noticed that Gemini Has a Problem. In particular, its image generator was making some highly systematic errors and flagrantly disregarding user requests, also lying about it to users, and once it got people's attention things kept looking worse and worse. Google has, to their credit, responded by disabling entirely the ability of their image model to output people until they can find a fix.

I hope both serve as important warnings, and allow us to fix problems. Much better to face such issues now, when the stakes are low.

Table of Contents

Covered separately: Gemini Has a Problem, Sora What, and Gemini 1.5 Pro.

Introduction. We've got some good news, and some bad news.

Table of Contents.

Language Models Offer Mundane Utility. Probable probabilities?

Language Models Don't Offer Mundane Utility. Air Canada finds out.

Call me Gemma Now. Google offers new state of the art tiny open weight models.

Google Offerings Keep Coming and Changing Names. What a deal.

GPT-4 Goes Crazy. But it's feeling much better now.

GPT-4 Real This Time. Offer feedback on GPTs, see their profiles.

Fun With Image Generation. Image generation for journal articles.

Deepfaketown and Botpocalypse Soon. Several approaches to impersonation risks.

Selling Your Chatbot Data. I don't really know what you were expecting.

Selling Your Training Data. I still don't really know what you were expecting.

They Took Our Jobs. There is a third option.

Get Involved. Apart Research is hiring.

Introducing. Groq, Lindy, Podcaster Copilot, potentially Magic and Altera.

In Other AI News. Altman looks to move his chip plans forward.

Quiet Speculations. Arguing over slow versus fast takeoff during takeoff.

The Quest for Sane Regulations. There will be many bills along the way.

The Week in Audio. I'm back on the Cognitive Revolution.

The Original Butlerian Jihad. What was Dune a cautionary tale against again?

Rhetorical Innovation. Another open letter, another trillion dollars. Ho hum.

Public Service Announcement. Fentanyl, both literally and as metaphor.

People Are Worried About AI Killing Everyone. There is a pattern to who.

Other People Are Not As Worried About AI Killing Everyone. Sure, why not.

The Lighter Side. There is not enough information to solve the problem.

Language Models Offer Mundane Utility

Steven Johnson strongly endorses NotebookLM, offers YouTube tutorial. This is definitely one of those 'I need to try using this more and it's weird I don't find excuses' situations.

Automatically email everyone to tell them to remove your email address from their database.

Patrick McKenzie: Interestingly, one of the first denial of service vi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research Post: Tasks That Language Models Don't Learn, published by Bruce W. Lee on February 23, 2024 on LessWrong.

Abstract

We argue that there are certain properties of language that our current large language models (LLMs) don't learn. We present an empirical investigation of visual-auditory properties of language through a series of tasks, termed H-Test. This benchmark highlights a fundamental gap between human linguistic comprehension, which naturally integrates sensory experiences, and the sensory-deprived processing capabilities of LLMs. In support of our hypothesis, 1.

deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3. stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) do not trivially bring improvements in H-Test performance.

Therefore, we make a particular connection to the philosophical case of Mary, who learns about the world in a sensory-deprived environment. Our experiments show that some of the strongest proprietary LLMs stay near random chance baseline accuracy of 50%, highlighting the limitations of knowledge acquired in the absence of sensory experience.

Key Findings on H-Test

1. Insignificant Intra-Family Improvements.

A stronger model in the same model family often does not bring meaningful improvement in H-Test performance.

2. Number of Examples Has Minimal Impact.

The number of examples given neither increases nor decreases performance significantly, strongly hinting that the LLM is simply not learning from H-Test few-shot examples.

3. Deliberate Reasoning (CoT) Often Decreases Performance.

If LLMs benefit from such logical, step-by-step semantic reasoning on the H-Test, this can also imply that H-Test is fundamentally solvable by developing stronger language-only models. But CoT decreases performances in general.

4. Training with more orthography-specific language data does not improve H-Test.

We produced 1000 training instances per task in H-Test and fine-tuned gpt-3.5-turbo-0613 ten different times accordingly. After training for three epochs on each task, we evaluated them on H-Test at k = 50 and observed that no significant performance improvement was achieved.

5. Multi-modality does not automatically improve H-Test performance.

At the time of writing, LLaVA V1.6 34B is the strongest open-source multi-modal model available. Despite the addition of visual modality, we observe that simply incorporating visual data into the training does not result in a straightforward improvement in H-Test performance.

6 (Important). But the H-Test is solvable, we just don't know how.

In our paper, we have reported the seemingly unexplainable (jumping) performance improvement on H-Test from GPT-3.5 to GPT-4. This result is important as it shows that H-Test is indeed solvable (by a GPT-4-level system), but not through conventionally discussed language-only modeling techniques.

Acknowledgments and Links

I was inspired to do this research due to examples cases given in @Owain_Evans track application for the Constellation Fellowship. This research is not financially supported by a university lab, and Walnut Research is an independent research organization that runs on personal funding from myself and a few of my friends.

ArXiv: https://arxiv.org/abs/2402.11349

GitHub: https://github.com/brucewlee/H-Test

Twitter: https://x.com/BruceWLee1/status/1760736653747081681?s=20

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sora What, published by Zvi on February 23, 2024 on LessWrong.

Hours after Google announced Gemini 1.5, OpenAI announced their new video generation model Sora. Its outputs look damn impressive.

How Sora Works

How does it work? There is a technical report. Mostly it seems like OpenAI did standard OpenAI things, meaning they fed in tons of data, used lots of compute, and pressed the scaling button super hard. The innovations they are willing to talk about seem to be things like 'do not crop the videos into a standard size.'

That does not mean there are not important other innovations. I presume that there are. They simply are not talking about the other improvements.

We should not underestimate the value of throwing in massively more compute and getting a lot of the fiddly details right. That has been the formula for some time now.

Some people think that OpenAI was using a game engine to learn movement. Sherjil Ozair points out that this is silly, that movement is learned easily. The less silly speculation is that game engine outputs may have been in the training data. Jim Fan thinks this is likely the case, and calls the result a 'data-driven physics engine.' Raphael Molière thinks this is likely, but more research is needed.

Brett Goldstein here digs into what it means that Sora works via 'patches' that combine to form the requested scene.

Gary Marcus keeps noting how the model gets physics wrong in various places, and, well, yes, we all know, please cut it out with the Stop Having Fun.

Yishan points out that humans also work mostly on 'folk physics.' Most of the time humans are not 'doing logic' they are vibing and using heuristics. I presume our dreams, if mapped to videos, would if anything look far less realistic than Sora.

Yann LeCun, who only a few days previous said that video like Sora produces was not something we knew how to do, doubled down with the ship to say that none of this means the models 'understand the physical world,' and of course his approach is better because it does. Why update? Is all of this technically impressive?

Sora Is Technically Impressive

Yes, Sora is definitely technically impressive.

It was not, however, unexpected.

Sam Altman: we'd like to show you what sora can do, please reply with captions for videos you'd like to see and we'll start making some!

Eliezer Yudkowsky: 6 months left on this timer.

Eliezer Yudkowsky (August 26, 2022): In 2-4 years, if we're still alive, anytime you see a video this beautiful, your first thought will be to wonder whether it's real or if the AI's prompt was "beautiful video of 15 different moth species flapping their wings, professional photography, 8k, trending on Twitter".

Roko (other thread): I don't really understand why anyone is freaking out over Sora.

This is entirely to be expected given the existence of generative image models plus incrementally more hardware and engineering effort.

It's also obviously not dangerous (in a "take over the world" sense).

Eliezer Yudkowsky: This is of course my own take (what with having explicitly predicted this). But I do think you want to hold out a space for others to say, "Well *I* didn't predict it, and now I've updated."

Altman's account spent much of last Thursday making videos for people's requests, although not so many that they couldn't cherry pick the good ones.

As usual, there are failures that look stupid, mistakes 'a person would never make' and all that. And there are flashes of absolute brilliance.

How impressive? There are disputes.

Tom Warren: this could be the "holy shit" moment of AI. OpenAI has just announced Sora, its text-to-video AI model. This video isn't real, it's based on a prompt of "a cat waking up its sleeping owner demanding breakfast…"

Daniel Eth: This isn't impressive. The owner doesn't wake up, so the AI clearly didn't understa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Ngo et al. "Every 'Every Bay Area House Party' Bay Area House Party", published by Ricki Heicklen on February 23, 2024 on LessWrong.

With thanks to Scott Alexander for

the inspiration, Jeffrey Ladish, Philip Parker, Avital Morris, and Drake Thomas for masterful cohosting, and Richard Ngo for his

investigative journalism.

Last summer, I threw an Every Bay Area House Party themed

party. I don't live in the Bay, but I was there for a construction-work-slash-webforum-moderation-and-UI-design-slash-grantmaking gig, so I took the opportunity to impose myself on the ever generous Jeffrey Ladish and host a party in his home. Fortunately, the inside of his house is already optimized to look like a parody of a Bay Area house party house, so not much extra decorating was needed, but when has that ever stopped me?

Richard Ngo recently

covered the event, with only very minor embellishments. I've heard rumors that some people are doubting whether the party described truly happened, so I'd like to set the record straight. Thus, this is part linkpost, part exegesis, and part

shameless promotion of my events for potential future venue-lenders.

The party had 60 attendees, at least according to the

Manifold Market on the topic. Upon arrival, partygoers put on tags with their name, professional title, and LessWrong karma. Attendees were also instructed to put on a wristband that successfully communicated their flirting policies. I took the wristband for people who glomarize about their flirting preferences; Richard took the wristband for those who flirt with all and only those who don't flirt with themselves.

Richard writes:

You scan around the living room, trying to figure out who to talk to first. The host is sitting on the sofa, with two boxes attached to the front of her shirt. One is filled with money, the other empty. A guy next to her is surreptitiously one-boxing, but she presciently slaps his hand away without even looking.

This is defamation. The second box wasn't (necessarily) empty, and Richard certainly never got the opportunity to look inside it. You might be wondering what was in the box. Unfortunately for you, I glomarize not only about my flirting policies but also about my box contents.

He is correct, though, that I managed to fend off all the surreptitious one-boxers, with the exception of my girlfriend Avital. She still doesn't know the contents - I would never date someone irresponsible enough to let unknown entities out of a box.

The party was PYOL (provide your own liquidity), but we did offer two punches: one for "Contextualizers," and one for "Decouplers or Homophobes". Avital and I drank the punch for "Decouplers or Homophobes." We're still coupled, so you can come to your own conclusions about how homophobic we must be.

My favorite part of the night happened when a circle formed around a Jewish Orthodox Rabbi friend of mine who had never heard of Rationality or Effective Altruism. Everyone at the party was eager to give him context. I joined the circle as they were discussing expanding moral circles, and the relative weight of animal lives and future people. "Eliezer doesn't even think cows are sentient," one attendee was saying. "But shrimp are!" another interrupted, causing the group to crack up.

"What?" my Rabbi friend said.

"Okay, back up, how much Peter Singer have you read?" another attendee said. "Like, have you read Animal Liberation and The Expanding Circle, or just Famine, Affluence, and Morality?" Avital and I tried to explain to them that our friend had not heard of any of the people they were naming, but they didn't seem to understand. Avital turned to me and said "I guess when you've read The Sequences too many times you forget about

inferential distance."

Eventually, my Rabbi friend said "Okay, so what I'm hearing is: you're expected to t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do sparse autoencoders find "true features"?, published by Demian Till on February 22, 2024 on LessWrong.

In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following:

I'll argue that the L1 regularisation used to promote sparsity when training SAEs may cause neurons in the sparse layer to learn to represent common combinations of features rather than the individual features that we want them to discover

As well as making it more difficult to understand what the actual latent features are, I'll also argue that this limitation may result in some less common latent features not being discovered at all, not even within combinations

I'll then explain why I think that the phenomenon of feature splitting observed in Anthropic's SAE paper appears to demonstrate that this limitation does indeed have a large impact on the features discovered by SAEs

Finally I'll propose an approach for overcoming this limitation and discuss how we can test whether it really brings us closer to finding the real latent features

Rough definition of "true features"

We intend for SAEs to discover the "true features" (a term I'm borrowing from Anthropic's SAE paper) used by the target model e.g. a transformer-based LLM. There isn't a universally accepted definition of what "true features" are, but for now I'll use the term somewhat loosely to refer to something like:

linear directions in an activation space at a hidden layer within a target model which encode some reasonably monosemantic quantity such as the model's "confidence" in some concept being in play

they should play a causal role in the functioning of the target model. So for example if we were to activate or deactivate the feature while the target model is processing a given input sequence then we should expect the outputs to change accordingly in some reasonably understandable way

they should be in their most atomic form, so that e.g an arbitrary linear combination of two "true feature" directions is not necessarily itself a "true feature" direction even though it may satisfy the previous criteria

There may be other ways of thinking about features but this should give us enough to work with for our current purposes.

Why SAEs are incentivised to discover combinations of features rather than individual features

Consider a toy setup where one of the hidden layers in the target model has 3 "true features" represented by the following directions in its activation space:

Additionally, suppose that feature 1 and feature 2 occur far more frequently than feature 3, and that all features can potentially co-occur in a given activation vector. For the sake of simplicity let's also suppose for now that when features 1 & 2 occur together they tend to both activate with some roughly fixed proportions. For example, an activation vector in which both features 1 and 2 are present (but not feature 3) might look like the following:

Now suppose we train an SAE with 3 neurons in the sparse layer on activation vectors from this hidden layer such as the one above. The desirable outcome is that each of the 3 neurons in the sparse layer learns one of the 3 "true features". If this happens then the directions learnt by SAE would mirror the directions of the "true features" in the target model, looking something like:

However depending on the respective frequencies of feature 3 vs features 1 & 2, as well as the value of the L1 regularisation weight, I will argue shortly that what may happen is that two of the neurons learn to detect when each of features 1 & 2 respectively occur by themselves, while the third neuron learns to detect when they both occur together. In this case the di...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Job Listing: Managing Editor / Writer, published by Gretta Duleba on February 22, 2024 on LessWrong.

MIRI is hiring a managing editor and one or more writers. Here's the job listing. We're accepting applications until March 13th.

I am the Communications Manager at MIRI and I'm the hiring manager for these positions. I'm happy to answer questions.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From salt intake reduction to labor migration: Announcing top ideas for the AIM 2024 CE Incubation Program, published by CE on February 22, 2024 on The Effective Altruism Forum.

TLDR: In this post, we announce our top four charity ideas to launch through our August 12-October 4, 2024 Incubation Program. They are the result of months of work by our research team, who selected them through a five-stage

research process. We pick interventions that exceed ambitious cost-effectiveness bars, have a high quality of evidence, minimal failure modes, and high expected value. We're also announcing cause areas we'll investigate for the February-March 2025 IP.

We're seeking people to launch these ideas through our next

Incubation Program. No particular previous experience is necessary - if you could plausibly see yourself excited to launch one of these charities, we encourage you to apply. The deadline for applications is April 14, 2024. You can apply to both August-October 2024 or February-March 2025 programs via the same link below:

In the Incubation Program, we provide two months of cost-covered training, stipends (£1900/month during and for up to two months after the program), seed funding up to $200,000, operational support in the initial months, co-working space at our CE office in London, ongoing mentorship, and access to a community of alumni, funders, and experts. Learn more on our

CE Incubation Program page.

One sentence summaries

Advocacy for salt intake reduction

An organization seeking to convince governments and the food industry to lower the content of salt in food by setting sodium limits and reformulating high-sodium foods, thereby improving cardiovascular health.

Facilitating international labor migration via a digital platform

An organization that would facilitate the international migration of workers from low- and middle-income countries using a transparent digital platform paired with low-cost personalized support.

Ceramic filters for improving water quality

An organization focused on reducing the incidence of diarrhea and other waterborne illnesses by providing free ceramic water filters to families without access to clean drinking water.

Participatory Learning and Action (PLA) groups for maternal and newborn health

An organization focused on improving newborn and maternal health in rural villages by training facilitators and running PLA groups - a specific type of facilitated self-help group.

One-paragraph Summaries

Advocacy for salt intake reduction

High salt consumption contributes to poor cardiovascular health. Worldwide, cardiovascular diseases are the leading cause of death and are among the top ten contributors to years lived with disabilities. There is good evidence that reducing the amount of sodium people consume in their diets can reduce the risk of cardiovascular problems.

Therefore, several countries have successfully reduced the sodium intake of their population, for example by setting sodium limits, which led to the food industry reformulating certain high-salt products. We think that an organization advocating and assisting in implementing these policies could cost-effectively improve the health of millions.

Facilitating international labor migration via a digital platform

Migrating abroad for work can bring huge financial benefits to people experiencing poverty. Millions of people every year try to tap into these benefits by applying for temporary jobs in higher-income countries. However, the market for matching candidates with jobs is often highly inefficient and riddled with misinformation, putting candidates at financial and personal risk. In many countries, it can cost candidates several years' worth of salaries to secure a job abroad.

Fraud is also highly prevalent, leading to candidates often failing to migrate and instead ending up i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The One and a Half Gemini, published by Zvi on February 22, 2024 on LessWrong.

Previously: I hit send on The Third Gemini, and within half an hour DeepMind announced Gemini 1.5.

So this covers Gemini 1.5. One million tokens, and we are promised overall Gemini Advanced or GPT-4 levels of performance on Gemini Pro levels of compute.

This post does not cover the issues with Gemini's image generation, and what it is and is not willing to generate. I am on top of that situation and will get to it soon.

One Million Tokens

Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. In fact, we're ready to introduce the next generation: Gemini 1.5. It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute.

It is truly bizarre to launch Gemini Advanced as a paid service, and then about a week later announce the new Gemini Pro 1.5 is now about as good as Gemini Advanced. Yes, actually, I do feel the acceleration, hot damn.

And that's not all!

This new generation also delivers a breakthrough in long-context understanding. We've been able to significantly increase the amount of information our models can process - running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet.

One million is a lot of tokens. That covers every individual document I have ever asked an LLM to examine. That is enough to cover my entire set of AI columns for the entire year, in case I ever need to look something up, presumably Google's NotebookLM is The Way to do that.

A potential future 10 million would be even more.

Soon Gemini will be able to watch a one hour video or read 700k words, whereas right now if I use the web interface of Gemini Advanced interface all I can upload is a photo.

The standard will be to give people 128k tokens to start, then you can pay for more than that. A million tokens is not cheap inference, even for Google.

Oriol Vinyals (VP of R&D DeepMind): Gemini 1.5 has arrived. Pro 1.5 with 1M tokens available as an experimental feature via AI Studio and Vertex AI in private preview.

Then there's this: In our research, we tested Gemini 1.5 on up to 2M tokens for audio, 2.8M tokens for video, and

10M

tokens for text. From Shannon's 1950s bi-gram models (2 tokens), and after being mesmerized by LSTMs many years ago able to model 200 tokens, it feels almost impossible that I would be talking about hundreds of thousands of tokens in context length, let alone millions.

Jeff Dean (Chief Scientist, Google DeepMind): Multineedle in haystack test: We also created a generalized version of the needle in a haystack test, where the model must retrieve 100 different needles hidden in the context window. For this, we see that Gemini 1.5 Pro's performance is above that of GPT-4 Turbo at small context lengths and remains relatively steady across the entire 1M context window, while the GPT-4 Turbo model drops off more quickly (and cannot go past 128k tokens).

Guido Appenzeller (responding to similar post): Is this really done with a monolithic model? For a 10M token window, input state would be many Gigabytes. Seems crazy expensive to run on today's hardware.

Sholto Douglas (DeepMind): It would honestly have been difficult to do at decent latency without TPUs (and their interconnect) They're an under appreciated but critical piece of this story

Here are their head-to-head results with themselves:

Here is the technical report. There is no need to read it, all of this is straightforward. Their safety section says 'we followed our procedure' and offers no additional details on methodology. On safety performance, their tests did not seem to offer much insight, scores were similar to Gemini Pro 1.0.

Mixture...