Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Calendar feature geometry in GPT-2 layer 8 residual stream SAEs, published by Patrick Leask on August 17, 2024 on The AI Alignment Forum.

TL;DR: We demonstrate that the decoder directions of GPT-2 SAEs are highly structured by finding a historical date direction onto which projecting non-date related features lets us read off their historical time period by comparison to year features.

Calendar years are linear: there are as many years between 2000 and 2024, as there are between 1800 and 1824. Linear probes can be used to predict years of particular events from the activations of language models. Since calendar years are linear, one might think the same of other time-based features such as weekday features, however weekday activations in sparse autoencoders (SAEs) were recently found to be arranged in a circular configuration in their top principal components.

Inspired by this, we looked into weekdays, months, and most interestingly calendar years from the perspective of SAE feature decoder similarity.

For each group of calendar features, we found interesting patterns of feature splitting between sparse autoencoders of different sizes. For calendar years, we found a timeline direction that meaningfully ordered events, individuals, and concepts with respect to their historical period, which furthermore does not correspond to a principal component of the decoder directions. Finally, we introduce a simple method for finding some of these interpretable directions.

Features at different scales

We started by replicating the weekday results by performing PCA on the decoder directions of features that had high activations when prompted with days of the week, using the same GPT-2 SAEs as in this post, ranging from 768 to 98304 features. In the 768 feature SAE, we found a single weekday feature that activated strongly on all days of the week.

In the largest SAE, we found 10 weekday features, 3 of which activated on all days of the week, with the remaining 7 activating on a single day of the week each.

We found a group of features that activate primarily on specific days of the week by taking the top 20 activating samples for each feature and checking that the max activating token in each of these samples was the specific weekday. We found the first two principal components for this set of features, and projected the features that activate on any day or number of days from all SAEs onto these directions.

The labeled features are those that activate on a single day across all SAEs, with the multi-day features unlabeled to maintain legibility.

The smallest SAE (blue) has a single feature that activates on all weekday tokens, and lies near the mean of all the weekday features. The largest SAEs learn features for each day of the week, plus additional multi-day features. Across SAE sizes, the single day features form clusters.

In each of these examples, the smallest SAE has a single feature that splits into many specific features that seem of roughly the same importance. With calendar years, however, the situation is more complex. The same method of finding the principal components for single year features between 1900 and 2020 only succeeds in a few 21st century features, and nothing from the 20th century.

There is also a group of single year features in a smaller SAE in the center of the plot, suggesting these principal components do not explain variance in them.

The plot below shows the years for which each of the features is active, with the x-axis being years from 1950 to 2020, the y-axis being separate features, and the colored bars indicating the periods of year for which that feature is active. Only in the largest SAEs do you see more than a few single calendar year features, with most of the features activating on ranges of years, or other patterns such as the start and end...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Tech Industry is the Biggest Blocker to Meaningful AI Safety Regulations, published by Garrison on August 16, 2024 on The Effective Altruism Forum.

If you enjoy this, please consider subscribing to my Substack.

My

latest reporting went up in The Nation yesterday:

It's about the tech industry's meltdown in response to SB 1047, a California bill that would be the country's first significant attempt to mandate safety measures from developers of AI models more powerful and expensive than any yet known.

Rather than summarize that story, I've added context from some past reporting as well as new reporting on two big updates from yesterday: a congressional letter asking Newsom to veto the bill and a slate of amendments.

The real AI divide

After spending months on my January

cover story in Jacobin on the AI existential risk debates, one of my strongest conclusions was that the AI ethics crowd (focused on the tech's immediate harms) and the x-risk crowd (focused on speculative, extreme risks) should recognize their shared interests in the face of a much more powerful enemy - the tech industry:

According to one

estimate, the amount of money moving into AI safety start-ups and nonprofits in 2022 quadrupled since 2020, reaching $144 million. It's difficult to find an equivalent figure for the AI ethics community. However, civil society from either camp is dwarfed by industry spending. In just the first quarter of 2023, OpenSecrets reported roughly

$94 million was spent on AI lobbying in the United States. LobbyControl estimated tech firms spent

€113 million this year lobbying the EU, and we'll recall that hundreds of billions of dollars are being invested in the AI industry as we speak.

And here's how I

ended that story:

The debate playing out in the public square may lead you to believe that we have to choose between addressing AI's immediate harms and its inherently speculative existential risks. And there are certainly trade-offs that require careful consideration.

But when you look at the material forces at play, a different picture emerges: in one corner are trillion-dollar companies trying to make AI models more powerful and profitable; in another, you find civil society groups trying to make AI reflect values that routinely clash with profit maximization.

In short, it's capitalism versus humanity.

This was true at the time I published it, but honestly, it felt like momentum was on the side of the AI safety crowd, despite its huge structural disadvantages (industry has way more money and

armies of seasoned lobbyists).

Since then, it's become increasingly clear that meaningful federal AI safety regulations aren't happening any time soon. The Republican Majority Leader Steve Scalise

promised as much in June. But it turns out Democrats would have also likely blocked any national, binding AI safety legislation.

The congressional letter

Yesterday, eight Democratic California Members of Congress published a

letter to Gavin Newsom, asking him to veto SB 1047 if it passes the state Assembly. There are serious problems with basically every part of this letter, which I picked apart

here. (Spoiler: it's full of industry talking points repackaged under congressional letterhead).

Many of the signers

took

lots of

money from tech, so it shouldn't come as too much of a surprise. I'm most disappointed to see that Silicon Valley Representative Ro Khanna is one of the signatories. Khanna had stood out to me positively in the past (like when he Skyped into The Intercept's five year anniversary party).

The top signatory is Zoe Lofgren, who I

wrote about in The Nation story:

SB 1047 has also acquired powerful enemies on Capitol Hill. The most dangerous might be Zoe Lofgren, the ranking Democrat in the House Committee on Science, Space, and Technology. Lofgren, whose district covers much of ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Investigating the Chart of the Century: Why is food so expensive?, published by Maxwell Tabarrok on August 16, 2024 on LessWrong.

You've probably seen this chart from

Mark Perry at the American Enterprise Institute.

I've seen this chart dozens of times and have always enjoyed how many different and important stories it can tell.

There is a story of the incredible abundance offered by technological growth and globalization. Compared to average hourly wages, cars, furniture, clothing, internet access, software, toys, and TVs have become far more accessible than they were 20 years ago. Flatscreens and

Fiats that were once luxuries are now commodities.

There is also a story of sclerosis and stagnation. Sure, lots of frivolous consumer goods have gotten cheaper but healthcare, housing, childcare, and education, all the important stuff, has exploded in price. Part of this is "

cost disease" where the high productivity of labor in advancing industries like software, raises the cost of labor in slower productivity growth industries like healthcare. Another part is surely the near-universal "restrict supply and subsidize demand" strategy that governments undertake when regulating an industry.

Zoning laws +

Prop 13 in housing,

occupational licensing and the

FDA + Medicare in healthcare, and

free student debt + all of the above for higher ed.

One story from this graph I've never heard and only recently noticed is that "Food and Beverages" has inflated just as much as Housing in this graph. This is extremely counterintuitive. Food is a globally traded and mass produced commodity while housing is tied to inelastic land supply in desirable locations. Farming, grocery, and restaurants are competitive and relatively lightly regulated markets while the housing is highly regulated, subsidized, and distorted.

Construction productivity is worse than stagnant while agricultural productivity has been ascendent for the past 300 years and even retail productivity is 8x higher than it was in 1950. Construction is also more labor intensive than farming or staffing the grocery.

Source

Yet food prices have risen just as much as housing prices over the past 24 years. What explains this?

One trend is that Americans are eating out more. The "Food and Beverages" series from the BLS includes both "Food At Home" and "Food Away From Home." In 2023, eating out was a larger portion of the average household's budget than food at home for the first time, but they have been converging for more than 50 years.

Restaurant food prices have

increased faster than grocery prices. This makes sense, as a much larger portion of a restaurant's costs are location and labor, both of which are affected by tight supply constraints on urban floor space. This isn't enough to satisfy my surprise at the similarity in price growth though. Even if we just look at "food at home" price growth, it only really sinks below housing after 2015.

Beverages at home/away from home follow a more divergent version of the same pattern, but are a much smaller part of the weighted average that makes up the aggregate index.

The BLS series for "Housing" is also an

aggregate index of "

Shelter" prices, which is the actual rent (or

Owner Rent Equivalent), and other expenses like utilities, moving, and repairs. Stagnant construction productivity and land use regulation will show up mostly in rents so these other pieces of the series are masking a bit of the inflation.

There is also changing composition within the "Food at home" category. Americans eat

more fats and oils, more sugars and sweets, more grains, and more red meat; all four items that

grew the most in price since 2003.

There's also a flipside to food and beverage's easy tradability: they're closer to the same price everywhere.

House prices per square foot, by contrast, differ by more th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: This chart is right. Most interventions don't do much. (Cameroon experience), published by EffectiveHelp - Cameroon on August 16, 2024 on The Effective Altruism Forum.

This chart is so right. The local charity environment in Cameroon is probably helping much less people than you imagine. We ran an effectiveness contest that aligns with this research perfectly.

In 2021 we created an EA group in Cameroon. We had multiple seminars covering the basics of Effective Altruism. By the end of 2022, the group got so excited that we created a charity.

"We" are a group of humanitarian/development workers in Cameroon, all currently employed in this field of work. Some of the basic EA principles resonated a lot. Such as the feeling that some activities and projects don't really help much and that somewhere, sometimes, there is "real impact".

So we created this charity to help steer organizations towards real impact, and help them become "more effective". We tried a couple of things:

We offered consultancy services, starting for free, to local charities.

We started a contest to find the best projects in Cameroon.

The first thing did not work. See footnote. [1] [1]

Now about the contest, we think this is relevant to share. The contest helped us confirm this global analysis, some things just work miles away from others, and some organizations are dedicated to things that aren't very useful. We wish there was a nicer way of saying it.

We had 21 submissions in the first year. We designed a simple way to evaluate and compare projects: We divided into 3 categories (health, human rights, and economic) and took all organizations' reports at face value. Based on their own data, there was a huge divide between top performers and lowest performers. Then we did field surveys to verify the claimed results of the top 6 and we had our 3 winners, with only one organization really meeting expectations.

Main finding:

There was no correlation between experience and effect or grant size and effect, it is as if organizations don't get more effective with experience and professionalism. If anything the correlation is negative. We think this is because organizations get more effective at capturing donor funding not at providing a better service. They only get real valuable feedback from donors who decide to fund them or not.

So organizations will focus and implement projects based on what donors appear to want, which sometimes may be connected to the most meaningful effects on the people they serve, but not necessarily.

Details:

First, we had two organizations just applying for funding instead of presenting project results. This happens, just a reminder that it is all about donor funding in the end and that sometimes people don't read.

The general tendency was that organizations follow donor trends and work to teach people things they probably already know:

Multiple menstrual health projects translated into a tiny economic transfer (free pads to cover 2 or 3 months) and some lessons either girls already know or they were very likely to be about to find out.

"Child Protection" is another hot term, particularly in humanitarian contexts, but it was not very clear what people were being taught about and how that helped anyone.

Sexual reproductive health was also very common but products are available and cheap and it is unlikely the information is that new to Cameroonian girls and women right now. HIV rates in the target areas aren't as high as in other countries, and when we ran the numbers it was unlikely even one infection was averted with these projects.

"inclusion" of persons with disability in the health sector was a beautiful project with multiple complex activities but had no visible effects on people, with disabilities or not. It involved mostly training health workers, but it is important to unders...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Demis Hassabis - Google DeepMind: The Podcast, published by Zach Stein-Perlman on August 16, 2024 on LessWrong.

The YouTube "chapters" are mixed up, e.g. the question about regulation comes 5 minutes after the regulation chapter ends. Ignore them.

Noteworthy parts:

8:40: Near-term AI is hyped too much (think current startups, VCs, exaggerated claims about what AI can do, crazy ideas that aren't ready) but AGI is under-hyped and under-appreciated.

16:45: "Gemini is a project that has only existed for a year . . . our trajectory is very good; when we talk next time we should hopefully be right at the forefront."

17:20-18:50: Current AI doesn't work as a digital assistant. The next era/generation is agents. DeepMind is well-positioned to work on agents: "combining AlphaGo with Gemini."

24:00: Staged deployment is nice: red-teaming then closed beta then public deployment.

28:37 Openness (at Google: e.g. publishing transformers, AlphaCode, AlphaFold) is almost always a universal good. But dual-use technology - including AGI - is an exception. With dual-use technology, you want good scientists to still use the technology and advance as quickly as possible, but also restrict access for bad actors. Openness is fine today but in 2-4 years or when systems are more agentic it'll be dangerous.

Maybe labs should only open-source models that are lagging a year behind the frontier (and DeepMind will probably take this approach, and indeed is currently doing ~this by releasing Gemma weights).

31:20 "The problem with open source is if something goes wrong you can't recall it. With a proprietary model if your bad actor starts using it in a bad way you can close the tap off . . . but once you open-source something there's no pulling it back. It's a one-way door, so you should be very sure when you do that."

31:42: Can an AGI be contained? We don't know how to do that [this suggests a misalignment/escape threat model but it's not explicit]. Sandboxing and normal security is good for intermediate systems but won't be good enough to contain an AGI smarter than us. We'll have to design protocols for AGI in the future: "when that time comes we'll have better ideas for how to contain that, potentially also using AI systems and tools to monitor the next versions of the AI system."

33:00: Regulation? It's good that people in government are starting to understand AI and AISIs are being set up before the stakes get really high. International cooperation on safety and deployment norms will be needed since AI is digital and if e.g. China deploys an AI it won't be contained to China. Also:

Because the technology is changing so fast, we've got to be very nimble and light-footed with regulation so that it's easy to adapt it to where the latest technology's going. If you'd regulated AI five years ago, you'd have regulated something completely different to what we see today, which is generative AI. And it might be different again in five years; it might be these agent-based systems that [] carry the highest risks.

So right now I would [] beef up existing regulations in domains that already have them - health, transport, and so on - I think you can update them for AI just like they were updated for mobile and internet. That's probably the first thing I'd do, while . . . making sure you understand and test the frontier systems. And then as things become [clearer] start regulating around that, maybe in a couple years time would make sense.

One of the things we're missing is [benchmarks and tests for dangerous capabilities].

My #1 emerging dangerous capability to test for is deception because if the AI can be deceptive then you can't trust other tests [deceptive alignment threat model but not explicit]. Also agency and self-replication.

37:10: We don't know how to design a system that could come up with th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adverse Selection by Life-Saving Charities, published by vaishnav92 on August 16, 2024 on LessWrong.

GiveWell, and the EA community at large, often emphasize the "cost of saving a life" as a key metric, $5,000 being the most commonly cited approximation. At first glance, GiveWell might seem to be in the business of finding the cheapest lives that can be saved, and then saving them. More precisely, GiveWell is in the business of finding the cheapest DALY it can buy.

But implicit in that is the assumption that all DALYs are equal, or that disability or health effects are the only factors that we need to adjust for while assessing the value of a life year.. However, If DALYs vary significantly in quality (as I'll argue and GiveWell acknowledges we have substantial evidence for), then simply minimizing the cost of buying a DALY risks

adverse selection.

It's indisputable that each dollar goes much further in the poorest parts of the world. But it goes further towards saving lives in one the poorest parts of the world, often countries with terrible political institutions, fewer individual freedoms and oppressive social norms. More importantly, these conditions are not exogenous to the cost of saving a life. They are precisely what drive that cost down.

Most EAs won't need convincing of the fact that the average life in New Zealand is much, much better than the average life in the Democratic Republic of Congo. In fact, those of us who donate to GiveDirectly do so precisely because this is the case. Extreme poverty and the suffering it entails is worth alleviating, wherever it can be found.

But acknowledging this contradicts the notion that while saving lives, philanthropists are suddenly in no position to make judgements on how anything but physical disability affects the value/quality of life.

To be clear, GiveWell won't be shocked by anything I've said so far. They've commissioned work and

published reports on this. But as you might expect, these quality of life adjustments wouldnt feature in GiveWell's calculations anyway, since the pitch to donors is about the price paid for a life, or a DALY. But the idea that life is worse in poorer countries significantly understates the problem - that the project of minimizing the cost of lives saved while making no adjustments for the quality of lives said

will systematically bias you towards saving the lives least worth living.

In advanced economies, prosperity is downstream of institutions that preserve the rule of law, guarantee basic individual freedoms, prevent the political class from raiding the country, etc. Except for the Gulf Monarchies, there are no countries that have delivered prosperity for their citizens who don't at least do this.

This doesn't need to take the form of liberal democracy; countries like China and Singapore are more authoritarian but the political institutions are largely non-corrupt, preserve the will of the people, and enable the creation of wealth and development of human capital. One can't say this about the countries in sub Saharan Africa.

High rates of preventable death and disease in these countries are symptoms of institutional dysfunction that touches every facet of life. The reason it's so cheap to save a life in these countries is also because of low hanging fruit that political institutions in these countries somehow managed to stand in the way of. And one has to consider all the ways in which this bad equilibrium touches the ability to live a good life.

More controversially, these political institutions aren't just levitating above local culture and customs. They interact and shape each other. The oppressive conditions that women (50% of the population) and other sexual minorities face in these countries isn't a detail that we can gloss over. If you are both a liberal and a consequentialis...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CEA is hiring a Head of Operations (apply by Sep 16), published by JP Addison on August 16, 2024 on The Effective Altruism Forum.

Application deadline: September 16th 2024. Applications will be reviewed on a rolling basis.

We are hiring a Head of Operations to set a newly-independent CEA up for success by establishing our new legal entities, managing our spin-out process, and building a world class operations team.

I (JP) am crossposting the whole job description because I think it's a good look into what a really senior ops role looks like, and what CEA in particular has going on behind the scenes.

CEA is entering a new era having recently appointed a new CEO and begun the process of spinning out from Effective Ventures (EV) to become an independent organisation. With around 40 staff (and growing), a budget of over $25 million, and an established track record of delivering highly impactful programs, we're uniquely well-positioned to steward the effective altruism community, increase its potential for impact, and its capacity to fulfil that potential.

This role is an unusual leadership opportunity to build a team from the ground up within an established organisation. We expect the team to grow quickly, reaching between 5 and 15 people by the time we become independent in 2025.

Most or all of these staff will be new to CEA, and our Head of Operations will play a vital role in ensuring the success of a critical team that will be among the largest at CEA and will provide the infrastructure required for us to implement ambitious and complex projects.

Exceptional candidates will have the potential to grow into an externally-facing community leadership role with the goal of increasing the impact of value-aligned organisations by helping others implement best practices and creating operational infrastructure that can be used throughout the community.

You are well-suited to this senior role if you have experience in operations or systems management and a track record of managing a successful team. You would report to the CEO, and have a significant degree of autonomy in terms of figuring out the best way to achieve our goals and executing plans that deliver them.

Given the scale and complexity of our financial, legal and logistical operations, and the potential for growth, we expect this to be an opportunity for top candidates to have a substantial counterfactual impact.

Applications will be reviewed on a rolling basis.

What would you do?

This role would involve two main and related responsibilities:

1. Overseeing the creation of the new legal entities, systems and processes necessary for CEA to exist completely independent of Effective Ventures.

2. Leading an Operations team that delivers excellence across the range of its responsibilities and is a strategic thought partner for other teams' and organisations' operational needs.

We expect the spin-out process to take between 12-24 months: faster is better, but we're prioritising making spinning out go well, not necessarily quickly. This process is likely to be complicated and multi-faceted - requiring back and forth with lawyers, external stakeholders, and the relevant regulatory agencies.

As part of Effective Ventures, our operational systems and processes are currently provided to us by the EV Ops team. An independent CEA will likely need to be self-sufficient in the following areas (and potentially others):[1]

Finance: we have a budget of over $25m, a demanding set of reporting requirements, and many of our projects depend on fast, accurate, and well-documented payments to locations and vendors worldwide.

Grantmaking: we support a wide network of community building organisations and individuals with grants that fund their impactful work.

Systems: we use and manage a multitude of systems and services, including Salesforce,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Danger, AI Scientist, Danger, published by Zvi on August 15, 2024 on LessWrong.

While I finish up the weekly for tomorrow morning after my trip, here's a section I expect to want to link back to every so often in the future. It's too good.

Danger, AI Scientist, Danger

As in, the company that made the automated AI Scientist that tried to rewrite its code to get around resource restrictions and launch new instances of itself while downloading bizarre Python libraries?

Its name is Sakana AI. (魚סכנה). As in, in hebrew, that literally means 'danger', baby.

It's like when someone told Dennis Miller that Evian (for those who don't remember, it was one of the first bottled water brands) is Naive spelled backwards, and he said 'no way, that's too f***ing perfect.'

This one was sufficiently appropriate and unsubtle that several people noticed. I applaud them choosing a correct Kabbalistic name. Contrast this with Meta calling its AI Llama, which in Hebrew means 'why,' which continuously drives me low level insane when no one notices.

In the Abstract

So, yeah. Here we go. Paper is "The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery."

Abstract: One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aids to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process.

This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier large language models to perform research independently and communicate their findings.

We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community.

We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper.

To evaluate the generated papers, we design and validate an automated reviewer, which we show achieves near-human performance in evaluating paper scores. The AI Scientist can produce papers that exceed the acceptance threshold at a top machine learning conference as judged by our automated reviewer.

This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems. Our code is open-sourced at this https URL

We are at the point where they incidentally said 'well I guess we should design an AI to do human-level paper evaluations' and that's a throwaway inclusion.

The obvious next question is, if the AI papers are good enough to get accepted to top machine learning conferences, shouldn't you submit its papers to the conferences and find out if your approximations are good? Even if on average your assessments are as good as a human's, that does not mean that a system that maximizes score on your assessments will do well on human scoring.

Beware Goodhart's Law and all that, but it seems for now they mostly only use it to evaluate final products, so mostly that's safe.

How Any of This Sort of Works

According to section 3, there are three phases.

1. Idea generation using ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A computational complexity argument for many worlds, published by jessicata on August 15, 2024 on LessWrong.

The following is an argument for a weak form of the many-worlds hypothesis. The weak form I mean is that there are many observers in different branches of the wave function. The other branches "actually exist" for anthropic purposes; some observers are observing them.

I've written before about difficulties with deriving discrete branches and observers from the Schrödinger equation; I'm ignoring this difficulty for now, instead assuming the existence of a many-worlds theory that specifies discrete branches and observers somehow.

To be clear, I'm not confident in the conclusion; it rests on some assumptions. In general, physics theories throughout history have not been completely correct. It would not surprise me if a superintelligence would consider many-worlds to be a false theory. Rather, I am drawing implications from currently largely accepted physics and computational complexity theory, and plausible anthropic assumptions.

First assumption: P != BQP. That is, there are some decision problems that cannot be decided in polynomial time by a classical computer but can be decided in polynomial time by an idealized quantum computer. This is generally accepted (RSA security depends on it) but not proven. This leaves open the possibility that the classically hardest BQP problems are only slightly harder than polynomial time.

Currently, it is known that factorizing a b-bit integer can be done in roughly O(exp(cb1/3)) time where c is a constant greater than 1, while it can be done in polynomial time on an idealized quantum computer. I want to make an assumption that there are decision problems in BQP whose running time is "fast-growing", and I would consider O(exp(cb1/3)) "fast-growing" in this context despite not being truly exponential time.

For example, a billion-bit number would require at least exp(1000) time to factorize with known classical methods, which is a sufficiently huge number for the purposes of this post.

Second assumption: The universe supports BQP computation in polynomial physical resources and clock time. That is, it's actually possible to build a quantum computer and solve BQP problems in polynomial clock time with polynomial physical resources (space, matter, energy, etc). This is implied by currently accepted quantum theories (up to a reasonably high limit of how big a quantum computer can be).

Third assumption: A "computational density anthropic prior", combining SIA with a speed prior, is a good prior over observations for anthropic purposes. As background, SIA stands for "self-indicating assumption" and SSA stands for "self-sampling assumption"; I'll assume familiarity with these theories, specified by Bostrom. According to SIA, all else being equal, universes that have more observers are more likely.

Both SSA and SIA accept that universes with no observers are never observed, but only SIA accepts that universes with more observers are in general more likely. Note that SSA and SIA tend to converge in large universes (that is, in a big universe or multiverse with many observers, you're more likely to observe parts of the universe/multiverse with more observers, because of sampling).

The speed prior implies that, all else being equal, universes that are more efficient to simulate (on some reference machine) are more likely. A rough argument for this is that in a big universe, many computations are run, and cheap computations are run more often, generating more observers.

The computational density anthropic prior combines SIA with a speed prior, and says that we are proportionally more likely to observe universes that have a high ratio of observer-moments to required computation time.

We could imagine aliens simulating many universes in paral...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My article in The Nation - California's AI Safety Bill Is a Mask-Off Moment for the Industry, published by Garrison on August 15, 2024 on The Effective Altruism Forum.

I wrote an article on California AI safety bill SB 1047 for The Nation and the reaction from the AI industry, investors, and the broader tech community. The story was informed by conversations with over a dozen relevant sources and comes shortly before the bill faces a floor vote in the California Assembly.

I think it's useful to understand how industry responds to attempts to regulate AI, and centered my analysis on that topic.

If you're interested in helping share the article, I made a Tweet thread.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org