LessWrong (Curated & Popular)

This is a link post. Someone I know wrote this very nice post explaining the core intuition around Shapley values (which play an important role in impact assessment) using Venn diagrams, and I think it's great. It might be the most intuitive explainer I've come across so far.

Incidentally, the post also won an honorable mention in 3blue1brown's Summer of Mathematical Exposition.

---

First published:
December 6th, 2024

Source:
https://www.lesswrong.com/posts/6dixnRRYSLTqCdJzG/understanding-shapley-values-with-venn-diagrams

---

Narrated by TYPE III AUDIO.

TL;DR: In September 2024, OpenAI released o1, its first "reasoning model". This model exhibits remarkable test-time scaling laws, which complete a missing piece of the Bitter Lesson and open up a new axis for scaling compute. Following Rush and Ritter (2024) and Brown (2024a, 2024b), I explore four hypotheses for how o1 works and discuss some implications for future scaling and recursive self-improvement.

The Bitter Lesson(s)

The Bitter Lesson is that "general methods that leverage computation are ultimately the most effective, and by a large margin." After a decade of scaling pretraining, it's easy to forget this lesson is not just about learning; it's also about search.

OpenAI didn't forget. Their new "reasoning model" o1 has figured out how to scale search during inference time. This does not use explicit search algorithms. Instead, o1 is trained via RL to get better at implicit search via chain of thought [...]

---

Outline:

(00:40) The Bitter Lesson(s)

(01:56) What we know about o1

(02:09) What OpenAI has told us

(03:26) What OpenAI has showed us

(04:29) Proto-o1: Chain of Thought

(04:41) In-Context Learning

(05:14) Thinking Step-by-Step

(06:02) Majority Vote

(06:47) o1: Four Hypotheses

(08:57) 1. Filter: Guess + Check

(09:50) 2. Evaluation: Process Rewards

(11:29) 3. Guidance: Search / AlphaZero

(13:00) 4. Combination: Learning to Correct

(14:23) Post-o1: (Recursive) Self-Improvement

(16:43) Outlook

---

First published:
December 9th, 2024

Source:
https://www.lesswrong.com/posts/byNYzsfFmb2TpYFPW/o1-a-technical-primer

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We present gradient routing, a way of controlling where learning happens in neural networks. Gradient routing applies masks to limit the flow of gradients during backpropagation. By supplying different masks for different data points, the user can induce specialized subcomponents within a model. We think gradient routing has the potential to train safer AI systems, for example, by making them more transparent, or by enabling the removal or monitoring of sensitive capabilities.

In this post, we:

• Show how to implement gradient routing.
• Briefly state the main results from our paper, on...
  • Controlling the latent space learned by an MNIST autoencoder so that different subspaces specialize to different digits;
  • Localizing computation in language models: (a) inducing axis-aligned features and (b) demonstrating that information can be localized then removed by ablation, even when data is imperfectly labeled; and
  • Scaling oversight to efficiently train a reinforcement learning policy even with [...]

---

Outline:

(01:48) Gradient routing

(03:02) MNIST latent space splitting

(04:31) Localizing capabilities in language models

(04:36) Steering scalar

(05:46) Robust unlearning

(09:06) Unlearning virology

(10:38) Scalable oversight via localization

(15:28) Key takeaways

(15:32) Absorption

(17:04) Localization avoids Goodharting

(18:02) Key limitations

(19:47) Alignment implications

(19:51) Robust removal of harmful capabilities

(20:19) Scalable oversight

(21:36) Specialized AI

(22:52) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:
December 6th, 2024

Source:
https://www.lesswrong.com/posts/nLRKKCTtwQgvozLTN/gradient-routing-masking-gradients-to-localize-computation

---

Narrated by TYPE III AUDIO.

---

Images from the article:

This is a brief summary of what we believe to be the most important takeaways from our new paper and from our findings shown in the o1 system card. We also specifically clarify what we think we did NOT show.

Paper: https://www.apolloresearch.ai/research/scheming-reasoning-evaluations

Twitter about paper: https://x.com/apolloaisafety/status/1864735819207995716

Twitter about o1 system card: https://x.com/apolloaisafety/status/1864737158226928124

What we think the most important findings are

Models are now capable enough to do in-context scheming reasoning

We say an AI system is “scheming” if it covertly pursues misaligned goals, hiding its true capabilities and

objectives. We think that in order to scheme, models likely need to be goal-directed, situationally aware, and capable enough to reason about scheming as a strategy. In principle, models might acquire situational awareness and stable long-term goals during training, and then scheme in pursuit of those goals. We only study the simpler case of “in-context scheming,” where [...]

---

Outline:

(00:28) What we think the most important findings are

(00:33) Models are now capable enough to do in-context scheming reasoning

(03:53) The scheming actions can sometimes be quite subtle

(05:05) Models sometimes double down on their deception

(06:35) Models sometimes scheme without any goal nudging

(07:34) Sonnet-3.5 and opus-3 can sandbag in pursuit of helpfulness

(08:22) Sonnet-3.5 can sandbag without relying on CoT reasoning

(09:28) What we are not claiming:

(10:33) Takeaways

(10:36) Chain-of-thought monitoring

(11:42) Chain-of-thought access

(13:27) Inability safety cases now have to be more specific

The original text contained 7 images which were described by AI.

---

First published:
December 5th, 2024

Source:
https://www.lesswrong.com/posts/8gy7c8GAPkuu6wTiX/frontier-models-are-capable-of-in-context-scheming

---

Narrated by TYPE III AUDIO.

---

Images from the article:

TLDR: LessWrong + Lighthaven need about $3M for the next 12 months. Donate here, or send me an email, DM or signal message (+1 510 944 3235) if you want to support what we do. Donations are tax-deductible in the US. Reach out for other countries, we can likely figure something out. We have big plans for the next year, and due to a shifting funding landscape we need support from a broader community more than in any previous year.

I've been running LessWrong/Lightcone Infrastructure for the last 7 years. During that time we have grown into the primary infrastructure provider for the rationality and AI safety communities. "Infrastructure" is a big fuzzy word, but in our case, it concretely means:

• We build and run LessWrong.com and the AI Alignment Forum.[1]
• We built and run Lighthaven (lighthaven.space), a ~30,000 sq. ft. campus in downtown Berkeley where we [...]

---

Outline:

(03:52) LessWrong

(06:36) Does LessWrong influence important decisions?

(09:37) Does LessWrong make its readers/writers more sane?

(11:37) LessWrong and intellectual progress

(19:08) Lighthaven

(22:04) The economics of Lighthaven

(24:26) How does Lighthaven improve the world?

(28:41) The relationship between Lighthaven and LessWrong

(30:36) Lightcone and the funding ecosystem

(35:17) Our work on funding infrastructure

(37:57) If its worth doing its worth doing with made-up statistics

(38:44) The OP GCR capacity building team survey

(42:09) Lightcone/LessWrong cannot be funded by just running ads

(43:55) Comparing LessWrong to other websites and apps

(45:00) Lighthaven event surplus

(47:13) The future of (the) Lightcone

(48:02) Lightcone culture and principles

(50:04) Things I wish I had time and funding for

(59:31) What do you get from donating to Lightcone?

(01:02:03) Tying everything together

The original text contained 22 footnotes which were omitted from this narration.

The original text contained 7 images which were described by AI.

---

First published:
November 30th, 2024

Source:
https://www.lesswrong.com/posts/5n2ZQcbc7r4R8mvqc/the-lightcone-is-nothing-without-its-people-lw-lighthaven-s-5

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Balsa Policy Institute chose as its first mission to lay groundwork for the potential repeal, or partial repeal, of section 27 of the Jones Act of 1920. I believe that this is an important cause both for its practical and symbolic impacts.

The Jones Act is the ultimate embodiment of our failures as a nation.

After 100 years, we do almost no trade between our ports via the oceans, and we build almost no oceangoing ships.

Everything the Jones Act supposedly set out to protect, it has destroyed.



Table of Contents

1. What is the Jones Act?
2. Why Work to Repeal the Jones Act?
3. Why Was the Jones Act Introduced?
4. What is the Effect of the Jones Act?
5. What Else Happens When We Ship More Goods Between Ports?
6. Emergency Case Study: Salt Shipment to NJ in [...]

---

Outline:

(00:38) What is the Jones Act?

(01:33) Why Work to Repeal the Jones Act?

(02:48) Why Was the Jones Act Introduced?

(03:19) What is the Effect of the Jones Act?

(06:52) What Else Happens When We Ship More Goods Between Ports?

(07:14) Emergency Case Study: Salt Shipment to NJ in the Winter of 2013-2014

(12:04) Why no Emergency Exceptions?

(15:02) What Are Some Specific Non-Emergency Impacts?

(18:57) What Are Some Specific Impacts on Regions?

(22:36) What About the Study Claiming Big Benefits?

(24:46) What About the Need to ‘Protect’ American Shipbuilding?

(28:31) The Opposing Arguments Are Disingenuous and Terrible

(34:07) What Alternatives to Repeal Do We Have?

(35:33) What Might Be a Decent Instinctive Counterfactual?

(41:50) What About Our Other Protectionist and Cabotage Laws?

(43:00) What About Potential Marine Highways, or Short Sea Shipping?

(43:48) What Happened to All Our Offshore Wind?

(47:06) What Estimates Are There of Overall Cost?

(49:52) What Are the Costs of Being American Flagged?

(50:28) What Are the Costs of Being American Made?

(51:49) What are the Consequences of Being American Crewed?

(53:11) What Would Happen in a Real War?

(56:07) Cruise Ship Sanity Partially Restored

(56:46) The Jones Act Enforcer

(58:08) Who Benefits?

(58:57) Others Make the Case

(01:00:55) An Argument That We Were Always Uncompetitive

(01:02:45) What About John Arnold's Case That the Jones Act Can’t Be Killed?

(01:09:34) What About the Foreign Dredge Act of 1906?

(01:10:24) Fun Stories

---

First published:
November 27th, 2024

Source:
https://www.lesswrong.com/posts/dnH2hauqRbu3GspA2/repeal-the-jones-act-of-1920

---

Narrated by TYPE III AUDIO.

---

Images from the article:

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race for Machine Superintelligence. Consider subscribing to stay up to date with my work.

An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence

The US-China AI rivalry is entering a dangerous new phase.

Earlier today, the US-China Economic and Security Review Commission (USCC) released its annual report, with the following as its top recommendation:

Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability. AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and [...]

---

Outline:

(00:28) An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence

(03:09) What China has said about AI

(06:14) Revealing technical errors

(08:29) Conclusion

The original text contained 1 image which was described by AI.

---

First published:
November 20th, 2024

Source:
https://www.lesswrong.com/posts/KPBPc7RayDPxqxdqY/china-hawks-are-manufacturing-an-ai-arms-race

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In contract law, there's this thing called a “representation”. Example: as part of a contract to sell my house, I might “represent that” the house contains no asbestos. How is this different from me just, y’know, telling someone that the house contains no asbestos? Well, if it later turns out that the house does contain asbestos, I’ll be liable for any damages caused by the asbestos (like e.g. the cost of removing it).

In other words: a contractual representation is a factual claim along with insurance against that claim being false.

I claim[1] that people often interpret everyday factual claims and predictions in a way similar to contractual representations. Because “representation” is egregiously confusing jargon, I’m going to call this phenomenon “assurance”.

Prototypical example: I tell my friend that I plan to go to a party around 9 pm, and I’m willing to give them a ride. My friend [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
October 20th, 2024

Source:
https://www.lesswrong.com/posts/p9rQJMRq4qtB9acds/information-vs-assurance

---

Narrated by TYPE III AUDIO.

Epistemic Status: 13 years working as a therapist for a wide variety of populations, 5 of them working with rationalists and EA clients. 7 years teaching and directing at over 20 rationality camps and workshops. This is an extremely short and colloquially written form of points that could be expanded on to fill a book, and there is plenty of nuance to practically everything here, but I am extremely confident of the core points in this frame, and have used it to help many people break out of or avoid manipulative practices.

TL;DR: Your wants and preferences are not invalidated by smarter or more “rational” people's preferences. What feels good or bad to someone is not a monocausal result of how smart or stupid they are.

Alternative titles to this post are "Two people are enough to form a cult" and "Red flags if dating rationalists," but this [...]

---

Outline:

(02:53) 1) You are not too stupid to know what you want.

(07:34) 2) Feeling hurt is not a sign of irrationality.

(13:15) 3) Illegible preferences are not invalid.

(17:22) 4) Your preferences do not need to fully match your communitys.

(21:43) Final Thoughts

The original text contained 3 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:
November 26th, 2024

Source:
https://www.lesswrong.com/posts/LifRBXdenQDiX4cu8/you-are-not-too-irrational-to-know-your-preferences-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.