LessWrong (Curated & Popular)

[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]

Widespread, misleading claims about AI forecasting

Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.

These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

• Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:
  
  
• A post on Marginal Revolution with the title and abstract [...]

---

Outline:

(00:24) Widespread, misleading claims about AI forecasting

(03:02) What does human-level or superhuman forecasting mean?

(04:08) Red flags for claims to (super)human AI forecasting accuracy

(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)

(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)

(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)

(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)

(15:19) Takeaways

(16:17) So how good are AI forecasters?

The original text contained 1 footnote which was omitted from this narration.

The original text contained 6 images which were described by AI.

---

First published:
September 12th, 2024

Source:
https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting

---

Narrated by TYPE III AUDIO.

---

Images from the article:

This is a link post. ---

First published:
September 12th, 2024

Source:
https://www.lesswrong.com/posts/bhY5aE4MtwpGf3LCo/openai-o1

---

Narrated by TYPE III AUDIO.

Epistemic status: these are my own opinions on AI risk communication, based primarily on my own instincts on the subject and discussions with people less involved with rationality than myself. Communication is highly subjective and I have not rigorously A/B tested messaging. I am even less confident in the quality of my responses than in the correctness of my critique.

If they turn out to be true, these thoughts can probably be applied to all sorts of communication beyond AI risk.

Lots of work has gone into trying to explain AI risk to laypersons. Overall, I think it's been great, but there's a particular trap that I've seen people fall into a few times. I'd summarize it as simplifying and shortening the text of an argument without enough thought for the information content. It comes in three forms. One is forgetting to adapt concepts for someone with a far [...]

---

Outline:

(01:14) Failure to Adapt Concepts

(03:41) Failure to Filter Information

(05:09) Failure to Sound Like a Human Being

(07:23) Summary

The original text contained 4 images which were described by AI.

---

First published:
September 10th, 2024

Source:
https://www.lesswrong.com/posts/CZQYP7BBY4r9bdxtY/the-best-lay-argument-is-not-a-simple-english-yud-essay

---

Narrated by TYPE III AUDIO.

---

Images from the article:

In my last post, I wrote that no resource out there exactly captured my model of epistemology, which is why I wanted to share a half-baked version of it.

But I do have one book which I always recommend to people who want to learn more about epistemology: Inventing Temperature by Hasok Chang.

To be very clear, my recommendation is not just to get the good ideas from this book (of which there are many) from a book review or summary — it's to actually read the book, the old-school way, one word at a time.

Why? Because this book teaches you the right feel, the right vibe for thinking about epistemology. It punctures the bubble of sterile non-sense that so easily pass for “how science works” in most people's education, such as the “scientific method”. And it does so by demonstrating how one actually makes progress in epistemology [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
September 8th, 2024

Source:
https://www.lesswrong.com/posts/TbaCa7sY3GxHBcXTd/my-number-1-epistemology-book-recommendation-inventing

---

Narrated by TYPE III AUDIO.

Our new video is an adaptation of That Alien Message, by @Eliezer Yudkowsky. This time, the text has been significantly adapted, so I include it below.

Part 1

Picture a world just like ours, except the people are a fair bit smarter: in this world, Einstein isn’t one in a million, he's one in a thousand. In fact, here he is now. He's made all the same discoveries, but they’re not quite as unusual: there have been lots of other discoveries. Anyway, he's out one night with a friend looking up at the stars when something odd happens. [visual: stars get brighter and dimmer, one per second. The two people on the hill look at each other, confused]

The stars are flickering. And it's just not a hallucination. Everyone's seeing it.

And so everyone immediately freaks out and panics! Ah, just kidding, the people of this world are [...]

---

Outline:

(00:16) Part 1

(06:22) Part 2

(09:53) Part 3

(11:58) Part 4

---

First published:
September 7th, 2024

Source:
https://www.lesswrong.com/posts/Q9omyL3qooXdjnyZn/that-alien-message-the-animation

---

Narrated by TYPE III AUDIO.

Personally, I suspect the alignment problem is hard. But even if it turns out to be easy, survival may still require getting at least the absolute basics right; currently, I think we're mostly failing even at that.

Early discussion of AI risk often focused on debating the viability of various elaborate safety schemes humanity might someday devise—designing AI systems to be more like “tools” than “agents,” for example, or as purely question-answering oracles locked within some kryptonite-style box. These debates feel a bit quaint now, as AI companies race to release agentic models they barely understand directly onto the internet.

But a far more basic failure, from my perspective, is that at present nearly all AI company staff—including those tasked with deciding whether new models are safe to build and release—are paid substantially in equity, the value of which seems likely to decline if their employers stop building and [...]

---

First published:
September 7th, 2024

Source:
https://www.lesswrong.com/posts/sMBjsfNdezWFy6Dz5/pay-risk-evaluators-in-cash-not-equity

---

Narrated by TYPE III AUDIO.

Intro

In April 2024, my colleague and I (both affiliated with Peking University) conducted a survey involving 510 students from Tsinghua University and 518 students from Peking University—China's two top academic institutions. Our focus was on their perspectives regarding the frontier risks of artificial intelligence.

In the People's Republic of China (PRC), publicly accessible survey data on AI is relatively rare, so we hope this report provides some valuable insights into how people in the PRC are thinking about AI (especially the risks). Throughout this post, I’ll do my best to weave in other data reflecting the broader Chinese sentiment toward AI.

For similar research, check out The Center for Long-Term Artificial Intelligence, YouGov, Monmouth University, The Artificial Intelligence Policy Institute, and notably, a poll conducted by Rethink Priorities, which closely informed our survey design.

You can read the full report published in the Jamestown Foundation's China Brief [...]

The original text contained 11 images which were described by AI.

---

First published:
September 2nd, 2024

Source:
https://www.lesswrong.com/posts/gxCGKHpX8G8D8aWy5/survey-how-do-elite-chinese-students-feel-about-the-risks-of

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Paging Gwern or anyone else who can shed light on the current state of the AI market—I have several questions.

Since the release of ChatGPT, at least 17 companies, according to the LMSYS Chatbot Arena Leaderboard, have developed AI models that outperform it. These companies include Anthropic, NexusFlow, Microsoft, Mistral, Alibaba, Hugging Face, Google, Reka AI, Cohere, Meta, 01 AI, AI21 Labs, Zhipu AI, Nvidia, DeepSeek, and xAI.

Since GPT-4's launch, 15 different companies have reportedly created AI models that are smarter than GPT-4. Among them are Reka AI, Meta, AI21 Labs, DeepSeek AI, Anthropic, Alibaba, Zhipu, Google, Cohere, Nvidia, 01 AI, NexusFlow, Mistral, and xAI.

Twitter AI (xAI), which seemingly had no prior history of strong AI engineering, with a small team and limited resources, has somehow built the third smartest AI in the world, apparently on par with the very best from OpenAI.

The top AI image [...]

---

First published:
August 28th, 2024

Source:
https://www.lesswrong.com/posts/yRjLY3z3GQBJaDuoY/things-that-confuse-me-about-the-current-ai-market

---

Narrated by TYPE III AUDIO.

If you ask the internet if breastfeeding is good, you will soon learn that YOU MUST BREASTFEED because BREAST MILK = OPTIMAL FOOD FOR BABY. But if you look for evidence, you’ll discover two disturbing facts.

First, there's no consensus about why breastfeeding is good. I’ve seen experts suggest at least eight possible mechanisms:

1. Formula can’t fully reproduce the complex blend of fats, proteins and sugars in breast milk.
2. Formula lacks various bio-active things in breast milk, like antibodies, white blood cells, oligosaccharides, and epidermal growth factor.
3. If local water is unhealthy, then the mother's body acts as a kind of “filter”.
4. Breastfeeding may have psychological/social benefits, perhaps in part by releasing oxytocin in the mother.
5. Breastfeeding decreases fertility, meaning the baby may get more time before resources are redirected to a younger sibling.
6. Breastfeeding may help mothers manage various post-birth health issues?
7. Infants are often given formula [...]

---

Outline:

(04:03) Except…

(05:07) The one big trial

(06:38) How much did breastfeeding increase?

(07:49) Did more breastfeeding lead to healthier babies?

(09:15) Did more breastfeeding lead to better long-term health?

(10:27) Intermission

(10:46) Did more breastfeeding lead to higher IQ?

(13:54) Did more breastfeeding lead to higher IQ later in life?

(15:37) Nursing doubts

The original text contained 3 images which were described by AI.

---

First published:
August 30th, 2024

Source:
https://www.lesswrong.com/posts/p7x3vvPR59WHuoQ2A/nursing-doubts

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Crossposted from https://williamrsaunders.substack.com/p/principles-for-the-agi-race

Why form principles for the AGI Race?

I worked at OpenAI for 3 years, on the Alignment and Superalignment teams. Our goal was to prepare for the possibility that OpenAI succeeded in its stated mission of building AGI (Artificial General Intelligence, roughly able to do most things a human can do), and then proceed on to make systems smarter than most humans. This will predictably face novel problems in controlling and shaping systems smarter than their supervisors and creators, which we don't currently know how to solve. It's not clear when this will happen, but a number of people would throw around estimates of this happening within a few years.

While there, I would sometimes dream about what would have happened if I’d been a nuclear physicist in the 1940s. I do think that many of the kind of people who get involved in the effective [...]

---

Outline:

(00:06) Why form principles for the AGI Race?

(03:32) Bad High Risk Decisions

(04:46) Unnecessary Races to Develop Risky Technology

(05:17) High Risk Decision Principles

(05:21) Principle 1: Seek as broad and legitimate authority for your decisions as is possible under the circumstances

(07:20) Principle 2: Don’t take actions which impose significant risks to others without overwhelming evidence of net benefit

(10:52) Race Principles

(10:56) What is a Race?

(12:18) Principle 3: When racing, have an exit strategy

(13:03) Principle 4: Maintain accurate race intelligence at all times.

(14:23) Principle 5: Evaluate how bad it is for your opponent to win instead of you, and balance this against the risks of racing

(15:07) Principle 6: Seriously attempt alternatives to racing

(16:58) Meta Principles

(17:01) Principle 7: Don’t give power to people or structures that can’t be held accountable.

(18:36) Principle 8: Notice when you can’t uphold your own principles.

(19:17) Application of my Principles

(19:21) Working at OpenAI

(24:19) SB 1047

(28:32) Call to Action

---

First published:
August 30th, 2024

Source:
https://www.lesswrong.com/posts/aRciQsjgErCf5Y7D9/principles-for-the-agi-race

---

Narrated by TYPE III AUDIO.