Sign up to save your podcasts
Or
Share “Base64Bench: How good are LLMs at base64, and why care about it?” by richbc
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Base64Bench: How good are LLMs at base64, and why care about it?” by richbc
This was a quick, short side-project produced during the MATS Research 8.1 extension. It's related to my group's main thread of work on black-box scheming monitoring through the connections to monitoring I explore below, but was time-boxed and pursued independently because I thought it was interesting!
Executive Summary
Figure 1. Accuracy vs. similarity threshold (0.95+) across 1700 pairs of encoding/decoding examples across a variety of datatypes and lengths. The accuracy is the proportion of the 3400 examples each model translated successfully (directly, with no reasoning or tools). Success for each task is defined by the normalised Levenshtein similarity of the answer/target pair hitting a given threshold, with a scoring requirement that model-encoded strings are decodable. Legend ordered by [email protected].- Introducing Base64Bench: a simple new benchmark for evaluating models on their ability to encode and decode base64.
- Base64 encoding and decoding are reasonably complex computational tasks to do perfectly [...]
---
Outline:
(00:31) Executive Summary
(03:07) An accidental (and surprising) discovery
(08:03) Have LLMs actually learned the algorithm?
(09:39) Introducing
(13:11) Accuracy vs. similarity threshold
(16:02) Encoding vs. decoding by model
(17:00) Task-level breakdown
(19:37) Why should we care?
(21:26) Monitoring implications
(23:51) Conclusion
(25:23) Appendix
(25:26) Zoomed-in threshold sweeps
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
October 5th, 2025
Source:
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64bench-how-good-are-llms-at-base64-and-why-care-about
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
View all episodes
By LessWrongThis was a quick, short side-project produced during the MATS Research 8.1 extension. It's related to my group's main thread of work on black-box scheming monitoring through the connections to monitoring I explore below, but was time-boxed and pursued independently because I thought it was interesting!
Executive Summary
Figure 1. Accuracy vs. similarity threshold (0.95+) across 1700 pairs of encoding/decoding examples across a variety of datatypes and lengths. The accuracy is the proportion of the 3400 examples each model translated successfully (directly, with no reasoning or tools). Success for each task is defined by the normalised Levenshtein similarity of the answer/target pair hitting a given threshold, with a scoring requirement that model-encoded strings are decodable. Legend ordered by [email protected].- Introducing Base64Bench: a simple new benchmark for evaluating models on their ability to encode and decode base64.
- Base64 encoding and decoding are reasonably complex computational tasks to do perfectly [...]
---
Outline:
(00:31) Executive Summary
(03:07) An accidental (and surprising) discovery
(08:03) Have LLMs actually learned the algorithm?
(09:39) Introducing
(13:11) Accuracy vs. similarity threshold
(16:02) Encoding vs. decoding by model
(17:00) Task-level breakdown
(19:37) Why should we care?
(21:26) Monitoring implications
(23:51) Conclusion
(25:23) Appendix
(25:26) Zoomed-in threshold sweeps
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
October 5th, 2025
Source:
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64bench-how-good-are-llms-at-base64-and-why-care-about
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,319 Listeners
Conversations with Tyler
2,452 Listeners
The Peter Attia Drive
8,529 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,176 Listeners
ManifoldOne
93 Listeners
Your Undivided Attention
1,601 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,920 Listeners
Machine Learning Street Talk (MLST)
95 Listeners
Dwarkesh Podcast
514 Listeners
Hard Fork
5,509 Listeners
The Ezra Klein Show
15,918 Listeners
Moonshots with Peter Diamandis
548 Listeners
No Priors: Artificial Intelligence | Technology | Startups
131 Listeners
Latent Space: The AI Engineer Podcast
93 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
466 Listeners