Sign up to save your podcasts
Or
Share “Base64Bench: How good are LLMs at base64, and why care about it?” by richbc
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Base64Bench: How good are LLMs at base64, and why care about it?” by richbc
This was a quick, short side-project produced during the MATS Research 8.1 extension. It's related to my group's main thread of work on black-box scheming monitoring through the connections to monitoring I explore below, but was time-boxed and pursued independently because I thought it was interesting!
Executive Summary
Figure 1. Accuracy vs. similarity threshold (0.95+) across 1700 pairs of encoding/decoding examples across a variety of datatypes and lengths. The accuracy is the proportion of the 3400 examples each model translated successfully (directly, with no reasoning or tools). Success for each task is defined by the normalised Levenshtein similarity of the answer/target pair hitting a given threshold, with a scoring requirement that model-encoded strings are decodable. Legend ordered by [email protected].- Introducing Base64Bench: a simple new benchmark for evaluating models on their ability to encode and decode base64.
- Base64 encoding and decoding are reasonably complex computational tasks to do perfectly [...]
---
Outline:
(00:31) Executive Summary
(03:07) An accidental (and surprising) discovery
(08:03) Have LLMs actually learned the algorithm?
(09:39) Introducing
(13:11) Accuracy vs. similarity threshold
(16:02) Encoding vs. decoding by model
(17:00) Task-level breakdown
(19:37) Why should we care?
(21:26) Monitoring implications
(23:51) Conclusion
(25:23) Appendix
(25:26) Zoomed-in threshold sweeps
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
October 5th, 2025
Source:
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64bench-how-good-are-llms-at-base64-and-why-care-about
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
View all episodes
By LessWrongThis was a quick, short side-project produced during the MATS Research 8.1 extension. It's related to my group's main thread of work on black-box scheming monitoring through the connections to monitoring I explore below, but was time-boxed and pursued independently because I thought it was interesting!
Executive Summary
Figure 1. Accuracy vs. similarity threshold (0.95+) across 1700 pairs of encoding/decoding examples across a variety of datatypes and lengths. The accuracy is the proportion of the 3400 examples each model translated successfully (directly, with no reasoning or tools). Success for each task is defined by the normalised Levenshtein similarity of the answer/target pair hitting a given threshold, with a scoring requirement that model-encoded strings are decodable. Legend ordered by [email protected].- Introducing Base64Bench: a simple new benchmark for evaluating models on their ability to encode and decode base64.
- Base64 encoding and decoding are reasonably complex computational tasks to do perfectly [...]
---
Outline:
(00:31) Executive Summary
(03:07) An accidental (and surprising) discovery
(08:03) Have LLMs actually learned the algorithm?
(09:39) Introducing
(13:11) Accuracy vs. similarity threshold
(16:02) Encoding vs. decoding by model
(17:00) Task-level breakdown
(19:37) Why should we care?
(21:26) Monitoring implications
(23:51) Conclusion
(25:23) Appendix
(25:26) Zoomed-in threshold sweeps
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
October 5th, 2025
Source:
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64bench-how-good-are-llms-at-base64-and-why-care-about
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,376 Listeners
Conversations with Tyler
2,429 Listeners
The Peter Attia Drive
8,187 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,155 Listeners
ManifoldOne
92 Listeners
Your Undivided Attention
1,553 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,799 Listeners
Machine Learning Street Talk (MLST)
89 Listeners
Dwarkesh Podcast
488 Listeners
Hard Fork
5,472 Listeners
The Ezra Klein Show
16,144 Listeners
Moonshots with Peter Diamandis
531 Listeners
No Priors: Artificial Intelligence | Technology | Startups
131 Listeners
Latent Space: The AI Engineer Podcast
96 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
510 Listeners