WeirdML website

Related posts:

How good are LLMs at doing ML on an unknown dataset?

o1-preview is pretty good at doing ML on an unknown dataset

Introduction

How good are Large Language Models (LLMs) at doing machine learning on novel datasets? The WeirdML benchmark presents LLMs with weird and unusual machine learning tasks, designed to require careful thinking and actual understanding to solve, and tests an LLM's ability to:

1. Actually understand the properties of the data and the problem
2. Come up with an appropriate ML architecture and training setup for the problem, and generate working PyTorch code that implements the solution
3. Debug and improve the solution over 5 iterations based on terminal output and the accuracy on the test set
4. Make good use of limited computational resources and time

Each task comes with a task prompt describing the problem precisely and some example code for loading data and [...]

---

Outline:

(00:18) Introduction

(01:24) Results

(01:47) Evaluation Setup

(02:28) System Architecture

(03:26) Tasks

(04:04) Shapes (Easy)

(05:53) Shapes (Hard)

(07:42) Image Patch Shuffling (Easy)

(09:46) Image Patch Shuffling (Hard)

(12:17) Chess Game Outcome Prediction

(14:28) Unsupervised Digit Recognition

(15:59) Further Analysis

(16:21) Failure Rate

(17:11) Model Performance by Number of Iterations

(18:13) Maximum of k First Submissions (max@k)

(20:11) Future Directions

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.