Little Pi's Tech Music Tips

Distillation is a hot word in the language model world, and music generation AI uses it too, but the same word means two different jobs. It comes from a 2015 paper by Geoffrey Hinton and colleagues: a big teacher model guides a small student, giving not just the right answer but soft labels like seventy percent violin, twenty percent viola, using the temperature from episode 41 to make the hints clearer. Language model distillation turns big into small: Hugging Face's DistilBERT (2019) is 40% smaller, 60% faster and keeps about 97% of BERT's language understanding; DeepSeek-R1 (2025) used about 800,000 worked examples to train distilled Qwen and Llama models from 1.5B to 70B. Music diffusion distillation turns many steps into few: progressive distillation (Tim Salimans and Jonathan Ho, Google, 2022) has the student do in one step what the teacher does in two, halving each round from 8192 steps down to 4; then came consistency distillation; and Presto! (Adobe and UC San Diego, 2024) distills both steps and layers, 10 to 18 times faster, 32 seconds of mono music in 0.23 seconds. Finally, a correction: episode 46 said Stable Audio Open Small relies on distillation, but it mainly uses adversarial post-training (ARC), which its authors stress is not distillation.

References:

Distilling the Knowledge in a Neural Network (Hinton et al. 2015): https://arxiv.org/abs/1503.02531

DistilBERT (Sanh et al. 2019): https://arxiv.org/abs/1910.01108

DeepSeek-R1: https://github.com/deepseek-ai/DeepSeek-R1

Progressive Distillation (Salimans & Ho 2022): https://arxiv.org/abs/2202.00512

Consistency Models (Song et al. 2023): https://arxiv.org/abs/2303.01469

Presto! (Novack et al. 2024): https://arxiv.org/abs/2410.05167

ARC / Stable Audio Open Small (2025): https://arxiv.org/abs/2505.08175

Episode 38 said diffusion models wipe noise away step by step until music appears. But how many steps, and how much noise per step? Little Pi explains the noise schedule and sampling steps. The noise schedule is the timetable for adding noise during training, like slowly watering down a glass of juice. In 2020 DDPM used a 1000-step linear schedule, but it adds noise too fast, so a long stretch near the end is pure noise and wasted work. In 2021 OpenAI's Alex Nichol and Prafulla Dhariwal proposed the cosine schedule, slowing down at both ends so every step counts. Generating with 1000 steps is too slow, so Stanford's DDIM (2020) lets the same model take big strides, 10 to 50 times faster in the paper's experiments. Too few steps leave grainy hiss, too many are slow with no audible gain; tools like Stable Audio Open usually use a few dozen to one or two hundred steps, with a choice of samplers. Finally: distillation, Consistency Models (2023), and Stable Audio Open Small (2025, Stability AI and Arm), officially generating 11 seconds of audio on a phone in under 8 seconds.

References:

DDPM (Ho et al. 2020): https://arxiv.org/abs/2006.11239

Improved DDPM (Nichol & Dhariwal 2021): https://arxiv.org/abs/2102.09672

DDIM (Song, Meng & Ermon 2020): https://arxiv.org/abs/2010.02502

Consistency Models (Song et al. 2023): https://arxiv.org/abs/2303.01469

Stable Audio Open Small: https://stability.ai/news/stability-ai-and-arm-release-stable-audio-open-small-enabling-real-world-deployment-for-on-device-audio-control

Episode 38 said Stable Audio Open does diffusion inside a compressed space. Where is that space? Little Pi introduces the latent space and the autoencoder: an encoder squeezes sound down, a decoder rebuilds it, and the narrow bottleneck in the middle only fits the essentials, like packing clothes in vacuum storage bags. The numbers in the bottleneck are like map coordinates, and similar sounds end up close together: flute and oboe are neighbors, drums live in another neighborhood, and the model organizes this by itself. In 2017 Google Magenta's NSynth was trained on about 300,000 single notes; taking the point halfway between a flute and an organ yields a brand-new tone color, later built into the NSynth Super hardware synth. In 2018 MusicVAE morphed gradually between melodies. The 2013 Variational Autoencoder (VAE) turns each sound into a small cloud, so the map has no holes. Stable Audio Open uses a VAE to squeeze 44.1 kHz stereo (88,200 numbers per second) into 21.5 frames per second of 64 numbers each, about 64 times smaller, then runs diffusion on that small map and decodes it back: latent diffusion. Finally, EnCodec (RVQ codes after the bottleneck, for token prediction) versus Stable Audio (continuous coordinates, for diffusion).

References:

Auto-Encoding Variational Bayes (Kingma & Welling 2013): https://arxiv.org/abs/1312.6114

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders (NSynth): https://arxiv.org/abs/1704.01279

MusicVAE (Roberts et al. 2018): https://arxiv.org/abs/1803.05428

High-Resolution Image Synthesis with Latent Diffusion Models: https://arxiv.org/abs/2112.10752

Stable Audio Open: https://arxiv.org/abs/2407.14358

When AI writes bar nine, how does it know to look back at bar one? Little Pi unpacks the attention mechanism, mentioned briefly in episode 37. Older models worked like a game of telephone, passing information along step by step and forgetting more the farther it went. Attention lets every step look straight back at all earlier steps and give each a score (adding up to 100%): the higher the score, the more attention. How are scores decided? Each step prepares a Query, a Key and a Value; the query is matched against every key, and the better the match, the more of that value gets read, like checking book spines in a library before deciding what to open. Multi-head attention runs several sets at once, each learning its own focus. Music looking at itself is self-attention; looking at text is cross-attention: MusicGen turns the prompt into numbers with T5, then glances back at it with cross-attention. Attention was first used for machine translation in 2014; the 2017 paper Attention Is All You Need gave birth to the Transformer; and in 2018 Google Magenta's Music Transformer wrote minute-long piano music with themes that answer each other. The price: double the steps, four times the comparisons.

References:

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho & Bengio 2014): https://arxiv.org/abs/1409.0473

Attention Is All You Need (Vaswani et al. 2017): https://arxiv.org/abs/1706.03762

Music Transformer (Huang et al. 2018): https://arxiv.org/abs/1809.04281

Simple and Controllable Music Generation (MusicGen paper): https://arxiv.org/abs/2306.05284

Why does AI use several codebooks to turn sound into tokens? Little Pi unpacks RVQ, Residual Vector Quantization, mentioned briefly in episode 36. The encoder cuts sound into short slices, turns each into a vector, and looks up the closest entry in a codebook. One codebook is either too rough or too huge: in the SoundStream paper's example, at 3 kbps and 100 slices per second, a single codebook would need about a billion entries, while 5 codebooks of 64 entries need only 320. In RVQ each codebook only fixes what the previous one left over (the residual), like paying with hundreds, then tens, then ones. EnCodec uses 75 slices per second and 1024 entries per codebook: 2 codebooks = 1.5 kbps, 32 = 24 kbps; MusicGen uses 4 codebooks at 50 slices per second. SoundStream's quantizer dropout randomly uses only the first few codebooks during training, so one model supports multiple quality levels and the first codebook carries the most important information. Multi-stage vector quantization goes back to Juang and Gray's 1982 speech coding work.

References:

SoundStream: An End-to-End Neural Audio Codec: https://arxiv.org/abs/2107.03312

SoundStream - Google Research blog: https://research.google/blog/soundstream-an-end-to-end-neural-audio-codec/

High Fidelity Neural Audio Compression (EnCodec): https://arxiv.org/abs/2210.13438

Simple and Controllable Music Generation (MusicGen paper): https://arxiv.org/abs/2306.05284

You type "jazz saxophone" and get a piano piece? Little Pi introduces CFG, Classifier-Free Guidance, the knob that makes AI follow your prompt. The older classifier guidance (Dhariwal and Nichol, 2021) needed a separate judge model. In 2021, Jonathan Ho and Tim Salimans had the same model calculate twice at every step, once with the prompt and once without, then amplify the difference (unconditional + scale x (conditional - unconditional)). During training the prompt is randomly dropped: MusicGen drops it 20% of the time, so one model does both jobs. How high to set it: MusicGen defaults to 3 (cfg_coef), and Stable Audio Open's official example uses 7. Too high leaves only one sound and can distort, and calculating twice roughly doubles generation time.

References:

Classifier-Free Diffusion Guidance (Ho & Salimans): https://arxiv.org/abs/2207.12598

Simple and Controllable Music Generation (MusicGen paper): https://arxiv.org/abs/2306.05284

Diffusion Models Beat GANs on Image Synthesis (classifier guidance): https://arxiv.org/abs/2105.05233

Stable Audio Open 1.0: https://huggingface.co/stabilityai/stable-audio-open-1.0

Same prompt, two different songs? Little Pi takes apart the dice behind AI music generation. At every step, an autoregressive model gives a list where every candidate has a probability (each MusicGen codebook has 1,024 candidates). Always picking number one (greedy decoding) is bland and repetitive, so models sample instead. Temperature divides the scores before turning them into probabilities: low is safe, high is risky. Top-k keeps only the top k candidates (MusicGen's default is 250). Top-p, or nucleus sampling (Holtzman et al., 2019), adds probabilities from the top until it passes p, so the number of candidates stretches and shrinks. MusicGen's defaults: temperature 1, top-k 250, top-p 0 (off).

References:

The Curious Case of Neural Text Degeneration (top-p paper): https://arxiv.org/abs/1904.09751

AudioCraft MusicGen source code: https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/musicgen.py

Softmax function - Wikipedia: https://en.wikipedia.org/wiki/Softmax_function

Can melodies evolve? Little Pi introduces the genetic algorithm, which John Holland turned into a computer method in 1975 based on Darwin's survival of the fittest. Start with a random crowd of melodies, where every note is a gene, and score each one (its fitness). Pick high scorers as parents, cut and join their halves (crossover), and once in a while randomly change a note (mutation). Repeat generation after generation, and good-sounding features pile up. Al Biles's GenJam (1994) evolved jazz solos with a human judge and even traded phrases with him on stage, but ran into the fitness bottleneck. DarwinTunes (2012) let 6,931 listeners rate music loops over 2,513 generations, and chords and rhythms emerged from noise. Plus a secret: Teacher Lin Na-Wei's work Salamander Quintet uses a genetic algorithm too.

References:

Genetic algorithm - Wikipedia: https://en.wikipedia.org/wiki/Genetic_algorithm

GenJam (Al Biles): https://genjam.org/

Evolution of music by public choice (DarwinTunes, PNAS 2012): https://pmc.ncbi.nlm.nih.gov/articles/PMC3409751

Salamander Quintet: https://zoelin.me/salamander/

How should four codebooks take turns? Little Pi takes a deep dive into MusicGen's delay pattern. EnCodec produces 50 frames per second, and with RVQ each frame has four codebooks, so every second has 200 tokens to guess. Pattern one, flattening, guesses one code at a time: most accurate but slowest, 2,000 steps for 10 seconds. Pattern two, parallel, guesses all four codes of a frame at once: only 500 steps, but the layers can't see each other, like coloring before the sketch is done. Pattern three, the delay pattern, starts each codebook one step later, like singing a canon, so finer codebooks can follow the coarse sketch, and 10 seconds takes only 503 steps. The 2023 MusicGen paper compared these patterns: the delay pattern came close to flattening in quality while running almost four times faster, so MusicGen chose it.

References:

Simple and Controllable Music Generation (MusicGen) - arXiv: https://arxiv.org/abs/2306.05284

AudioCraft / MusicGen - GitHub: https://github.com/facebookresearch/audiocraft

High Fidelity Neural Audio Compression (EnCodec) - arXiv: https://arxiv.org/abs/2210.13438

Music can be carved out of noise! Little Pi introduces diffusion models: during training, music gets a little more noise at each step, like ink spreading in water, and the AI learns to guess the noise added at each step. To generate, it runs in reverse, starting from pure noise and subtracting noise step by step, guided by a text description. The 2020 DDPM paper made this method a big hit. Riffusion (late 2022) used the image model Stable Diffusion to draw spectrograms, showing that sound can be drawn. Stable Audio Open (2024) denoises in a compressed latent space, using T5 for text and a Transformer for denoising, generating up to 47 seconds of 44.1 kHz stereo audio, trained on about 480,000 Creative Commons licensed recordings from Freesound and the Free Music Archive.

References:

Denoising Diffusion Probabilistic Models (DDPM) - arXiv: https://arxiv.org/abs/2006.11239

Diffusion model - Wikipedia: https://en.wikipedia.org/wiki/Diffusion_model

Riffusion - Wikipedia: https://en.wikipedia.org/wiki/Riffusion

Stable Audio Open 1.0 - Hugging Face: https://huggingface.co/stabilityai/stable-audio-open-1.0

Stable Audio Open paper - arXiv: https://arxiv.org/abs/2407.14358