Word error rate has a formula that fits on a napkin, yet it's the number that ends up in every headline about speech recognition. This episode digs into the mechanics: how edit distance at the word level gets computed, why WER has no upper bound, and when character error rate is the more honest lens. Then it goes after everything a single score buries — normalization choices that swing results by forty points, a Whisper normalizer that strips vowel signs out of Indic scripts, inverse text normalization for numbers, named entities, code-switching, punctuation, and speaker attribution. Along the way: why WER agrees with human judgment less than any other metric tested, and what semantic metrics are doing about it.
Key sources:
• HuggingFace Audio Transformers Course, ch. 5 Evaluation metrics for ASR — https://github.com/huggingface/audio-transformers-course/blob/main/chapters/en/chapter5/evaluation.mdx
• Wikipedia, Word error rate — https://en.wikipedia.org/wiki/Word_error_rate
• TorchMetrics, WordErrorRate docs — https://torchmetrics.readthedocs.io/en/v1.1.2/text/word_error_rate.html
Episode 5962 — myweirdprompts.com/n/5962 (permanent short link: myweirdprompts.com/398196)