Astral Codex Ten Podcast

Gwern's AI-Generated Poetry


Listen Later

Gwern has answered my prayers and taught GPT-2 poetry.

GPT-2 is the language processing system that OpenAI announced a few weeks ago. They are keeping the full version secret, but have released a smaller prototype version. Gwern retrained it on the Gutenberg Poetry Corpus, a 117 MB collection of pre-1923 English poetry, to create a specialized poetry AI.

I previously tested the out-of-the-box version of GPT-2 and couldn’t make it understand rhyme and meter. I wrongly assumed this was a fundamental limitation: “obviously something that has never heard sound can’t derive these complex rhythms just from meaningless strings of letters.” I was wrong; it just didn’t have enough training data. Gwern’s retrained version gets both of these right, and more too. For example:

Thou know’st how Menoetiades the swift Was dragged, of Hector and the fierce compeers And Phrygian warriors. So, we will dispatch Your bodies, then, yourselves to burn the ships In sacrifice; with torches and with bells To burn them, and with oxen to replace Your gallant friends for ever. But I wish That no man living has so long endured The onset of his foes, as I have power To burn or storm; for mighty Hector erst Was slain, and now returns his safe return

This is all perfect iambic pentameter. I know AP English students who can’t write iambic pentameter as competently as this.

(by the way, both “compeers” and “erst” are perfectly cromulent words from the period when people wrote poems like this; both show up in Shelley)

It has more trouble with rhymes – my guess is a lot of the poetry it was trained on was blank verse. But when it decides it should be rhyming, it can keep it up for a little while. From its Elegy Written in a Country Churchyardfanfic:

...more
View all episodesView all episodes
Download on the App Store

Astral Codex Ten PodcastBy Jeremiah

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

123 ratings


More shows like Astral Codex Ten Podcast

View all
EconTalk by Russ Roberts

EconTalk

4,233 Listeners

Robert Wright's Nonzero by Nonzero

Robert Wright's Nonzero

584 Listeners

Conversations with Tyler by Mercatus Center at George Mason University

Conversations with Tyler

2,395 Listeners

Odd Lots by Bloomberg

Odd Lots

1,789 Listeners

Future of Life Institute Podcast by Future of Life Institute

Future of Life Institute Podcast

105 Listeners

ChinaTalk by Jordan Schneider

ChinaTalk

269 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

89 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

88 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

426 Listeners

Clearer Thinking with Spencer Greenberg by Spencer Greenberg

Clearer Thinking with Spencer Greenberg

128 Listeners

Joe Lonsdale: American Optimist by Joe Lonsdale

Joe Lonsdale: American Optimist

164 Listeners

"Moment of Zen" by Erik Torenberg, Dan Romero, Antonio Garcia Martinez

"Moment of Zen"

91 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

75 Listeners

"Econ 102" with Noah Smith and Erik Torenberg by Turpentine

"Econ 102" with Noah Smith and Erik Torenberg

146 Listeners

Complex Systems with Patrick McKenzie (patio11) by Patrick McKenzie

Complex Systems with Patrick McKenzie (patio11)

123 Listeners