Connor Leahy has a wide-ranging chat with Jim about the state & future of Deep Learning...

Connor Leahy has a wide-ranging chat with Jim about the state & future of Deep Learning. They cover the history of EleutherAI, how GPT-3 works, the dynamics & power of scaling laws, ideal sampling rates & sizes for models, data sets, EleutherAI's opensource GTP-Neo & GTP-NeoX, PyTorch vs TensorFlow, TPU's vs GPU's, the challenge of benchmarking & evaluations, quadradic bottlenecks, broad GTP-3 applications, Connors thoughts on Jim's proposed GPT-3 research project, untapped GPT-3 potential, OpenAI's move away from opensource, alignment, AI safety, the unknown future, and much more.

Episode Transcript

Connor on Twitter

Jared Kaplan

The Pile

CoreWeave

Connor Leahy is an AI Researcher at German startup Aleph Alpha and founding member of the loose AI research collective EleutherAI. Eleuther is best known for their ongoing efforts to produce a full open source GPT-3 sized language model. Connor currently researches large, general purpose language models and how to align them to human values.