Sign up to save your podcasts
Or
Share Episode 40: What Every LLM Developer Needs to Know About GPUs
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 40: What Every LLM Developer Needs to Know About GPUs
Hugo speaks with Charles Frye, Developer Advocate at Modal and someone who really knows GPUs inside and out. If you’re a data scientist, machine learning engineer, AI researcher, or just someone trying to make sense of hardware for LLMs and AI workflows, this episode is for you.
Charles and Hugo dive into the practical side of GPUs—from running inference on large models, to fine-tuning and even training from scratch. They unpack the real pain points developers face, like figuring out:
- How much VRAM you actually need.
- Why memory—not compute—ends up being the bottleneck.
- How to make quick, back-of-the-envelope calculations to size up hardware for your tasks.
- And where things like fine-tuning, quantization, and retrieval-augmented generation (RAG) fit into the mix.
One thing Hugo really appreciate is that Charles and the Modal team recently put together the GPU Glossary—a resource that breaks down GPU internals in a way that’s actually useful for developers. We reference it a few times throughout the episode, so check it out in the show notes below.
🔧 Charles also does a demo during the episode—some of it is visual, but we talk through the key points so you’ll still get value from the audio. If you’d like to see the demo in action, check out the livestream linked below.
This is the "Building LLM Applications for Data Scientists and Software Engineers" course that Hugo is teaching with Stefan Krawczyk (ex-StitchFix) in January (https://maven.com/s/course/d56067f338). Charles is giving a guest lecture at on hardware for LLMs, and Modal is giving all students $1K worth of compute credits (use the code VG25 for $200 off).
LINKS
The livestream on YouTube (https://www.youtube.com/live/INryb8Hjk3c?si=0cbb0-Nxem1P987d)
The GPU Glossary (https://modal.com/gpu-glossary) by the Modal team
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) by Charles and friends
Charles on twitter (https://x.com/charles_irl)
Hugo on twitter (https://x.com/hugobowne)
Vanishing Gradients on twitter (https://x.com/vanishingdata)
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hugobowne.substack.com
Charles and Hugo dive into the practical side of GPUs—from running inference on large models, to fine-tuning and even training from scratch. They unpack the real pain points developers face, like figuring out:
- How much VRAM you actually need.
- Why memory—not compute—ends up being the bottleneck.
- How to make quick, back-of-the-envelope calculations to size up hardware for your tasks.
- And where things like fine-tuning, quantization, and retrieval-augmented generation (RAG) fit into the mix.
One thing Hugo really appreciate is that Charles and the Modal team recently put together the GPU Glossary—a resource that breaks down GPU internals in a way that’s actually useful for developers. We reference it a few times throughout the episode, so check it out in the show notes below.
🔧 Charles also does a demo during the episode—some of it is visual, but we talk through the key points so you’ll still get value from the audio. If you’d like to see the demo in action, check out the livestream linked below.
This is the "Building LLM Applications for Data Scientists and Software Engineers" course that Hugo is teaching with Stefan Krawczyk (ex-StitchFix) in January (https://maven.com/s/course/d56067f338). Charles is giving a guest lecture at on hardware for LLMs, and Modal is giving all students $1K worth of compute credits (use the code VG25 for $200 off).
LINKS
The livestream on YouTube (https://www.youtube.com/live/INryb8Hjk3c?si=0cbb0-Nxem1P987d)
The GPU Glossary (https://modal.com/gpu-glossary) by the Modal team
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) by Charles and friends
Charles on twitter (https://x.com/charles_irl)
Hugo on twitter (https://x.com/hugobowne)
Vanishing Gradients on twitter (https://x.com/vanishingdata)
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hugobowne.substack.com
View all episodes
By Hugo Bowne-Anderson
5
1111 ratings
Hugo speaks with Charles Frye, Developer Advocate at Modal and someone who really knows GPUs inside and out. If you’re a data scientist, machine learning engineer, AI researcher, or just someone trying to make sense of hardware for LLMs and AI workflows, this episode is for you.
Charles and Hugo dive into the practical side of GPUs—from running inference on large models, to fine-tuning and even training from scratch. They unpack the real pain points developers face, like figuring out:
- How much VRAM you actually need.
- Why memory—not compute—ends up being the bottleneck.
- How to make quick, back-of-the-envelope calculations to size up hardware for your tasks.
- And where things like fine-tuning, quantization, and retrieval-augmented generation (RAG) fit into the mix.
One thing Hugo really appreciate is that Charles and the Modal team recently put together the GPU Glossary—a resource that breaks down GPU internals in a way that’s actually useful for developers. We reference it a few times throughout the episode, so check it out in the show notes below.
🔧 Charles also does a demo during the episode—some of it is visual, but we talk through the key points so you’ll still get value from the audio. If you’d like to see the demo in action, check out the livestream linked below.
This is the "Building LLM Applications for Data Scientists and Software Engineers" course that Hugo is teaching with Stefan Krawczyk (ex-StitchFix) in January (https://maven.com/s/course/d56067f338). Charles is giving a guest lecture at on hardware for LLMs, and Modal is giving all students $1K worth of compute credits (use the code VG25 for $200 off).
LINKS
The livestream on YouTube (https://www.youtube.com/live/INryb8Hjk3c?si=0cbb0-Nxem1P987d)
The GPU Glossary (https://modal.com/gpu-glossary) by the Modal team
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) by Charles and friends
Charles on twitter (https://x.com/charles_irl)
Hugo on twitter (https://x.com/hugobowne)
Vanishing Gradients on twitter (https://x.com/vanishingdata)
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hugobowne.substack.com
Charles and Hugo dive into the practical side of GPUs—from running inference on large models, to fine-tuning and even training from scratch. They unpack the real pain points developers face, like figuring out:
- How much VRAM you actually need.
- Why memory—not compute—ends up being the bottleneck.
- How to make quick, back-of-the-envelope calculations to size up hardware for your tasks.
- And where things like fine-tuning, quantization, and retrieval-augmented generation (RAG) fit into the mix.
One thing Hugo really appreciate is that Charles and the Modal team recently put together the GPU Glossary—a resource that breaks down GPU internals in a way that’s actually useful for developers. We reference it a few times throughout the episode, so check it out in the show notes below.
🔧 Charles also does a demo during the episode—some of it is visual, but we talk through the key points so you’ll still get value from the audio. If you’d like to see the demo in action, check out the livestream linked below.
This is the "Building LLM Applications for Data Scientists and Software Engineers" course that Hugo is teaching with Stefan Krawczyk (ex-StitchFix) in January (https://maven.com/s/course/d56067f338). Charles is giving a guest lecture at on hardware for LLMs, and Modal is giving all students $1K worth of compute credits (use the code VG25 for $200 off).
LINKS
The livestream on YouTube (https://www.youtube.com/live/INryb8Hjk3c?si=0cbb0-Nxem1P987d)
The GPU Glossary (https://modal.com/gpu-glossary) by the Modal team
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) by Charles and friends
Charles on twitter (https://x.com/charles_irl)
Hugo on twitter (https://x.com/hugobowne)
Vanishing Gradients on twitter (https://x.com/vanishingdata)
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hugobowne.substack.com
More shows like Vanishing Gradients
View all
The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
538 Listeners
Making Sense with Sam Harris
26,332 Listeners
The a16z Show
1,087 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
302 Listeners
Founders
2,169 Listeners
The Prof G Pod with Scott Galloway
5,545 Listeners
Machine Learning Street Talk (MLST)
95 Listeners
Dwarkesh Podcast
501 Listeners
No Priors: Artificial Intelligence | Technology | Startups
131 Listeners
AI + a16z
35 Listeners
Prof G Markets
1,430 Listeners
Lightcone Podcast
21 Listeners
Training Data
39 Listeners
High Signal: Data Science | Career | AI
19 Listeners
Cheeky Pint
48 Listeners