Jeff and Seymour mention audio generation technology which leads to a larger conversation about how we can tell if audio, text, images, and other media is a Deepfake. They end with questions about media gatekeepers, social polarization, and how do we know what's real?

Links:

• Vim is Jeff's favorite text editor and 10 reasons you should learn it.
• Andrej Karpthy's YouTube video on building a transformer from scratch.
• Simon Willison on AI empowering him to build more software side projects.
• Open Letter for 6-month moratorium on generative AI development.
• Italian regulators use GDPR rules to ban use of ChatGPT in Italy.
• Daring Fireball "It's Game Over For Deep Fakes" which links toJohn H. Meyer's impressive Steve Jobs chatbot video.
• Google/DeepMind Wavenet and related 2016 paper.
• Deepfake of Pope Francis in trendy white puffer coat. 
• Deepfake of Donald Trump on his way to jail. At time of recording (3/30/2023), Trump has been indicted but not yet been arrested or arraigned, so these images are faked.
• Good article that covers earlier fake/misleading photographs. Includes: (1) Alexander Gardner's famous 1863 Gettysburg photo of a dead Confederate soldier and (2) Robert Capa's 1936 photo of a Spanish Republican soldier mid-fall in apparent battle from the Spanish Civil war.

Send questions/comments to [email protected] and find us on the web at www.stepfunction.org

The news that Gordon Moore passed away broke last Friday just as we were recording the previous episode. Today, Jeff and Seymour discuss Moore's legacy, the past and future of Moore's Law, and how advancements in microchips have been the foundation of deep learning for more than a decade. Links:

• NY Times obituary for Gordon Moore and a remembrance by Walden Kirsch at Intel.
• Apple's latest iPhone has an A16 Bionic system-on-a-chip with 16 billion transistors, compared to 1993's then ground-breaking Intel Pentium which had 3 million transistors.
• Articles on Moore's Law and Dennard Scaling
  • Excellent piece by The Computer History Museum in Mountain View, CA.
  • 2018 piece by Micron on Moore's Law and Dennard Scaling (aka MOSFET scaling).
  • 2007 piece in IEEE on 30 years of Dennard/MOSFET scaling (PDF).
• Types of chips
  • AI Accelerators and Deep Learning Processors (DLP).
  • CPUs, FPGAs, and ASICs.
  • Google's Tensor Processing Units (TPUs).
• Accleration of AI progress above Moore's Law
  • Seymour refers to the chart in this 2021 IEEE Spectrum article.
  • 2022 article in Discover magazine by the Physics arXiv Blog.

Send questions/comments to [email protected] and find us on the web at www.stepfunction.org.

Jeff and Seymour debate the significance of the generative AI revolution relative to the invention of the WIMP interface, the web, and even the Industrial and Agricultural Revolutions.  Prometheus refers to the humans learning how to control fire; is gen AI bigger than that?! Articles referenced:

• Bill Gates "The Age of AI Has Begun" 
• Thomas Friedman "Our New Promethean Moment" (NY Times)
• Rodney Brooks on GPT and related LLM technology 

Send questions/comments to [email protected] and find us on the web at www.stepfunction.org

Jeff and Seymour start with another major week of news, including GPT-4, Microsoft and Google adding generative AI to their office suites, and LLaMA breaking out into the wild and onto Raspbery Pi and smartphones.

• Simon Willison's 3-day history of LLaMA developments this past weekend. Potentially bigger news than the dramatic collapse of Silicon Valley Bank.
• Two relevant articles from Elad Gil: Startups vs Incumbents and Why weren't there any big startup winners from the last 10 years of deep learning?
• Coca-Cola commercial set in an art museum using Stable Diffusion
• PCs disrupting mainframes in the 1980s in the context of Clay Christiensen's Innovator's Dilemma and theory of disruptive vs sustaining innovation
• Runway ML applying Stable Diffusion to generative AI for video 

Questions and feedback? Let us know.

Jeff and Seymour dig further into image generators like Midjourney, DALL-E, and Stable Diffusion. What is the role of the visual artist as generative AI tools continue to evolve? How might it replicate the relationship between a great writer and a trusted magazine editor? Additional links:

• Adam Gopnik's piece on DALL-E 2 in a recent issue of The New Yorker
• The Girl with The Pearl Earring by Johannes Vermeer and related movie and novel
• The Last Supper by Leonardo da Vinci
• Prediction Machines by Ajay Agrawal, Joshua Gans, Avi Goldfarb at the University of Toronto. Quote about introduction of cheap electric light from page 19

Questions and comments? Talk to us.

Jeff and Seymour discuss the launch of a new marketing company using generative AI called Typeface. This leads to a conversation about copyright consent, compensation, and credit. We discuss how the history of photography, Napster, and Paul Cézanne's The Card Players might help us understand the future of image generators like DALL-E, Midjourney, and Stable Diffusion.

Questions and comments? Talk to us.

Jeff and Seymour use stories and analogies to explain the two main approaches to AI: Bottom-Up and Top-Down. The recent wave of AI success is mostly based on the bottom up  path which includes machine learning, neural networks, and deep learning. Related links:

• The case of the construction worker with a nail in his boot
• Murray Shanahan's excellent December 2022 paper Talking About Large Language Models
• Meta / Facebook AI research's September 2020 blog post on Retrieval Augmented Generation and Segmentation
• DeepMind / Google AI's December 2021 paper on using a Retrieval-Enhanced Transformer (aka Retro) and a database of 2-trillion tokens for improved LLM capabilities. 

Questions and comments? Talk to us.

Jeff and Seymour discuss the unexpected impact of ChatGPT and how Bing Chat may not be ready for prime time. Did OpenAI unintentionally open Pandora's Box because they were worried someone else would beat them to it? Plus some reassurances that Sydney is definitely not sentient or emotional.

• Kevin Roose's article in The New York Times and transcript of his long chat session with Bing / Sydney.
• Podcast episode detailing the behind the scenes at OpenAI in the weeks leading up to the launch of ChatGPT in November, 2022 as discussed by Roose and Casey Newton on their show Hard Fork.

Questions and comments? Talk to us.

Jeff and Seymour kick off the podcast with an exploration of ChatGPT. What is it and how might it impact our careers and lives? They use ChatGPT and large language models (LLMs) as an entry point to the larger topic of generative AI.

Questions and comments? Talk to us.