Machine Learning Street Talk (MLST)

NLP is not NLU and GPT-3 - Walid Saba


Listen Later

#machinelearning

This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic Kilcher speak with veteran NLU expert Dr. Walid Saba. 

Walid is an old-school AI expert. He is a polymath, a neuroscientist, psychologist, linguist,  philosopher, statistician, and logician. He thinks the missing information problem and lack of a typed ontology is the key issue with NLU, not sample efficiency or generalisation. He is a big critic of the deep learning movement and BERTology. We also cover GPT-3 in some detail in today's session, covering Luciano Floridi's recent article "GPT‑3: Its Nature, Scope, Limits, and Consequences" and a commentary on the incredible power of GPT-3 to perform tasks with just a few examples including the Yann LeCun commentary on Facebook and Hackernews. 

Time stamps on the YouTube version

0:00:00 Walid intro 

00:05:03 Knowledge acquisition bottleneck 

00:06:11 Language is ambiguous 

00:07:41 Language is not learned 

00:08:32 Language is a formal language 

00:08:55 Learning from data doesn’t work  

00:14:01 Intelligence 

00:15:07 Lack of domain knowledge these days 

00:16:37 Yannic Kilcher thuglife comment 

00:17:57 Deep learning assault 

00:20:07 The way we evaluate language models is flawed 

00:20:47 Humans do type checking 

00:23:02 Ontologic 

00:25:48 Comments On GPT3 

00:30:54 Yann lecun and reddit 

00:33:57 Minds and machines - Luciano 

00:35:55 Main show introduction 

00:39:02 Walid introduces himself 

00:40:20 science advances one funeral at a time 

00:44:58 Deep learning obsession syndrome and inception 

00:46:14 BERTology / empirical methods are not NLU 

00:49:55 Pattern recognition vs domain reasoning, is the knowledge in the data 

00:56:04 Natural language understanding is about decoding and not compression, it's not learnable. 

01:01:46 Intelligence is about not needing infinite amounts of time 

01:04:23 We need an explicit ontological structure to understand anything 

01:06:40 Ontological concepts 

01:09:38 Word embeddings 

01:12:20 There is power in structure 

01:15:16 Language models are not trained on pronoun disambiguation and resolving scopes 

01:17:33 The information is not in the data 

01:19:03 Can we generate these rules on the fly? Rules or data? 

01:20:39 The missing data problem is key 

01:21:19 Problem with empirical methods and lecunn reference 

01:22:45 Comparison with meatspace (brains) 

01:28:16 The knowledge graph game, is knowledge constructed or discovered 

01:29:41 How small can this ontology of the world be? 

01:33:08 Walids taxonomy of understanding 

01:38:49 The trend seems to be, less rules is better not the othe way around? 

01:40:30 Testing the latest NLP models with entailment 

01:42:25 Problems with the way we evaluate NLP 

01:44:10 Winograd Schema challenge 

01:45:56 All you need to know now is how to build neural networks, lack of rigour in ML research 

01:50:47 Is everything learnable 

01:53:02  How should we elevate language systems? 

01:54:04 10 big problems in language (missing information) 

01:55:59 Multiple inheritance is wrong 

01:58:19 Language is ambiguous 

02:01:14 How big would our world ontology need to be? 

02:05:49 How to learn more about NLU 

02:09:10 AlphaGo 


Walid's blog: https://medium.com/@ontologik

LinkedIn: https://www.linkedin.com/in/walidsaba/

...more
View all episodesView all episodes
Download on the App Store

Machine Learning Street Talk (MLST)By Machine Learning Street Talk (MLST)

  • 4.7
  • 4.7
  • 4.7
  • 4.7
  • 4.7

4.7

84 ratings


More shows like Machine Learning Street Talk (MLST)

View all
Data Skeptic by Kyle Polich

Data Skeptic

481 Listeners

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

440 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

298 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

323 Listeners

Machine Learning Guide by OCDevel

Machine Learning Guide

765 Listeners

Practical AI by Practical AI LLC

Practical AI

189 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

87 Listeners

Google DeepMind: The Podcast by Hannah Fry

Google DeepMind: The Podcast

199 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

372 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

122 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

199 Listeners

Unsupervised Learning by by Redpoint Ventures

Unsupervised Learning

40 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

76 Listeners

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

441 Listeners

Training Data by Sequoia Capital

Training Data

36 Listeners