Machine Learning Street Talk (MLST)

YT version: https://youtu.be/DxBZORM9F-8

Patreon: https://www.patreon.com/mlst 

Discord: https://discord.gg/ESrGqhf5CB

Prof. Ken Stanley argued in his book that our world has become saturated with objectives. The process of setting an objective, attempting to achieve it, and measuring progress along the way has become the primary route to achievement in our culture. He’s not saying that objectives are bad per se, especially if they’re modest, but he thinks that when goals are ambitious then the search space becomes deceptive.

Is the key to artificial intelligence really related to intelligence? Does taking a job with a higher salary really bring you closer to being a millionaire? The problem is that the stepping stones which lead to ambitious objectives tend to be pretty strange, they don't resemble the final end state at all. Vaccum tubes led to computers for example and Youtube started as a dating website. 

What fascinated us about this conversation with Ken is that we got a much deeper understanding of his philosophy. He lead by saying that he thought it's worth questioning whether artificial intelligence is even a science or not. Ken thinks that the secret to future progress is for us to embrace more subjectivity. 

[00:00:00] Tim Intro

[00:12:54] Intro

[00:17:08] Seeing ideas everywhere - AI and art are highly connected

[00:28:40] Creativity in Mathematics

[00:30:14] Where is the intelligence in art?

[00:38:49] Is AI disappointingly simple to mechanise?

[00:42:48] Slightly conscious

[00:46:27] Do we have subjective experience?

[00:50:23] Fear of the unknown

[00:51:48] Free Will

[00:54:22] Chalmers

[00:55:08] What's happening now in open-endedness

[00:58:31] Generalisation

[01:06:34] Representation primitives and what it means to understand

[01:12:37] Appeal to definitions, knowledge itself blocks discovery

Make sure you buy Kenneth's book!

Why Greatness Cannot Be Planned: The Myth of the Objective [Stanley, Lehman]

https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

Abandoning Objectives: Evolution through the

Search for Novelty Alone [Lehman, Stanley]

https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehman_ecj11.pdf

Twitter

https://twitter.com/kenneth0stanley

Special discount link for Zak's GNN course - https://bit.ly/3uqmYVq

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/ESrGqhf5CB

YT version: https://youtu.be/jAGIuobLp60 (there are lots of helper graphics there, recommended if poss)

Want to sponsor MLST!? Let us know on Linkedin / Twitter. 

[00:00:00] Preamble

[00:03:12] Geometric deep learning

[00:10:04] Message passing

[00:20:42] Top down vs bottom up

[00:24:59] All NN architectures are different forms of information diffusion processes (squashing and smoothing problem)

[00:29:51] Graph rewiring

[00:31:38] Back to information diffusion 

[00:42:43] Transformers vs GNNs

[00:47:10] Equivariant subgraph aggregation networks + WL test

[00:55:36] Do equivariant layers aggregate too?

[00:57:49] Zak's GNN course

Exhaustive list of references on the YT show URL (https://youtu.be/jAGIuobLp60)

Today we are having a discussion with Letitia Parcalabescu from the AI Coffee Break youtube channel! We discuss linguistics, symbolic AI and our respective Youtube channels. Make sure you subscribe to her channel! In the first 15 minutes Tim dissects the recent article from Gary Marcus "Deep learning has hit a wall". 

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/ESrGqhf5CB

YT: https://youtu.be/p2D2duT-R2E

[00:00:00] Comments on Gary Marcus Article / Symbolic AI

[00:14:57] Greetings

[00:17:40] Introduction

[00:18:48] A shared journey towards computation

[00:22:10] A linguistics outsider

[00:24:11] Is computational linguistics AI?

[00:28:23] swinging pendulums of dogma and resource allocation

[00:31:16] the road less travelled

[00:34:35] pitching grants with multimodality ... and then the truth

[00:40:50] some aspects of language are statistically learnable

[00:44:58] ... and some aspects of language are dimensionally cursed

[00:48:24] it's good to have both approaches to machine intelligence

[00:51:14] the world runs on symbols

[00:54:28] there is much more to learn biology

[00:59:26] Letitia's creation process

[01:02:23] don't overfit content, instead publish and iterate

[01:07:48] merging the big picture arrow from the small direction arrows

[01:11:02] use passion to drive through failure to success

[01:12:56] stay positive

[01:16:02] closing remarks

Today we are speaking with Dr. Thomas Lux, a research scientist at Meta in Silicon Valley. 

In some sense, all of supervised machine learning can be framed through the lens of geometry. All training data exists as points in euclidean space, and we want to predict the value of a function at all those points. Neural networks appear to be the modus operandi these days for many domains of prediction. In that light; we might ask ourselves — what makes neural networks better than classical techniques like K nearest neighbour from a geometric perspective. Our guest today has done research on exactly that problem, trying to define error bounds for approximations in terms of directions, distances, and derivatives.  

The insights from Thomas's work point at why neural networks are so good at problems which everything else fails at, like image recognition. The key is in their ability to ignore parts of the input space, do nonlinear dimension reduction, and concentrate their approximation power on important parts of the function. 

[00:00:00] Intro to Show

[00:04:11] Intro to Thomas (Main show kick off)

[00:04:56] Interpolation of Sparse High-Dimensional Data

[00:12:19] Where does one place the basis functions to partition the space, the perennial question

[00:16:20] The sampling phenomenon -- where did all those dimensions come from?

[00:17:40] The placement of the MLP basis functions, they are not where you think they are

[00:23:15] NNs only extrapolate when given explicit priors to do so, CNNs in the translation domain

[00:25:31] Transformers extrapolate in the permutation domain

[00:28:26] NN priors work by creating space junk everywhere

[00:36:44] Are vector spaces the way to go? On discrete problems

[00:40:23] Activation functioms

[00:45:57] What can we prove about NNs? Gradients without backprop

Interpolation of Sparse High-Dimensional Data [Lux]

https://tchlux.github.io/papers/tchlux-2020-NUMA.pdf

A Spline Theory of Deep Learning [_Balestriero_]

https://proceedings.mlr.press/v80/balestriero18b.html

Gradients without Backpropagation ‘22

https://arxiv.org/pdf/2202.08587.pdf

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/HNnAwSduud

YT version: https://youtu.be/pMtk-iUaEuQ

Dr. Walid Saba is an old-school polymath. He has a background in cognitive  psychology, linguistics, philosophy, computer science and logic and he’s is now a Senior Scientist at Sorcero.

Walid is perhaps the most outspoken critic of BERTOLOGY, which is to say trying to solve the problem of natural language understanding with application of large statistical language models. Walid thinks this approach is cursed to failure because it’s analogous to memorising infinity with a large hashtable. Walid thinks that the various appeals to infinity by some deep learning researchers are risible.

[00:00:00] MLST Housekeeping

[00:08:03] Dr. Walid Saba Intro

[00:11:56] AI Cannot Ignore Symbolic Logic, and Here’s Why

[00:23:39] Main show - Proposition: Statistical learning doesn't work

[01:04:44] Discovering a sorting algorithm bottom-up is hard

[01:17:36] The axioms of nature (universal cognitive templates)

[01:31:06] MLPs are locality sensitive hashing tables

References;

The Missing Text Phenomenon, Again: the case of Compound Nominals

https://ontologik.medium.com/the-missing-text-phenomenon-again-the-case-of-compound-nominals-abb6ece3e205

A Spline Theory of Deep Networks

https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf

The Defeat of the Winograd Schema Challenge

https://arxiv.org/pdf/2201.02387.pdf

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

https://twitter.com/yasaman_razeghi/status/1495112604854882304?s=21

https://arxiv.org/abs/2202.07206

AI Cannot Ignore Symbolic Logic, and Here’s Why

https://medium.com/ontologik/ai-cannot-ignore-symbolic-logic-and-heres-why-1f896713525b

Learnability can be undecidable

http://gtts.ehu.es/German/Docencia/1819/AC/extras/s42256-018-0002-3.pdf

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

https://arxiv.org/pdf/2112.11446.pdf

DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning

https://arxiv.org/abs/2006.08381

On the Measure of Intelligence [Chollet]

https://arxiv.org/abs/1911.01547

A Formal Theory of Commonsense Psychology: How People Think People Think

https://www.amazon.co.uk/Formal-Theory-Commonsense-Psychology-People/dp/1107151007

Continuum hypothesis

https://en.wikipedia.org/wiki/Continuum_hypothesis

Gödel numbering + completness theorems

https://en.wikipedia.org/wiki/G%C3%B6del_numbering

https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems

Concepts: Where Cognitive Science Went Wrong [Jerry A. Fodor]

https://oxford.universitypressscholarship.com/view/10.1093/0198236360.001.0001/acprof-9780198236368

We engage in a bit of epistemic foraging with Prof. Karl Friston! In this show; we discuss the free energy principle in detail, also emergence, cognition, consciousness and Karl's burden of knowledge!

YT: https://youtu.be/xKQ-F2-o8uM

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/HNnAwSduud

[00:00:00] Introduction to FEP/Friston

[00:06:53] Cheers to Epistemic Foraging!

[00:09:17] The Burden of Knowledge Across Disciplines

[00:12:55] On-show introduction to Friston

[00:14:23] Simple does NOT mean Easy

[00:21:25] Searching for a Mathematics of Cognition

[00:26:44] The Low Road and The High Road to the Principle

[00:28:27] What's changed for the FEP in the last year

[00:39:36] FEP as stochastic systems with a pullback attractor

[00:44:03] An attracting set at multiple time scales and time infinity

[00:53:56] What about fuzzy Markov boundaries?

[00:59:17] Is reality densely or sparsely coupled?

[01:07:00] Is a Strong and Weak Emergence distinction useful?

[01:13:25] a Philosopher, a Zombie, and a Sentient Consciousness walk into a bar ... 

[01:24:28] Can we recreate consciousness in silico? Will it have qualia?

[01:28:29] Subjectivity and building hypotheses

[01:34:17] Subject specific realizations to minimize free energy

[01:37:21] Free will in a deterministic Universe

The free energy principle made simpler but not too simple

https://arxiv.org/abs/2201.06387

We have a chat with Alexander Mattick aka ZickZack from Yannic's Discord community. Alex is one of the leading voices in that community and has an impressive technical depth. Don't forget MLST has now started it's own Discord server too, come and join us! We are going to run regular events, our first big event on Wednesday 9th 1700-1900 UK time. 

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/HNnAwSduud

YT version: https://youtu.be/rGOOLC8cIO4

[00:00:00] Introduction to Alex 

[00:02:16] Spline theory of NNs 

[00:05:19] Do NNs abstract? 

[00:08:27] Tim's exposition of spline theory of NNs

[00:11:11] Semantics in NNs 

[00:13:37] Continuous vs discrete 

[00:19:00] Open-ended Search

[00:22:54] Inductive logic programming

[00:25:00] Control to gain knowledge and knowledge to gain control

[00:30:22] Being a generalist with a breadth of knowledge and knowledge transfer

[00:36:29] Causality

[00:43:14] Discrete program synthesis + theorem solvers

Note: there are no politics discussed in this show and please do not interpret this show as any kind of a political statement from us.  We have decided not to discuss politics on MLST anymore due to its divisive nature. 

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/HNnAwSduud

[00:00:00] Intro

[00:01:36] What we all need to understand about machine learning

[00:06:05] The Master Algorithm Target Audience

[00:09:50] Deeply Connected Algorithms seen from Divergent Frames of Reference

[00:12:49] There is a Master Algorithm; and it's mine!

[00:14:59] The Tribe of Evolution

[00:17:17] Biological Inspirations and Predictive Coding

[00:22:09] Shoe-Horning Gradient Descent

[00:27:12] Sparsity at Training Time vs Prediction Time

[00:30:00] World Models and Predictive Coding

[00:33:24] The Cartoons of System 1 and System 2

[00:40:37] AlphaGo Searching vs Learning

[00:45:56] Discriminative Models evolve into Generative Models

[00:50:36] Generative Models, Predictive Coding, GFlowNets

[00:55:50] Sympathy for a Thousand Brains

[00:59:05] A Spectrum of Tribes

[01:04:29] Causal Structure and Modelling

[01:09:39] Entropy and The Duality of Past vs Future, Knowledge vs Control

[01:16:14] A Discrete Universe?

[01:19:49] And yet continuous models work so well

[01:23:31] Finding a Discretised Theory of Everything

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/HNnAwSduud

YT: https://www.youtube.com/watch?v=ZDY2nhkPZxw

We have a chat with Prof. Gary Marcus about everything which is currently top of mind for him, consciousness 

[00:00:00] Gary intro

[00:01:25] Slightly conscious

[00:24:59] Abstract, compositional models

[00:32:46] Spline theory of NNs

[00:36:17] Self driving cars / algebraic reasoning 

[00:39:43] Extrapolation

[00:44:15] Scaling laws

[00:49:50] Maximum likelihood estimation

References:

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

https://arxiv.org/abs/2201.02177

DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT

https://arxiv.org/pdf/1912.02292.pdf

Bayesian Deep Learning and a Probabilistic Perspective of Generalization

https://arxiv.org/pdf/2002.08791.pdf

We are now sponsored by Weights and Biases! Please visit our sponsor link: http://wandb.me/MLST

Patreon: https://www.patreon.com/mlst

For Yoshua Bengio, GFlowNets are the most exciting thing on the horizon of Machine Learning today. He believes they can solve previously intractable problems and hold the key to unlocking machine abstract reasoning itself. This discussion explores the promise of GFlowNets and the personal journey Prof. Bengio traveled to reach them.

Panel:

Dr. Tim Scarfe

Dr. Keith Duggar

Dr. Yannic Kilcher

Our special thanks to: 

- Alexander Mattick (Zickzack)

References:

Yoshua Bengio @ MILA (https://mila.quebec/en/person/bengio-yoshua/)

GFlowNet Foundations (https://arxiv.org/pdf/2111.09266.pdf)

Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (https://arxiv.org/pdf/2106.04399.pdf)

Interpolation Consistency Training for Semi-Supervised Learning (https://arxiv.org/pdf/1903.03825.pdf)

Towards Causal Representation Learning (https://arxiv.org/pdf/2102.11107.pdf)

Causal inference using invariant prediction: identification and confidence intervals (https://arxiv.org/pdf/1501.01332.pdf)