Machine Learning Street Talk (MLST)

This week on Machine Learning Street Talk, Dr. Tim Scarfe, Dr. Keith Duggar, Alex Stenlake and Yannic Kilcher have a conversation with the founder and principal researcher at the Montreal AI Ethics institute -- Abhishek Gupta. We cover several topics from the Social Dilemma film and AI Ethics in general. 

00:00:00 Introduction

00:03:57 Overcome our weaknesses

00:14:30 threat landscape blind spots  

00:18:35 differential reality vs universal shaping  

00:24:21 shared reality incentives and tools  

00:32:01 transparency and knowledge to avoid pathology  

00:40:09 federated informed autonomy    

00:49:48 diversity is a metric, inclusion is a strategy  

00:59:58 locally aligned pockets can stabilize global diversity 

01:10:58 making inclusion easier with tools 

01:23:35 enabling community feedback  

01:26:16 open source the algorithms  

01:33:02 the N+1 cost of inclusion  

01:38:08 broader impact statement

https://atg-abhishek.github.io

https://www.linkedin.com/in/abhishekguptamcgill/

In this first part of our three part series on the Social Dilemma Netflix film, Dr. Tim Scarfe, Yannic "Lightspeed" Kilcher and Zak Jost gang up with Cybersecurity expert Andy Smith. We give you our take on the film. We are super excited to get your feedback on this one! Hope you enjoy. 

00:00:00 Introduction

00:06:11 Moral hypocrisy  

00:12:38 Road to hell is paved with good intentions, attention economy

00:15:04 They know everything about you

00:18:02 Addiction

00:21:22 Differential realities

00:26:12 Self determination and Monetisation

00:29:08 AI: Overwhelm human strengths undermine human vulnerabilities

00:31:51 Conspiracy theory / fake news

00:34:23 Overton window / polarisation

00:39:12 Short attention span / convergent behaviour

00:41:26 Is social media good for you

00:45:17 Your attention time is linear, the things you can pay attention to are a volume, anonymity 

00:51:32 Andy question on security: social engineering

00:56:32 Is it a security risk having your information in social media

00:58:02 Retrospective judgement

01:03:06 Free speech and censorship 

01:06:06 Technology accelerator

In today's episode, Dr. Keith Duggar, Alex Stenlake and Dr. Tim Scarfe chat about the education chapter in Kenneth Stanley's "Greatness cannot be planned" book, and we relate it to our Algoshambes conversation a few weeks ago. We debate whether objectives in education are a good thing and whether they cause perverse incentives and stifle creativity and innovation. Next up we dissect capsule networks from the top down! We finish off talking about fast algorithms and quantum computing.

00:00:00 Introduction

00:01:13 Greatness cannot be planned / education 

00:12:03 Perverse incentives

00:19:25 Treasure hunting 

00:30:28 Capsule Networks

00:46:08 Capsules As Compositional Networks

00:52:45 Capsule Routing

00:57:10 Loss and Warps

01:09:55 Fast Algorithms and Quantum Computing

This week Dr. Tim Scarfe, Dr. Keith Duggar, Yannic "Lightspeed" Kilcher have a conversation with Microsoft Senior Software Engineer Sachin Kundu. We speak about programming languages including which our favourites are and functional programming vs OOP. Next we speak about software engineering and the intersection of software engineering and machine learning. We also talk about applications of ML and finally what makes an exceptional software engineer and tech lead. Sachin is an expert in this field so we hope you enjoy the conversation!

Spoiler alert, how many of you have read the Mythical Man-Month by Frederick P. Brooks?!  

00:00:00 Introduction

00:06:37 Programming Languages

00:53:41 Applications of ML

01:55:59 What makes an exceptional SE and tech lead

01:22:08 Outro 

This week Dr. Keith Duggar, Alex Stenlake and Dr. Tim Scarfe discuss the theory of computation, intelligence, Bayesian model selection, the intelligence explosion and the the phenomenon of "interactive articles". 

00:00:00 Intro

00:01:27 Kernels and context-free grammars

00:06:04 Theory of computation

00:18:41 Intelligence

00:22:03 Bayesian model selection

00:44:05 AI-IQ Measure / Intelligence explosion

00:52:09 Interactive articles

01:12:32 Outro

Today Yannic Lightspeed Kilcher and I spoke with Alex Stenlake about Kernel Methods. What is a kernel? Do you remember those weird kernel things which everyone obsessed about before deep learning? What about Representer theorem and reproducible kernel hilbert spaces? SVMs and kernel ridge regression? Remember them?! Hope you enjoy the conversation!

00:00:00 Tim Intro

00:01:35 Yannic clever insight from this discussion 

00:03:25 Street talk and Alex intro 

00:05:06 How kernels are taught

00:09:20 Computational tractability

00:10:32 Maths 

00:11:50 What is a kernel? 

00:19:39 Kernel latent expansion 

00:23:57 Overfitting 

00:24:50 Hilbert spaces 

00:30:20 Compare to DL

00:31:18 Back to hilbert spaces

00:45:19 Computational tractability 2

00:52:23 Curse of dimensionality

00:55:01 RBF: infinite taylor series

00:57:20 Margin/SVM 

01:00:07 KRR/dual

01:03:26 Complexity compute kernels vs deep learning

01:05:03 Good for small problems? vs deep learning)

01:07:50 Whats special about the RBF kernel

01:11:06 Another DL comparison

01:14:01 Representer theorem

01:20:05 Relation to back prop

01:25:10 Connection with NLP/transformers

01:27:31 Where else kernels good

01:34:34 Deep learning vs dual kernel methods

01:33:29 Thoughts on AI

01:34:35 Outro

This week Dr. Tim Scarfe and Dr. Keith Duggar discuss Explainability, Reasoning, Priors and GPT-3. We check out Christoph Molnar's book on intepretability, talk about priors vs experience in NNs, whether NNs are reasoning and also cover articles by Gary Marcus and Walid Saba critiquing deep learning. We finish with a brief discussion of Chollet's ARC challenge and intelligence paper. 

00:00:00 Intro

00:01:17 Explainability and Christoph Molnars book on Intepretability

00:26:45 Explainability - Feature visualisation

00:33:28 Architecture / CPPNs

00:36:10 Invariance and data parsimony, priors and experience, manifolds

00:42:04 What NNs learn / logical view of modern AI (Walid Saba article)

00:47:10 Core knowledge

00:55:33 Priors vs experience 

00:59:44 Mathematical reasoning 

01:01:56 Gary Marcus on GPT-3 

01:09:14 Can NNs reason at all? 

01:18:05 Chollet intelligence paper/ARC challenge

This week Dr. Tim Scarfe, Yannic Lightspeed Kicher, Sayak Paul and Ayush Takur interview Mathilde Caron from Facebook Research (FAIR).

We discuss Mathilde's paper which she wrote with her collaborators "SWaV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments" @ https://arxiv.org/pdf/2006.09882.pdf 

This paper is the latest unsupervised contrastive visual representations algorithm and has a new data augmentation strategy and also a new online clustering strategy. 

Note; Other authors; Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

Sayak Paul -  @RisingSayak / https://www.linkedin.com/in/sayak-paul/

Ayush Thakur - @ayushthakur0

 / https://www.linkedin.com/in/ayush-thakur-731914149/

The article they wrote;

https://app.wandb.ai/authors/swav-tf/reports/Unsupervised-Visual-Representation-Learning-with-SwAV--VmlldzoyMjg3Mzg

00:00:00 Yannic probability challenge (CAN YOU SOLVE IT?)

00:01:29 Intro topic (Tim)

00:08:18 Yannic take

00:09:33 Intro show and guests

00:11:29 SWaV elevator pitch 

00:17:31 Clustering approach in general

00:21:17 Sayak and Ayush's article on SWaV 

00:23:49 Optional transport problem / Sinkhorn-Knopp algorithm

00:31:43 Is clustering a natural approach for this?

00:44:19 Image augmentations 

00:46:20 Priors vs experience (data)

00:48:32 Life at FAIR 

00:52:33 Progress of image augmentation 

00:56:10 When things do not go to plan with research

01:01:04 Question on architecture

01:01:43 SWaV Results

01:06:26 Reproducing Matilde's code

01:14:51 Do we need the whole dataset to set clustering loss

01:16:40 Self-supervised learning and transfer learning

01:23:25 Link to attention mechanism)

01:24:41 Sayak final thought why unsupervised better

01:25:56 Outro

Abstract; 

"Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a “swapped” prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks."

This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic "Lightspeed" Kilcher respond to the "Algoshambles" exam fiasco in the UK where the government were forced to step in to standardise the grades which were grossly inflated by the schools.  The schools and teachers are all paid on metrics related to the grades received by students, what could possibly go wrong?! The result is that we end up with grades which have lost all their value and students are coached for the exams and don't actually learn the subject.   We also cover the second Francois Chollet interview on the Lex Fridman podcast. We cover GPT-3, Neuralink, and discussion of intelligence.

00:00:00 Algoshambles 

00:45:40 Lex Fridman/Chollet: Intro 

00:55:21 Lex Fridman/Chollet: Neuralink 

01:06:28 Lex Fridman/Chollet: GPT-3 

01:23:43 Lex Fridman/Chollet: Intelligence discussion

This week we spoke with Sayak Paul, who is extremely active in the machine learning community. We discussed the AI landscape in India, unsupervised representation learning, data augmentation and contrastive learning, explainability, abstract scene representations and finally pruning and the recent super positions paper. I really enjoyed this conversation and I hope you folks do too!

00:00:00 Intro to Sayak

00:17:50 AI landscape in India

00:24:20 Unsupervised representation learning

00:26:11 DATA AUGMENTATION/Contrastive learning

00:59:20 EXPLAINABILITY

01:12:10 ABSTRACT SCENE REPRESENTATIONS

01:14:50 PRUNING and super position paper