Machine Learning Street Talk (MLST)

SWaV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (Mathilde Caron)


Listen Later

This week Dr. Tim Scarfe, Yannic Lightspeed Kicher, Sayak Paul and Ayush Takur interview Mathilde Caron from Facebook Research (FAIR).

We discuss Mathilde's paper which she wrote with her collaborators "SWaV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments" @ https://arxiv.org/pdf/2006.09882.pdf 

This paper is the latest unsupervised contrastive visual representations algorithm and has a new data augmentation strategy and also a new online clustering strategy. 

Note; Other authors; Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

Sayak Paul -  @RisingSayak / https://www.linkedin.com/in/sayak-paul/

Ayush Thakur - @ayushthakur0

 / https://www.linkedin.com/in/ayush-thakur-731914149/

The article they wrote;

https://app.wandb.ai/authors/swav-tf/reports/Unsupervised-Visual-Representation-Learning-with-SwAV--VmlldzoyMjg3Mzg


00:00:00 Yannic probability challenge (CAN YOU SOLVE IT?)

00:01:29 Intro topic (Tim)

00:08:18 Yannic take

00:09:33 Intro show and guests

00:11:29 SWaV elevator pitch 

00:17:31 Clustering approach in general

00:21:17 Sayak and Ayush's article on SWaV 

00:23:49 Optional transport problem / Sinkhorn-Knopp algorithm

00:31:43 Is clustering a natural approach for this?

00:44:19 Image augmentations 

00:46:20 Priors vs experience (data)

00:48:32 Life at FAIR 

00:52:33 Progress of image augmentation 

00:56:10 When things do not go to plan with research

01:01:04 Question on architecture

01:01:43 SWaV Results

01:06:26 Reproducing Matilde's code

01:14:51 Do we need the whole dataset to set clustering loss

01:16:40 Self-supervised learning and transfer learning

01:23:25 Link to attention mechanism)

01:24:41 Sayak final thought why unsupervised better

01:25:56 Outro


Abstract; 

"Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a “swapped” prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks."

...more
View all episodesView all episodes
Download on the App Store

Machine Learning Street Talk (MLST)By Machine Learning Street Talk (MLST)

  • 4.7
  • 4.7
  • 4.7
  • 4.7
  • 4.7

4.7

84 ratings


More shows like Machine Learning Street Talk (MLST)

View all
Data Skeptic by Kyle Polich

Data Skeptic

480 Listeners

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

441 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

298 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

322 Listeners

Machine Learning Guide by OCDevel

Machine Learning Guide

763 Listeners

Practical AI by Practical AI LLC

Practical AI

192 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

87 Listeners

Google DeepMind: The Podcast by Hannah Fry

Google DeepMind: The Podcast

198 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

389 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

121 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

201 Listeners

Unsupervised Learning by by Redpoint Ventures

Unsupervised Learning

39 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

75 Listeners

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

462 Listeners

Training Data by Sequoia Capital

Training Data

43 Listeners