Machine Learning Street Talk (MLST)

In this episode of Machine Learning Street Talk, we chat with Jonathan Frankle, author of The Lottery Ticket Hypothesis. Frankle has continued researching Sparse Neural Networks, Pruning, and Lottery Tickets leading to some really exciting follow-on papers! This chat discusses some of these papers such as Linear Mode Connectivity, Comparing and Rewinding and Fine-tuning in Neural Network Pruning, and more (full list of papers linked below). We also chat about how Jonathan got into Deep Learning research, his Information Diet, and work on developing Technology Policy for Artificial Intelligence! 

This was a really fun chat, I hope you enjoy listening to it and learn something from it!

Thanks for watching and please subscribe!

Huge thanks to everyone on r/MachineLearning who asked questions!

Paper Links discussed in the chat:

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks: https://arxiv.org/abs/1803.03635

Linear Mode Connectivity and the Lottery Ticket Hypothesis: https://arxiv.org/abs/1912.05671

Dissecting Pruned Neural Networks: https://arxiv.org/abs/1907.00262

Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs: https://arxiv.org/abs/2003.00152

What is the State of Neural Network Pruning? https://arxiv.org/abs/2003.03033

The Early Phase of Neural Network Training: https://arxiv.org/abs/2002.10365

Comparing Rewinding and Fine-tuning in Neural Network Pruning: https://arxiv.org/abs/2003.02389

(Also Mentioned)

Block-Sparse GPU Kernels: https://openai.com/blog/block-sparse-gpu-kernels/

Balanced Sparsity for Efficient DNN Inference on GPU: https://arxiv.org/pdf/1811.00206.pdf

Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLP: https://arxiv.org/pdf/1906.02768.pdf

r/MachineLearning question list: https://www.reddit.com/r/MachineLearning/comments/g9jqe0/d_lottery_ticket_hypothesis_ask_the_author_a/ (edited) 

#machinelearning #deeplearning

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten chat about Large-scale Transfer Learning in Natural Language Processing. The Text-to-Text Transfer Transformer (T5) model from Google AI does an exhaustive survey of what’s important for Transfer Learning in NLP and what’s not. In this conversation, we go through the key takeaways of the paper, text-to-text input/output format, architecture choice, dataset size and composition, fine-tuning strategy, and how to best use more computation.

Beginning with these topics, we diverge into exciting ideas such as embodied cognition, meta-learning, and the measure of intelligence. We are still beginning our podcast journey and really appreciate any feedback from our listeners. Is the chat too technical? Do you prefer group discussions, interviewing experts, or chats between the three of us? Thanks for watching and if you haven’t already, Please Subscribe!

Paper Links discussed in the chat:

Text-to-Text Transfer Transformer: https://arxiv.org/abs/1910.10683

Experience Grounds Language (relevant to divergent discussion about embodied cognition): https://arxiv.org/pdf/2004.10151.pdf

On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Train Large, Then Compress: https://arxiv.org/pdf/2002.11794.pdf

Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361.pdf

The Illustrated Transformer: http://jalammar.github.io/illustrated...

ELECTRA: https://arxiv.org/pdf/2003.10555.pdf

Transformer-XL: https://arxiv.org/pdf/1901.02860.pdf

Reformer: The Efficient Transformer: https://openreview.net/pdf?id=rkgNKkHtvB

The Evolved Transformer: https://arxiv.org/pdf/1901.11117.pdf

DistilBERT: https://arxiv.org/pdf/1910.01108.pdf

How to generate text (HIGHLY RECOMMEND): https://huggingface.co/blog/how-to-ge...

Tokenizers: https://blog.floydhub.com/tokenization-nlp/

According to Yann Le Cun, the next big thing in machine learning is unsupervised learning. Self-supervision has changed the entire game in the last few years in deep learning, first transforming the language world with word2vec and BERT -- but now it's turning computer vision upside down. 

This week Yannic, Connor and I spoke with one of the authors, Aravind Srinivas who recently co-led the hot-off-the-press CURL: Contrastive Unsupervised Representations for Reinforcement Learning alongside Michael (Misha) Laskin. CURL has had an incredible reception in the ML community in the last month or so. Remember the Deep Mind paper which solved the Atari games using the raw pixels? Aravind's approach uses contrastive unsupervised learning to featurise the pixels before applying RL. CURL is the first image-based algorithm to nearly match the sample-efficiency and performance of methods that use state-based features! This is a huge step forwards in being able to apply RL in the real world. 

We explore RL and self-supervision for computer vision in detail and find out about how Aravind got into machine learning. 

Original YouTube Video: https://youtu.be/1MprzvYNpY8

Paper:

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Aravind Srinivas, Michael Laskin, Pieter Abbeel

https://arxiv.org/pdf/2004.04136.pdf

Yannic's analysis video: https://www.youtube.com/watch?v=hg2Q_O5b9w4 

#machinelearning #reinforcementlearning #curl #timscarfe #yannickilcher #connorshorten

Music credit; https://soundcloud.com/errxrmusic/in-my-mind

Three YouTubers; Tim Scarfe - Machine Learning Dojo (https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA), Connor Shorten - Henry AI Labs (https://www.youtube.com/channel/UCHB9VepY6kYvZjj0Bgxnpbw) and Yannic Kilcher (https://www.youtube.com/channel/UCZHmQk67mSJgfCCTn7xBfew). We made a new YouTube channel called Machine Learning Street Talk. Every week we will talk about the latest and greatest in AI. Subscribe now!

Special guests this week; Dr. Mathew Salvaris (https://www.linkedin.com/in/drmathewsalvaris/), Eric Craeymeersch (https://www.linkedin.com/in/ericcraeymeersch/), Dr. Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/),  Dmitri Soshnikov (https://www.linkedin.com/in/shwars/)

We discuss the new concept of an open-ended, or "AI-Generating" algorithm. Open-endedness is a class of algorithms which generate problems and solutions to increasingly complex and diverse tasks. These algorithms create their own curriculum of learning. Complex tasks become tractable because they are now the final stepping stone in a lineage of progressions. In many respects, it's better to trust the machine to develop the learning curriculum, because the best curriculum might be counter-intuitive. These algorithms can generate a radiating tree of evolving challenges and solutions just like natural evolution. Evolution has produced an eternity of diversity and complexity and even produced human intelligence as a side-effect! Could AI-generating algorithms be the next big thing in machine learning?

Wang, Rui, et al. "Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions." arXiv preprint arXiv:2003.08536 (2020). https://arxiv.org/abs/2003.08536

Wang, Rui, et al. "Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions." arXiv preprint arXiv:1901.01753 (2019). https://arxiv.org/abs/1901.01753

Watch Yannic’s video on POET: https://www.youtube.com/watch?v=8wkgDnNxiVs

#reinforcementlearning #machinelearning #uber #deeplearning #rl #timscarfe #connorshorten #yannickilcher