Machine Learning Street Talk (MLST)

We speak with Robert Lange!

Robert is a PhD student at the Technical University Berlin. His research combines Deep Multi-Agent Reinforcement Learning and Cognitive Science to study the learning dynamics of large collectives. He has a brilliant blog where he distils and explains cutting edge ML research. We spoke about his story, economics, multi-agent RL, intelligence and AGI, and his recent article summarising the state of the art in neural network pruning. 

Robert's article on pruning in NNs https://roberttlange.github.io/posts/2020/06/lottery-ticket-hypothesis/

00:00:00 Intro

00:04:17 Show start and intro to Robert

00:11:39 Economics background 

00:27:20 Intrinsic motivation 

00:33:22 Intelligence/consciousness

00:48:16 Lottery ticket/pruning article discussion

01:43:21 Robert's advice for younger self and state of deep learning

Robert's LinkedIn: https://www.linkedin.com/in/robert-tjarko-lange-19539a12a/

@RobertTLange

#machinelearning #deeplearning

We welcome Zak Jost from the WelcomeAIOverlords channel. Zak is an ML research scientist at Amazon. He has a great blog at http://blog.zakjost.com and also a Discord channel at https://discord.gg/xh2chKX

WelcomeAIOverlords: https://www.youtube.com/channel/UCxw9_WYmLqlj5PyXu2AWU_g 

00:00:00 INTRO START

00:01:07 MAIN SHOW START

00:01:59 ZAK'S STORY

00:05:06 YOUTUBE DISCUSSION

00:24:12 UNDERSTANDING PAPERS

00:29:53 CONTRASTIVE LEARNING INTRO

00:33:00 BRING YOUR OWN LATENT PAPER

01:03:13 GRAPHS IN ML AND KNOWLEDGE GRAPHS 

01:21:36 GRAPH USE CASES - FRAUD

01:30:15 KNOWLEDGE GRAPHS

01:34:22 GRAPHS IN ML

01:38:53 AUTOMATED ML

01:57:32 OUTRO

In this episode of Machine Learning Street Talk Dr. Tim Scarfe, Yannic Kilcher and Connor Shorten spoke with Marie-Anne Lachaux, Baptiste Roziere and Dr. Guillaume Lample from Facebook Research (FAIR) in Paris. They recently released the paper "Unsupervised Translation of Programming Languages" which was an exciting new approach to learned translation of programming languages (learned transcoder) using an unsupervised encoder trained on individual monolingual corpora i.e. no parallel language data needed. The trick they used what that there is significant token overlap when using word-piece embeddings. It was incredible to talk with this talented group of researchers and I hope you enjoy the conversation too. 

Yannic's video on this got watched over 120K times! Check it out too https://www.youtube.com/watch?v=xTzFJIknh7E

Paper https://arxiv.org/abs/2006.03511; 

Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample

Abstract;

"A transcompiler, also known as source-to-source translator, is a system that converts source code from a high-level programming language (such as C++ or Python) to another. Transcompilers are primarily used for interoperability, and to port codebases written in an obsolete or deprecated language (e.g. COBOL, Python 2) to a modern one. They typically rely on handcrafted rewrite rules, applied to the source code abstract syntax tree. Unfortunately, the resulting translations often lack readability, fail to respect the target language conventions, and require manual modifications in order to work properly. The overall translation process is timeconsuming and requires expertise in both the source and target languages, making code-translation projects expensive. Although neural models significantly outperform their rule-based counterparts in the context of natural language translation, their applications to transcompilation have been limited due to the scarcity of parallel data in this domain. In this paper, we propose to leverage recent approaches in unsupervised machine translation to train a fully unsupervised neural transcompiler. We train our model on source code from open source GitHub projects, and show that it can translate functions between C++, Java, and Python with high accuracy. Our method relies exclusively on monolingual source code, requires no expertise in the source or target languages, and can easily be generalized to other programming languages. We also build and release a test set composed of 852 parallel functions, along with unit tests to check the correctness of translations. We show that our model outperforms rule-based commercial baselines by a significant margin."

We cover Francois Chollet's recent paper.

Abstract; To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model. With the help of Microsoft’s ZeRO-2 / DeepSpeed optimiser, OpenAI trained an 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning.

00:00:00 Intro

00:00:54 ZeRO1+2 (model + Data parallelism) (Connor)

00:03:17 Recent history of NLP (Tim)

00:06:04 Yannic "Light-speed" Kilcher's brief overview of GPT-3

00:14:25 Reviewing Yannic's YT comments on his GPT-3 video (Tim)

00:20:26 Main show intro

00:23:03 Is GPT-3 reasoning? 

00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)

00:36:18 Utility of GPT-3 in industry

00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)

00:51:03 Generalisation

00:56:48 Esoterics of language models

00:58:46 Architectural trade-offs

01:07:37 Memorization machines and intepretability

01:17:16 Nearest neighbour probes / watermarks

01:20:03 YouTube comments on GPT-3 video 

01:21:50 GPT-3 news article generation issue

01:27:36 Sampling data for language models / bias / fairness / politics

01:51:12 Outro

These paradigms of task adaptation are divided into zero, one, and few shot learning. Zero-shot learning is a very extreme case where we expect a language model to perform a task such as sentiment classification or extractive question answering, without any additional supervision. One and Few-shot learning provide some examples to the model. However, GPT-3s definition of this diverges a bit from the conventional literature. GPT-3 provides one and few-shot examples in the form of “In-Context Learning”. Instead of fine-tuning the model on a few examples, the model has to use the input to infer the downstream task. For example, the GPT-3 transformer has an input sequence of 2048 tokens, so demonstrations of a task such as yelp sentiment reviews, would have to fit in this input sequence as well as the new review.

Thanks for watching! Please Subscribe!

Paper Links:

GPT-3: https://arxiv.org/abs/2005.14165

ZeRO: https://arxiv.org/abs/1910.02054

ZeRO (Blog Post): https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/

ZeRO-2 (Blog Post): https://www.microsoft.com/en-us/research/blog/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale/?OCID=msr_blog_deepspeed2_build_tw

#machinelearning #naturallanguageprocessing #deeplearning #gpt3

This week we had a super insightful conversation with  Jordan Edwards, Principal Program Manager for the AzureML team!  Jordan is on the coalface of turning machine learning software engineering into a reality for some of Microsoft's largest customers. 

ML DevOps is all about increasing the velocity of- and orchastrating the non-interactive phase of- software deployments for ML. We cover ML DevOps and Microsoft Azure ML. We discuss model governance, testing, intepretability, tooling. We cover the age-old discussion of the dichotomy between science and engineering and how you can bridge the gap with ML DevOps. We cover Jordan's maturity model for ML DevOps. 

We also cover off some of the exciting ML announcments from the recent Microsoft Build conference i.e. FairLearn, IntepretML, SEAL, WhiteNoise, OpenAI code generation, OpenAI GPT-3. 

00:00:04 Introduction to ML DevOps and Microsoft Build ML Announcements

00:10:29 Main show kick-off

00:11:06 Jordan's story

00:14:36 Typical ML DevOps workflow

00:17:38 Tim's articulation of ML DevOps

00:19:31 Intepretability / Fairness

00:24:31 Testing / Robustness

00:28:10 Using GANs to generate testing data

00:30:26 Gratuitous DL?

00:33:46 Challenges of making an ML DevOps framework / IaaS

00:38:48 Cultural battles in ML DevOps

00:43:04 Maturity Model for Ml DevOps

00:49:19 ML: High interest credit card of technical debt paper

00:50:19 ML Engineering at Microsoft

01:01:20 ML Flow

01:03:05 Company-wide governance 

01:08:15 What's coming next

01:12:10 Jordan's hillarious piece of advice for his younger self

Super happy with how this turned out, this is not one to miss folks! 

#deeplearning #machinelearning #devops #mldevops

*Note this is an episode from Tim's Machine Learning Dojo YouTube channel. 

Join Eric Craeymeersch on a wonderful discussion all about ML engineering, computer vision, siamese networks, contrastive loss, one shot learning and metric learning. 

00:00:00 Introduction 

00:11:47 ML Engineering Discussion

00:35:59 Intro to the main topic

00:42:13 Siamese Networks

00:48:36 Mining strategies

00:51:15 Contrastive Loss

00:57:44 Trip loss paper

01:09:35 Quad loss paper

01:25:49 Eric's Quadloss Medium Article 

02:17:32 Metric learning reality check

02:21:06 Engineering discussion II

02:26:22 Outro

In our second paper review call, Tess Ferrandez covered off the FaceNet paper from Google which was a one-shot siamese network with the so called triplet loss. It was an interesting change of direction for NN architecture i.e. using a contrastive loss instead of having a fixed number of output classes. Contrastive architectures have been taking over the ML landscape recently i.e. SimCLR, MOCO, BERT. 

Eric wrote an article about this at the time: https://medium.com/@crimy/one-shot-learning-siamese-networks-and-triplet-loss-with-keras-2885ed022352 

He then discovered there was a new approach to one shot learning in vision using a quadruplet loss and metric learning. Eric wrote a new article and several experiments on this @ https://medium.com/@crimy/beyond-triplet-loss-one-shot-learning-experiments-with-quadruplet-loss-16671ed51290?source=friends_link&sk=bf41673664ad8a52e322380f2a456e8b

Paper details: 

Beyond triplet loss: a deep quadruplet network for person re-identification

https://arxiv.org/abs/1704.01719 (Chen at al '17)

"Person re-identification (ReID) is an important task in wide area video surveillance which focuses on identifying people across different cameras. Recently, deep learning networks with a triplet loss become a common framework for person ReID. However, the triplet loss pays main attentions on obtaining correct orders on the training set. It still suffers from a weaker generalization capability from the training set to the testing set, thus resulting in inferior performance. In this paper, we design a quadruplet loss, which can lead to the model output with a larger inter-class variation and a smaller intra-class variation compared to the triplet loss. As a result, our model has a better generalization ability and can achieve a higher performance on the testing set. In particular, a quadruplet deep network using a margin-based online hard negative mining is proposed based on the quadruplet loss for the person ReID. In extensive experiments, the proposed network outperforms most of the state-of-the-art algorithms on representative datasets which clearly demonstrates the effectiveness of our proposed method."

Original facenet paper; 

https://arxiv.org/abs/1503.03832

#deeplearning #machinelearning

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten interviewed Harri Valpola, CEO and Founder of Curious AI. We continued our discussion of System 1 and System 2 thinking in Deep Learning, as well as miscellaneous topics around Model-based Reinforcement Learning. Dr. Valpola describes some of the challenges of modelling industrial control processes such as water sewage filters and paper mills with the use of model-based RL. Dr. Valpola and his collaborators recently published “Regularizing Trajectory Optimization with Denoising Autoencoders” that addresses some of the concerns of planning algorithms that exploit inaccuracies in their world models!

00:00:00 Intro to Harri and Curious AI System1/System 2

00:04:50 Background on model-based RL challenges from Tim

00:06:26 Other interesting research papers on model-based RL from Connor

00:08:36 Intro to Curious AI recent NeurIPS paper on model-based RL and denoising autoencoders from Yannic

00:21:00 Main show kick off, system 1/2

00:31:50 Where does the simulator come from?

00:33:59 Evolutionary priors

00:37:17 Consciousness

00:40:37 How does one build a company like Curious AI?

00:46:42 Deep Q Networks

00:49:04 Planning and Model based RL

00:53:04 Learning good representations

00:55:55 Typical problem Curious AI might solve in industry

01:00:56 Exploration

01:08:00 Their paper - regularizing trajectory optimization with denoising

01:13:47 What is Epistemic uncertainty

01:16:44 How would Curious develop these models

01:18:00 Explainability and simulations

01:22:33 How system 2 works in humans

01:26:11 Planning

01:27:04 Advice for starting an AI company

01:31:31 Real world implementation of planning models

01:33:49 Publishing research and openness

We really hope you enjoy this episode, please subscribe!

Regularizing Trajectory Optimization with Denoising Autoencoders: https://papers.nips.cc/paper/8552-regularizing-trajectory-optimization-with-denoising-autoencoders.pdf

Pulp, Paper & Packaging: A Future Transformed through Deep Learning: https://thecuriousaicompany.com/pulp-paper-packaging-a-future-transformed-through-deep-learning/

Curious AI: https://thecuriousaicompany.com/

Harri Valpola Publications: https://scholar.google.com/citations?user=1uT7-84AAAAJ&hl=en&oi=ao

Some interesting papers around Model-Based RL:

GameGAN: https://cdn.arstechnica.net/wp-content/uploads/2020/05/Nvidia_GameGAN_Research.pdf

Plan2Explore: https://ramanans1.github.io/plan2explore/

World Models: https://worldmodels.github.io/

MuZero: https://arxiv.org/pdf/1911.08265.pdf

PlaNet: A Deep Planning Network for RL: https://ai.googleblog.com/2019/02/introducing-planet-deep-planning.html

Dreamer: Scalable RL using World Models: https://ai.googleblog.com/2020/03/introducing-dreamer-scalable.html

Model Based RL for Atari: https://arxiv.org/pdf/1903.00374.pdf

In this episode of Machine Learning Street Talk, Tim Scarfe, Connor Shorten and Yannic Kilcher react to Yoshua Bengio’s ICLR 2020 Keynote “Deep Learning Priors Associated with Conscious Processing”. Bengio takes on many future directions for research in Deep Learning such as the role of attention in consciousness, sparse factor graphs and causality, and the study of systematic generalization. Bengio also presents big ideas in Intelligence that border on the line of philosophy and practical machine learning. This includes ideas such as consciousness in machines and System 1 and System 2 thinking, as described in Daniel Kahneman’s book “Thinking Fast and Slow”. Similar to Yann LeCun’s half of the 2020 ICLR keynote, this talk takes on many challenging ideas and hopefully this video helps you get a better understanding of some of them! Thanks for watching! 

Please Subscribe for more videos!

Paper Links:

Link to Talk: https://iclr.cc/virtual_2020/speaker_7.html

The Consciousness Prior: https://arxiv.org/abs/1709.08568

Thinking Fast and Slow: https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555

Systematic Generalization: https://arxiv.org/abs/1811.12889

CLOSURE: Assessing Systematic Generalization of CLEVR Models: https://arxiv.org/abs/1912.05783

Neural Module Networks: https://arxiv.org/abs/1511.02799

Experience Grounds Language: https://arxiv.org/pdf/2004.10151.pdf

Benchmarking Graph Neural Networks: https://arxiv.org/pdf/2003.00982.pdf

On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Please check out our individual channels as well!

Machine Learning Dojo with Tim Scarfe: https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA

Yannic Kilcher: https://www.youtube.com/channel/UCZHmQk67mSJgfCCTn7xBfe

Henry AI Labs: https://www.youtube.com/channel/UCHB9VepY6kYvZjj0Bgxnpbw

00:00:00 Tim and Yannics takes

00:01:37 Intro to Bengio

00:03:13 System 2, language and Chomsky

00:05:58 Cristof Koch on conciousness

00:07:25 Francois Chollet on intelligence and consciousness

00:09:29 Meditation and Sam Harris on consciousness

00:11:35 Connor Intro

00:13:20 Show Main Intro

00:17:55 Priors associated with Conscious Processing

00:26:25 System 1 / System 2

00:42:47 Implicit and Verbalized Knowledge [DONT MISS THIS!]

01:08:24 Inductive Priors for DL 2.0

01:27:20 Systematic Generalization

01:37:53 Contrast with the Symbolic AI Program

01:54:55 Attention

02:00:25 From Attention to Consciousness

02:05:31 Thoughts, Consciousness, Language

02:06:55 Sparse Factor graph

02:10:52 Sparse Change in Abstract Latent Space

02:15:10 Discovering Cause and Effect

02:20:00 Factorize the joint distribution

02:22:30 RIMS: Modular Computation

02:24:30 Conclusion

#machinelearning #deeplearning

This week Connor Shorten, Yannic Kilcher and Tim Scarfe reacted to Yann LeCun's keynote speech at this year's ICLR conference which just passed. ICLR is the number two ML conference and was completely open this year, with all the sessions publicly accessible via the internet. Yann spent most of his talk speaking about self-supervised learning, Energy-based models (EBMs) and manifold learning. Don't worry if you hadn't heard of EBMs before, neither had we!

Thanks for watching! Please Subscribe!

Paper Links:

ICLR 2020 Keynote Talk: https://iclr.cc/virtual_2020/speaker_7.html

A Tutorial on Energy-Based Learning: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf

Concept Learning with Energy-Based Models (Yannic's Explanation): https://www.youtube.com/watch?v=Cs_j-oNwGgg

Concept Learning with Energy-Based Models (Paper): https://arxiv.org/pdf/1811.02486.pdf

Concept Learning with Energy-Based Models (OpenAI Blog Post): https://openai.com/blog/learning-concepts-with-energy-functions/

#deeplearning #machinelearning #iclr #iclr2020 #yannlecun