Data Science at Home

As a continuation of the previous episode in this one I cover the topic about compressing deep learning models and explain another simple yet fantastic approach that can lead to much smaller models that still perform as good as the original one.

Don't forget to join our Slack channel and discuss previous episodes or propose new ones.

This episode is supported by Pryml.io

Comparing Rewinding and Fine-tuning in Neural Network Pruning

Using large deep learning models on limited hardware or edge devices is definitely prohibitive. There are methods to compress large models by orders of magnitude and maintain similar accuracy during inference.

In this episode I explain one of the first methods: knowledge distillation

 Come join us on Slack

• Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531

Codiv-19 is an emergency. True. Let's just not prepare for another emergency about privacy violation when this one is over.

Join our new Slack channel

This episode is supported by Proton. You can check them out at protonmail.com or protonvpn.com

Whenever people reason about probability of events, they have the tendency to consider average values between two extremes. 

We are moving our community to Slack. See you there!

In this episode I briefly explain the concept behind activation functions in deep learning. One of the most widely used activation function is the rectified linear unit (ReLU). 

This episode is supported by pryml.io. At pryml we let companies share confidential data. Visit our website.

Don't forget to join us on discord channel to propose new episode or discuss the previous ones. 

Dynamic ReLU https://arxiv.org/abs/2003.10027

One of the best features of neural networks and machine learning models is to memorize patterns from training data and apply those to unseen observations. That's where the magic is. 

Think about a language generator that discloses the passwords, the credit card numbers and the social security numbers of the records it has been trained on. Or more generally, think about a synthetic data generator that can disclose the training data it is trying to protect. 

In this episode I explain why unintended memorization is a real problem in machine learning. Except for differentially private training there is no other way to mitigate such a problem in realistic conditions.

This episode is supported by Harmonizely. 

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

In this episode I explain a very effective technique that allows one to infer the membership of any record at hand to the (private) training dataset used to train the target model. The effectiveness of such technique is due to the fact that it works on black-box models of which there is no access to the data used for training, nor model parameters and hyperparameters. Such a scenario is very realistic and typical of machine learning as a service APIs. 

This episode is supported by pryml.io, a platform I am personally working on that enables data sharing without giving up confidentiality. 

As promised below is the schema of the attack explained in the episode.

Membership Inference Attacks Against Machine Learning Models

Masking, obfuscating, stripping, shuffling. 

There are very good reasons why a financial institution should never share their data. Actually, they should never even move their data. Ever.

Building reproducible models is essential for all those scenarios in which the lead developer is collaborating with other team members. Reproducibility in machine learning shall not be an art, rather it should be achieved via a methodical approach. 

Enjoy the show!