Papers Read on AI

This paper studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Visual Transformers (ViT). We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects.

2021: Xinlei Chen, Saining Xie, Kaiming He

https://arxiv.org/pdf/2104.02057v4.pdf

In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at extrapolation tests where traditional neural networks fail. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural networks.

2021: Andrea Banino, Jan Balaguer, C. Blundell

https://arxiv.org/pdf/2107.05407v1.pdf

Codex, a GPT language model fine tuned on publicly available code from GitHub, and study its Python code-writing capabilities.

2021: Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, J. Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea. Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, F. Such, D. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, I. Babuschkin, S. Balaji, Shantanu Jain, A. Carr, J. Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, M. Knight, Miles Brundage, Mira Murati, Katie Mayer, P. Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba

https://arxiv.org/pdf/2107.03374.pdf

All existing approaches to uncertainty modeling can be expressed as ENNs, and any ENN can be identified with a Bayesian neural network. We propose a clear and simple metric for progress in ENNs: the KL-divergence with respect to a target distribution.

2021: Ian Osband, Zheng Wen, M. Asghari, Morteza Ibrahimi, Xiyuan Lu, Benjamin Van Roy

https://arxiv.org/pdf/2107.08924v1.pdf

We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement.

2021: Stephan R. Richter, Hassan Abu Alhaija, V. Koltun

https://arxiv.org/pdf/2105.04619.pdf

The proposed framework contains two important ingredients: Smoothness regularization and Bregman proximal point optimization. Our experiments show that the proposed framework achieves new state-of-the-art performance on a number of NLP tasks including GLUE, SNLI, SciTail and ANLI. 2020: Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, T. Zhao Keywords: Language model, Overfitting, Experiment, Manifold regularization, Matrix regularization https://arxiv.org/pdf/1911.03437v4.pdf

In this work, we design a novel Transformer-style module, i.e., Contextual Transformer (CoT) block, for visual recognition. Such design fully capitalizes on the contextual information among input keys to guide the learning of dynamic attention matrix and thus strengthens the capacity of visual representation. Technically, CoT block first contextually encodes input keys via a convolution, leading to a static contextual representation of inputs.

2021: Yehao Li, Ting Yao, Yingwei Pan, Tao Mei

Keywords: Instance segmentation, Object detection

https://arxiv.org/pdf/2107.12292v1.pdf

Delivering malware covertly and detection-evadingly is critical to advanced malware campaigns. In this paper, we present a method that delivers malware covertly and detection evadingly through neural network models. Neural network models are poorly explainable and have a good generalization ability. By embedding malware into the neurons, malware can be delivered covertly with minor or even no impact on the performance of neural networks. Meanwhile, since the structure of the neural network models remains unchanged, they can pass the security scan of antivirus engines.

2021: Zhi Wang, Chaoge Liu, Xiang Cui

Keywords: Malware, Virus

https://arxiv.org/pdf/2107.08590v1.pdf

Socially influenceable paper at time 25 july, 2021

Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability.

2020: Shen Li, Yanli Zhao, R. Varma, Omkar Salpekar, P. Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, Soumith Chintala

Data parallelism, Deep learning, Gradient, Computation, Computational resource, Scalability, Parallel computing, Computational science, Norm (social), Iteration, Graphics processing unit

https://arxiv.org/pdf/2006.15704v1.pdf

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised auto curriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps.

2020: Bowen Baker, I. Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch

Reinforcement learning, Multi-agent system, Emergent, Machine learning, Algorithm

https://arxiv.org/pdf/1909.07528.pdf