Papers Read on AI

Papers Read on AI

By RobNewsTech News
Download on the App Store

Papers Read on AI episodes

  • Black-Box Tuning for Language-Model-as-a-Service
    Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service, allowing users to design taskspecific prompts to query the PTMs through some black-box APIs. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), gradients of the PTMs are usually not available. Can we optimize the task prompts by only accessing the model inference APIs? Based on recent observations that large PTMs have a very low intrinsic dimensionality, this work proposes the Black-Box Tuning to optimize PTMs through derivativefree algorithms.
    2022: Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, Xipeng Qiu
    https://arxiv.org/pdf/2201.03514v1.pdf
    30 min
  • Vision Transformer with Deformable Attention
    We propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features.
    2022: Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, Gao Huang
    Ranked #1 on Object Detection on COCO test-dev (AP metric)
    https://arxiv.org/pdf/2201.00520v1.pdf
    31 min
  • R-Drop: Regularized Dropout for Neural Networks
    Dropout is a powerful and widely used technique to regularize the training of deep neural networks. Though effective and performing well, the randomness introduced by dropout causes unnegligible inconsistency between training and inference. In this paper, we introduce a simple consistency training strategy to regularize dropout, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other.
    2021: Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu
    Ranked #2 on Abstractive Text Summarization on CNN / Daily Mail
    https://arxiv.org/pdf/2106.14448v2.pdf
    38 min
  • Vision Transformer with Deformable Attention
    We propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features.
    2022: Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, Gao Huang
    Ranked #1 on Object Detection on COCO test-dev (AP metric)
    https://arxiv.org/pdf/2201.00520v1.pdf
    31 min
  • Language-driven Semantic Segmentation
    We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., “grass” or “building”) together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space.
    2022: Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, V. Koltun, René Ranftl
    Ranked #2 on Few-Shot Semantic Segmentation on FSS-1000
    https://arxiv.org/pdf/2201.03546v1.pdf
    24 min
  • Vision Transformer with Deformable Attention
    We propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features.
    2022: Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, Gao Huang Ranked
    #1 on Object Detection on COCO test-dev (AP metric)
    https://arxiv.org/pdf/2201.00520v1.pdf
    31 min
  • A ConvNet for the 2020s
    The “Roaring 20s” of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks.
    2022: Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie
    Ranked #1 on Domain Generalization on ImageNet-Sketch (using extra training data)
    https://arxiv.org/pdf/2201.03545v1.pdf
    36 min
  • Black-Box Tuning for Language-Model-as-a-Service
    Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service, allowing users to design taskspecific prompts to query the PTMs through some black-box APIs. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), gradients of the PTMs are usually not available. Can we optimize the task prompts by only accessing the model inference APIs? Based on recent observations that large PTMs have a very low intrinsic dimensionality, this work proposes the Black-Box Tuning to optimize PTMs through derivativefree algorithms.
    2022: Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, Xipeng Qiu
    https://arxiv.org/pdf/2201.03514v1.pdf
    30 min
  • Transfer Learning for Pose Estimation of Illustrated Characters
    Human pose information is a critical component in many downstream image processing tasks, such as activity recognition and motion tracking. Likewise, a pose estimator for the illustrated character domain would provide a valuable prior for assistive content creation tasks, such as reference pose retrieval and automatic character animation.
    2021: Shuhong Chen, Matthias Zwicker
    https://arxiv.org/pdf/2108.01819v3.pdf
    35 min
  • Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

    In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of “grokking” a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting.

    2022: Alethea Power, Yuri Burda, Harrison Edwards, I. Babuschkin, Vedant Misra

    https://arxiv.org/pdf/2201.02177v1.pdf

    13 min

About Papers Read on AI

From the publisher's feed

Keeping you up to date with the latest trends and best performing architectures in this fast evolving field in computer science.