AI Post Transformers

This 2019 paper, "Language Models are Unsupervised Multitask Learners," introduces GPT-2, a large language model designed for zero-shot learning, meaning it can perform tasks without explicit, task-specific training. The research highlights the model's ability to learn various natural language processing (NLP) tasks, such as question answering, summarization, and translation, by being trained on a diverse and extensive dataset called WebText, composed of millions of high-quality webpages. The paper demonstrates that increasing the model's capacity significantly improves performance across these tasks, often achieving state-of-the-art results in a zero-shot setting. While showing promising results, the authors acknowledge that GPT-2's practical applications are still developing, particularly in areas like summarization and translation where performance remains rudimentary compared to human benchmarks

This 2020 paper outlines the development and evaluation of GPT-3, a large language model, exploring its performance across various natural language processing tasks under zero-shot, one-shot, and few-shot learning conditions, which involve providing minimal to no task-specific examples during inference. It details the model's architecture, training methodology, including its use of a massive dataset, and analyzes its limitations and broader impacts, such as the potential for misuse and the presence of biases related to gender, race, and religion inherited from its training data. The document also discusses the challenges of data contamination and the computational resources required for training such a large model.

This paper introduces Grouped-Query Attention (GQA), a novel approach designed to enhance the inference efficiency of large language models. It addresses the limitations of Multi-Query Attention (MQA), which, while fast, can compromise model quality, and Multi-Head Attention (MHA), which offers high quality but is slower due to memory bandwidth overhead. The authors propose uptraining existing MHA models to either MQA or GQA with minimal additional computational cost. GQA acts as an interpolation between MHA and MQA, utilizing an intermediate number of key-value heads to strike a balance, achieving near-MHA quality with speeds comparable to MQA, making it a favorable trade-off for larger models.

This 2023 paper, GPT-4 Technical Report from OpenAI introduces GPT-4, a multimodal AI model capable of processing both image and text inputs to produce text outputs, demonstrating human-level performance on various professional and academic benchmarks, such as the bar exam. The report highlights the predictable scaling of the model's performance and its improved factual accuracy and adherence to desired behaviors achieved through post-training alignment processes like Reinforcement Learning from Human Feedback (RLHF). It also addresses the model's limitations, including "hallucinations" and potential for misuse, detailing the safety evaluations and mitigation strategies implemented to reduce harmful outputs.

This 2022 paper explores the significant negative impact of repeated data on the performance of large language models, even when such repetitions constitute a small fraction of the total training data. The authors observe a "double descent" phenomenon, where model performance initially improves, then degrades at a specific repetition frequency, and finally improves again with excessive repetition, suggesting a trade-off between generalization and memorization. This performance degradation is disproportionately linked to the impairment of the model's copying ability and crucial internal structures called induction heads, which are vital for in-context learning. The study bridges scaling laws—predictable relationships between hyperparameters and performance—with mechanistic interpretability, aiming to understand how these microscopic changes in the model's internal computations lead to macroscopic performance shifts. Ultimately, the research offers practical insights for diagnosing and mitigating data-repetition issues in large language model training, highlighting how repeated data can hinder effective generalization.

The first source introduces Layer Normalization (LN), a technique designed to accelerate and stabilize the training of deep neural networks, particularly recurrent neural networks, by normalizing summed inputs across a layer for each training case. It contrasts LN with Batch Normalization, highlighting LN's independence from mini-batch size and its consistent computation during both training and testing. The second source, published later, builds upon this foundation by proposing Dual PatchNorm (DPN) for Vision Transformers (ViTs). DPN involves applying two Layer Normalization layers – one before and one after the patch embedding layer – demonstrating improved accuracy and training stability across various computer vision tasks. This subsequent research indicates that while traditional Layer Normalization placements within the Transformer block are effective, additional normalization in the initial patch embedding stage yields further benefits.

This paper outlines the advancements in Optical Character Recognition (OCR), particularly focusing on handwritten character and word recognition using Neural Networks. The authors, affiliated with AT&T Labs-Research, detail various machine learning techniques, including Gradient-Based Learning and Convolutional Neural Networks (CNNs) like LeNet-5, highlighting their effectiveness in handling high-dimensional inputs and generating intricate decision functions. A significant portion of the paper is dedicated to Graph Transformer Networks (GTNs), a multi-module system designed to interpret sequences of characters by leveraging graph-based representations and global training methods to reduce errors. The paper also describes the creation and use of the MNIST dataset, a benchmark for handwritten digit recognition, and discusses the practical application of these technologies in commercial check reading systems and online handwriting recognition.

This academic paper introduces Linformer, a novel approach to address the computational bottleneck of Transformer models in natural language processing. The authors demonstrate that the self-attention mechanism, which is a core component of Transformers and typically incurs quadratic time and space complexity with respect to sequence length, can be approximated by a low-rank matrix. By exploiting this finding, the Linformer reduces this complexity to linear time and space (O(n)), making it significantly more efficient for long sequences. The research provides both theoretical proofs and empirical evidence that the Linformer performs comparably to standard Transformers while offering substantial speed and memory improvements during both training and inference.

This paper introduces Longformer, a novel Transformer-based model designed to overcome the limitations of traditional Transformers in processing exceptionally long sequences. Unlike prior models with quadratic scaling, Longformer employs an attention mechanism that scales linearly with sequence length, making it efficient for documents containing thousands of tokens. This innovative architecture combines local windowed attention with task-motivated global attention, enhancing its ability to capture both immediate and overarching contextual information. The authors demonstrate Longformer's superior performance in character-level language modeling and its effectiveness across various downstream natural language processing (NLP) tasks, including question answering and coreference resolution. Furthermore, they present Longformer-Encoder-Decoder (LED), a variant for sequence-to-sequence tasks, showcasing its proficiency in long document summarization.

This 2000 paper introduces a novel solution to a weakness found in Long Short-Term Memory (LSTM) networks, specifically when processing continuous data streams without predefined segmentation. The core problem addressed is the unbounded growth of internal cell states within standard LSTM networks, which can lead to performance degradation. The authors propose and implement "forget gates", an adaptive mechanism that allows LSTM cells to learn when to reset their internal memory at appropriate times, thus managing resources effectively. Through experiments with complex, continual versions of benchmark problems, the paper demonstrates that LSTMs equipped with these forget gates successfully overcome limitations faced by standard LSTMs and other recurrent neural networks. Ultimately, the work highlights the importance of adaptive forgetting for neural networks dealing with ongoing, unsegmented input.