The paper presents Llama 3, a new family of foundation language models developed by Meta, featuring models with 8B, 70B, and a flagship 405B parameters. These models natively support multilinguality, coding, reasoning, and tool usage, with the 405B model capable of processing information in a context window of up to 128K tokens.

The development of Llama 3 focuses on optimizing data, scale, and complexity:

• Pre-training: The models were pre-trained on a massive corpus of 15.6 trillion tokens, which is substantially larger and higher quality than the data used for Llama 2.
• Post-training: The models underwent rigorous alignment using supervised finetuning (SFT), rejection sampling, and direct preference optimization (DPO) to better follow instructions and ensure helpfulness and harmlessness.

Extensive empirical and human evaluations demonstrate that the flagship 405B model performs on par with leading closed-source models like GPT-4 across a wide variety of tasks, while the 8B and 70B models establish best-in-class performance compared to alternative models of similar sizes.

The paper also highlights robust safety measures, including the release of Llama Guard 3 for system-level input and output safety. Finally, the authors detail ongoing, unreleased experiments integrating image, video, and speech capabilities into Llama 3 using a compositional approach, which has shown competitive results against state-of-the-art multimodal models.

The provided text is a blog post introducing Command R+, Cohere's highly scalable large language model (LLM) specifically designed for enterprise workloads.

Here is a short summary of its key features:

• Advanced RAG Capabilities: The model features a 128k-token context window and is highly optimized for Retrieval Augmented Generation (RAG). It improves response accuracy and provides in-line citations to help mitigate hallucinations, allowing businesses to reliably use their own proprietary data.
• Robust Multilingual Support: Command R+ is built to excel in 10 key languages for global business operations. It also features a custom tokenizer that compresses non-English text much more efficiently than competitors, reducing token costs by up to 57%.
• Multi-Step Tool Use: The model can act as a reasoning engine to automate complex business workflows. It can string together multiple tools over several steps to accomplish difficult tasks and can even self-correct if it encounters a bug or tool failure.
• Enterprise Security and Availability: Built with a strong commitment to data privacy, Command R+ does not access customer data without permission and offers copyright assurance. It is available first on Microsoft Azure, followed by Oracle Cloud Infrastructure (OCI) and Cohere's hosted API.

Kolmogorov-Arnold Networks (KANs) are proposed as a promising and mathematically grounded alternative to standard Multi-Layer Perceptrons (MLPs). Unlike MLPs, which apply fixed activation functions on nodes (neurons), KANs place learnable activation functions on the edges (weights) of the network. In a KAN, every weight parameter is replaced by a univariate function parameterized as a spline, meaning the networks contain no linear weight matrices at all.

This architectural shift allows KANs to significantly outperform MLPs in two key areas:

• Accuracy and Scaling: Smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and function regression tasks. Theoretically and empirically, KANs exhibit faster neural scaling laws and can successfully overcome the "curse of dimensionality" by combining external compositional structures with highly accurate internal spline approximations. Additionally, because splines are local, KANs demonstrate local plasticity, allowing them to avoid catastrophic forgetting in continual learning tasks.
• Interpretability and Interactivity: Because of their structure, KANs can be intuitively visualized and easily simplified through techniques like sparsification, pruning, and symbolification. This high level of transparency allows human users to easily interact with the network, test hypotheses, and extract exact symbolic formulas from the model's learned weights.

Ultimately, the authors position KANs as highly effective foundational models for small-scale AI + Science tasks. Through extensive experiments, KANs are shown to act as valuable scientific "collaborators," successfully helping researchers (re)discover complex mathematical laws in knot theory, map phase boundaries in physics (Anderson localization), and efficiently solve partial differential equations (PDEs).

Jamba is a new large language model developed by AI21 Labs that introduces a novel hybrid architecture. The model interleaves traditional Transformer layers with Mamba (a state-space model) layers, and integrates a Mixture-of-Experts (MoE) module to increase capacity without proportionally increasing compute requirements.

This hybrid approach addresses the fundamental limitations of pure Transformer models, which suffer from high memory and compute requirements for long contexts due to the growing key-value (KV) cache. It also improves upon pure Mamba models, which can sometimes struggle to match the in-context learning capabilities of Transformers.

Key highlights of the Jamba model include:

• Massive Context Window: Jamba supports an impressive context length of up to 256K tokens. By replacing some attention layers with Mamba layers, it requires an 8x smaller KV cache than a vanilla Transformer (needing only 4GB for a 256K context, compared to 32GB for Mixtral).
• High Efficiency: The model features 52B total parameters but only 12B active parameters per token due to its MoE routing. This allows it to easily fit on a single 80GB GPU.
• Superior Throughput: Jamba boasts up to 3x the throughput (tokens per second) of comparable models like Mixtral-8x7B, especially when processing long contexts.
• State-of-the-Art Performance: Across standard academic benchmarks and long-context evaluations, Jamba performs comparably to leading models of similar or larger sizes, such as Mixtral-8x7B and Llama-2 70B.

To encourage further community research and exploration into hybrid Attention-Mamba architectures, AI21 Labs has made the base model publicly available under a permissive Apache 2.0 license.

DBRX is a new state-of-the-art, open, general-purpose large language model (LLM) developed by Databricks.

Key Highlights:

• Architecture and Scale: DBRX uses a fine-grained mixture-of-experts (MoE) architecture. It contains 132 billion total parameters, but only 36 billion are active for any given input. The model was pre-trained on 12 trillion tokens of carefully curated text and code, supporting a maximum context length of 32k tokens.
• Superior Performance: DBRX establishes a new standard for open models, outperforming peers like LLaMA2-70B, Mixtral, and Grok-1 across composite benchmarks, with particular strengths in programming (HumanEval) and mathematics (GSM8k). It also exceeds the capabilities of GPT-3.5 and is highly competitive with closed models like Gemini 1.0 Pro and Mistral Medium.
• High Efficiency: Thanks to its MoE architecture, DBRX achieves significant efficiency gains. It is highly compute-efficient to train and delivers inference throughput that is up to 2x faster than LLaMA2-70B. Databricks notes that their overall end-to-end training pipeline has become nearly 4x more compute-efficient compared to their previous MPT models.

Ultimately, DBRX is designed to provide the open community and enterprises with the capability to build and control their own world-class foundation models, matching the quality of closed APIs.

The provided document is a model card introducing Anthropic's Claude 3 family of large multimodal models, which consists of three models: Opus (the most capable), Sonnet (a balance of skills and speed), and Haiku (the fastest and most affordable).

Key highlights from the paper include:

• Core Capabilities & Vision: The Claude 3 family sets new industry benchmarks in reasoning, mathematics, coding, and multilingual understanding. A major new feature is their multimodal vision capabilities, which allow users to upload and analyze visual data such as images, charts, and diagrams alongside text. Opus achieves state-of-the-art results on standard evaluations like GPQA, MMLU, and MMMU.
• Long Context and Recall: The models are offered with a 200,000-token context window (though they are capable of reaching 1 million tokens). In evaluations like the "Needle In A Haystack" test, Claude 3 Opus demonstrated near-perfect recall, consistently extracting specific information from dense documents with over 99% accuracy.
• Behavioral Improvements: Anthropic focused heavily on behavioral design. The Claude 3 models demonstrate improved factual accuracy, better instruction following, and a more nuanced understanding of prompts. Notably, the models exhibit a significant reduction in unnecessary refusals, meaning they are much less likely to unhelpfully refuse benign or harmless prompts compared to previous generations.
• Safety and Catastrophic Risk Assessments: Guided by Anthropic's Responsible Scaling Policy, the models underwent extensive automated and red-teaming evaluations for catastrophic risks, including autonomous replication, biological threats, and cyber capabilities. The evaluations found no indicators of catastrophic risk, classifying the models at the ASL-2 risk level. The report also outlines ongoing mitigations for Trust & Safety, bias, and discrimination.

The provided paper introduces Gemma, a family of lightweight, open-weights language models developed by Google DeepMind. Built upon the foundational research, architecture, and training methodologies of Google's Gemini models, Gemma is available in two sizes: 2 billion and 7 billion parameters.

Key highlights of the paper include:

• State-of-the-Art Performance: Gemma models outperform similarly sized open-source alternatives (such as LLaMA 2 and Mistral) across a wide range of academic benchmarks. They demonstrate particularly strong capabilities in mathematics, coding, and reasoning tasks.
• Model Variants: Google has released both raw, pretrained checkpoints and instruction-tuned versions that have been refined for dialogue, helpfulness, and safety.
• Advanced Architecture: The models are built on a transformer decoder architecture enhanced with improvements like Multi-Query Attention, RoPE Embeddings, and GeGLU Activations. They were trained on up to 6 trillion tokens of primarily English web documents, math, and code.
• Safety and Responsible Deployment: A major focus of the paper is minimizing AI risks. The developers extensively filtered the pre-training data to remove harmful content and personal information, resulting in no observed memorization of sensitive data. Furthermore, the fine-tuned models utilize supervised fine-tuning and reinforcement learning from human feedback (RLHF) to ensure safe and reliable outputs.

Ultimately, the release of Gemma aims to provide developers and researchers with equitable access to frontier AI technology, encouraging innovation while balancing the risks of open model deployment.

The paper introduces OLMo, a state-of-the-art, truly open language model designed to accelerate the scientific study of large language models. While the commercial value of language models has led to the most powerful models being closed off or only partially released (e.g., releasing only weights or inference code), OLMo provides the research community with full access to its entire development framework.

Key highlights of the paper include:

• A Fully Open Framework: The release encompasses the complete pipeline, including the model weights (in 1B and 7B variants), the open pretraining dataset called Dolma, training and evaluation code, detailed training logs, and hundreds of intermediate model checkpoints. All code and weights are released under a permissive Apache 2.0 license.
• Competitive Performance: The OLMo-7B model, trained on at least 2 trillion tokens, is highly competitive with other similarly sized models like LLaMA-7B, Llama-2-7B, MPT-7B, and Falcon-7B across various zero-shot downstream tasks and perplexity benchmarks.
• Adaptation and Alignment: The authors also fine-tuned OLMo using instruction tuning (SFT) and Direct Preference Optimization (DPO). The adapted models showed significant improvements in general chat capabilities, safety, and truthfulness, proving that OLMo serves as a very strong base model for downstream applications.
• Research Goal: By releasing everything from the exact datasets used to the intermediate training steps, the creators of OLMo aim to help researchers study poorly understood aspects of language models, such as how training data impacts model capabilities, the effects of hyperparameter choices, and the models' underlying biases and risks.

The paper "Textbooks Are All You Need II: phi-1.5 technical report" investigates the capabilities of small language models (SLMs) when trained on highly curated datasets.

Here is a short summary of the key findings:

• Textbook-Quality Data: The researchers developed phi-1.5, a 1.3 billion parameter model, by training it primarily on synthetically generated "textbook-like" data rather than traditional, raw web data. This dataset included roughly 20 billion tokens specifically crafted to teach general world knowledge and common sense reasoning.
• Surprising Performance: Despite its small size, phi-1.5 achieves performance on natural language tasks that is comparable to models five times larger. Furthermore, it vastly exceeds other models of similar size on complex multi-step reasoning tasks, such as grade-school mathematics and basic Python coding.
• Reduced Toxicity: By relying on synthetic textbook data instead of standard internet scrapes, the model demonstrated a much lower propensity for generating toxic, biased, or harmful content.
• Challenging Scaling Laws: Ultimately, the research challenges the prevailing assumption that language models must rely solely on massive scale (hundreds of billions of parameters) to achieve high capabilities. Instead, it proves that data quality is an equally, if not more, crucial factor.

The paper introduces Gemini, a new family of natively multimodal models developed by Google, designed to seamlessly understand and reason across text, image, audio, and video.

The Gemini 1.0 family is built to accommodate different computational limitations and is released in three sizes: Ultra for highly complex reasoning tasks, Pro for efficient deployability at scale, and Nano for on-device applications.

Key highlights of the paper include:

• State-of-the-Art Performance: Gemini Ultra advances the state-of-the-art in 30 out of 32 prominent benchmarks across text, reasoning, image, video, and speech. Notably, it is the first model to achieve human-expert performance on the MMLU (Massive Multitask Language Understanding) exam benchmark, scoring over 90%.
• Native Multimodality: Unlike previous models that stitch together separate vision and language components, Gemini is jointly trained across modalities from the very beginning. This allows it to exhibit impressive cross-modal reasoning, natively ingesting interleaved sequences of audio, images, and text, and directly outputting images using discrete image tokens.
• Complex Reasoning and Coding: The models show strong proficiencies in mathematics, science, and coding. When combined with search and tool-use mechanisms, Gemini powers advanced agents like AlphaCode 2, which performs in the top 15% of entrants in competitive programming.
• Safety and Responsible Deployment: After large-scale pre-training, the models undergo rigorous post-training—utilizing Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—to improve overall quality, ensure alignment, and mitigate potential harms.

The models are deployed through two main variants: Gemini Apps models (optimized for conversational AI services like Gemini Advanced) and Gemini API models (optimized for developers building applications via Google AI Studio and Cloud Vertex AI).