The document describes the development of MMIE, a large-scale benchmark designed to evaluate the performance of Large Vision-Language Models (LVLMs) in interleaved multimodal comprehension and generation tasks. MMIE comprises a dataset of 20,000 meticulously curated multimodal queries across various domains, including mathematics, coding, and literature, which are designed to challenge LVLMs to produce and interpret both images and text in arbitrary sequences. The authors also propose a reliable automated evaluation metric for MMIE, leveraging a scoring model fine-tuned with human-annotated data and systematic evaluation criteria. Extensive experiments demonstrate the effectiveness of the benchmark and metrics, revealing significant room for improvement in the development of interleaved LVLMs. The paper provides detailed insights into the benchmark's construction, evaluation methods, and error analysis, offering valuable guidance for future research in multimodal learning.

This research paper proposes a novel method called Thought Preference Optimization (TPO) to train large language models (LLMs) to "think" before responding to user instructions. TPO utilizes a preference-based training framework where LLMs generate internal thoughts alongside their responses, and these thoughts are then optimized based on the quality of the resulting responses. The authors argue that this approach, unlike previous methods relying on direct supervision, allows LLMs to develop thinking abilities for a broader range of tasks beyond traditional reasoning and problem-solving. They demonstrate the effectiveness of TPO on benchmark datasets and observe that LLMs trained with TPO show improvements even in non-reasoning categories like language and translation, marketing, and health, highlighting the potential for thinking-based LLMs in diverse applications.

The paper, "VIT-LENS: Towards Omni-modal Representations," introduces a novel approach to enable Artificial Intelligence (AI) agents to perceive information from various modalities beyond just vision and language. It proposes a method that leverages a pre-trained visual transformer (ViT) to efficiently encode information from diverse modalities, such as 3D point clouds, depth, audio, tactile, and electroencephalograms (EEG). By aligning these modalities with a shared embedding space, VIT-LENS unlocks a range of capabilities for AI agents, including any-modality captioning, question answering, and image generation. The paper presents extensive experimental results demonstrating that VIT-LENS achieves state-of-the-art performance on various benchmark datasets and outperforms prior methods in understanding and interacting with diverse modalities.

This research explores how the architecture of pre-trained language models influences their base capabilities, specifically focusing on the FFN-Wider Transformer architecture. The study identifies a key factor in model performance: the contribution ratio of the Multi-Head Attention (MHA) layer, which acts as a combination function that reflects the model's ability to combine linguistic features. The authors demonstrate that FFN-Wider Transformers reduce the contribution ratio of this combination function, leading to a decline in base capabilities. To address this issue, they propose a Combination Enhanced Architecture (CEA) that redistributes the wider FFN layer, enhancing the combination function and ultimately improving base capabilities. The effectiveness of CEA is further validated by its successful application to Mixture of Experts (MoE) Transformers, highlighting its potential for broader architecture improvement.

This research paper proposes a new method called Reinforcement Learning from Execution Feedback (RLEF) to improve the ability of large language models (LLMs) to generate code that successfully completes tasks. The authors demonstrate the effectiveness of RLEF by training LLMs on a challenging competitive programming benchmark called CodeContests. RLEF trains the models to iteratively generate code based on the feedback received from running their code against test cases. The results show that RLEF significantly improves solve rates and reduces the number of code samples needed compared to previous approaches, achieving state-of-the-art performance. The paper also investigates the inference-time behavior of RLEF-trained LLMs, highlighting their ability to effectively learn from feedback and make targeted improvements over multiple code generations.

This research paper investigates how variations in the phrasing of prompts impact the performance of large language models (LLMs) across 120 tasks and five models. The study systematically analyzes six families of paraphrase types, including morphology, syntax, lexicon, lexico-syntax, discourse, and others, to determine their influence on model outputs. The findings demonstrate a potential for significant performance gains when prompts are adapted using specific paraphrase types, particularly morphology and lexicon changes. The research also considers factors like prompt complexity, temperature, and proximity to training data, concluding that smaller models are more sensitive to paraphrase changes and can potentially achieve comparable performance to larger models through prompt engineering.

LLaMA-Omni, designed to improve the seamless interaction between speech and large language models (LLMs). This model integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder, allowing it to generate text and speech responses directly from speech instructions with minimal latency. To enhance the model's performance, the authors create a speech instruction dataset called InstructS2S-200K containing 200,000 speech instructions and corresponding speech responses. Experimental results demonstrate that LLaMA-Omni provides superior responses in both content and style compared to previous speech-language models, achieving a response latency of 226 milliseconds. Furthermore, the model's training process is efficient, requiring less than 3 days on 4 GPUs.

"Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models," details the development of a new family of multimodal language models (VLMs) called Molmo. Molmo is notable for its open-weight and open-data approach, meaning the model's weights, training data, and code are publicly available. This contrasts with the current trend of proprietary VLMs which keep their models closed. Molmo achieves state-of-the-art performance by utilizing a novel image captioning dataset called PixMo, collected from human annotators using speech-based descriptions. This approach avoids reliance on synthetic data generated by proprietary systems, enabling the creation of performant VLMs without the need for distilling closed models. The authors highlight Molmo's potential for various tasks, including question answering and image-based navigation.

This technical paper proposes a novel technique called Low-Rank Adaptation (LoRA) for adapting large language models (LLMs) to specific downstream tasks. LoRA addresses the challenge of fine-tuning LLMs, which requires updating all model parameters, by injecting low-rank decomposition matrices into each layer of the Transformer architecture. This significantly reduces the number of trainable parameters, resulting in a substantial decrease in storage requirements, memory usage, and training time. The paper shows that LoRA performs comparably or even better than fine-tuning on various tasks, including natural language understanding (NLU) and generation (NLG), while providing additional benefits such as efficient task switching and lower hardware barrier to entry. The paper concludes by investigating the low-rank structure of model updates, providing insights into the effectiveness of LoRA and the underlying mechanisms of model adaptation.