This research presents a novel training method for Vision Language Models (VLMs) focused on improving their ability to both assign scores to images and provide natural language explanations for those scores. By leveraging an existing image scoring dataset and an instruction-tuned VLM, the approach utilizes self-training without requiring additional external data or models. A key innovation is the creation of a dataset using Direct Preference Optimization (DPO) to enhance the alignment between predicted scores and generated text justifications. Through an iterative process of training the VLM on two self-generated datasets and then merging the resulting models, the system demonstrably improves both the accuracy of image scoring and the consistency of its accompanying explanations.

This academic paper presents MANTIS, a new approach to training large multimodal models (LMMs) to handle interleaved text and images. Instead of relying on massive, potentially noisy pre-training datasets, the researchers developed MANTIS-INSTRUCT, a focused dataset of 721K instances designed to improve multi-image understanding. The paper evaluates MANTIS on several multi-image and single-image benchmarks, demonstrating that this instruction-tuning method achieves state-of-the-art performance on multi-image tasks with significantly less computational effort compared to previous methods. The research highlights the importance of a vision encoder and a well-structured text-image format for effectively processing multiple images.

We explore how Vision-Language Models (VLMs) are revolutionizing ad click prediction by processing both ad images and detailed user personas. It explains the architecture of VLMs, highlighting the dual-encoder structure and the importance of a shared embedding space and attention mechanisms in understanding the interplay between visual and textual information. The text discusses key VLM models like CLIP, ALIGN, Flamingo, BLIP-2, LLaVA, GPT-4V, and Gemini, outlining their innovations. Ultimately, it describes how VLMs use the persona as a "lens" to personalize understanding and predict click likelihood, emphasizing the impact on personalized marketing, the associated challenges, and the exciting future directions of this technology.

The research introduces LLaVA-Critic, a new open-source large multimodal model specifically designed to evaluate the performance of other multimodal models. Trained on a specialized dataset, it functions effectively in two primary ways: first, as an LMM-as-a-Judge, providing reliable scores comparable to or better than commercial models like GPT, and second, for Preference Learning, generating reward signals that improve model alignment. This work highlights the potential of open-source models for self-critique and scalable evaluation in the multimodal domain. The text details the dataset creation process, model architecture, and experimental results supporting LLaVA-Critic's capabilities.

We examine the potential feasibility of using GPT-4V, a multimodal AI, to predict an individual's likelihood of clicking on an ad image based on a detailed psychological profile ("persona") rather than just historical behavior. The analysis breaks down how GPT-4V could process ad images and extensive persona text to infer connections, noting the complexity of matching visual elements and abstract psychological traits. While recognizing GPT-4V's relevant capabilities, the discussion highlights significant challenges like reasoning opacity, potential biases, and data privacy concerns. The author concludes that while theoretically plausible, achieving reliable, scalable, and ethical predictions is currently uncertain, suggesting future research directions focusing on prompt engineering, explainability, and hybrid models.

This paper explores how the rise of algorithms, particularly machine learning, is profoundly changing the field of economics, going beyond just impacting the economy itself. It argues that unlike previous technological innovations, algorithms represent a fundamental shift in how decisions are made, which is the core of economics. The author highlights that traditional econometric tools are designed for estimating parameters, whereas machine learning excels at predicting outcomes. This distinction leads to new areas of research, such as "prediction policy problems", where decisions hinge on predictive inferences rather than causal ones. The text further discusses how algorithms can aid in scientific discovery by generating hypotheses and identifying anomalies that challenge existing economic theories. Finally, it emphasizes that as algorithms become increasingly integrated into economic decisions, they will become a necessity for economic modeling and policy design.

This document explores how Google's implementation of the BERT algorithm in October 2019 impacted the sponsored search market. Analyzing changes in bidder competition and cost-per-click (CPC), the authors found that BERT generally increased the number of bidders but had varied effects on CPC depending on search query length. Shorter queries saw increased CPC, while longer queries experienced a decrease despite a rise in competition. A theoretical model is introduced to explain these findings, suggesting that the algorithm's ability to understand multi-dimensional query types (topic and context) is key, with better context identification potentially leading to lower CPC by improving advertiser differentiation and relevancy scoring.

This paper explores a novel framework called LR-Recsys, which enhances black-box recommender systems by integrating Large Language Model (LLM)-based contrastive explanations. Instead of relying solely on past user behavior or explicit product details, LR-Recsys uses LLMs to generate both positive and negative reasons why a user might or might not like a product. These generated explanations, converted into numerical representations, are then fed into a deep neural network, improving the system's ability to predict user preferences more accurately while also providing valuable insights to users, sellers, and the platform. The authors demonstrate empirically and theoretically that these performance gains stem primarily from the LLMs' reasoning capabilities, particularly for more challenging or uncertain recommendation scenarios.

We detail research into how platforms like Mercari can guide sellers' pricing decisions. The research, a collaboration with Mercari, involved a large-scale field experiment where sellers were shown different suggested prices or no suggestions at all. Key findings indicate that suggested prices significantly influence seller listing prices, particularly for inexperienced sellers or those listing used items. Furthermore, lower suggested prices led to improved selling outcomes, such as a higher likelihood of sale and increased revenue for both sellers and the platform, without negatively impacting the number of listings. The study addresses potential interference issues in the experimental design and finds that the results, particularly the benefit of lower suggested prices, appear to generalize to platform-wide implementation.

This paper examines how AI systems are likely to incorporate advertising, similar to web search and social media, but with new complexities due to their dynamic, personalized nature. The authors discuss potential methods for delivering commercial content, differentiating between static ads and "generative advertisements" seamlessly integrated into AI responses through product recommendations, placements, or paid visibility. The paper outlines key considerations for designing commercially-influenced AI systems, emphasizing faithfulness to sponsor information, maintaining core utility, respecting user privacy, and ensuring provenance of content. It also explores strategies users might employ to identify and mitigate commercial biases, including direct debiasing and multi-sampling, while acknowledging limitations and open questions regarding regulation, transparency, user autonomy, and the long-term impact on model quality.