Github: https://github.com/microsoft/LLM2CLIP

The paper introduces LLM2CLIP, a method to improve the visual representation learning capabilities of CLIP by integrating large language models (LLMs). LLM2CLIP addresses CLIP's limitations with long and complex text by fine-tuning the LLM to enhance its textual discriminability, effectively using the LLM's knowledge to guide CLIP's visual encoder. Experiments demonstrate significant performance improvements across various image-text retrieval tasks and benchmarks, including cross-lingual retrieval. The approach is efficient, requiring minimal additional computational cost compared to training the original CLIP model. The improved model shows enhanced understanding of long and complex text semantics, exceeding the performance of state-of-the-art CLIP models.

ai , computer vision , cv , peking university , artificial intelligence , arxiv , research , paper , publication , lvm , large visual models