In the rapidly evolving landscape of artificial intelli-gence, multi-modal large language models are emerging asa significant area of interest. These models, which combinevarious forms of data input, are becoming increasingly pop-ular. However, understanding their internal mechanisms re-mains a complex task. Numerous advancements have beenmade in the field of explainability tools and mechanisms,yet there is still much to explore. In this work, we present anovel interactive application aimed towards understandingthe internal mechanisms of large vision-language models.Our interface is designed to enhance the interpretability ofthe image patches, which are instrumental in generating ananswer, and assess the efficacy of the language model ingrounding its output in the image. With our application, auser can systematically investigate the model and uncoversystem limitations, paving the way for enhancements in sys-tem capabilities. Finally, we present a case study of how ourapplication can aid in understanding failure mechanisms ina popular large multi-modal model: LLaVA