Apple AI Research focuses on how LLMs can resolve references not only within conversational text but also about on-screen entities (such as buttons or text in an app) and background information (like an app running on a device). 

Traditionally, this problem has been approached by separating the tasks into different modules or using models specific to each type of reference. However, the authors propose a unified model that treats reference resolution as a language modeling problem, capable of handling various reference types effectively. The link to the research paper is https://arxiv.org/pdf/2403.20329.pdf

Apple researchers have unveiled a breakthrough AI system named ReALM, designed to enhance how technology interprets on-screen content, conversational cues, and active background tasks. This innovative system translates on-screen information into text, streamlining the process by eliminating the need for complex image recognition technology.