The research paper "Word Meanings in Transformer Language Models" by Jumbly and Peter Grindrod investigates how transformer-based large language models (LLMs) represent word meanings. Specifically, the authors explore whether these models possess a "lexical store" where words inherently hold semantic information, as opposed to meaning being solely derived from context. Through two studies involving clustering token embeddings from the RoBERTa-base model, they found strong evidence that semantic, morphological, syntactic, and even "worldly" information is encoded within these static embeddings. This challenges the "meaning eliminativist" hypothesis, suggesting that LLMs do rely on a form of invariant semantic knowledge for individual words as part of their text comprehension process.

https://arxiv.org/pdf/2508.12863