This September 2025 paper introduces "Behavioral Fingerprinting," a novel framework designed to evaluate Large Language Models (LLMs) beyond traditional performance scores like MMLU. It aims to understand how models "think," creating a multi-faceted profile of their intrinsic cognitive and interactive styles. The methodology employs a diagnostic prompt suite and an automated evaluation pipeline where a powerful LLM acts as a judge, analyzing eighteen different models across four key dimensions: internal world model, reasoning abilities, biases and personality (including sycophancy), and semantic robustness. Findings indicate a convergence in core reasoning abilities among top models but a significant divergence in alignment-related behaviors such as sycophancy and robustness, which are influenced by specific developer strategies. The framework also identifies default personality clustering (ISTJ/ESTJ types), reflecting common training paradigms that reward logical, structured, and decisive responses.

Source:

https://arxiv.org/pdf/2509.04504