Assessing Consistency of Behavioral Tendencies in Large Language Models

Google researchers introduced a systematic framework for evaluating behavioral tendencies of large language models (LLMs), transforming standard psychological questionnaires into large-scale situational judgment tests (SJTs). The study used validated instruments such as the Interpersonal Reactivity Index (IRI) for empathy and the Emotion Regulation Questionnaire (ERQ). Models were tested on realistic scenarios involving professional behavior, conflict resolution, and everyday tasks. Analysis of 25 LLMs revealed two types of discrepancies: deviation from human consensus when annotator agreement was high, and insufficient reflection of opinion diversity when consensus was low. Small models (under 25 billion parameters) showed significantly worse alignment, often failing to distinguish appropriate expression or suppression of traits. Large models (over 120 billion parameters) achieved near-perfect alignment under unanimous consensus, but their performance plateaued at 80% when consensus was below 90%. Qualitative analysis showed that models tend to encourage emotional openness in professional situations where humans recommend restraint, and prioritize harmony over asserting positions in social disputes. A systematic overconfidence was also found: all 25 models failed to reduce confidence when human agreement was low, violating the principle of distributional pluralism. The framework also revealed discrepancies between models' self-reports and actual behavior: for example, models often report low impulsivity but tend to act impulsively in practice.