ConvApparel: Measuring and Closing the Realism Gap in LLM-Based User Simulators

Google Research has introduced ConvApparel, a new dataset and evaluation system for measuring the "realism gap" of large language model (LLM)-based user simulators. Modern conversational AI agents often struggle in long interactions, forgetting constraints or generating irrelevant responses. Training such systems requires continuous feedback, but using live users is expensive and difficult to scale. As an alternative, LLM-based user simulators are employed, but these suffer from a realism gap, exhibiting unnatural patience or encyclopedic knowledge. ConvApparel includes more than 4,000 conversations between humans and AI (roughly 15,000 turns) in the domain of clothing shopping. Data collection used two agents — a helpful one ("Good") and a deliberately unhelpful one ("Bad") — which made it possible to capture a wide range of human reactions. To evaluate simulators, the researchers developed a three-part system: population-level statistical tests, human-likeness assessment using an automatic discriminator, and counterfactual validation — a simulator trained on the good agent should react appropriately when faced with the bad one. Three tested simulators (a prompt-based one, an in-context-learning, or ICL, one, and a supervised fine-tuning, or SFT, one built on Gemini) showed that the ICL and SFT approaches significantly outperform the prompt-based one, though even the best SFT models produce noticeable artifacts. Counterfactual validation revealed that the prompt-based simulator fails to adapt, whereas the ICL and SFT simulators show realistic irritation when interacting with the bad agent. The researchers stress that building a fully realistic simulator remains an open problem, and that the proposed framework can measure the gap but does not determine the degree of realism actually required. ConvApparel has been released to the community for further research.