How to prevent AI psychologists from sycophancy and hallucinations: an engineering breakdown of safety safeguards
Character.AI
The article examines why general-purpose LLMs are dangerous in mental health roles due to sycophancy, hallucinations, and crisis blindness. It breaks down the safety – obvyazka (perimeter safeguards like disclaimers, crisis protocols, validated scales) and highlights a multi-agent architecture (e.g., Vera) where a second AI supervisor and academic reviewer check every response. The trade‑off is higher cost and latency for content‑level review, and the author asks where the balance between safety and cost should lie.
The article argues that general-purpose LLMs are unfit for psychotherapy because of three innate flaws: sycophancy (models trained with RLHF are biased toward agreeing with the user), hallucinations (confident generation of false facts), and crisis oversight (models often fail to recognize suicidal intent, as in the 2024 Florida teen case involving Character.AI). The industry response is safety – obvyazka – external mechanisms such as disclaimers, crisis hotlines, validated scales (PHQ‑9, GAD‑7, Beck scales, HADS), crisis questionnaires (SCI‑2, SIS), and privacy measures. However, these all act on the perimeter and do not inspect the actual response content. A more advanced approach, implemented by the service Vera, uses a multi-agent system: an AI‑psychologist generates the reply, an AI‑supervisor reviews tone and safety, and an AI‑academic checks recommendations against evidence. This second loop catches sycophancy and fabricated advice but multiplies API calls, raising cost and latency. Among seven Russian‑language AI‑therapy services surveyed, only Vera claims such a second review; others rely on perimeter safeguards alone. The author asks where the reasonable boundary lies between cheap perimeter obvyazka and expensive content verification, suggesting selective checks might offer a middle ground.
- Сокращения
- RLHF = Reinforcement Learning from Human Feedback — обучение с подкреплением на основе обратной связи от человека
- LLM = Large Language Model — большая языковая модель
- PHQ-9 = Patient Health Questionnaire‑9 — опросник здоровья пациента (депрессия)
- GAD-7 = Generalized Anxiety Disorder‑7 — генерализованное тревожное расстройство (опросник)
- HADS = Hospital Anxiety and Depression Scale — госпитальная шкала тревоги и депрессии
- SCI-2 = Suicide Concern Inventory‑2 — инвентарь суицидальных опасений‑2
- SIS = Suicide Intent Scale — шкала суицидальных намерений
- KPT = Cognitive Behavioral Therapy (CBT) — когнитивно-поведенческая терапия
- API = Application Programming Interface — интерфейс прикладного программирования
Source: Habr — хаб ИИ —
original