Self-Distilled Reasoning for Amazon Nova Fine-Tuning
Amazon Web Services
Amazon explores self-distilled reasoning (SDR) to generate chain-of-thought traces for supervised fine-tuning (SFT) when such traces are missing. SDR reuses the base Amazon Nova 2 Lite model's own reasoning, mitigating catastrophic forgetting and improving target performance without human annotation.
When fine-tuning a model with supervised fine-tuning (SFT), creating high-quality chain-of-thought (CoT) reasoning traces is often impractical. Without reasoning traces, the model may lose its reasoning ability due to the reasoning suppression problem. Amazon proposes Self-Distilled Reasoning (SDR), which uses the base Amazon Nova 2 Lite model to generate reasoning traces for each SFT example, then prepends them to the original outputs and fine-tunes with reasoning mode on. SDR requires no human annotation, is applicable across domains, and preserves reasoning capability more effectively than model merging. Experiments on benchmarks like ToolACE, CoCoHD, GovReport, Invoice-OCR, and CaptionGen show SDR mitigates catastrophic forgetting, maintains math performance (70% vs 68% with model merging), and improves target performance by over 6.5% on average.
- Сокращения
- SFT = Supervised Fine-Tuning — обучение с учителем
- CoT = chain-of-thought — цепочка рассуждений
- SDR = Self-Distilled Reasoning — самодистиллированное рассуждение
- RFT = Reinforcement Fine-Tuning — обучение с подкреплением
- LoRA = Low-Rank Adaptation — низкоранговая адаптация
- RLHF = Reinforcement Learning from Human Feedback — обучение с подкреплением на основе человеческой обратной связи
Source: AWS ML blog —
original