Building Your Own Warhammer 40K GPT-like LLM from Scratch. Part 4: Supervised Fine-Tuning on Question-Answer
Hugging Face
The fourth part of a series on creating a decoder-only LLM from scratch, focusing on supervised fine-tuning (SFT) on a question-answer dataset to enable dialogue. The author extends the tokenizer with special role tokens, builds a QA dataset, and fine-tunes the model, observing that while loss decreases, generation quality may degrade due to overfitting. The checkpoint is selected based on generation quality rather than minimal loss.
The article describes the SFT stage of training a small decoder-only LLM for Warhammer 40K lore. After pre-training, the base model can continue text but cannot answer questions. The author collects question-answer pairs from forums, articles, and synthetic data, then builds a ChatQADataset that masks loss only on assistant tokens. The tokenizer is extended with special tokens <|user|> and <|assistant|>. The model's embedding and output layers are resized to accommodate the new tokens. Fine-tuning uses a lower learning rate (1e-5) and runs for about 20 minutes. Although loss drops from 3.7 to 2.1, the author notes overfitting and recommends selecting checkpoints based on generation quality. A comparison shows the base model outputs dialogue-like text while the fine-tuned model attempts to answer the question.
- Сокращения
- SFT = Supervised Fine-Tuning — контролируемая донастройка
- LLM = Large Language Model — большая языковая модель
- QA = Question Answering — вопрос-ответ
- EOS = End of Sequence — конец последовательности
- PAD = Padding — заполнение
- LR = Learning Rate — скорость обучения
- SFT = Supervised Fine-Tuning — контролируемая донастройка
Source: Habr — хаб ИИ —
original