Integrating a Custom Warhammer 40K LLM with Hugging Face
Hugging Face
The article describes how to package a custom decoder-only LLM (LinguaLaboratoriumMechanicus) trained on Warhammer 40K lore into Hugging Face format. It explains creating a configuration class and a model wrapper inheriting from transformers.PretrainedConfig and transformers.PreTrainedModel, then saving and uploading to the Hub for easy use with AutoModelForCausalLM.
The author, Vladimir, is writing a series on building a small decoder-only LLM from scratch. In part 5, he integrates the model with Hugging Face by creating a configuration class (LinguaLaboratoriumMechanicusConfig) that stores hyperparameters and a model wrapper (LLMForCausalLM) that inherits from PreTrainedModel and GenerationMixin. The wrapper defines forward and prepare_inputs_for_generation methods, and the weights are converted from a PyTorch checkpoint. Finally, the model is saved with auto-class registration and can be loaded via AutoModelForCausalLM with the trust_remote_code flag.
- Abbreviations
- LLM = Large Language Model — большая языковая модель
- HF = Hugging Face — Hugging Face
Source: Habr — хаб ML —
original