Building LLMs from Scratch: Full Course by Sebastian Raschka

Sebastian Raschka, author of the book "Build a Large Language Model (From Scratch)", has released an extended course on building an LLM from scratch, consisting of several videos totaling about 15 hours. The course begins with setting up the development environment using uv, then covers working with text data, including tokenization, Byte Pair Encoding (BPE), and data loaders. It then explains coding attention mechanisms (self-attention, causal attention, multi-head attention) from scratch, followed by demonstrating the construction of the LLM architecture. The course also shows pretraining the model on unlabeled data, fine-tuning for classification (using spam as an example), and instruction finetuning. As a bonus for paid subscribers, a non-technical video lasting 2.5 hours is available, discussing the LLM landscape in 2025 and changes since 2018. Raschka notes that building an LLM from scratch is the best way to understand how they work, drawing an analogy to assembling a race car starting from a go-kart. The videos were originally created as supplementary material for the book but also function as standalone content.