Four Main Approaches to Evaluating LLMs: From Benchmarks to Judge Models
This article provides an overview of four main methods for evaluating large language models (LLMs): multiple-choice evaluation (MMLU), verifiers, leaderboards, and LLM-as-a-judge. Each method is described in detail and illustrated with code examples based on the Qwen3 0.6B model. The material is based on an article by Sebastian Raschka.
Sebastian Raschka's article describes four main approaches to evaluating large language models (LLMs): multiple-choice evaluation (e.g., the MMLU benchmark), verifiers, leaderboards, and LLM-as-a-judge. Multiple-choice evaluation is a benchmark-oriented method where the model answers questions with answer choices, and accuracy is measured as the proportion of correct answers. The article demonstrates an implementation of this method using the Qwen3 0.6B model and PyTorch code: a pre-trained model is loaded, a prompt with the question and options is constructed, an answer is generated, and it is compared to the correct one. For the example from the high_school_mathematics subset of MMLU, the model gave an incorrect answer. It is noted that in practice, methods based on log probabilities are also used, and different approaches are applied for reasoning models. Verifiers, leaderboards, and LLM-as-a-judge are discussed in detail in the author's book 'Build a Reasoning Model (From Scratch).'
Source: Sebastian Raschka —
original