One RTX 5090, Prices from 1₽ to 20,726₽ per Million Tokens, and Why API Is Still Cheaper
NVIDIA
An ML CTO breaks down the real cost of running your own RTX 5090 for AI inference, showing that the price per million tokens varies from 1 ruble to over 20,000 rubles depending on model size, quantization, and concurrent load. The article argues that despite these numbers, API services often remain cheaper for most teams due to the breakeven point being high.
The author, an ML CTO, addresses the recurring debate about buying a GPU for self-hosted inference versus paying for cloud APIs. They calculate the cost of running an RTX 5090 (32GB) in Russia, including amortization, electricity, and system costs, arriving at about 17 rubles per hour or 12,435 rubles per month for a 24/7 operation. The article shows that the price per million tokens on the same card varies wildly from 1 ruble to 20,726 rubles depending on the model and workload. They compare various GPUs by price per VRAM gigabyte, noting that the RTX 3090 is far cheaper per GB, and discuss multi-GPU builds, which are often inefficient due to PCIe bottlenecks. Key performance measurements in llama.cpp show token generation speeds ranging from 61 to 234 tokens per second for different models, while vLLM with continuous batching on one 5090 can achieve about 4,570 tokens per second aggregate across hundreds of concurrent requests. Despite this, the breakeven point against seven API services is high, often exceeding realistic monthly token usage, leading the author to conclude that APIs are usually more cost-effective for most teams.
- Abbreviations
- CTO = Chief Technology Officer — технический директор
- ML = Machine Learning — машинное обучение
- API = Application Programming Interface — программный интерфейс приложения
- RTX = Ray Tracing Texel eXtreme — линейка видеокарт NVIDIA
- TDP = Thermal Design Power — расчетная тепловая мощность
- GB = Gigabyte — гигабайт
- NVMe = Non-Volatile Memory Express — протокол доступа к твердотельным накопителям
- MSRP = Manufacturer's Suggested Retail Price — рекомендованная розничная цена
- GDDR7 = Graphics Double Data Rate 7 — тип видеопамяти
- HBM = High Bandwidth Memory — высокопроизводительная память
- VRAM = Video Random Access Memory — видеопамять
- PCIe = Peripheral Component Interconnect Express — шина ввода-вывода
- NVLink = NVIDIA Link — интерфейс связи между GPU
- MoE = Mixture of Experts — смесь экспертов
- RAG = Retrieval-Augmented Generation — генерация с дополнением по поиску
- vLLM = Virtual Large Language Model — система инференса больших языковых моделей
- AWQ = Activation-aware Weight Quantization — квантование весов с учётом активаций
- W4A16 = Weights 4-bit, Activations 16-bit — квантование: веса 4 бита, активации 16 бит
- FP32 = Floating Point 32-bit — числа с плавающей точкой 32 бита
- FP16 = Floating Point 16-bit — числа с плавающей точкой 16 бит
- KV = Key-Value — ключ-значение
Source: Habr — хаб ИИ —
original