Deploying Kimi K3 on AWS
Moonshot AI
NVIDIA
Moonshot AI released Kimi K3, a 2.8 trillion parameter MoE model, open-weight. The AWS blog post describes deploying it on SageMaker HyperPod or Amazon EKS using p6-b300 instances with NVIDIA B300 Blackwell Ultra GPUs, vLLM serving, and MXFP4 quantization.
On July 27, 2026, Moonshot AI released Kimi K3, a 2.8 trillion parameter Mixture of Experts (MoE) model with open weights. It uses architecture innovations like Kimi Delta Attention and Gated Multi Head Latent Attention, activating 16 of 896 experts per token (104B active parameters). The model supports a 1 million token context window, native multimodal input (text+vision), tool calling, and structured output. AWS offers two deployment options: Amazon SageMaker HyperPod with Inference Operator, which abstracts orchestration, and a self-managed Amazon EKS cluster using EC2 Capacity Blocks. Both require p6-b300.48xlarge instances (8 NVIDIA B300 Blackwell Ultra GPUs). The weights are available in MXFP4 format on Hugging Face, and vLLM is the recommended serving engine. The endpoint exposes an OpenAI-compatible API.
- Сокращения
- MoE = Mixture of Experts — Смесь экспертов
- GPU = Graphics Processing Unit — Графический процессор
- EKS = Amazon Elastic Kubernetes Service — Amazon Elastic Kubernetes Service
- KDA = Kimi Delta Attention — Kimi Delta Attention
- MLA = Multi Head Latent Attention — Многоголовое латентное внимание
- API = Application Programming Interface — Интерфейс программирования приложений
- IAM = Identity and Access Management — Управление идентификацией и доступом
- VPC = Virtual Private Cloud — Виртуальное частное облако
- EC2 = Amazon Elastic Compute Cloud — Amazon Elastic Compute Cloud
- MXFP4 = Microscaling Floating Point 4-bit — Микро-масштабируемый формат с плавающей точкой 4 бита
- S3 = Amazon Simple Storage Service — Amazon Simple Storage Service
- FTP = Flexible Training Plan — Гибкий план обучения
- Helm = Helm (package manager for Kubernetes) — Helm (менеджер пакетов для Kubernetes)
- Terraform = Terraform (infrastructure as code tool) — Terraform (инструмент инфраструктуры как код)
Source: AWS ML blog —
original