Google Accelerates Gemini Nano Models on Pixel with Frozen Multi-Token Prediction
Google/DeepMind
Google DeepMind
Google has introduced a method to retrofit Multi-Token Prediction (MTP) onto frozen production models like Gemini Nano v3, enabling faster on-device inference on Pixel 9 and 10 devices. The MTP head attaches to the main model's final layers, leveraging its hidden states and KV cache to generate multiple tokens per inference pass without separate drafting models, achieving speedups of 50% or more and reducing memory consumption by 130MB per instance.
Google Research has announced a new architecture for accelerating on-device language models such as Gemini Nano v3 on Pixel smartphones. The approach retrofits Multi-Token Prediction (MTP) onto frozen base models by appending a lightweight Transformer head to the final layers. This MTP head directly uses the main model's existing key-value cache via cross-attention, eliminating the need for a standalone drafter model and reducing memory overhead by 130MB per instance. In production tasks like AI Notification Summaries and Proofread, MTP produces nearly two additional tokens per inference pass, leading to speed improvements of over 50% on Pixel 9 devices. The final output is bit-identical to the base model, ensuring safety alignment and backward compatibility. The method has been rolled out to Pixel 9 and 10 series, and Google plans to explore parallel decoding and verification leniency for further efficiency gains.
- Сокращения
- MTP = Multi-Token Prediction — многотокенное предсказание
- LLM = Large Language Model — большая языковая модель
- RAM = Random Access Memory — оперативная память
- KV = key-value — ключ-значение
Source: Google Research —
original