Car-GPT: Could LLMs finally make self-driving cars happen?
OpenAI
Meta
Wayve
xAI
The article explores the potential of Large Language Models (LLMs) to revolutionize autonomous driving, drawing a parallel to Alexander Fleming's accidental discovery of penicillin. It explains how LLMs, through tokenization, transformers, and next-word prediction, can be adapted for self-driving tasks such as perception, planning, and generation. However, trust and black-box issues remain unresolved, and it's too early to tell if LLMs will be the key to fully autonomous vehicles.
The article compares the current state of self-driving car research to the accidental discovery of penicillin, suggesting that LLMs might be the unexpected breakthrough. It explains that traditional autonomous driving uses a modular approach (perception, localization, planning, control), while end-to-end learning replaces all modules with a single neural network, creating a black-box problem. LLMs work via tokenization (converting text to numbers), transformers (attention-based architecture), and next-word prediction. For self-driving, inputs can be images or sensor data tokenized similarly, and tasks include perception (detection, prediction, tracking), planning (decision-making, navigation), and generation (creating training data or scenarios). Examples like Talk2BEV and DriveGPT use LLMs to enhance BEV perception and plan trajectories, while GAIA-1 and MagicDrive generate videos or scenes. However, concerns about hallucinations and lack of determinism make trust in LLMs for real-time driving uncertain, and it's too early to conclude.
- Сокращения
- LLM = Large Language Model — Large Language Model
- BEV = Bird's Eye View — вид сверху
- GPT = Generative Pre-trained Transformer — Generative Pre-trained Transformer
Source: The Gradient —
original