Profiling in PyTorch (Part 3): Attention is all you profile
PyTorch
NVIDIA
This post profiles attention mechanisms in PyTorch, comparing naive attention, in-place optimization, and Scaled Dot Product Attention (SDPA) backends. It reveals that the math backend of SDPA is slower than naive attention due to Tensor Core underutilization, mask reconstruction, and safe softmax overhead.
The third post in the Profiling in PyTorch series profiles attention mechanisms. It starts with a naive attention module (matmul, scaling, masking, softmax, matmul) and its trace shows 5 GPU kernels per forward. Replacing out-of-place masked_fill with in-place masked_fill_ eliminates a memory copy kernel, saving time and memory. Next, Scaled Dot Product Attention (SDPA) is introduced; its math backend, while safer (handling NaN cases) and more precise (FP32), is 3.7x slower than naive attention because it launches 20 GPU kernels, does not use Tensor Cores (uses sgemm instead of bf16 Tensor Core matmul), rebuilds the causal mask each call, and uses _safe_softmax. The post notes that SDPA normally selects the fastest backend automatically.
- Сокращения
- GPU = Graphics Processing Unit — графический процессор
- CUDA = Compute Unified Device Architecture — архитектура параллельных вычислений от NVIDIA
- SDPA = Scaled Dot Product Attention — масштабированное скалярное произведение внимания
- SM = Streaming Multiprocessor — потоковый мультипроцессор
- FP32 = Floating Point 32-bit — 32-разрядная плавающая точка
Source: Hugging Face blog —
original