PEVA: Whole-Body Motion Guided Ego-Centric Video Prediction

Researchers from BAIR (Berkeley Artificial Intelligence) have developed PEVA (Predicting Ego-centric Video from human Actions), a model that predicts the next frame of egocentric video based on past frames and a sequence of human actions specified through changes in 3D pose. The goal is to create a world model for embodied agents that accounts for complex, hierarchical whole-body control (48+ degrees of freedom) and an egocentric view. PEVA uses an autoregressive conditional diffusion transformer trained on the Nymeria dataset, which contains synchronized egocentric videos and body motion capture. The model encodes actions as vectors of dimension 48 (3 for body position and 45 for relative rotations of 15 upper body joints) and uses random temporal masking, sequence-level training, and action embeddings. At test time, PEVA generates frames autoregressively: given context frames and the current action, it predicts the next frame, which is added to the context while the oldest is removed. The model can generate atomic actions (hand and body movements), simulate counterfactual scenarios, and produce consistent 16-second videos. For planning, PEVA uses cross-entropy method to optimize action sequences, evaluating them by similarity to a target image (LPIPS metric). Quantitative results show that PEVA outperforms baseline models in perceptual metrics (FID) and scales well with model size. Limitations include the lack of explicit task goal awareness and the need for closed-loop control.