RL without TD Learning: New Divide and Conquer Algorithm
Berkeley Artificial Intelligence Research (BAIR)
Researchers propose Transitive RL (TRL), a new divide-and-conquer reinforcement learning algorithm that avoids temporal difference learning. TRL scales to long-horizon tasks by recursively splitting trajectories, achieving state-of-the-art results on challenging benchmarks without needing to tune the step parameter n.
A new reinforcement learning algorithm called Transitive RL (TRL) has been introduced, based on a divide-and-conquer paradigm instead of traditional temporal difference (TD) learning. TD learning suffers from error accumulation over long horizons, and while n-step TD reduces this linearly, it requires careful tuning of n. TRL reduces the number of Bellman recursions logarithmically by recursively splitting trajectories into equal-length segments and combining their values. Practically, TRL restricts subgoal search to dataset states and uses expectile regression to avoid overestimation. Evaluated on OGBench's hardest humanoidmaze and puzzle tasks (up to 3000 steps), TRL achieved the best performance among strong baselines and matched the best individually tuned n-step TD without requiring n tuning. The method currently assumes deterministic dynamics and is limited to goal-conditioned RL, but future work may extend to reward-based tasks and stochastic environments.
- Сокращения
- RL = Reinforcement Learning — обучение с подкреплением
- TD = Temporal Difference — временные разности
- MC = Monte Carlo — Монте-Карло
- TRL = Transitive RL — транзитивное обучение с подкреплением
Source: BAIR (Berkeley AI) —
original