Asynchronous Methods for Deep Reinforcement Learning

A3C replaced one agent's replay memory with many fresh, out-of-phase actor-learners updating a shared model. Follow how parallel experience decorrelates training data, how short n-step rollouts update both policy and value, and why later synchronous A2C changed the verdict on asynchrony itself.

Reading focus: Why independent actors can perform the stabilizing, decorrelating role that experience replay played in DQN. How an A3C worker turns a short rollout into an advantage estimate and updates shared policy and value parameters without a synchronization barrier. What the 57-game Atari and thread-scaling results establish, which comparison caveat matters, and why A2C later separated parallelism from asynchrony.

ICML 2016. Volodymyr Mnih et al.. 13 min read, hard difficulty.