Bellman optimality equation

Defines optimal action value recursively through one observed transition and the best possible continuation.

Optimal Q of state s action a equals the expected next reward plus gamma times the maximum over next actions a prime of optimal Q at the next state and action, given the current state and action.

The value of acting now equals the average immediate reward plus the discounted value of choosing perfectly afterward.

Expectation is outermost, with an inner maximum expressing optimal choice after the environment reveals the next state.

This is a fixed-point definition. Value iteration and Q-learning turn it into computational update rules.