A Short Introduction To QV-learning - Hado van Hasselt

A Short Introduction To Some Reinforcement Learning Algorithms

By Hado van Hasselt

QV-learning

Previous -- Up -- Next

QV-learning is a natural extension of Q-learning and Sarsa to the case where we also use state values. Its equation is:

$Q_{t+1}(s_t,a_t) \overset{\alpha_t}{\longleftarrow} r_t + \gamma V_t(s_{t+1})$

Neutral characteristics

It is on-policy.
It learns state-action values (Q values).

Advantages

Using state values decreases the variance compared to Q-learning and Sarsa.
Using state values often speeds up learning.
State values are easily extendable to eligibility traces.

Disadvantages

Cannot handle continuous action spaces.

Algorithm

The QV-learning algorithm in schematic form:

QV-learning algorithm

Comparing this algorithm to Q-learning we see that the state value takes the place of the value of the highest valued action. Amongst other things, this makes the algorithm on-policy and it makes it easier to use eligibility traces.

Selected relevant publications:

M.A. Wiering. QV(lambda)-learning: A New On-policy Reinforcement Learning Algorithm. Proceedings of the 7th European Workshop on Reinforcement Learning, D. Leone (editor), pages 17-18, 2005.

M.A. Wiering and H. van Hasselt. Ensemble Algorithms in Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B, Volume 38, 4, 930-936, 2008.

M.A. Wiering and H. van Hasselt. The QV Family Compared to Other Reinforcement Learning Algorithms. Proceedings of IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL), Nashville, USA, 2009.

Quick links:

Previous -- Up -- Next

Notation
Using only state-action values:

Also using state values:

Using continuous actions:

Cacla

Contact

My contact data can be found here.