Hado van Hasselt
| Home | My publications | Source code | Dissertation |
July 1, 2011
You can now download the chapter on reinforcement learning in continuous spaces that I wrote for the upcoming "Reinforcement Learning: State of the Art" book from my publications. This chapter surveys the available literature on learning in MDPs with continuous states and/or continuous actions and presents some new results.
In addition to a PDF, the chapter is also available in HTML, to be viewed online. (Follow that link to view the abstract.)
May 13, 2011
Our paper on Best-Match equations was accepted for publication at JMLR. The abstract is given below and a preprint of the paper can be found in my publications.
Exploiting Best-Match Equations for Efficient Reinforcement Learning
by Harm van Seijen, Shimon Whiteson, Hado van Hasselt and Marco Wiering
Abstract This article presents and evaluates best-match learning, a new approach to reinforcement learning that trades off the sample efficiency of model-based methods with the space efficiency of model-free methods. Best-match learning works by approximating the solution to a set of best-match equations, which combine a sparse model with a model-free Q-value function constructed from samples not used by the model. We prove that, unlike regular sparse model-based methods, best-match learning is guaranteed to converge to the optimal Q-values in the tabular case. Empirical results demonstrate that best-match learning can substantially outperform regular sparse model-based methods, as well as several model- free methods that strive to improve the sample efficiency of temporal-difference methods. In addition, we demonstrate that best-match learning can be successfully combined with function approximation.
December 26, 2010
On January 17, 2011, I will defend my dissertations "Insights in Reinforcement Learning".
September 9, 2010
My paper on Double Q-learning was accepted at NIPS. Update: download the poster presentation.
August 1, 2010
As of today, I am employed as a project member at the CWI (the Centrum Wiskunde & Informatica, which translates to the center for mathematics and computer science).
September 21, 2009
Small bugs have been fixed on the implementations. Please use the newest version of the code.
August 4, 2009
The implementations are now online.
July 28, 2009
I decided to restructure the implementation to make the code more readable and extendable. Specifically, any algorithm will be able to handle discrete and continuous state spaces. Also, there will be support for MDPs that use a discrete or a continuous state space in combination with a discrete or continuous action space. Hopefully, this will make it quite easy to implement a problem as an MDP and run any (or all) of the algorithms on it. Also, it should be easy to include new algorithms, as long as they adhere to some simple interface restrictions.
The implementation should be ready in the next few days. I will then shortly test it and put it online, so expect more updates in the near future.
July 14, 2009
Some small updates to A Short Introduction To Some Reinforcement Learning Algorithms, including - most notably - schematic representations of each algorithm.
July 9, 2009
I have finished the first version for a page on different reinforcement learning algorithms. The implementations will follow in the next couple of weeks.
A Short Introduction To Some Reinforcement Learning Algorithms
June 18, 2009
I will be putting some of my software online in the next couple of weeks. This software will include the following:
- An implementation of various reinforcement learning algorithms, including:
- Q-learning
- Sarsa
- QV-learning
- Acla (Actor Critic Learning Automaton)
- Cacla (Continuous Actor Critic Learning Automaton)
- Examples of use of these algorithms, such as:
- Problems with discrete states and actions that are solved with tabular versions of the algorithms
- Problems where neural networks are used as function approximators on problems with larger or continuous state spaces
- Problems with continuous states and continuous actions, where neural networks are used as the building blocks of the Cacla algorithms
- A neural network implementation to be used as a function approximator for use with the aforementioned algorithms
The algorithms will be available in both python and C++. The neural network implementation is in C++. I can give no guarantees on the correct and bug-free status of the software, but it has been tested to some extent and is available for all non-commercial purposes.
I will also put some documentation on the usage of the software online, but in case you have any questions, please contact me. Also, I am interested to know if you intend to use my software, so if this is the case, please let me know.
Contact
My contact data can be found at CWI.