Deep Reinforcement Learning with Sarsa and Q-Learning: A Hybrid Approach

Zhi-xiong XU  Lei CAO  Xi-liang CHEN  Chen-xi LI  Yong-liang ZHANG  Jun LAI  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E101-D   No.9   pp.2315-2322
Publication Date: 2018/09/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2017EDP7278
Type of Manuscript: PAPER
Category: Artificial Intelligence, Data Mining
Keyword: 
deep reinforcement learning,  Deep Q Network,  overestimation,  double estimator,  Sarsa,  

Full Text: PDF(1.7MB)
>>Buy this Article


Summary: 
The commonly used Deep Q Networks is known to overestimate action values under certain conditions. It's also proved that overestimations do harm to performance, which might cause instability and divergence of learning. In this paper, we present the Deep Sarsa and Q Networks (DSQN) algorithm, which can considered as an enhancement to the Deep Q Networks algorithm. First, DSQN algorithm takes advantage of the experience replay and target network techniques in Deep Q Networks to improve the stability of neural networks. Second, double estimator is utilized for Q-learning to reduce overestimations. Especially, we introduce Sarsa learning to Deep Q Networks for removing overestimations further. Finally, DSQN algorithm is evaluated on cart-pole balancing, mountain car and lunarlander control task from the OpenAI Gym. The empirical evaluation results show that the proposed method leads to reduced overestimations, more stable learning process and improved performance.