Reinforcement Learning with Orthonormal Basis Adaptation Based on Activity-Oriented Index Allocation

Hideki SATOH  

Publication
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E91-A   No.4   pp.1169-1176
Publication Date: 2008/04/01
Online ISSN: 1745-1337
DOI: 10.1093/ietfec/e91-a.4.1169
Print ISSN: 0916-8508
Type of Manuscript: PAPER
Category: Nonlinear Problems
Keyword: 
orthonormal basis,  function approximation,  non-linear,  reinforcement learning,  activity,  

Full Text: PDF>>
Buy this Article




Summary: 
An orthonormal basis adaptation method for function approximation was developed and applied to reinforcement learning with multi-dimensional continuous state space. First, a basis used for linear function approximation of a control function is set to an orthonormal basis. Next, basis elements with small activities are replaced with other candidate elements as learning progresses. As this replacement is repeated, the number of basis elements with large activities increases. Example chaos control problems for multiple logistic maps were solved, demonstrating that the method for adapting an orthonormal basis can modify a basis while holding the orthonormality in accordance with changes in the environment to improve the performance of reinforcement learning and to eliminate the adverse effects of redundant noisy states.