A State Space Compression Method Based on Multivariate Analysis for Reinforcement Learning in High-Dimensional Continuous State Spaces

Hideki SATOH  

Publication
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E89-A   No.8   pp.2181-2191
Publication Date: 2006/08/01
Online ISSN: 1745-1337
DOI: 10.1093/ietfec/e89-a.8.2181
Print ISSN: 0916-8508
Type of Manuscript: PAPER
Category: Nonlinear Problems
Keyword: 
reinforcement learning,  curse of dimensionality,  multivariate analysis,  approximation,  nonlinear,  

Full Text: PDF>>
Buy this Article




Summary: 
A state space compression method based on multivariate analysis was developed and applied to reinforcement learning for high-dimensional continuous state spaces. First, useful components in the state variables of an environment are extracted and meaningless ones are removed by using multiple regression analysis. Next, the state space of the environment is compressed by using principal component analysis so that only a few principal components can express the dynamics of the environment. Then, a basis of a feature space for function approximation is constructed based on orthonormal bases of the important principal components. A feature space is thus autonomously construct without preliminary knowledge of the environment, and the environment is effectively expressed in the feature space. An example synchronization problem for multiple logistic maps was solved using this method, demonstrating that it solves the curse of dimensionality and exhibits high performance without suffering from disturbance states.