Natural Gradient Descent of Complex-Valued Neural Networks Invariant under Rotations

Jun-ichi MUKUNO  Hajime MATSUI  

Publication
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E102-A   No.12   pp.1988-1996
Publication Date: 2019/12/01
Online ISSN: 1745-1337
DOI: 10.1587/transfun.E102.A.1988
Type of Manuscript: PAPER
Category: Neural Networks and Bioengineering
Keyword: 
complex number,  Fisher information matrix,  projected natural gradient,  data augmentation,  online character recognition,  

Full Text: PDF(1011.9KB)>>
Buy this Article




Summary: 
The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.