Latent Variable Based Anomaly Detection in Network System Logs

Kazuki OTOMO  Satoru KOBAYASHI  Kensuke FUKUDA  Hiroshi ESAKI  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E102-D   No.9   pp.1644-1652
Publication Date: 2019/09/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2018OFP0007
Type of Manuscript: Special Section PAPER (Special Section on Log Data Usage Technology and Office Information Systems)
Category: Network Operation Support
Keyword: 
network operation,  system logs,  syslog,  anomaly detection,  latent variable analysis,  variational autoencoder,  

Full Text: PDF(1.3MB)>>
Buy this Article




Summary: 
System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.