Summary
|
For Full-Text PDF, please login, if you are a member of IEICE, or go to Pay Per View on menu list, if you are a nonmember of IEICE. |
|
Latent Variable Based Anomaly Detection in Network System Logs Kazuki OTOMO Satoru KOBAYASHI Kensuke FUKUDA Hiroshi ESAKI Publication IEICE TRANSACTIONS on Information and Systems Vol.E102-D No.9 pp.1644-1652 Publication Date: 2019/09/01 Publicized: 2019/06/07 Online ISSN: 1745-1361 DOI: 10.1587/transinf.2018OFP0007 Type of Manuscript: Special Section PAPER (Special Section on Log Data Usage Technology and Office Information Systems) Category: Network Operation Support Keyword: network operation, system logs, syslog, anomaly detection, latent variable analysis, variational autoencoder, Full Text: PDF(1.3MB)>>
Summary: System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults. | ||||||||||
| ||||||||||