Software Abnormal Behavior Detection Based on Function Semantic Tree

Yingxu LAI  Wenwen ZHANG  Zhen YANG  

IEICE TRANSACTIONS on Information and Systems   Vol.E98-D   No.10   pp.1777-1787
Publication Date: 2015/10/01
Publicized: 2015/07/03
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2015EDP7098
Type of Manuscript: PAPER
Category: Software System
software behavior,  system call,  state graph,  semantic analysis,  deviation density,  function semantic rules,  

Full Text: PDF(1.1MB)>>
Buy this Article

Current software behavior models lack the ability to conduct semantic analysis. We propose a new model to detect abnormal behaviors based on a function semantic tree. First, a software behavior model in terms of state graph and software function is developed. Next, anomaly detection based on the model is conducted in two main steps: calculating deviation density of suspicious behaviors by comparison with state graph and detecting function sequence by function semantic rules. Deviation density can well detect control flow attacks by a deviation factor and a period division. In addition, with the help of semantic analysis, function semantic rules can accurately detect application layer attacks that fail in traditional approaches. Finally, a case study of RSS software illustrates how our approach works. Case study and a contrast experiment have shown that our model has strong expressivity and detection ability, which outperforms traditional behavior models.