ROPminer: Learning-Based Static Detection of ROP Chain Considering Linkability of ROP Gadgets

Toshinori USUI
Tomonori IKUSE

IEICE TRANSACTIONS on Information and Systems   Vol.E103-D    No.7    pp.1476-1492
Publication Date: 2020/07/01
Publicized: 2020/04/07
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2019ICP0016
Type of Manuscript: Special Section PAPER (Special Section on Information and Communication System Security)
Category: Network and System Security
return-oriented programming,  static detection,  machine learning,  

Full Text: PDF(2.3MB)>>
Buy this Article

Return-oriented programming (ROP) has been crucial for attackers to evade the security mechanisms of recent operating systems. Although existing ROP detection approaches mainly focus on host-based intrusion detection systems (HIDSes), network-based intrusion detection systems (NIDSes) are also desired to protect various hosts including IoT devices on the network. However, existing approaches are not enough for network-level protection due to two problems: (1) Dynamic approaches take the time with second- or minute-order on average for inspection. For applying to NIDSes, millisecond-order is required to achieve near real time detection. (2) Static approaches generate false positives because they use heuristic patterns. For applying to NIDSes, false positives should be minimized to suppress false alarms. In this paper, we propose a method for statically detecting ROP chains in malicious data by learning the target libraries (i.e., the libraries that are used for ROP gadgets). Our method accelerates its inspection by exhaustively collecting feasible ROP gadgets in the target libraries and learning them separated from the inspection step. In addition, we reduce false positives inevitable for existing static inspection by statically verifying whether a suspicious byte sequence can link properly when they are executed as a ROP chain. Experimental results showed that our method has achieved millisecond-order ROP chain detection with high precision.

open access publishing via