  This paper presents HeNet, a hierarchical ensemble neural network, applied to
classify hardware-generated control flow traces for malware detection. Deep
learning-based malware detection has so far focused on analyzing executable
files and runtime API calls. Static code analysis approaches face challenges
due to obfuscated code and adversarial perturbations. Behavioral data collected
during execution is more difficult to obfuscate but recent research has shown
successful attacks against API call based malware classifiers. We investigate
control flow based characterization of a program execution to build robust deep
learning malware classifiers. HeNet consists of a low-level behavior model and
a top-level ensemble model. The low-level model is a per-application behavior
model, trained via transfer learning on a time-series of images generated from
control flow trace of an execution. We use Intel$^\circledR$ Processor Trace
enabled processor for low overhead execution tracing and design a lightweight
image conversion and segmentation of the control flow trace. The top-level
ensemble model aggregates the behavior classification of all the trace segments
and detects an attack. The use of hardware trace adds portability to our system
and the use of deep learning eliminates the manual effort of feature
engineering. We evaluate HeNet against real-world exploitations of PDF readers.
HeNet achieves 100\% accuracy and 0\% false positive on test set, and higher
classification accuracy compared to classical machine learning algorithms.

Cryptography and Security;Machine Learning