Identifying Symmetric-Key Algorithms Using CNN in Intel Processor Trace

Abstract

Malware and ransomware are often encrypted to protect their own code, making it challenging to apply reverse engineering to analyze them. Recently, various studies have been underway to identify cryptography algorithms in malware or ransomware that use anti-reversing technology via deep-learning technology. In particular, CNNs (convolution neural networks) are deep-learning algorithms with superior performance, as compared to existing machine-learning algorithms in image classification. In the cases of malicious files to which anti-debugging techniques or anti-DBI (dynamic binary instrumentation) techniques are applied, if the traces are extracted using various debuggers or DBI, the traces are cut off due to these techniques. The IPT (Intel processor trace) has the advantage of extracting an accurate trace of a program by bypassing the anti-debugging or anti-DBI technique. This paper presents a novel method by which to identify the symmetric-key algorithms by applying a CNN to the traces extracted from the IPT. The IPT minimally interrupts software execution. First, the trace encrypted by the symmetric-key algorithms is extracted using the IPT. Then it is converted into an image to be an input into the CNN. The experiments were carried out with two different datasets. The first dataset contained traces extracted by different types of symmetric-key algorithms, and the training results were classified into nine classes with 100% accuracy. The second dataset contained traces that included the various bit sizes of the security keys and the block-cipher modes for each type of symmetric-key algorithm. Training results were classified into 36 classes with an accuracy of 70.55%. While previous studies have identified the types of encryption algorithms, this study employed a CNN to identify the number of key bits and the block-cipher modes as well.