Hybrid emulation for bypassing anti-reversing techniques and analyzing malware

Abstract

Malware uses a variety of anti-reverse engineering techniques, which makes its analysis difficult. Dynamic analysis tools, e.g., debuggers, DBI (Dynamic Binary Instrumentation), and CPU emulators, do not provide both accuracy and convenience when analyzing complex malware, which utilizes diverse anti-reversing techniques. Debuggers are convenient, but are easily detected by anti-debugging techniques. DBI tools are better for bypassing anti-reversing techniques than debuggers, but cannot execute complex programs correctly. Emulators are not designed for precise malware analysis. To address the problem fundamentally, we developed a new approach completely different from the previous works. We present a new dynamic analysis scheme for malware, which includes automatic detection and evasion of various anti-reversing techniques. This approach combines a CPU simulator and actual code execution, i.e., machine instructions are simulated with the CPU simulator, whereas API functions are directly executed when they are called. In this method, the CPU simulator can precisely execute code without modifying the code chunks for trampolines. Moreover, our method takes advantage of the OS functionalities, including thread management or interrupt handling. We conducted experiments on 16 widely used protectors, which show that our method outperforms conventional tools: Pin, DynamoRIO, Apate, and OllyAdvanced. Our scheme can unpack 15 protectors and bypass the anti-debugging techniques associated with them.