Efficient Automatic Original Entry Point Detection

Abstract

Malware authors employ sophisticated anti-reverse engineering techniques such as packing, encryption, polymorphism, etc. For a packed file, when launched, the packed executable will reconstruct the code of the original program. The OEP (Original Entry Point) is the address indicating the beginning point of the original code. Previous work or conventional unpacking tools provide a relatively large set of OEP candidates and sometimes OEP is missing among candidates. In this paper, we present an efficient OEP detection scheme for x86 Windows environments. This scheme is designed to find exact one OEP by using three methods. First, we enhanced Isawa et al.'s work by examining branch instructions. Our second method is to track the system parameters relevant to the main function in stack memory to refine OEP candidates. Our third method is that we track the startup function calls to find the installation routine for exception handling. To evaluate feasibility, we implemented our algorithm and then conducted experiments on 16 commercial representative packers and 6 previous unpacking tools/schemes. Experimental results show that even though our scheme produces a single OEP candidate for each packed file, accuracy is the highest (up to 14 times higher than the previous work).