A study on similarity calculation method for API invocation sequences

Abstract

Malware variants have been developed and spread in the Internet, and the number of new malware variants is increases every year. Recently, malware is applied with obfuscation and mutation techniques to hide its existence, and malware variants are developed with various automatic tools that transform the properties of existing malware to avoid static analysis based malware detection systems. It is difficult to detect such obfuscated malware with static-based signatures, so we have designed a detection system based on dynamic analysis. In this paper, we propose a dynamic analysis based system that uses the API invocation sequences to compare behaviors of suspicious software with behaviors of existing malware.