Study of Natural Language Processing for Collecting Cyber Threat Intelligence Using SyntaxNet

Abstract

The importance of cyberattack analysis has increased for responding quickly and effectively to cyber threats, which are becoming more intelligent. Analyzing cyberattacks requires examining the resources (malicious code, IP, domain, vulnerability, etc.) used in the cyberattack, similarity between the resources, attack technique, attack target, and activity time. It is also necessary to collect the data to be used in the analysis of a cyberattack. The formatted data shared through a specific format can be collected according to that format. However, it is difficult to collect the data when the cyberattack analyst generates the analysis result in unformatted data in the form of a report. As a way to solve this problem, this paper proposes the technique of using natural language processing technology to collect the Indicator of Compromise (IoC) in the form of a report. We have outlined the technologies and designed the processing procedure needed to extract resource data (IoC) abused in cyberattacks and the attack techniques (TTPs) included in the report based on the natural language processing model (SyntaxNet) disclosed as open source by Google. Extracting 345,364 token data based on 190 malware(and cyberattack) analysis reports and testing of them by dividing them into learning and test data in the ratio of about 9:1 resulted in extraction of IoC data at an average f1-score of 76%. © 2019, Springer Nature Switzerland AG.