Deep-Learning-Enhanced Entity Resolution for Cyber Threat Intelligence Using OSINT Sources

Abstract

Cyber Threat Intelligence (CTI) relies on open-source intelligence (OSINT) more as a means to detect, link and assign malicious cyber activity. Another ongoing difficulty with OSINT-based CTI is entity resolution, in which heterogeneous mentions of the same real-world entity, e.g. IP addresses, malware families, and threat actor alias, need to be correctly and reliably identified across heterogeneous and noisy data sources. This is specifically challenging as naming conventions are inconsistent and when one writes in more than one language, and when rivals intentionally misname it. Conventional, rule based, and shallow learning methods are not very strong in these conditions. This paper suggests an entity resolution framework of CTI based on deep-learning to produce context-aware representations of cyber threat entities of unstructured OSINT data using transformer-based architectures. This framework defines entity resolution as a similarity learning and clustering problem with the use of multilingual embeddings and graph-based entity linking. Curated OSINT data experimental testing shows that the proposed solution performs very well when comparing the obtained results with baseline techniques in terms of precision, recall, F1-score, and quality of clustering. The findings also demonstrate better resiliency to alias ambiguity and textual unstructured variations. On the whole, the paper has shown that entity resolution using transformers is a cornerstone to the right correlation of threats, campaign analysis, and automated pipelines of CTI.