Honeypot's Best Friend? Investigating ChatGPT's Ability to Evaluate Honeypot Logs

Honeypots can gather substantial data from intruders, but many honeypots lack the necessary features to analyse and explain the nature of these potential attacks. Typically, honeypot analysis reports only highlight the attacking IP addresses and the malicious requests. As such, analysts might miss out on the more useful insights that can be derived from the honeypot data, such as the attackers’ plan or emerging threats. Meanwhile, recent advances in large language models (LLM) – such as ChatGPT – have opened up the possibility of using artificial intelligence (AI) to comprehend honeypot data better, for instance, to perform an automated and intelligent log analysis that can explain consequences, provide labels, and deal with obfuscation. In this study, we probed ChatGPT’s proficiency in understanding and explaining honeypot logs from actual recorded attacks on our honeypots. Our data encompassed 627 requests to Elasticsearch honeypots and 73 attacks detected by SSH honeypots, collected over a two-week period. Our analysis was focused on evaluating ChatGPT’s explanation ability regarding the potential consequences of each attack, in alignment with the MITRE ATT&CK Framework, and whether ChatGPT can identify any obfuscation techniques that might be used by attackers. We found that ChatGPT achieved a 96.65% accuracy in correctly explaining the consequences of the attack targeting Elasticsearch servers. Furthermore, ChatGPT achieved a 72.46% accuracy in matching a given attack to one or more techniques listed by the MITRE ATT&CK Framework. Similarly, ChatGPT was excellent in identifying obfuscation techniques employed by attackers and offering deobfuscation solutions. However, 30.46% of the request body and 7.5% of the targeted URI were falsely identified as obfuscated, leading to a very high score of false positive for obfuscation. With the SSH honeypot data, we achieved a 97.26% accuracy while explaining the consequences of the attacks and a 98.84% accuracy for correctly mapping to MITRE ATT&CK Framework techniques. Based on these results, we can say that ChatGPT has shown great potential for automating the process of analysing honeypot data. Its proficiency in explaining attack consequences and in managing obfuscation through implementing MITRE ATT&CK techniques is impressive. Nevertheless, it is essential to be mindful of the possibility of high false positive rates, which can cause some issues. This needs to be addressed in future research, for example by leveraging the advanced fine-tuning techniques that were recently introduced to ChatGPT, but not available at the time of writing of this paper.