Using LLM Embeddings with Similarity Search for Botnet TLS Certificate Detection

Modern botnets leverage TLS encryption to mask C&C server communications. TLS certificates used by botnets could exhibit subtle characteristics that facilitate detection. In this paper we investigate whether text features from TLS certificates can be represented by open-source and 3rd party vendor LLM text embeddings in a projected vector space, for the purpose of building a classifier to detect botnet certificates. Our method extracts informative features, generating vector representations for effective identification, creating a projected space that can be queried with test certificates via similarity search. Using a balanced dataset consisting of the publicly available SSLBL botnet certificates and TLS certificates used by popular websites, our evaluations show that C-BERT, an open-source model, emerges as the preferred choice within our proposed system rather than a vendor solution. C-BERT achieves a competitive F1 score of 0.994 on unseen test data, 97.9% accuracy on data gathered several months after an initial projected embedding space was created, and maintains performance in a simulated zero-day evaluation against four C&C groups, with an average F1 score of 0.946. Further evaluation on a random sample of 150,000 real-world certificates collected from a full internet scan between Jan 2024 to May 2024 predicts 13 potential botnet certificates, among which one was confirmed to be malicious by VirusTotal. Comparing with the scenario where no such tool exists, we randomly selected 1,300 certificates from these 150,000 certificates and ran them through VirusTotal, and none were confirmed to be malicious. This translates to 100 fold effort reduction in identifying botnet certificates in the wild.