Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Enhanced Malicious Traffic Detection in Encrypted Communication Using TLS Features and a Multi-class Classifier Ensemble

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

The use of encryption for network communication leads to a significant challenge in identifying malicious traffic. The existing malicious traffic detection techniques fail to identify malicious traffic from the encrypted traffic without decryption. The current research focuses on feature extraction and malicious traffic classification from the encrypted network traffic without decryption. In this paper, we propose an ensemble model using Deep Learning (DL), Machine Learning (ML), and self-attention-based methods. Also, we propose novel TLS features extracted from the network and perform experimentation on the ensemble model. The experimental results demonstrated that the ML-based (RF, LGBM, XGB) ensemble model achieved a significant accuracy of 94.85% whereas the other ensemble model using RF, LSTM, and Bi-LSTM with self-attention technique achieved an accuracy of 96.71%. To evaluate the efficacy of our proposed models, we curated datasets encompassing both phishing, legitimate and malware websites, leveraging features extracted from TLS 1.2 and 1.3 traffic without decryption.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The dataset will be made available based on a reasonable request to the corresponding author.

Notes

  1. https://apwg.org/trendsreports/

  2. https://transparencyreport.google.com/https/overview?hl=en

  3. https://scikit-learn.org/stable/

  4. https://www.phishtank.com/developer

  5. https://mcfp.weebly.com/the-ctu-13-dataset

  6. https://www.alexa.com/topsites

References

  1. Rescorla, E.: SSL and TLS Designing ADN Building Secure Systems. Addison-Wesley, Reading (2002)

    Google Scholar 

  2. Velan, P., Čermák, M., Čeleda, P., Drašar, M.: A survey of methods for encrypted traffic classification and analysis. Int. J. Netw. Manag. 25, 355–374 (2015)

    Article  Google Scholar 

  3. Cao, Z., Xiong, G., Zhao, Y., Li, Z., Guo, L.: A survey on encrypted traffic classification. In: Applications and Techniques in Information Security: 5th International Conference, ATIS 2014, Melbourne, VIC, Australia, November 26–28, 2014. Proceedings 5, pp. 73–81 (2014)

  4. Chen, J., Lv, T., Cai, S., Song, L., Yin, S.: A novel detection model for abnormal network traffic based on bidirectional temporal convolutional network. Inf. Softw. Technol. 157, 107166 (2023)

    Article  Google Scholar 

  5. Kumar, M., Kondaiah, C., Pais, A., Rao, R.: Machine learning models for phishing detection from TLS traffic. Cluster Comput. 1–15 (2023)

  6. Zheng, J., Zeng, Z., Feng, T.: GCN-ETA: high-efficiency encrypted malicious traffic detection. Secur. Commun. Netw. 1–11 (2022)

  7. Weiss, K., Khoshgoftaar, T.: Detection of phishing webpages using heterogeneous transfer learning. In: 2017 IEEE 3rd International Conference on Collaboration and Internet Computing (CIC), pp. 190–197 (2017)

  8. Zhang, W., Jiang, Q., Chen, L., Li, C.: Two-stage ELM for phishing web pages detection using hybrid features. World Wide Web. 20, 797–813 (2017)

    Article  Google Scholar 

  9. Thabtah, F., Kamalov, F.: Phishing detection: a case analysis on classifiers with rules using machine learning. J. Inf. Knowl. Manag. 16, 1750034 (2017)

    Article  Google Scholar 

  10. Zhang, J., Li, X.: Phishing detection method based on borderline-smote deep belief network. In: Security, Privacy, and Anonymity in Computation, Communication, and Storage: SpaCCS 2017 International Workshops, Guangzhou, China, December 12–15, 2017, Proceedings 10, pp. 45–53 (2017)

  11. Rao, R., Vaishnavi, T., Pais, A.: CatchPhish: detection of phishing websites by inspecting URLs. J. Ambient Intell. Hum. Comput. 11, 813–825 (2020)

    Article  Google Scholar 

  12. Sonowal, G., Kuppusamy, K.: PhiDMA-A phishing detection model with multi-filter approach. J. King Saud Univ. Comput. Inf. Sci. 32, 99–112 (2020)

    Google Scholar 

  13. Zhu, E., Chen, Z., Cui, J., Zhong, H.: MOE/RF: a novel phishing detection model based on revised multiobjective evolution optimization algorithm and random forest. IEEE Trans. Netw. Serv. Manag. 19, 4461–4478 (2022)

    Article  Google Scholar 

  14. Marchal, S., François, J., State, R., Engel, T.: PhishStorm: detecting phishing with streaming analytics. IEEE Trans. Netw. Serv. Manag. 11, 458–471 (2014)

    Article  Google Scholar 

  15. Moghimi, M., Varjani, A.: New rule-based phishing detection method. Expert Syst. Appl. 53, 231–242 (2016)

    Article  Google Scholar 

  16. Rao, R., Pais, A.: Detection of phishing websites using an efficient feature-based machine learning framework. Neural Comput. Appl. 31, 3851–3873 (2019)

    Article  Google Scholar 

  17. Mohammad, R., Thabtah, F., McCluskey, L.: Predicting phishing websites based on self-structuring neural network. Neural Comput. Appl. 25, 443–458 (2014)

    Article  Google Scholar 

  18. Feng, J., Zhang, Y., Qiao, Y.: A detection method for phishing web page using DOM-based Doc2Vec model. J. Comput. Inf. Technol. 28, 19–31 (2020)

    Article  Google Scholar 

  19. Marchal, S., Saari, K., Singh, N., Asokan, N.: Know your phish: novel techniques for detecting phishing sites and their targets. In: 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS), pp. 323–333 (2016)

  20. Marchal, S., Armano, G., Gröndahl, T., Saari, K., Singh, N., Asokan, N.: Off-the-hook: an efficient and usable client-side phishing prevention application. IEEE Trans. Comput. 66, 1717–1733 (2017)

    Article  MathSciNet  Google Scholar 

  21. Rao, R., Pais, A.: An enhanced blacklist method to detect phishing websites. In: Information Systems Security: 13th International Conference, ICISS 2017, Mumbai, India, December 16–20, 2017, Proceedings 13, pp. 323–333 (2017)

  22. Cui, Q., Jourdan, G., Bochmann, G., Couturier, R., Onut, I.: Tracking phishing attacks over time. In: Proceedings of the 26th International Conference on World Wide Web, pp. 667–676 (2017)

  23. Tanaka, S., Matsunaka, T., Yamada, A., Kubota, A.: Phishing site detection using similarity of website structure. In: 2021 IEEE Conference on Dependable and Secure Computing (DSC), pp. 1–8 (2021)

  24. Mao, J., Li, P., Wei, T., et al.: Phishing-alarm: robust and efficient phishing detection via page component similarity. IEEE Access 5, 17020–17030 (2017)

    Article  Google Scholar 

  25. Zhou, X., Verma, R.: Phishing sites detection from a web developer’s perspective using machine learning. In: HICSS, pp. 1–10 (2020)

  26. Liu, D., Lee, J.: A cnn-based sia screenshot method to visually identify phishing websites. J. Netw. Syst. Manag. 32, 8 (2024)

    Article  Google Scholar 

  27. Wang, M., Song, L., Li, L., Zhu, Y., Li, J.: Phishing webpage detection based on global and local visual similarity. Expert Syst. Appl. 252, 124120 (2024)

    Article  Google Scholar 

  28. Trinh, N., Phan, T., Pham, V.: Leveraging deep learning image classifiers for visual similarity-based phishing website detection. In: Proceedings of the 11th International Symposium on Information and Communication Technology, pp. 134–141 (2022)

  29. Chiew, K., Chang, E., Tiong, W.: Utilisation of website logo for phishing detection. Comput. Secur. 54, 16–26 (2015)

    Article  Google Scholar 

  30. Xiang, G., Hong, J., Rose, C., Cranor, L.: Cantina+ a feature-rich machine learning framework for detecting phishing web sites. ACM Trans. Inf. Syst. Secur. 14, 1–28 (2011)

    Article  Google Scholar 

  31. Tan, C., Chiew, K., Wong, K.: PhishWHO: phishing webpage detection via identity keywords extraction and target domain name finder. Decis. Support Syst. 88, 18–27 (2016)

    Article  Google Scholar 

  32. Rao, R., Pais, A.: Jail-Phish: an improved search engine based phishing detection system. Comput. Secur. 83, 246–267 (2019)

    Article  Google Scholar 

  33. Zhang, Y., Hong, J., Cranor, L.: Cantina: a content-based approach to detecting phishing web sites. In: Proceedings of the 16th International Conference on World Wide Web, pp. 639–648 (2007)

  34. Mohammad, R., Thabtah, F., McCluskey, L.: An assessment of features related to phishing websites using an automated technique. In: 2012 International Conference for Internet Technology and Secured Transactions, pp. 492–497 (2012)

  35. Mohammad, R., Thabtah, F., McCluskey, L.: Intelligent rule-based phishing websites classification. IET Inf. Secur. 8, 153–160 (2014)

    Article  Google Scholar 

  36. Pham, C., Nguyen, L., Tran, N., Huh, E., Hong, C.: Phishing-aware: a neuro-fuzzy approach for anti-phishing on fog networks. IEEE Trans. Netw. Serv. Manag. 15, 1076–1089 (2018)

    Article  Google Scholar 

  37. Basnet, R., Sung, A., Liu, Q.: Rule-based phishing attack detection. In: International Conference on Security and Management (SAM 2011), Las Vegas, NV (2011)

  38. Han, W., Cao, Y., Bertino, E., Yong, J.: Using automated individual white-list to protect web digital identities. Expert Syst. Appl. 39, 11861–11869 (2012)

    Article  Google Scholar 

  39. Jain, A., Gupta, B.: A novel approach to protect against phishing attacks at client side using auto-updated white-list. In: EURASIP Journal on Information Security, pp. 1–11 (2016)

  40. Rao, R., Pais, A.: Two level filtering mechanism to detect phishing sites using lightweight visual similarity approach. J. Ambient Intell. Hum. Comput. 11, 3853–3872 (2020)

    Article  Google Scholar 

  41. Sahingoz, O.K., Buber, E., Demir, O., Diri, B.: Machine learning based phishing detection from URLs. Expert Syst. Appl. 117, 345–357 (2019)

    Article  Google Scholar 

  42. Li, Y., Yang, Z., Chen, X., Yuan, H., Liu, W.: A stacking model using URL and HTML features for phishing webpage detection. Future Gener. Comput. Syst. 94, 27–39 (2019)

    Article  Google Scholar 

  43. Zhao, H., Chen, Z., Yan, R.: Malicious domain names detection algorithm based on statistical features of URLs. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 11–16 (2022)

  44. Bharadwaj, R., Bhatia, A., Chhibbar, L., Tiwari, K., Agrawal, A.: Is this URL safe: detection of malicious URLs using global vector for word representation. In: 2022 International Conference on Information Networking (ICOIN), pp. 486–491 (2022)

  45. Gupta, B., Yadav, K., Razzak, I., Psannis, K., Castiglione, A., Chang, X.: A novel approach for phishing URLs detection using lexical based machine learning in a real-time environment. Comput. Commun. 175, 47–57 (2021)

    Article  Google Scholar 

  46. Chaitanya, V., Deo, S., Thilagam, P.: User interest drift identification using contextual factors in implicit feedback-based recommender systems. In: International Conference on Pattern Recognition and Machine Intelligence, pp. 340–347 (2023)

  47. Sinha, B., Dhanalakshmi, R., Chaitanya, V.: Diversifying the predictions in the recommender systems. Int. J. Bus. Inf. Syst. 38, 168–178 (2021)

    Google Scholar 

  48. Anderson, B., McGrew, D.: Identifying encrypted malware traffic with contextual flow data. In: Proceedings of The 2016 ACM Workshop on Artificial Intelligence and Security, pp. 35–46 (2016)

  49. Anderson, B., Paul, S., McGrew, D.: Deciphering malware’s use of TLS (without decryption). J. Comput. Virol. Hacking Tech. 14, 195–211 (2018)

    Article  Google Scholar 

  50. Shekhawat, A., Di Troia, F., Stamp, M.: Feature analysis of encrypted malicious traffic. Expert Syst. Appl. 125, 130–141 (2019)

    Article  Google Scholar 

  51. Ramiz, M., Sh, M.: Classification ensemble based anomaly detection in network traffic. Rev. Comput. Eng. Res. 6, 12–23 (2019)

    Article  Google Scholar 

  52. Yao, H., Liu, C., Zhang, P., Wu, S., Jiang, C., Yu, S.: Identification of encrypted traffic through attention mechanism based long short term memory. IEEE Trans. Big Data 8, 241–252 (2019)

    Article  Google Scholar 

  53. Kashyap, H., Pais, A., Kondaiah, C.: Machine learning-based malware detection and classification in encrypted TLS traffic. In: International Conference on Security, Privacy and Data Analytics, pp. 247–262 (2022)

  54. Wang, Z., Fok, K., Thing, V.: Machine learning for encrypted malicious traffic detection: approaches, datasets and comparative study. Comput. Secur. 113, 102542 (2022)

    Article  Google Scholar 

  55. Hranickỳ, R., Horák, A., Polišenskỳ, J., Jeřábek, K., Ryšavỳ, O.: Unmasking the phishermen: phishing domain detection with machine learning and multi-source intelligence. In: NOMS 2024–2024 IEEE Network Operations and Management Symposium, pp. 1–5 (2024)

  56. McGrew, D., Joy, B.: Cisco JOY (2016). https://github.com/cisco/joy

  57. Garcia, S., Grill, M., Stiborek, J., Zunino, A.: An empirical comparison of botnet detection methods. Comput. Secur. 45, 100–123 (2014)

    Article  Google Scholar 

  58. Saha, I., Sarma, D., Chakma, R., Alam, M., Sultana, A., Hossain, S.: Phishing attacks detection using deep learning approach. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), pp. 1180–1185 (2020)

  59. Barlow, L., Bendiab, G., Shiaeles, S., Savage, N.: A novel approach to detect phishing attacks using binary visualisation and machine learning. In: 2020 IEEE World Congress on Services (SERVICES), pp. 177–182 (2020)

  60. Raj, R., Babu, S., Varalatchoumy, M., Kathirvel, C.: Implementing multiclass classification to find the optimal machine learning model for forecasting malicious URLs. In: 2022 6th International Conference on Computing Methodologies and Communication (ICCMC), pp. 1127–1130 (2022)

  61. Mahesh, A., Dheepthi.: Using machine learning to detect and classify URLs: a phishing detection approach. In: 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC), pp. 1285–1291 (2023)

Download references

Funding

The author declares that there is no funding for this paper.

Author information

Authors and Affiliations

  1. Information Security Research Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, Karnataka, 575025, India

    Cheemaladinne Kondaiah & Alwyn Roshan Pais

  2. Department of Computer Science and Engineering, GITAM (Deemed to be University), Visakhapatnam, Andhra Pradesh, 530045, India

    Routhu Srinivasa Rao

Authors
  1. Cheemaladinne Kondaiah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alwyn Roshan Pais
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Routhu Srinivasa Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Chhemaladinne Kondaiah: Conceptualized, Performed experimentation, Wrote the manuscript. Alwyn Roshan Pais: Supervised, Reviewed, and Edited the manuscript. Routhu Srinivasa Rao: Supervised, Reviewed, and Edited the manuscript.

Corresponding author

Correspondence to Cheemaladinne Kondaiah.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kondaiah, C., Pais, A.R. & Rao, R.S. Enhanced Malicious Traffic Detection in Encrypted Communication Using TLS Features and a Multi-class Classifier Ensemble. J Netw Syst Manage 32, 76 (2024). https://doi.org/10.1007/s10922-024-09847-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-024-09847-3

Keywords

Search

Navigation