It Is All about Data: A Survey on the Effects of Data on Adversarial Robustness
Authors: 
Peiyu Xiong, Michael Tegegn, Jaskeerat Singh Sarin, Shubhraneel Pal, Julia RubinAuthors Info & Claims
Peiyu Xiong, Michael Tegegn, Jaskeerat Singh Sarin, Shubhraneel Pal, Julia RubinAuthors Info & ClaimsArticle No.: 174, Pages 1 - 41
Abstract
Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to confuse the model into making a mistake. Such examples pose a serious threat to the applicability of machine learning-based systems, especially in life- and safety-critical domains. To address this problem, the area of adversarial robustness investigates mechanisms behind adversarial attacks and defenses against these attacks. This survey reviews a particular subset of this literature that focuses on investigating properties of training data in the context of model robustness under evasion attacks. It first summarizes the main properties of data leading to adversarial vulnerability. It then discusses guidelines and techniques for improving adversarial robustness by enhancing the data representation and learning procedures, as well as techniques for estimating robustness guarantees given particular data. Finally, it discusses gaps of knowledge and promising future research directions in this area.
References
[1]
ACM Computing Surveys Journal. [n.d.]. Retrieved from https://dl.acm.org/journal/csur
[2]
CORE ranking (Conference Portal). [n.d.]. Retrieved from http://portal.core.edu.au/conf-ranks/
[3]
Journal Citation Reports (JCR). [n.d.]. Retrieved from https://jcr.clarivate.com/jcr/home
[4]
Semantic Scholar Academic APIs. [n.d.]. Retrieved from https://www.semanticscholar.org/product/api
[5]
Albert Ahumada and Heidi Peterson. 1992. Luminance-model-based DCT quantization for color image compression. Human Vision, Visual Process Display III 1666 (Feb.1992).
[6]
Naveed Akhtar and Ajmal Mian. 2018. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 6 (2018), 14410–14430.
[7]
Naveed Akhtar, Ajmal S. Mian, Navid Kardan, and Mubarak Shah. 2021. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access 9 (2021), 155161–155196.
[8]
Laurent Amsaleg, James Bailey, Amélie Barbe, Sarah M. Erfani, Teddy Furon, Michael E. Houle, Miloš Radovanović, and Xuan Vinh Nguyen. 2021. High intrinsic dimensionality facilitates adversarial attack: Theoretical evidence. IEEE Trans. Info. Forensics Secur. 16 (2021), 854–865.
[9]
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. 2019. Intrinsic dimension of data representations in deep neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 6111–6122.
[10]
Pranjal Awasthi, Himanshu Jain, Ankit Singh Rawat, and Aravindan Vijayaraghavan. 2020. Adversarial robustness via robust low rank representations. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 11391–11403.
[11]
Ms. Aayushi Bansal, Dr. Rewa Sharma, and Dr. Mamta Kathuria. 2021. A systematic review on data scarcity problem in deep learning: Solution and applications. Comput. Surveys 54, 208 (2021), 1–29.
[12]
Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2022. A survey on data augmentation for text classification. Comput. Surveys (2022).
[13]
Shai Ben-David, Nicolò Cesa-Bianchi, David Haussler, and Philip M. Long. 1995. Characterizations of learnability for classes of (0, ..., n)-valued functions. J. Comput. System Sci. 50, 1 (1995), 74–86.
[14]
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. 2019. Lower bounds on adversarial robustness from optimal transport. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19).
[15]
Robi Bhattacharjee and Kamalika Chaudhuri. 2020. When are non-parametric methods robust? In Proceedings of the International Conference on Machine Learning (ICML’20). 832–841.
[16]
Robi Bhattacharjee, Somesh Jha, and Kamalika Chaudhuri. 2021. Sample complexity of robust linear classification on separated data. In Proceedings of the Conference on Learning Theory (COLT’21). 884–893.
[17]
Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recogn. 84 (2018), 317–331.
[18]
Avrim Blum, Travis Dick, Naren Manoj, and Hongyang Zhang. 2020. Random smoothing might be unable to certify \(L_{\infty }\) robustness for high-dimensional images. Journal of Machine Learning Research 21, 211 (2020), 8726–8746.
[19]
Giuseppe Bonaccorso. 2017. Machine Learning Algorithms: A Reference Guide to Popular Algorithms for Data Science and Machine Learning. Packt Publishing.
[20]
Christer Borell. 1975. The Brunn-Minkowski inequality in Gauss space. Inventiones Mathematicae 30 (1975), 207–216.
[21]
Anh Bui, Trung Le, He Zhao, Paul Montague, Oliver deVel, Tamas Abraham, and Dinh Phung. 2020. Improving adversarial robustness by enforcing local and global compactness. In Proceedings of the European Conference on Computer Vision (ECCV’20). 209–223.
[22]
Ginevra Carbone, Matthew Wicker, Luca Laurenti, Andrea Patane, Luca Bortolussi, and Guido Sanguinetti. 2020. Robustness of Bayesian neural networks to gradient-based attacks. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS’20). 15602–15613.
[23]
Nicholas Carlini and David A. Wagner. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the Symposium on Security and Privacy (SP’17). 39–57.
[24]
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy S. Liang. 2019. Unlabeled data improves adversarial robustness. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19).
[25]
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2018. Adversarial attacks and defences: A survey. ArXiv: https://arxiv.org/abs/1810.00069
[26]
Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. 2021. Amplitude-phase recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’21). 458–467.
[27]
Pin-Yu Chen and Cho-Jui Hsieh. 2023. Adversarial Robustness for Machine Learning. Academic Press.
[28]
Yiting Chen, Qibing Ren, and Junchi Yan. 2022. Rethinking and improving robustness of convolutional neural networks: A shapley value-based approach in frequency domain. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22).
[29]
Wuxinlin Cheng, Chenhui Deng, Zhiqiang Zhao, Yaohui Cai, Zhiru Zhang, and Zhuo Feng. 2021. SPADE: A spectral method for black-box adversarial robustness evaluation. In Proceedings of the International Conference on Machine Learning (ICML’21). 1814–1824.
[30]
Zhen Cheng, Fei Zhu, Xu-Yao Zhang, and Cheng-Lin Liu. 2023. Adversarial training with distribution normalization and margin balance. Pattern Recogn. 136 (2023), 109182.
[31]
Fan. R. K. Chung. 1997. Spectral Graph Theory. American Mathemetical Society.
[32]
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019. Certified adversarial robustness via randomized smoothing. In Proceedings of the International Conference on Machine Learning (ICML’19). 1310–1320.
[33]
J. S. Cramer. 2002. The Origins of Logistic Regression. Technical Report 2002-119/4. Tinbergen Institute.
[34]
Francesco Croce and Matthias Hein. 2020. Minimally distorted adversarial examples with a fast adaptive boundary attack. In Proceedings of the International Conference on Machine Learning (ICML’20). 2196–2205.
[35]
Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. 2018. PAC-learning in the presence of evasion adversaries. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 228–239.
[36]
Chen Dan, Yuting Wei, and Pradeep Ravikumar. 2020. Sharp statistical guarantees for adversarially robust Gaussian classification. In Proceedings of the International Conference on Machine Learning (ICML’20). 2345–2355.
[37]
Amit Daniely and Hadas Schacham. 2020. Most ReLU networks suffer from L2 adversarial perturbations. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 6629–6636.
[38]
David H. Haussler, Nick Littlestone, and Manfred K. Warmuth. 1994. Predicting 0, 1-functions on randomly drawn points. Info. Comput. 115, 2 (1994), 248–292.
[39]
Giacomo De Palma, Bobak Kiani, and Seth Lloyd. 2021. Adversarial robustness guarantees for random deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML’21). 2522–2534.
[40]
Luca Demetrio, Scott E. Coull, Battista Biggio, Giovanni Lagorio, Alessandro Armando, and Fabio Roli. 2021. Adversarial EXEmples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection. ACM Trans. Priv. Secur. 24, 4 (2021), 1–31.
[41]
Ambra Demontis, Marco Melis, Battista Biggio, Davide Maiorca, Dan Arp, Konrad Rieck, Igino Corona, Giorgio Giacinto, and Fabio Roli. 2019. Yes, machine learning can be more secure! a case study on android malware detection. IEEE Trans. Depend. Secure Comput. 16, 4 (2019), 711–724.
[42]
Yingpeng Deng and Lina J. Karam. 2020. Frequency-tuned universal adversarial perturbations. In Proceedings of the Computer Vision Workshops (ECCV’20). 494–510.
[43]
Yingpeng Deng and Lina J. Karam. 2022. Frequency-tuned universal adversarial attacks on texture recognition. IEEE Trans. Image Process. 31 (2022), 5856–5868.
[44]
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. 2019. On the sensitivity of adversarial robustness to input data distributions. In Proceedings of the International Conference on Learning Representations (ICLR’19).
[45]
Dimitrios I. Diochnos, Saeed Mahloujifar, and Mohammad Mahmoody. 2018. Adversarial risk and robustness: General definitions and implications for the uniform distribution. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 10380–10389.
[46]
Elvis Dohmatob. 2019. Generalized no free lunch theorem for adversarial robustness. In Proceedings of the International Conference on Machine Learning (ICML’19). 1646–1654.
[47]
Chengyu Dong, Liyuan Liu, and Jingbo Shang. 2022. Label noise in adversarial training: A novel perspective to study robust overfitting. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22).
[48]
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 1625–1634.
[49]
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. 2018. Adversarial vulnerability for any classifier. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 1186–1195.
[50]
Sid Ahmed Fezza, Yassine Bakhti, Wassim Hamidouche, and Olivier Déforges. 2019. Perceptual evaluation of adversarial attacks for CNN-based image classification. In Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX’19). 1–6.
[51]
Benoit Frenay and Michel Verleysen. 2014. Classification in the presence of label noise: A survey. IEEE Trans. Neural Netw. Learn. Syst. 25, 5 (2014), 845–869.
[52]
Shivam Garg, Vatsal Sharan, Brian Hu Zhang, and Gregory Valiant. 2018. A spectral view of adversarially robust features. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 10159–10169.
[53]
Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’19). 3681–3688.
[54]
Partha Ghosh, Arpan Losalka, and Micheal. J. Black. 2019. Resisting adversarial attacks using Gaussian mixture variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’19). 541–548.
[55]
Justin Gilmer, Ryan P. Adams, Ian J. Goodfellow, David G. Andersen, and George E. Dahl. 2018. Motivating the rules of the game for adversarial example research. ArXiv abs/1807.06732. http://arxiv.org/abs/1807.06732
[56]
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian J. Goodfellow. 2018. Adversarial spheres. In Proceedings of the International Conference on Learning Representations (ICLR’18).
[57]
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. 2023. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Trans. Pattern Anal. Mach. Intell. 45, 2 (2023), 1563–1580.
[58]
Ian Goodfellow, Nicolas Papernot, Sandy Huang, Rocky Duan, Pieter Abbeel, and Jack Clark. 2017. Attacking Machine Learning with Adversarial Examples. Retrieved from https://openai.com/blog/adversarial-example-research/
[59]
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR’14).
[60]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR’15).
[61]
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. 2021. Regularization of neural networks by enforcing lipschitz continuity. Mach. Learn. 110 (2021), 393–416.
[62]
Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. 2021. On the hardness of robust classification. J. Mach. Learn. Res. 22, 273 (2021), 12521–12549.
[63]
Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. 2022. Sample complexity bounds for robustly learning decision lists against evasion attacks. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’22). 3022–3028.
[64]
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy Mann. 2021. Improving robustness using generated data. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’21). 4218–4233.
[65]
Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing. 2018. LEMNA: Explaining deep learning-based security applications. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’18). 364–379.
[66]
Yiwen Guo, Long Chen, Yurong Chen, and Changshui Zhang. 2021. On connections between regularizations for improving DNN robustness. IEEE Trans. Pattern Anal. Mach. Intell. 43, 12 (2021), 4469–4476.
[67]
Sicong Han, Chenhao Lin, Chao Shen, Qian Wang, and Xiaohong Guan. 2023. Interpreting adversarial examples in deep learning: A review. Comput. Surveys 55, 14s (2023), 1–38.
[68]
Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21, 9 (2009), 1263–1284.
[69]
Xinlei He and Yang Zhang. 2021. Quantifying and mitigating privacy risks of contrastive learning. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS). 845–863.
[70]
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. Comput. Surveys 54, 235 (2022), 1–37.
[71]
Huawei Huang, Wei Kong, Sicong Zhou, Zibin Zheng, and Song Guo. 2021. A survey of state-of-the-art on blockchains: Theories, modelings, and tools. Comput. Surveys 54, 44 (2021), 1–42.
[72]
Ingo Höntsch and Lina. J. Karam. 2002. Adaptive image coding with perceptual distortion control. IEEE Trans. Image Process. 11, 3 (2002), 213–222.
[73]
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Mądry. 2019. Adversarial examples are not bugs, they are features. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 125–136.
[74]
Rauf Izmailov, Shridatt Sugrim, Ritu Chadha, Patrick McDaniel, and Ananthram Swami. 2018. Enablers of adversarial attacks in machine learning. In Proceedings of the IEEE Military Communications Conference (MILCOM’18). 425–430.
[75]
Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani. 2020. Precise tradeoffs in adversarial training for linear regression. In Proceedings of the International Conference on Learning Theory (COLT’20). 2034–2078.
[76]
Jongheon Jeong and Jinwoo Shin. 2020. Consistency regularization for certified robustness of smoothed classifiers. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 10558–10570.
[77]
Xi Wu and Jiefeng Chen. 2019. Robust Attribution Regularization. Retrieved from https://www.altacognita.com/robust-attribution/
[78]
Ian T. Jolliffe. 2002. Principal Component Analysis. Springer.
[79]
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. CIFAR-10 and CIFAR-100 Datasets. Retrieved from https://www.cs.toronto.edu/ kriz/cifar.html
[80]
Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. 2020. Curse of dimensionality on randomized smoothing for certifiable robustness. In Proceedings of the International Conference on Machine Learning (ICML’20). 5458–5467.
[81]
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial machine learning at scale. In Proceedings of the International Conference on Learning Representations (ICLR’17).
[82]
Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. 1998. The MNIST Database of Handwritten Digits. Retrieved from http://yann.lecun.com/exdb/mnist/
[83]
Guang-He Lee, Yang Yuan, Shiyu Chang, and Tommi Jaakkola. 2019. Tight certificates of adversarial robustness for randomly smoothed classifiers. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19).
[84]
Saehyung Lee, Hyungyu Lee, and Sungroh Yoon. 2020. Adversarial vertex mixup: Toward better adversarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). 269–278.
[85]
Paul Lévy. 1951. Problémes Concrets D’analyse Fonctionnelle. Gauthier-Villers.
[86]
Binghui Li, Jikai Jin, Han Zhong, John E. Hopcroft, and Liwei Wang. 2022. Why robust generalization in deep learning is difficult: Perspective of expressive power. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22).
[87]
Deqiang Li, Qianmu Li, Yanfang (Fanny) Ye, and Shouhuai Xu. 2021. Arms race in adversarial malware detection: A survey. Comput. Surveys 55, 1 (2021), 1–35.
[88]
Xiang Ling, Shouling Ji, Jiaxu Zou, Jiannan Wang, Chunming Wu, Bo Li, and Ting Wang. 2019. DEEPSEC: A uniform platform for security analysis of deep learning model. In Proceedings of the IEEE Symposium on Security and Privacy (SP’19). 673–690.
[89]
Jinxin Liu, Michele Nogueira, Johan Fernandes, and Burak Kantarci. 2022. Adversarial machine learning: A multilayer review of the state-of-the-art and challenges for wireless and mobile systems. IEEE Commun. Surveys Tutor. 24, 1 (2022), 123–159.
[90]
Xuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, and Cho-Jui Hsieh. 2019. A unified framework for data poisoning attack to graph-based semi-supervised learning. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 9780–9790.
[91]
Yue Liu, Chakkrit Tantithamthavorn, Li Li, and Yepang Liu. 2022. Deep learning for android malware defenses: A systematic literature review. Comput. Surveys (2022).
[92]
Victoria López, Alberto Fernández, Salvador García, Vasile Palade, and Francisco Herrera. 2013. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. Info. Sci. 250 (2013), 113–141.
[93]
Ana C. Lorena, Luís P. F. Garcia, Jens Lehmann, Marcilio C. P. Souto, and Tin Kam Ho. 2020. How complex is your classification problem? A survey on measuring classification complexity. Comput. Surveys 52, 107 (2020), 1–34.
[94]
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. 2021. Understanding adversarial attacks on deep learning-based medical image analysis systems. Pattern Recogn. 110 (2021), 107332.
[95]
Gabriel Resende Machado, Eugênio Silva, and Ronaldo Ribeiro Goldschmidt. 2021. Adversarial machine learning in image classification: A survey toward the defender’s perspective. Comput. Surveys 55, 8 (2021), 1–38.
[96]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations (ICLR’18).
[97]
Saeed Mahloujifar, Dimitrios I. Diochnos, and Mohammad Mahmoody. 2019. The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’19). 4536–4543.
[98]
Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, and David Evans. 2019. Empirically measuring concentration: Fundamental limits on intrinsic robustness. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 5209–5220.
[99]
Davide Maiorca, Battista Biggio, and Giorgio Giacinto. 2019. Towards adversarial malware detection: Lessons learned from PDF-based attacks. Comput. Surveys 52, 78 (2019), 1–36.
[100]
Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray, Shuran Song, Junfeng Yang, and Carl Vondrick. 2020. Multitask learning strengthens adversarial robustness. In Proceedings of the European Conference on Computer Vision (ECCV’20). 158–174.
[101]
Mohammad Mehrabi, Adel Javanmard, Ryan A. Rossi, Anup Rao, and Tung Mai. 2021. Fundamental tradeoffs in distributionally adversarial training. In Proceedings of the International Conference on Machine Learning (ICML’21). 7544–7554.
[102]
Ali H. Mezher, Yingpeng Deng, and Lina J. Karam. 2022. Visual quality assessment of adversarially attacked images. In Proceedings of the European Workshop on Visual Information Processing (EUVIP’22). 1–5.
[103]
Eric Mintun, Alexander Kirillov, and Saining Xie. 2021. On interaction between augmentations and corruptions in natural corruption robustness. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’21). 3571–3583.
[104]
Omar Montasser, Steve Hanneke, and Nathan Srebro. 2022. Adversarially robust learning: A generic minimax optimal learner and characterization. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22).
[105]
Jose Garcia Moreno-Torres, Troy Raeder, Rocío Alaíz-Rodríguez, N. Chawla, and Francisco Herrera. 2012. A unifying view on dataset shift in classification. Pattern Recogn. 45 (2012), 521–530.
[106]
Aamir Mustafa, Salman Khan, Munawar Hayat, Roland Goecke, Jianbing Shen, and Ling Shao. 2019. Adversarial defense by restricting the hidden space of deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’19). 3384–3393.
[107]
Aamir Mustafa, Salman H. Khan, Munawar Hayat, Roland Goecke, Jianbing Shen, and Ling Shao. 2020. Deeply supervised discriminative learning for adversarial defense. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 9 (2020), 3154–3166.
[108]
Vasileios Mygdalis and Ioannis Pitas. 2022. Hyperspherical class prototypes for adversarial robustness. Pattern Recognition (2022).
[109]
Amir Najafi, Shin ichi Maeda, Masanori Koyama, and Takeru Miyato. 2019. Robustness to adversarial perturbations in learning from incomplete data. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 5541–5551.
[110]
Mahum Naseer, Bharath Srinivas Prabakaran, Osman Hasan, and Muhammad Shafique. 2023. UnbiasedNets: A dataset diversification framework for robustness bias alleviation in neural networks. Mach. Learn. (2023), 1–28.
[111]
Balas K. Natarajan. 2004. On learning sets and functions. Mach. Learn. 4 (2004), 67–97.
[112]
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning.
[113]
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. 2018. Adversarial robustness toolbox v1.2.0. ArXiv. http://arxiv.org/abs/1807.01069
[114]
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2020. Hold me tight! influence of discriminative features on deep network boundaries. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 2935–2946.
[115]
Tianyu Pang, Chao Du, and Jun Zhu. 2018. Max-mahalanobis linear discriminant analysis networks. In Proceedings of the 35th International Conference on Machine Learning (ICML’18). 4013–4022.
[116]
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. 2020. Rethinking softmax cross-entropy loss for adversarial robustness. In Proceedings of the International Conference on Learning Representations (ICLR’20).
[117]
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. 2019. Improving adversarial robustness via promoting ensemble diversity. In Proceedings of the International Conference on Machine Learning (ICML’19). 4970–4979.
[118]
Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Jun Zhu, and Hang Su. 2020. Boosting adversarial training with hypersphere embedding. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 7779–7792.
[119]
Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. 2020. Intriguing properties of adversarial ML attacks in the problem space. In Proceedings of the Symposium on Security and Privacy (SP’20).
[120]
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. 2021. The intrinsic dimension of images and its impact on learning. In Proceedings of the International Conference on Learning Representations (ICLR’21).
[121]
Jack Prescott, Xiao Zhang, and David Evans. 2021. Improved estimation of concentration under lp-norm distance metrics using half spaces. In Proceedings of the International Conference on Learning Representations (ICLR’21).
[122]
Muni Sreenivas Pydi and Varun Jog. 2020. Adversarial risk via optimal transport and optimal couplings. In Proceedings of the International Conference on Machine Learning (ICML’20). 7814–7823.
[123]
Muni Sreenivas Pydi and Varun Jog. 2021. The many faces of adversarial risk. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’21). 10000–10012.
[124]
Mohammadreza Qaraei and Rohit Babbar. 2022. Adversarial examples for extreme multilabel text classification. Mach. Learn. (2022), 1–25.
[125]
Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. 2019. Does data augmentation lead to positive margin? In Proceedings of the International Conference on Machine Learning (ICML’19). 5321–5330.
[126]
S. J. Raudys and A. K. Jain. 1991. Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Trans. Pattern Anal. Mach. Intell. 13, 3 (1991), 252–264.
[127]
Mohammad Rezaeirad, Brown Farinholt, Hitesh Dharmdasani, Paul Pearce, Kirill Levchenko, and Damon McCoy. 2018. Schrödinger’s RAT: Profiling the stakeholders in the remote access trojan ecosystem. In Proceedings of the USENIX Security Symposium. 1043–1060.
[128]
Eitan Richardson and Yair Weiss. 2021. A Bayes-optimal view on adversarial examples. J. Mach. Learn. Res. 22, 221 (2021), 10076–10103.
[129]
Ishai Rosenberg, Asaf Shabtai, Yuval Elovici, and Lior Rokach. 2021. Adversarial machine learning attacks and defense methods in the cyber security domain. Comput. Surveys 54, 5 (2021), 1–36.
[130]
Miriam Santos, Pedro Henriques Abreu, Nathalie Japkowicz, Alberto Fernández, Carlos Soares, Szymon Wilk, and Joao Santos. 2022. On the joint-effect of class imbalance and overlap: A critical review. Artific. Intell. Rev. (2022), 1–69.
[131]
Amartya Sanyal, Puneet K. Dokania, Varun Kanade, and Philip Torr. 2021. How benign is benign overfitting? In Proceedings of the International Conference on Learning Representations (ICLR’21).
[132]
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. 2018. Adversarially robust generalization requires more data. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 5019–5031.
[133]
H. Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Info. Theory 11, 3 (1965), 363–371.
[134]
Alex Serban, Erik Poll, and Joost Visser. 2020. Adversarial examples on object recognition: A comprehensive survey. Comput. Surveys 53, 3 (2020), 1–38.
[135]
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’18). 6106–6116.
[136]
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. 2019. Are adversarial examples inevitable? In Proceedings of the International Conference on Learning Representations (ICLR’19).
[137]
Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.
[138]
Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. 2019. A simple explanation for the existence of adversarial examples with small hamming distance. Retrieved from https://arxiv.org/abs/1901.10861
[139]
Claude E. Shannon. 1949. The Mathematical Theory of Communication. University of Illinois Press.
[140]
Mahmood. Sharif, Lujo. Bauer, and Michael. K. Reiter. 2018. On the suitability of lp-norms for creating and preventing adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’18). 1686–16868.
[141]
Yucheng Shi, Yahong Han, Yu-an Tan, and Xiaohui Kuang. 2022. Decision-based black-box attack against vision transformers via patch-wise adversarial removal. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22). 12921–12933.
[142]
Carl-Johann Simon-Gabriel, Yann Ollivier, Bernhard Schölkopf, Léon Bottou, and David Lopez-Paz. 2019. First-order adversarial vulnerability of neural networks and input dimension. In Proceedings of the International Conference on Machine Learning (ICML’19). 5809–5817.
[143]
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2022. Learning from noisy labels with deep neural networks: A survey. IEEE Trans. Neural Netw. Learn. Syst. (2022), 1–19.
[144]
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. 2018. PixelDefend: Leveraging generative models to understand and defend against adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR’18).
[145]
Lichao Sun, Yingtong Dou, Carl Yang, Kai Zhang, Ji Wang, S. Yu Philip, Lifang He, and Bo Li. 2022. Adversarial attack and defense on graph data: A survey. IEEE Trans. Knowl. Data Eng. (2022).
[146]
Jiachen Sun, Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, Dan Hendrycks, Jihun Hamm, and Z. Morley Mao. 2022. A spectral view of randomized smoothing under common corruptions: Benchmarking and improving certified robustness. In Proceedings of the European Conference on Computer Vision (ECCV’22). 654–671.
[147]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR’14).
[148]
Michel Talagrand. 1996. A new look at independence. Ann. Probab. 24, 1 (Jan. 1996), 1–34.
[149]
Michel Talagrand. 1996. Transportation cost for Gaussian and other product measures. Geom. Funct. Anal. 6 (1996), 587–600.
[150]
Mingtian Tan, Junpeng Wan, Zhe Zhou, and Zhou Li. 2021. Invisible probe: Timing attacks with PCIe congestion side-channel. In Proceedings of the IEEE Symposium on Security and Privacy (SP’21). 322–338.
[151]
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. 2022. A comprehensive survey on poisoning attacks and countermeasures in machine learning. Comput. Surveys (2022).
[152]
Liang Tong, Bo Li, Chen Hajaj, Chaowei Xiao, Ning Zhang, and Yevgeniy Vorobeychik. 2019. Improving robustness of ML classifiers against realizable evasion attacks using conserved features. In Proceedings of the USENIX Conference on Security Symposium (USENIX Security’19). 285–302.
[153]
Antonio Torralba, Rob Fergus, and William T. Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 30, 11 (2008), 1958–1970.
[154]
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NY.
[155]
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In Proceedings of the International Conference on Learning Representations (ICLR’19).
[156]
Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. 2019. Are labels required for improving adversarial robustness? In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 12214–12223.
[157]
viso.ai. 2022. What is Adversarial Machine Learning? Attack Methods in 2022. Retrieved from https://viso.ai/deep-learning/adversarial-machine-learning/
[158]
Weitao Wan, Jiansheng Chen, Cheng Yu, Tong Wu, Yuanyi Zhong, and Ming-Hsuan Yang. 2022. Shaping deep feature space towards Gaussian mixture for visual classification. IEEE Trans. Pattern Anal. Mach. Intell. 45, 2 (2022), 2430–2444.
[159]
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018. CosFace: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’18). 5265–5274.
[160]
H. Wang, X. Wu, Z. Huang, and E. P. Xing. 2020. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). 8681–8691.
[161]
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. ICLR Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models. https://openreview.net/forum?id=uw6HSkgoM29
[162]
Tianfeng Wang, Zhisong Pan, Guyu Hu, Yexin Duan, and Yu Pan. 2022. Understanding universal adversarial attack and defense on graph. Int. J. Semant. Web Info. Syst. 18, 1 (2022), 1–21.
[163]
Wentao Wang, Han Xu, Xiaorui Liu, Yaxin Li, Bhavani Thuraisingham, and Jiliang Tang. 2021. Imbalanced adversarial training with reweighting. ArXiv, abs/2107.13639. https://arxiv.org/abs/2107.13639
[164]
Yizhen Wang, Somesh Jha, and Kamalika Chaudhuri. 2018. Analyzing the robustness of nearest neighbors to adversarial examples. In Proceedings of the International Conference on Machine Learning (ICML’18). 5133–5142.
[165]
Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya K. Menon, and Sanjiv Kumar. 2020. Robust large-margin learning in hyperbolic space. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’20). 17863–17873.
[166]
Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. 2021. Adversarial robustness under long-tailed distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 8655–8664.
[167]
Yue Xing, Qifan Song, and Guang Cheng. 2022. Why do artificially generated data help adversarial robustness. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22), Vol. 35. 954–966.
[168]
Peiyu Xiong, Michael Tegegn, Jaskeerat Singh Sarin, and Shubhraneel Pal. 2023. Supplementary Materials. Retrieved from https://resess.github.io/artifacts/DataForMLRobustness
[169]
Jingyuan Xu and Weiwei Liu. 2022. On robust multiclass learnability. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’22). 32412–32423.
[170]
Kaidi Xu, Hongge Chen, Sijia Liu, Pin-Yu Chen, Tsui-Wei Weng, Mingyi Hong, and Xue Lin. 2019. Topology attack and defense for graph neural networks: An optimization perspective. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’19). 3961–3967.
[171]
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. 2021. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 14378–14387.
[172]
Lu Yang, He Jiang, Qing Song, and Jun Guo. 2022. A survey on long-tailed visual recognition. Int. J. Comput. Vision 130, 7 (2022), 1837–1872.
[173]
Shuo Yang, Zeyu Feng, Pei Du, Bo Du, and Chang Xu. 2021. Structure-aware stabilization of adversarial robustness with massive contrastive adversaries. In Proceedings of the IEEE International Conference on Data Mining (ICDM’21). 807–816.
[174]
Shuo Yang, Tianyu Guo, Yunhe Wang, and Chang Xu. 2021. Adversarial robustness through disentangled representations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’21). 3145–3153.
[175]
Yao-Yuan Yang, Cyrus Rashtchian, Yizhen Wang, and Kamalika Chaudhuri. 2020. Robustness for non-parametric classification: A generic attack and defense. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’20). 941–951.
[176]
Dong Yin, Ramchandran Kannan, and Peter Bartlett. 2019. Rademacher complexity for adversarially robust generalization. In Proceedings of the International Conference on Machine Learning (ICML’19). 7085–7094.
[177]
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer. 2019. A fourier perspective on model robustness in computer vision. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’19). 13276–13286.
[178]
Xiaoyan Yin, Wanyu Lin, Kexin Sun, Chun Wei, and Yanjiao Chen. 2023. A2S2-GNN: Rigging GNN-based social status by adversarial attacks in signed social networks. IEEE Trans. Info. Forens. Secur. 18 (2023), 206–220.
[179]
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. 2019. Adversarial examples: Attacks and defenses for deep learning. IEEE Trans. Neural Netw. Learn. Syst. 30, 9 (2019), 2805–2824.
[180]
Chaoning Zhang, Philipp Benz, Adil Karjauv, and In So Kweon. 2021. Universal adversarial perturbations through the lens of deep steganography: Towards a fourier perspective. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’21). 3296–3304.
[181]
Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. 2021. Delving deep into label smoothing. IEEE Trans. Image Process. 30 (2021), 5984–5996.
[182]
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit Dhillon, and Cho-Jui Hsieh. 2019. The limitations of adversarial training and the blind-spot attack. In Proceedings of the International Conference on Learning Representations (ICLR’19).
[183]
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the International Conference on Machine Learning (ICML’19). 7472–7482.
[184]
Hengtong Zhang, Tianhang Zheng, Jing Gao, Chenglin Miao, Lu Su, Yaliang Li, and Kui Ren. 2019. Data poisoning attack against knowledge graph embedding. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’19). 4853–4859.
[185]
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Trans. Intell. Syst. Technol. 11, 3 (2020), 1–41.
[186]
Xiao Zhang and David Evans. 2022. Incorporating label uncertainty in understanding adversarial robustness. In Proceedings of the International Conference on Learning Representations (ICLR’22).
[187]
Shuai Zhou, Chi Liu, Dayong Ye, Tianqing Zhu, Wanlei Zhou, and Philip S. Yu. 2022. Adversarial attacks and defenses in deep learning: From a perspective of cybersecurity. Comput. Surveys 55, 8 (2022), 1–39.
[188]
Hangyu Zhu and Yaochu Jin. 2020. Multi-objective evolutionary federated learning. IEEE Trans. Neural Netw. Learn. Syst. 31, 4 (2020), 1310–1322.
[189]
Yao Zhu, Jiacheng Sun, and Zhenguo Li. 2022. Rethinking adversarial transferability from a data distribution perspective. In Proceedings of the International Conference on Learning Representations (ICLR’22).
Index Terms
- It Is All about Data: A Survey on the Effects of Data on Adversarial Robustness
Recommendations
Defending malware detection models against evasion based adversarial attacks
Highlights- We developed twenty distinct malware detection models and investigated their adversarial robustness and evasion resistance.
AbstractThe last decade has witnessed a massive malware boom in the Android ecosystem. Literature suggests that artificial intelligence/machine learning based malware detection models can potentially solve this problem. But, these detection ...
A novel method for improving the robustness of deep learning-based malware detectors against adversarial attacks
AbstractMalware is constantly evolving with rising concern for cyberspace. Deep learning-based malware detectors are being used as a potential solution. However, these detectors are vulnerable to adversarial attacks. The adversarial attacks manipulate ...
Graphical abstractDisplay Omitted
Highlights- An approach to combining adversarial attacks is proposed to analyse the robustness of malware detectors against attacks.
- Ten adversarial attacks are created to generate binary-encoded malicious samples, including the proposed combined ...
Adversarial machine learning for cybersecurity and computer vision: Current developments and challenges
AbstractWe provide a comprehensive overview of adversarial machine learning focusing on two application domains, that is, cybersecurity and computer vision. Research in adversarial machine learning addresses a significant threat to the wide application of ...
Poisoning attack contaminates the training data to render a classifier useless; evasion attack generates adversarial samples at test time; membership inference attack and model inversion attack aim to infer information about data points used in the ...
Comments
Information & Contributors
Information
Published In
July 2024
1006 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3613612
- Editors:
- David Atienza,
- Michela Milano
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 April 2024
Online AM: 18 October 2023
Accepted: 26 September 2023
Revised: 12 July 2023
Received: 26 October 2022
Published in CSUR Volume 56, Issue 7
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 689Total Downloads
- Downloads (Last 12 months)689
- Downloads (Last 6 weeks)66
Reflects downloads up to 18 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in