research-article

A Self-Supervised Feature Map Augmentation (FMA) Loss and Combined Augmentations Finetuning to Efficiently Improve the Robustness of CNNs

Authors:

Jonas Löhdefink,

Roland Zimmerman,

Serin Varghese,

Peter Schlicht,

Tim FingscheidtAuthors Info & Claims

CSCS '20: Proceedings of the 4th ACM Computer Science in Cars Symposium

Article No.: 3, Pages 1 - 8

https://doi.org/10.1145/3385958.3430477

Published: 04 December 2020 Publication History

Abstract

Deep neural networks are often not robust to semantically-irrelevant changes in the input. In this work we address the issue of robustness of state-of-the-art deep convolutional neural networks (CNNs) against commonly occurring distortions in the input such as photometric changes, or the addition of blur and noise. These changes in the input are often accounted for during training in the form of data augmentation. We have two major contributions: First, we propose a new regularization loss called feature-map augmentation (FMA) loss which can be used during finetuning to make a model robust to several distortions in the input. Second, we propose a new combined augmentations (CA) finetuning strategy, that results in a single model that is robust to several augmentation types at the same time in a data-efficient manner. We use the CA strategy to improve an existing state-of-the-art method called stability training (ST). Using CA, on an image classification task with distorted images, we achieve an accuracy improvement of on average 8.94% with FMA and 8.86% with ST absolute on CIFAR-10 and 8.04% with FMA and 8.27% with ST absolute on ImageNet, compared to 1.98% and 2.12%, respectively, with the well known data augmentation method, while keeping the clean baseline performance.

References

[1]

A. Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. https://www.cs.toronto.edu/~kriz/cifar.html. Accessed: 2019-10-10.

[2]

Aharon Azulay and Yair Weiss. 2018. Why Do Deep Convolutional Networks Generalize so Poorly to Small Image Transformations?arXiv1805.12177 (May 2018). arxiv:1805.12177

[3]

James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research (JMLR) 13, 1 (March 2012), 281–305.

[4]

Charlotte Bunne, Lukas Rahmann, and Thomas Wolf. 2018. Studying Invariances of Trained Convolutional Neural Networks. arXiv1803.05963 (March 2018). arxiv:1803.05963

[5]

Andreas Bär, Fabian Hüger, Peter Schlicht, and Tim Fingscheidt. 2019. On the Robustness of Teacher-Student Frameworks for Semantic Segmentation. In Proc. of CVPR - Workshops. Long Beach, CA, USA, 1–9.

[6]

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. 2017. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv1706.05587 (June 2017). arxiv:1706.05587

[7]

Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. 2020. Randaugment: Practical Automated Data Augmentation with a Reduced Search Space. In Proc. of CVPR. Seattle, WA, USA, 702–703.

[8]

Ekin D. Cubuk, Barret Zoph†, Dandelion Man, Vijay Vasudevan, and Quoc V. Le. 2019. AutoAugment: Learning Augmentation Strategies from Data. In Proc. of CVPR. Long Beach, CA, USA, 113–123.

[9]

Logan Engstrom, Brandon Tran, Dimitris Tsipras, and Ludwig Schmidt. 2019. A Rotation and a Translation Suffice : Fooling CNNs with Simple Transformations. In Proc. of ICLR. New Orleans, LA, USA, 1–21.

[10]

Alhussein Fawzi and Pascal Frossard. 2015. Manitest: Are Classifiers Really Invariant?arXiv1507.06535 (July 2015). arxiv:1507.06535

[11]

Alhussein Fawzi, Seyed Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2017. The Robustness of Deep Networks: A Geometrical Perspective. IEEE Signal Processing Magazine 34, 6 (Nov. 2017), 50–62.

[12]

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019. ImageNet-trained CNNs are Biased Towards Texture; Increasing Shape Bias Improves Accuracy and Robustness. In Proc. of ICLR. Addis Ababa, Ethopia, 1–15.

[13]

Ross Girshick. 2015. Fast R-CNN. In Proc. of ICCV. Las Condes, Chile, 1–21.

[14]

K. He, G. Gkioxari, P. Dollár, and R. Girshick. 2017. Mask R-CNN. In Proc. of ICCV. Venice, Italy, 2980–2988.

[15]

K. He, X. Zhang, S. Ren, and J. Sun. 2015. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 37, 9 (Sept. 2015), 1904–1916.

Digital Library

[16]

Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In Proc. of ICLR. New Orleans, LA, USA, 1–15.

[17]

Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2020. Augmix: A Simple Data Processing Method To Improve Robustness and Uncertainty. In Proc. of ICLR. Addis Ababa, Ethopia, 1–15.

[18]

Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. 2019. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. In Proc. of ICML. Long Beach, CA, USA, 1–14.

[19]

Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. 2017. Measuring Catastrophic Forgetting in Neural Networks. In Proc. of AAAI. San Francisco, CA, USA, 1–15.

[20]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proc. of NIPS. Lake Tahoe, USA, 1097–1105.

Digital Library

[21]

J. Löhdefink, A. Bär, N. M. Schmidt, F. Hüger, P. Schlicht, and T. Fingscheidt. 2019. On Low-Bitrate Image Compression for Distributed Automotive Perception: Higher Peak SNR Does Not Mean Better Semantic Segmentation. In Proc. of IV. Paris, France, 352–359.

[22]

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. 2018. Exploring the Limits of Weakly Supervised Pretraining. In Proc. of ECCV. Munich, Germany, 1–23.

Digital Library

[23]

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2015. You Only Look Once: Unified, Real-Time Object Detection. In Proc. of CVPR. Boston, MA, USA, 779–788.

[24]

Erik Rodner, Marcel Simon, Robert B. Fisher, and Joachim Denzler. 2016. Fine-grained Recognition in the Noisy Wild: Sensitivity Analysis of Convolutional Neural Networks Approaches. In Proc. of BMVC. York, UK, 1–13.

[25]

Andras Rozsa, Manuel Gunther, and Terrance E. Boult. 2018. Towards Robust Deep Neural Networks with BANG. In Proc. of IEEE WACV. Lake Tahoe, NV, USA, 1–9.

[26]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks For Large-Scale Image Recognition. arXiv1409.1556 (Sept. 2014). arxiv:1409.1556

[27]

Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. 2016. Examining the Impact of Blur on Recognition by Convolutional Networks. arXiv1611.05760 (Nov. 2016). arxiv:1611.05760

[28]

Jason Wang and Luis Perez. 2017. The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv1712.04621 (Dec. 2017). arxiv:1712.04621

[29]

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. 2019. Deep High-Resolution Representation Learning for Visual Recognition. In Proc. of CVPR. Long Beach, CA, USA, 1–17.

[30]

Sebastian C. Wong, Adam Gatt, Victor Stamatescu, and Mark D. McDonnell. 2016. Understanding Data Augmentation for Classification: When to Warp?. In Proc. of DICTA. Gold Coast, QLD, Australia, 1–14.

[31]

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proc. of CVPR. Salt Lake City, UT, USA, 586–595.

[32]

Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. 2016. Improving The Robustness of Deep Neural Networks via Stability Training. In Proc. of CVPR. Las Vegas, NV, USA, 4480–4488.

[33]

Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. 2019. Improving Semantic Segmentation via Video Propagation and Label Relaxation. In Proc. of CVPR. Long Beach, CA, USA, 1–14.

Cited By

Samal SZhang YSaez JWang SBalabantaray BNayak R(2024)EAYv3-CF$C^{3}$: Ensemble Learning With Attention-Based Yv3 Combined With CF$C^{3}$ Loss for Obscenity DetectionIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33205538:2(1097-1101)Online publication date: Apr-2024
https://doi.org/10.1109/TETCI.2023.3320553
Ballas ADiou C(2023)CNN Feature Map Augmentation for Single-Source Domain Generalization2023 IEEE Ninth International Conference on Big Data Computing Service and Applications (BigDataService)10.1109/BigDataService58306.2023.00024(127-131)Online publication date: Jul-2023
https://doi.org/10.1109/BigDataService58306.2023.00024
Houben SAbrecht SAkila MBär ABrockherde FFeifel PFingscheidt TGannamaneni SGhobadi SHammam AHaselhoff AHauser FHeinzemann CHoffmann MKapoor NKappel FKlingner MKronenberger JKüppers FLöhdefink JMlynarski MMock MMualla FPavlitskaya SPoretschkin MPohl ARavi-Kumar VRosenzweig JRottmann MRüping SSämann TSchneider JSchulz ESchwalbe GSicking JSrivastava TVarghese SWeber MWirkert SWirtz TWoehrle M(2022)Inspect, Understand, Overcome: A Survey of Practical Methods for AI SafetyDeep Neural Networks and Data for Automated Driving10.1007/978-3-031-01233-4_1(3-78)Online publication date: 18-Jun-2022
https://doi.org/10.1007/978-3-031-01233-4_1

Recommendations

LDN-RC: a lightweight denoising network with residual connection to improve adversarial robustness
Abstract
Deep neural networks (DNNs) are prone to produce incorrect prediction results under the attack of adversarial samples. To cope with this problem, some defense methods are presented. However, most of them are based on adversarial training, which ...
Rethinking data augmentation for adversarial robustness
Abstract
Recent work has proposed novel data augmentation methods to improve the adversarial robustness of deep neural networks. In this paper, we re-evaluate such methods through the lens of different metrics that characterize the augmented manifold, ...
Highlights
- Augmentation methods for adversarial robustness are often not tested in isolation.
- They are often tested on one single value of augmentation probability.
- They improve robustness only when combined with classical augmentations.
- ...
Data Augmentation via Variational Auto-Encoders
Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications
Abstract
Data augmentation is a widely considered technique to improve the performance of Convolutional Neural Networks during training. This step consists in synthetically generate new labeled data by perturbing the samples of the training set, which is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCS '20: Proceedings of the 4th ACM Computer Science in Cars Symposium

December 2020

115 pages

ISBN:9781450376211

DOI:10.1145/3385958

Editors:
Björn Brücher
Intel Germany
,
Oliver Wasenmüller
University for Applied Science Mannheim, Germany
,
Mario Fritz
CISPA Helmholtz Center for Information Security, Germany
,
Hans-Joachim Hof
Technical University of Ingolstadt, Germany
,
Christoph Krauß
Fraunhofer Institute for Secure Information Technology (SIT), Germany

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Bundesministerium für Wirtschaft und Energie

Conference

CSCS '20

Sponsor:

SIGGRAPH

CSCS '20: Computer Science in Cars Symposium

December 2, 2020

Feldkirchen, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Samal SZhang YSaez JWang SBalabantaray BNayak R(2024)EAYv3-CF$C^{3}$: Ensemble Learning With Attention-Based Yv3 Combined With CF$C^{3}$ Loss for Obscenity DetectionIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33205538:2(1097-1101)Online publication date: Apr-2024
https://doi.org/10.1109/TETCI.2023.3320553
Ballas ADiou C(2023)CNN Feature Map Augmentation for Single-Source Domain Generalization2023 IEEE Ninth International Conference on Big Data Computing Service and Applications (BigDataService)10.1109/BigDataService58306.2023.00024(127-131)Online publication date: Jul-2023
https://doi.org/10.1109/BigDataService58306.2023.00024
Houben SAbrecht SAkila MBär ABrockherde FFeifel PFingscheidt TGannamaneni SGhobadi SHammam AHaselhoff AHauser FHeinzemann CHoffmann MKapoor NKappel FKlingner MKronenberger JKüppers FLöhdefink JMlynarski MMock MMualla FPavlitskaya SPoretschkin MPohl ARavi-Kumar VRosenzweig JRottmann MRüping SSämann TSchneider JSchulz ESchwalbe GSicking JSrivastava TVarghese SWeber MWirkert SWirtz TWoehrle M(2022)Inspect, Understand, Overcome: A Survey of Practical Methods for AI SafetyDeep Neural Networks and Data for Automated Driving10.1007/978-3-031-01233-4_1(3-78)Online publication date: 18-Jun-2022
https://doi.org/10.1007/978-3-031-01233-4_1

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents