Survey on Normalization Techniques

This repo is for our paper survey paper on normalization techniques in training deep neural networks:

Normalization Techniques in Training DNNs: Methodology, Analysis and Application,

Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu and Ling Shao.

arXiv preprint arXiv:2009.12836

We hope this repo provide a more friendly way for readers to review/follow the related works.

=========================Update: ===============================

2021-06-18. V2: Update for the Tutorial of Normalization

2020-11-28. V1: The initial version for the Survey paper

Table of content

1. Methodology
2 Analysis
3. Application
Contact

Methodology

Normalizing Activations by Population Statistics

Efficient BackProp. Neural Networks: Tricks of the Trade, 1998. paper.
Accelerated Gradient Descent by Factor-Centering Decomposition. Technical Report, 1998. paper .
Deep Boltzmann Machines and the Centering Trick. Neural Networks: Tricks of the trade, 2012. paper.
Deep learning made easier by linear transformations in perceptrons. AISTATS, 2012. paper .
Mean-normalized stochastic gradient for large-scale deep learning. ICASSP, 2014. paper.
Natural Neural Networks. NeurIPS, 2015. paper.
Learning Deep Architectures via Generalized Whitened Neural Networks. ICML, 2017. paper

Normalizing Activations as Functions

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, 2015. paper , code.
Knowledge matters: Importance of prior information for optimization. JMLR, 2016. paper .
Recurrent Batch Normalization. ICLR, 2017. paper , code.
Batch normalized recurrent neural networks. ICASSP, 2016. paper.

Normalization Area Partitioning

Layer Normalization. arXiv:1607.06450, 2016. paper .
Group Normalization. ECCV, 2018. paper , code.
Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv:1607.08022, 2016. paper , code.
Positional Normalization. NeurIPS, 2019. paper , code.
Four Things Everyone Should Know to Improve Batch Normalization. ICLR, 2020. paper , code.
Local Context Normalization: Revisiting Local Normalization. CVPR, 2020. paper , code.
What is the best multi-stage architecture for object recognition?. ICCV, 2009. paper .
ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS, 2012. paper .
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes. ICLR, 2017. paper , code.

Normalization Operation

Decorrelated Batch Normalization. CVPR, 2018. paper , code.
Iterative Normalization: Beyond Standardization towards Efficient Whitening. CVPR, 2019. paper , code.
Whitening and Coloring transform for GANs. ICLR, 2019. paper , code.
An Investigation into the Stochasticity of Batch Whitening. CVPR, 2020. paper , code.
Network Deconvolution. ICLR, 2020. paper , code.
Channel Equilibrium Networks for Learning Deep Representation. ICML, 2020. paper , code.
Concept Whitening for Interpretable Image Recognition. arXiv:2002.01650, 2020. paper , code.
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization. arXiv:2005.02178, 2020. paper.
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning. arXiv:1610.06160, 2016. paper.
L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks. arXiv:1802.09769, 2018. paper .
Norm matters: efficient and accurate normalization schemes in deep networks. NeurIPS, 2018. paper , code.
Generalized Batch Normalization: Towards Accelerating Deep Neural Networks. AAAI, 2019. paper .
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. NeurIPS, 2016. paper , code.
Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization. ICLR, 2020. paper , code.
PowerNorm: Rethinking Batch Normalization in Transformers. ICML, 2020. paper , code.
Progressive Growing of GANs for Improved Quality, Stability, and Variation. ICLR, 2018. paper , code.
Root Mean Square Layer Normalization. NeurIPS, 2019. paper , code.
Online Normalization for Training Neural Networks. NeurIPS, 2019. paper , code.
Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction. arXiv:2006.12753, 2020. paper.

Normalization Representation Recovery

Whitening and Coloring transform for GANs. ICLR, 2019. paper , code.
Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition. INTERSPEECH, 2017. paper.
Multimodal Unsupervised Image-to-Image Translation. ECCV, 2018. paper , code.
U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation. ICLR, 2020. paper , code.
Instance-Level Meta Normalization. CVPR, 2019. paper , code.
Semantic Image Synthesis with Spatially-Adaptive Normalization. CVPR, 2019. paper , code.
Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise. AAAI, 2020. paper , code.
Attentive Normalization. ECCV, 2020. paper , code.
Understanding and Improving layer normalization. NeurIPS, 2019. paper , code.
Modulating early visual processing by language. NeurIPS, 2017. paper , code.
A Learned Representation For Artistic Style. ICLR, 2017. paper , code.

Multi-Mode

Training Faster by Separating Modes of Variation in Batch-normalized Models. TPAMI, 2019. paper , code.
Mode Normalization. ICLR, 2019. paper , code.

Combinational Normalization

Differentiable learning-to-normalize via switchable normalization. ICLR, 2019. paper , code.
SSN: Learning Sparse Switchable Normalization via SparsestMax. CVPR, 2019. paper , code.
Switchable Whitening for Deep Representation Learning. ICCV, 2019. paper , code.
Exemplar Normalization for Learning Deep Representation. CVPR, 2020. paper.
Differentiable Dynamic Normalization for Learning Deep Representation. ICML, 2019. paper .
Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks. NeurIPS, 2018. paper , code.
U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation. ICLR, 2020. paper , code.
TaskNorm: Rethinking Batch Normalization for Meta-Learning. ICML, 2020. paper , code.
Rethinking Normalization and Elimination Singularity in Neural Networks. arXiv:1911.09738, 2019. paper , code.
Evolving Normalization-Activation Layers. arXiv:2004.02967, 2020. paper , code.
Representative Batch Normalization with Feature Calibration. CVPR, 2021. paper
Exploiting Invariance in Training Deep Neural Networks. arXiv:2103.16634, 2021. paper

BN for More Robust Estimation

Kalman Normalization: Normalizing Internal Representations Across Network Layers. NeurIPS, 2018. paper , code.

Normalization as Functions Combining Population Statistics

Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models. NeurIPS, 2017. paper .
Density estimation using Real NVP. ICLR, 2017. paper , code.
Convergence Analysis of Batch Normalization for Deep Neural Nets. arXiv:1705.08011, 2017. paper .
Revisit Batch Normalization: New Understanding and Refinement via Composition Optimization. AISTATS, 2019. paper .
Online Normalization for Training Neural Networks. NeurIPS, 2019. paper , code.
Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization. ICLR, 2020. paper , code.
PowerNorm: Rethinking Batch Normalization in Transformers. ICML, 2020. paper , code.
Momentum Batch Normalization for Deep Learning with Small Batch Size. ECCV, 2020. paper .
Double forward propagation for memorized batch normalization. AAAI, 2018. paper .
Cross-iteration batch normalization. arXiv:2002.05712, 2020. paper , code.
Stochastic Normalization. NeurIPS, 2020. paper

Robust Inference Methods for BN

EvalNorm: Estimating Batch Normalization Statistics for Evaluation. ICCV, 2019. paper .
Four Things Everyone Should Know to Improve Batch Normalization. ICLR, 2020. paper , code.
An Investigation into the Stochasticity of Batch Whitening. CVPR, 2020. paper , code.
Rethinking" Batch" in BatchNorm. arXiv:2105.07576, 2021. paper

Normalizing Weights

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. NeurIPS, 2016. paper , code.
Centered Weight Normalization in Accelerating Training of Deep Neural Networks. ICCV, 2017. paper , code.
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. AAAI, 2018. paper , code.
Spectral normalization for generative adversarial networks. ICLR, 2018. paper , code.
Cosine normalization: Using cosine similarity instead of dot product in neural networks. ICANN, 2018. paper .
Weight standardization. arXiv:1903.10520, 2019. paper , code.
Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization. ICLR, 2020. paper , code.
Characterizing signal propagation to close the performance gap in unnormalized ResNets. ICLR, 2021. paper

(Approximating ) Orthogonality constraints
Unitary Evolution Recurrent Neural Networks. ICML, 2016. paper .
Full-Capacity Unitary Recurrent Neural Networks. NeurIPS, 2016. paper , code.
DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation. arXiv:1612.04035, 2016. paper .
On orthogonality and learning recurrent networks with long term dependencies. ICML, 2017. paper , code.
Learning Unitary Operators with Help From u(n). AAAI, 2017. paper , code.
Gated Orthogonal Recurrent Units: On Learning to Forget. arXiv:1706.02761, 2017. paper , code.
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. AAAI, 2018. paper , code.
Orthogonal Recurrent Neural Networks with Scaled {C}ayley Transform. ICML, 2018. paper , code.
Fine-grained Optimization of Deep Neural Networks. NeurIPS, 2019. paper.
Orthogonal deep neural networks. TPAMI, 2019. paper .
Orthogonal Convolutional Neural Networks. CVPR, 2020. paper , code.
Deep Isometric Learning for Visual Recognition. ICML, 2020. paper , code.
Controllable Orthogonalization in Training DNNs. CVPR, 2020. paper , code.
Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?. NeurIPS, 2018. paper , code.
Parseval Networks: Improving Robustness to Adversarial Examples. ICML, 2017. paper , code.
Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR, 2019. paper , code.
Efficient Riemannian optimization on the Stiefel manifold via the Cayley transform. ICLR, 2020. paper , code.

Normalizing Gradients

Block-normalized gradient method: An empirical study for training deep neural network. arXiv:1707.04822, 2017. paper.
Large batch training of convolutional networks. arXiv:1708.03888, 2017. paper , code.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. ICLR, 2020. paper , code.
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes. arXiv:2006.13484, 2020. paper .
Large Batch Training Does Not Need Warmup. arXiv:2002.01576, 2020. paper .
Gradient Centralization: A New Optimization Technique for Deep Neural Networks. ECCV, 2020. paper , code.
High-Performance Large-Scale Image Recognition Without Normalization. ICML, 2021. paper

Analysis

Scale Invariance in Stabilizing Training

Layer Normalization. arXiv:1607.06450, 2016. paper.
Data-Dependent Path Normalization in Neural Networks. ICLR, 2016. paper .
Riemannian approach to batch normalization. NeurIPS, 2017. paper , code.
New Interpretations of Normalization Methods in Deep Learning. AAAI, 2020. paper .
Norm matters: efficient and accurate normalization schemes in deep networks. NeurIPS, 2018. paper , code.
Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs. ECCV, 2020. paper , code.
Towards Accelerating Training of Batch Normalization: A Manifold Perspective. arXiv:2101.02916, 2021. paper
A spherical analysis of Adam with Batch Normalization. arXiv:2006.13382, 2020. paper
AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights. ICLR, 2021. paper

Learning Rate Auto-tuning
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization. ICLR, 2019. paper.
Spherical Perspective on Learning with Batch Norm. arXiv:2006.13382, 2020. paper , code.
A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent. ICML, 2019. paper.
Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory. arXiv:2002.10674, 2020. paper .
L2 regularization versus batch and weight normalization. arXiv:1706.05350, 2017. paper .
Projection Based Weight Normalization for Deep Neural Networks. arXiv:1710.02338, 2017. paper , code.
Three Mechanisms of Weight Decay Regularization. ICLR, 2019. paper , code.
An Exponential Learning Rate Schedule For Batch Normalized Networks. ICLR, 2020. paper .
Spherical Motion Dynamics of Deep Neural Networks with Batch Normalization and Weight Decay. arXiv:2006.08419, 2020. paper .

Improved Conditioning in Optimization

Second Order Properties of Error Surfaces. NeurIPS, 1990. paper .
Decorrelated Batch Normalization. CVPR, 2018. paper , code.
How Does Batch Normalization Help Optimization?. NeurIPS, 2018. paper .
An Exponential Learning Rate Schedule For Batch Normalized Networks. ICLR, 2020. paper .
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. ICML, 2019. paper , code.
Understanding Batch Normalization. NeurIPS, 2018. paper .
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks. NeurIPS, 2019. paper.
Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs. ECCV, 2020. paper , code.
Theoretical Understanding of Batch-normalization: A Markov Chain Perspective. arXiv:2003.01652, 2020. paper.
A Mean Field Theory of Batch Normalization. ICLR, 2019. paper .
Mean-field Analysis of Batch Normalization. arXiv:1903.02606, 2019. paper .
Characterizing Well-Behaved vs. Pathological Deep Neural Networks. ICML, 2019. paper , code.
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization. AISTATS, 2019. paper .
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers. ICML, 2020. paper.
Implicit Regularization and Convergence for Weight Normalization. NeurIPS, 2020. paper
Inductive Bias of Gradient Descent for Exponentially Weight Normalized Smooth Homogeneous Neural Nets. arXiv:2010.12909, 2020. paper

Stochasticity for Generalization

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks. ICML, 2018. paper .
Stochastic Normalizations as Bayesian Learning. ACCV, 2018. paper .
Uncertainty Estimation via Stochastic Batch Normalization. ICLR Workshop, 2018. paper.
Iterative Normalization: Beyond Standardization towards Efficient Whitening. CVPR, 2019. paper , code.
An Investigation into the Stochasticity of Batch Whitening. CVPR, 2020. paper , code.
Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise. AAAI, 2020. paper , code.
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift. arXiv:2006.10963, 2020. paper .

Representation

Group Whitening: Balancing Learning Efficiency and Representational Capacity. CVPR, 2021. paper, code
Training BatchNorm and Only BatchNorm: On the Expressivity of Random Features in CNNs. ICLR, 2021. paper

Application

Domain Adaptation

Revisiting Batch Normalization For Practical Domain Adaptation. arXiv:1603.04779, 2016. paper .
AutoDIAL: Automatic DomaIn Alignment Layers. ICCV, 2017. paper , code.
Domain-Specific Batch Normalization for Unsupervised Domain Adaptation. CVPR, 2019. paper , code.
A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation. WACV, 2019. paper , code.
Adversarial Examples Improve Image Recognition. CVPR, 2020. paper .
Unsupervised Domain Adaptation Using Feature-Whitening and Consensus Loss. CVPR, 2019. paper , code.
Transferable Normalization: Towards Improving Transferability of Deep Neural Networks. NeurIPS, 2019. paper , code.
Learning to Optimize Domain Specific Normalization for Domain Generalization. ECCV, 2020. paper .
Sandwich Batch Normalization. arXiv:2102.11382, 2021. paper

Domain Generalization

Prepare for the Worst: Generalizing across Domain Shifts with Adversarial Batch Normalization. arXiv:2009.08965, 2020. paper
Batch Normalization Embeddings for Deep Domain Generalization. arXiv:2011.12672, 2020. paper
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening. CVPR, 2021. paper

For Corruption Robustness

Improving robustness against common corruptions by covariate shift adaptation. NeurIPS, 2020. paper
Revisiting Batch Normalization for Improving Corruption Robustness. WACV, 2021. paper
Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR, 2021. paper
Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics. arXiv:2101.10842, 2021. paper
Adversarially Robust Classifier with Covariate Shift Adaptation. arXiv:2102.05096, 2021. paper

For Adversarial Robustness

Towards Defending Multiple Adversarial Perturbations via Gated Batch Normalization. arXiv:2012.01654, 2020. paper

For Person Re-identification

Rethinking the Distribution Gap of Person Re-identification with Camera-based Batch Normalization. ECCV 2020. paper.
Bridging the Distribution Gap of Visible-Infrared Person Re-identification with Modality Batch Normalization. arXiv:2103.04778, 2021. paper.
A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification. arXiv:1906.08332, 2019. paper

For Stereo

Domain-invariant stereo matching networks. ECCV, 2020. paper.

For Face Landmark/Recognition

Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training. arXiv:2101.06663, 2021. paper
Decomposed Meta Batch Normalization for Fast Domain Adaptation in Face Recognition. TNNLS, 2021.

Learning Universal Representations

Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv:1701.07275, 2017. paper.
Interpolating Convolutional Neural Networks Using Batch Normalization. ECCV, 2018. paper .
Efficient Multi-Domain Learning by Covariance Normalization. CVPR, 2019. paper , code.
K for the price of 1: Parameter efficient multi-task and transfer learning. ICLR, 2019. paper.

Style Transfer

Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv:1607.08022, 2016. paper , code.
A Learned Representation For Artistic Style. ICLR, 2017. paper , code.
Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. ICCV, 2017. paper , code.
FET-GAN: Font and Effect Transfer via K-shot Adaptive Instance Normalization. AAAI, 2020. paper , code.
Dynamic Instance Normalization for Arbitrary Style Transfer. AAAI, 2020. paper .
Universal style transfer via feature transforms. NeurIPS, 2017. paper , code.
Understanding Generalized Whitening and Coloring Transform for Universal Style Transfer. ICCV, 2019. paper .
Avatar-net: Multi-scale zero-shot style transfer by feature decoration. CVPR, 2018. paper , code.
Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization. arXiv:2103.11784, 2021. paper

Image Translation

Multimodal Unsupervised Image-to-Image Translation. ECCV, 2018. paper , code.
Image-to-image translation via group-wise deep whitening-and-coloring transformation. CVPR, 2019. paper .
Unpaired Image Translation via Adaptive Convolution-based Normalization. arXiv:1911.13271, 2019. paper .
Region Normalization for Image Inpainting. AAAI, 2020. paper , code.
Attentive Normalization for Conditional Image Generation. CVPR, 2020. paper , code.

Training GANs

Spectral normalization for generative adversarial networks. ICLR, 2018. paper , code.
Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR, 2019. paper , code.
Controllable Orthogonalization in Training DNNs. CVPR, 2020. paper , code.
Modulating early visual processing by language. NeurIPS, 2017. paper , code.
A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR, 2019. paper , code.
On Self Modulation for Generative Adversarial Networks. ICLR, 2019. paper , code.
An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation. arXiv:1908.00061, 2019. paper .

Efficient Deep Models

Learning Efficient Convolutional Networks through Network Slimming. ICCV, 2017. paper , code.
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers. ICLR, 2018. paper , code.
EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning. ECCV, 2020. paper , code.
Slimmable Neural Networks. ICLR, 2019. paper , code.
Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks. arXiv:2005.06828, 2020. paper .
Scalable methods for 8-bit training of neural networks. NeurIPS, 2018. paper , code.
Low-precision batch-normalized activations. arXiv:1702.08231, 2017. paper.
Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond. arXiv:2008.13128, 2020. paper .
Learning Recurrent Binary/Ternary Weights. ICLR, 2019. paper , code.
Normalization Helps Training of Quantized LSTM. NeurIPS, 2019. paper , code.
How Does Batch Normalization Help Binary Training. arXiv:1909.09139, 2019. paper .
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks. NeurIPS, 2019. paper.
Post-Training BatchNorm Recalibration. arXiv:2010.05625, 2020. paper
BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening. arXiv:2105.06423, 2021. paper
Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization. arXiv:2007.03230, 2020. paper

Meta learning

On first-order meta-learning algorithms. arXiv:1803.02999, 2018. paper , code.
Meta-Learning Probabilistic Inference for Prediction. ICLR, 2019. paper , code.
TaskNorm: Rethinking Batch Normalization for Meta-Learning. ICML, 2020. paper , code.
MetaNorm: Learning to Normalize Few-Shot Batches Across Domains . ICLR, 2021. paper

Reinforcement learning

Learning values across many orders of magnitude. NeurIPS, 2016. paper , code.
Crossnorm: Normalization for off-policy td reinforcement learning. arXiv:1902.05605, 2019. paper.
Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling. ICML, 2020. paper , code.

Unsupervised/Semi-supervised representation learning

Momentum Contrast for Unsupervised Visual Representation Learning. CVPR, 2020. paper , code.
Unsupervised Batch Normalization. CVPR Workshops, 2020. paper.
Exploring Simple Siamese Representation Learning. arXiv:2011.10566, 2020. paper.
Whitening for Self-Supervised Representation Learning. arXiv:2007.06346, 2020. paper
On Feature Decorrelation in Self-Supervised Learning. arXiv:2105.00470, 2021. paper
Barlow Twins: Self-Supervised Learning via Redundancy Reduction. arXiv:2103.03230, 2021. paper

Graph Neural Networks

PairNorm: Tackling Oversmoothing in GNNs. ICLR, 2020. paper, code
Towards Deeper Graph Neural Networks with Differentiable Group Normalization. NeurIPS, 2020. paper
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training. ICML, 2021. paper, code
Learning Graph Normalization for Graph Neural Networks. arXiv:2009.11746, 2020. paper, code

Applied in miscellaneous networks.

Learning to find good correspondences. CVPR, 2018. paper , code.
Attentive context normalization for robust permutation-equivariant learning. CVPR, 2020. paper , code.
Towards Understanding Normalization in Neural ODEs. arXiv:2004.09222, 2020. paper .
Riemannian batch normalization for SPD neural networks. NeurIPS, 2019. paper .
Batch Normalization is a Cause of Adversarial Vulnerability. arXiv:1905.02161, 2019. paper .
Towards an Adversarially Robust Normalization Approach. arXiv:2006.11007, 2020. paper , code.
Intriguing Properties of Adversarial Training at Scale. ICLR, 2020. paper , code.
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. ICLR, 2021. paper
Passport-aware Normalization for Deep Model Protection. NeurIPS, 2020. paper

Contact

Lei Huang - huanglei36060520 [at] gmail.com

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
LICENSE		LICENSE
README.md		README.md

License

huangleiBuaa/NormalizationSurvey

Folders and files

Latest commit

History

Repository files navigation