General nonlinearities in SO(2)-equivariant CNNs
Article No.: 695, Pages 9086 - 9098
Abstract
Invariance under symmetry is an important problem in machine learning. Our paper looks specifically at equivariant neural networks where transformations of inputs yield homomorphic transformations of outputs. Here, steerable CNNs have emerged as the standard solution. An inherent problem of steerable representations is that general nonlinear layers break equivariance, thus restricting architectural choices. Our paper applies harmonic distortion analysis to illuminate the effect of nonlinearities on Fourier representations of SO(2). We develop a novel FFT-based algorithm for computing representations of non-linearly transformed activations while maintaining band-limitation. It yields exact equivariance for polynomial (approximations of) nonlinearities, as well as approximate solutions with tunable accuracy for general functions. We apply the approach to build a fully E(3)-equivariant network for sampled 3D surface data. In experiments with 2D and 3D data, we obtain results that compare favorably to the state-of-the-art in terms of accuracy while permitting continuous symmetry and exact equivariance.
References
[1]
M. Atzmon, H. Maron, and Y. Lipman. Point convolutional neural networks by extension operators. ACM Trans. Graph., 37(4):71:1-71:12, 2018. URLhttps://doi.org/10.1145/3197517.3201301.
[2]
B. Charlier, J. Feydy, J. A. Glaunès, F.-D. Collin, and G. Durif. Kernel operations on the GPU, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1-6, 2021. URL http://jmlr.org/papers/v22/20-275.html.
[3]
C. Chen, G. Li, R. Xu, T. Chen, M. Wang, and L. Lin. ClusterNet: Deep hierarchical cluster network with rigorously rotation-invariant representation for point cloud analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[4]
X. Cheng, Q. Qiu, A. R. Calderbank, and G. Sapiro. RotDCF: Decomposition of convolutional filters for rotation-equivariant deep networks. In 7th International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=HlgTEj09FX.
[5]
D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by Exponential Linear Units (ELUs). In 4th International Conference on Learning Representations (ICLR), 2016. URL http://arxiv.org/abs/1511.07289.
[6]
T. Cohen and M. Welling. Group equivariant convolutional networks. In Proceedings of the 33nd International Conference on Machine Learning (ICML), 2016. URL http://proceedings.mlr. press/v48/cohencl6.html.
[7]
T. S. Cohen and M. Welling. Steerable CNNs. In 5th International Conference on Learning Representations, (ICLR), 2017. URL https://openreview.net/forum?id=rjqKYt511.
[8]
P. de Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In 9th International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=Jnspzp-oIZE.
[9]
C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. Vector Neurons: A general framework for SO(3)-equivariant networks. CoRR, abs/2104.12229, 2021. URL https://arxiv.org/abs/2104.12229.
[10]
S. Dieleman, J. D. Fauw, and K. Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. In Proceedings of the 33nd International Conference on Machine Learning (ICML), 2016. URL http://proceedings.mlr.press/v48/dieleman16.html.
[11]
C. Esteves, C. Allen-Blanchette, A. Makadia, and K. Daniilidis. 3D object classification and retrieval with Spherical CNNs. CoRR, abs/1711.06721, 2017. URL http://arxiv.org/abs/1711.06721.
[12]
R. P. Feynman, R. B. Leighton, and M. Sands. The Feynman lectures on physics; Vol. I. American Journal of Physics, 33(9):750-752, 1965.
[13]
J. Fox, B. Zhao, S. Rajamanickam, R. Ramprasad, and L. Song. Concentric spherical GNN for 3D representation learning. CoRR, abs/2103.10484, 2021. URL https://arxiv.org/abs/2103.10484.
[14]
A. S. Glassner. Principles of Digital Image Synthesis. Morgan Kaufmann Publishers, 1995.
[15]
A. Glielmo, P. Sollich, and A. De Vita. Accurate interatomic force fields via machine learning with covariant kernels. Phys. Rev. B, 95, Jun2017.
[16]
V Gottemukkula. Polynomial activation functions. In 8th International Conference on Learning Representations (ICLR), retracted paper, 2020. URL https://openreview.net/forum?id=rkxsgkHKvH.
[17]
R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-Or. MeshCNN: a network with an edge. ACM Trans. Graph., 38(4):90:1-90:12, 2019.
[18]
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In IEEE International Conference on Computer Vision, (ICCV), 2015.
[19]
E. Hoogeboom, J. W. T. Peters, T. S. Cohen, and M. Welling. HexaConv. In 6th International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=rlvuQG-CW.
[20]
S. Ioffe and C. Szegedy. Batch Normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015. URL http://proceedings.mlr.press/v37/ioffe15.html.
[21]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), 2015. URL http://arxiv.org/abs/1412.6980.
[22]
R. Kondor, H. T. Son, H. Pan, B. M. Anderson, and S. Trivedi. Covariant compositional networks for learning graphs. In 6th International Conference on Learning Representations, (ICLR), 2018. URL https://openreview.net/forum?id=SkIv3MAUf.
[23]
D. Laptev, N. Savinov, J. M. Buhmann, and M. Pollefeys. TI-POOLING: transformation-invariant pooling for feature learning in convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[24]
X. Li, R. Li, G. Chen, C. Fu, D. Cohen-Or, and P. Heng. A rotation-invariant framework for deep point cloud analysis. CoRR, abs/2003.07238, 2020. URL https://arxiv.org/abs/2003.07238.
[25]
Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. PointCNN: Convolution on X-transformed points. In Advances in Neural Information Processing Systems 31 (NeurIPS), 2018. URL https://proceedings.neurips.cc/paper/2018/hash/f5f8590cd58a54e94377e6ae2eded4d9-Abstract.html.
[26]
C. H. X. A. Mehmeti-Göpel, D. Hartmann, and M. Wand. Ringing ReLUs: Harmonic distortion analysis of nonlinear feedforward networks. In 9th International Conference on Learning Representations, ICLR, 2021. URL https://openreview.net/forum?id=TaYhv-qlXit.
[27]
H. Pfister, M. Zwicker, J. van Baar, and M. H. Gross. Surfels: surface elements as rendering primitives. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), 2000.
[28]
A. Poulenard, M. Rakotosaona, Y. Ponty, and M. Ovsjanikov. Effective rotation-invariant point CNN with spherical harmonics kernels. In 7th International Conference on 3D Vision (3DV), 2019.
[29]
C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[30]
C. R. Qi, L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. URL https://proceedings.neurips.cc/paper/2017/hash/d8bf84be3800dl2f74d8b05e9b89836f-Abstract.html.
[31]
Y. Rao, J. Lu, and J. Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[32]
K. Schütt, P. Kindermans, H. E. S. Felix, S. Chmiela, A. Tkatchenko, and K. Müller. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. URL https://proceedings.neurips.cc/paper/2017/hash/303ed4c69846ab36c2904d3ba8573050-Abstract.html.
[33]
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929–1958, 2014. URL http://dl.acm.org/citation.cfm?id=2670313.
[34]
H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3D shape recognition. In IEEE International Conference on Computer Vision (ICCV), 2015.
[35]
H. Thomas. Rotation-invariant point convolution with multiple equivariant alignments. In V. Struc and F. G. Fernández, editors, 8th International Conference on 3D Vision (3DV), 2020.
[36]
H. Thomas, C. R. Qi, J. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas. KPConv: Flexible and deformable convolution for point clouds. In IEEE International Conference on Computer Vision (ICCV), 2019.
[37]
N. Thomas, T. Smidt, S. M. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor Field Networks: Rotation- and translation-equivariant neural networks for 3D point clouds. CoRR, abs/1802.08219, 2018. URL http://arxiv.org/abs/1802.08219.
[38]
Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph CNN for learning on point clouds. ACMTrans. Graph., 38(5):146:1-146:12, 2019.
[39]
M. Weiler and G. Cesa. General E(2)-equivariant Steerable CNNs. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. URL https://proceedings.neurips.cc/paper/2019/hash/45d6637b718d0f24a237069fe41b0db4-Abstract.html.
[40]
M. Weiler, M. Geiger, M. Welling, W Boomsma, and T. Cohen. 3D Steerable CNNs: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems 31 (NeurIPS), 2018. URL https://proceedings.neurips.cc/paper/2018/hash/488e4104520c6aab692863ccldba45af-Abstract.html.
[41]
M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant CNNs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[42]
R. Wiersma, E. Eisemann, and K. Hildebrandt. CNNs on surfaces using rotation-equivariant features. ACM Trans. Graph., 39(4):92, 2020.
[43]
D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow. Harmonic Networks: Deep translation and rotation equivariance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[44]
Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[45]
Z. Zhang, B. Hua, D. W. Rosen, and S. Yeung. Rotation invariant convolutions for 3D point clouds deep learning. In 7th International Conference on 3D Vision (3DV), 2019.
[46]
Z. Zhang, B. Hua, W. Chen, Y. Tian, and S. Yeung. Global context aware convolutions for 3D point cloud understanding. In V. Struc and F. G. Fernández, editors, 8th International Conference on 3D Vision (3DV), 2020.
[47]
C. Zhao, J. Yang, X. Xiong, A. Zhu, Z. Cao, and X. Li. Rotation invariant point cloud classification: Where local geometry meets global topology. CoRR, abs/1911.00195, 2019. URL http://arxiv.org/abs/1911.00195.
Recommendations
General E(2)-equivariant steerable CNNs
NIPS'19: Proceedings of the 33rd International Conference on Neural Information Processing SystemsThe big empirical success of group equivariant networks has led in recent years to the sprouting of a great variety of equivariant network architectures. A particular focus has thereby been on rotation and reflection equivariant CNNs for planar images. ...
Rotation Equivariant CNNs for Digital Pathology
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018AbstractWe propose a new model for digital pathology segmentation, based on the observation that histopathology images are inherently symmetric under rotation and reflection. Utilizing recent findings on rotation equivariant CNNs, the proposed model ...
3D-Rotation-Equivariant Quaternion Neural Networks
Computer Vision – ECCV 2020AbstractThis paper proposes a set of rules to revise various neural networks for 3D point cloud processing to rotation-equivariant quaternion neural networks (REQNNs). We find that when a neural network uses quaternion features, the network feature ...
Comments
Information & Contributors
Information
Published In
December 2021
30517 pages
ISBN:9781713845393
Copyright © 2021 Neural Information Processing Systems Foundation, Inc.
Publisher
Curran Associates Inc.
Red Hook, NY, United States
Publication History
Published: 10 June 2024
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024