research-article

Rotation-equivariant correspondence matching based on a dual-activation mixer

Authors:

Chengju Liu, and

Qijun ChenAuthors Info & Claims

Volume 568, Issue C

https://doi.org/10.1016/j.neucom.2023.127053

Published: 14 March 2024 Publication History

Abstract

Learning-based correspondence matching methods have become the mainstream techniques in many computer vision and robotics applications due to their robustness to large illumination and viewpoint changes. However, it is difficult for conventional convolutional neural networks (CNNs) to extract rotation-equivariant local features. Recent work has shown that CNNs combined with group-equivariant architectures are surprisingly effective at matching correspondences even when the images are rotated to a dramatic extent. However, the inherent shape (square) of convolution kernels causes the performance bottleneck of such rotation-equivariant CNNs. To address this issue, we propose an adaptive dual rotation-equivariant correspondence matching algorithm, which performs stably at all angles. We mathematically analyze the effectiveness of our proposed rotation-equivariant correspondence matching approach and its performance with respect to different convolution kernels. Extensive experiments on the Rotated-HPatches, SIM2E, and MegaDepth datasets demonstrate the effectiveness of our proposed algorithm.

References

[1]

Ma J., et al., Image matching from handcrafted to deep features: A survey, Int. J. Comput. Vis. 129 (2021) 23–79.

Digital Library

[2]

Jiang X., et al., A review of multimodal image matching: Methods and applications, Inf. Fusion 73 (2021) 22–71.

Digital Library

[3]

Ma J., et al., LMR: Learning a two-class classifier for mismatch removal, IEEE Trans. Image Process. 28 (8) (2019) 4045–4059.

[4]

Jiang X., et al., Robust feature matching using spatial clustering with heavy outliers, IEEE Trans. Image Process. 29 (2019) 736–746.

[5]

Jiang X., et al., Robust feature matching for remote sensing image registration via linear adaptive filtering, IEEE Trans. Geosci. Remote Sens. 59 (2) (2020) 1577–1591.

[6]

Jiang X., et al., Learning for mismatch removal via graph attention networks, ISPRS J. Photogramm. Remote Sens. 190 (2022) 181–195.

[7]

Fan A., et al., Efficient deterministic search with robust loss functions for geometric model fitting, IEEE Trans. Pattern Anal. Mach. Intell. 44 (11) (2021) 8212–8229.

[8]

Lowe D.G., Object recognition from local scale-invariant features, in: Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 2, IEEE, 1999, pp. 1150–1157.

[9]

Lowe D.G., Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2) (2004) 91–110.

Digital Library

[10]

Leutenegger S., et al., BRISK: Binary robust invariant scalable keypoints, in: 2011 International Conference on Computer Vision, IEEE, 2011, pp. 2548–2555.

[11]

Guo J., et al., Learning for feature matching via graph context attention, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1–14.

[12]

Shi Z., et al., JRA-Net: Joint representation attention network for correspondence learning, Pattern Recognit. 135 (2023).

[13]

Chen S., et al., SSL-Net: Sparse semantic learning for identifying reliable correspondences, Pattern Recognit. (2023).

[14]

Liu X., et al., Pgfnet: Preference-guided filtering network for two-view correspondence learning, IEEE Trans. Image Process. 32 (2023) 1367–1378.

[15]

D. DeTone, et al., Superpoint: Self-supervised interest point detection and description, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 224–236.

[16]

Revaud J., et al., R2d2: Reliable and repeatable detector and descriptor, in: Advances in Neural Information Processing Systems, Vol. 32, 2019.

[17]

Cohen T., et al., Group equivariant convolutional networks, in: International Conference on Machine Learning, PMLR, 2016, pp. 2990–2999.

[18]

Peri A., et al., ReF–Rotation equivariant features for local feature matching, 2022, arXiv preprint arXiv:2203.05206.

[19]

Liu Y., et al., Gift: Learning transformation-invariant dense visual descriptors via group cnns, Adv. Neural Inf. Process. Syst. 32 (2019).

[20]

Parihar U.S., et al., RoRD: Rotation-robust descriptors and orthographic views for local feature matching, in: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, IEEE, 2021, pp. 1593–1600.

[21]

P.-E. Sarlin, et al., Superglue: Learning feature matching with graph neural networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4938–4947.

[22]

J. Xu, et al., SGMNet: Learning rotation-invariant point cloud representations via sorted Gram matrix, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10468–10477.

[23]

Esteves C., et al., Polar transformer networks, 2017, arXiv preprint arXiv:1709.01889.

[24]

D. Marcos, et al., Rotation equivariant vector field networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5048–5057.

[25]

Cohen T.S., et al., Spherical CNNS, 2018, arXiv preprint arXiv:1801.10130.

[26]

Weiler M., et al., General e (2)-equivariant steerable CNNS, Adv. Neural Inf. Process. Syst. 32 (2019).

[27]

Finzi M., et al., Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data, in: International Conference on Machine Learning, PMLR, 2020, pp. 3165–3176.

[28]

He L., et al., Efficient equivariant network, Adv. Neural Inf. Process. Syst. 34 (2021) 5290–5302.

[29]

G. Bökman, et al., A case for using rotation invariant features in state of the art feature matchers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5110–5119.

[30]

M. Weiler, et al., Learning steerable filters for rotation equivariant CNNS, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 849–858.

[31]

Bagad P., et al., C-3PO: Towards rotation equivariant feature detection and description, in: 3rd Visual Inductive Priors for Data-Efficient Deep Learning Workshop, 2022, URL: https://openreview.net/forum?id=dXouQ9ubkPJ.

[32]

Y. Tian, et al., L2-net: Deep learning of discriminative patch descriptor in Euclidean space, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 661–669.

[33]

Yu F., et al., Multi-scale context aggregation by dilated convolutions, 2015, arXiv preprint arXiv:1511.07122.

[34]

Z. Wang, et al., Smoothed dilated convolutions for improved dense prediction, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 2486–2495.

[35]

Sattler T., et al., Image retrieval for image-based localization revisited, in: BMVC, Vol. 1, no. 2, 2012, p. 4.

[36]

T. Sattler, et al., Benchmarking 6dof outdoor visual localization in changing conditions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 8601–8610.

[37]

V. Balntas, et al., HPatches: A benchmark and evaluation of handcrafted and learned local descriptors, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5173–5182.

[38]

Su S., et al., SIM2E: Benchmarking the group equivariant capability of correspondence matching algorithms, 2022, arXiv preprint arXiv:2208.09896.

[39]

Z. Li, et al., Megadepth: Learning single-view depth prediction from internet photos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2041–2050.

[40]

J.L. Schonberger, et al., Structure-from-motion revisited, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4104–4113.

Index Terms

Rotation-equivariant correspondence matching based on a dual-activation mixer
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
2. Theory of computation
  1. Design and analysis of algorithms
    1. Data structures design and analysis
      1. Pattern matching

Index terms have been assigned to the content through auto-classification.

Recommendations

SIM2E: Benchmarking the Group Equivariant Capability of Correspondence Matching Algorithms
Computer Vision – ECCV 2022 Workshops
Abstract
Correspondence matching is a fundamental problem in computer vision and robotics applications. Solving correspondence matching problems using neural networks has been on the rise recently. Rotation-equivariance and scale-equivariance are both ...
Read More
Stereo Correspondence Matching Using Multiwavelets
ICDT '10: Proceedings of the 2010 Fifth International Conference on Digital Telecommunications

this paper presents a novel multiwavelet-based stereo correspondence matching technique. A multiwavelet transform is first applied to a pair of stereo images to decorrelate the images into a number of approximation (baseband) and detail subbands. ...
Read More
Deep Rotation Equivariant Network
Abstract
Recently, learning equivariant representations has attracted considerable research attention. Dieleman et al. introduce four operations which can be inserted into convolutional neural network to learn deep representations equivariant ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Neurocomputing

Neurocomputing Volume 568, Issue C

Feb 2024

249 pages

ISSN:0925-2312

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 14 March 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents