research-article

Fully pipelined FPGA-based architecture for real-time SIFT extraction

Authors:

John Vourvoulakis,

John Kalomiros,

John LygourasAuthors Info & Claims

Microprocessors & Microsystems, Volume 40, Issue C

Pages 53 - 73

https://doi.org/10.1016/j.micpro.2015.11.013

Published: 01 February 2016 Publication History

Abstract

A fully pipelined FPGA-based architecture for real-time SIFT detector and descriptor is proposed.Each descriptor vector is extracted in one clock cycle, i.e. 46 ns considering Cyclone IV technology.The overall frame rate is independent of the number of detected features.Measures of response, repeatability and matching capability are maintained very high.Efficient resource usage allows the architecture to be hosted in a mid-range FPGA device. Image feature extraction constitutes a fundamental task in robotic vision applications. Scale-Invariant Feature Transform (SIFT) has been widely used as a robust method for detecting and matching features. Nevertheless, SIFT algorithm is computationally demanding and its implementation in an embedded system requires a subtle approach. In this paper, an optimized and fully pipelined architecture is proposed for real-time detection of SIFT keypoints and extraction of SIFT descriptors. The system is suitable to target robotic vision applications and it is pipelined on pixel basis. The architecture is hosted in a medium-scale Cyclone IV FPGA device clocked at 21.7 MHz and is capable of extracting a feature with its descriptor at every clock cycle, i.e. in 46 ns. This processing speed is independent of the number of features detected in the input image and it therefore represents a very high SIFT throughput, adequate for the most demanding SIFT-based robotic applications. The system can process 70 fps in VGA resolution, while it keeps power dissipation at low levels. Moreover, the proposed implementation achieves high response and repeatability values and its matching ability is directly comparable with floating point software-based SIFT implementations. Design details are given for the combinational and RAM-based circuits forming the SIFT datapath. Display Omitted

References

[1]

H.P. Moravec, Towards automatic visual obstacle avoidance, in: Proceedings of 5th International Joint Conference on Artificial Intelligence, IJCAI'77, 1977.

[2]

H.P. Moravec, Rover visual obstacle avoidance, in: Proceedings of Seventh International Joint Conference on Artificial Intelligence, 1981, pp. 785-790.

[3]

C. Harris, M. Stephens, A combined corner and edge detector, in: Proceedings of Alvey Vision Conference 1988, Alvey Vision Club, 1988, pp. 147-151.

[4]

D.G. Lowe, Object recognition from local scale-invariant features, in: Proceedings of IEEE International Joint Conference on Computer Vision, IEEE, 1999, pp. 150-1157.

[5]

D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis., 60 (2004) 91-110.

Digital Library

[6]

H. Bay, T. Tuytelaars, L. Van Gool, SURF: speeded up robust features, in: Lecture Notes Computer Science (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), Berlin, Heidelberg, 2006: pp. 404-417.

[7]

H. Bay, A. Ess, T. Tuytelaars, L. Van Gool, Speeded-up robust features, Comput. Vis. Image Underst., 110 (2008) 346-359.

Digital Library

[8]

S. Leutenegger, M. Chli, R.Y. Siegwart, BRISK: Binary Robust invariant scalable keypoints, in: Proceedings of IEEE International Joint Conference on Computer Vision, IEEE, 2011, pp. 2548-2555.

Digital Library

[9]

M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, P. Fua, BRIEF: computing a local binary descriptor very fast, IEEE Trans. Pattern Anal. Mach. Intell., 34 (2011) 1281-1298.

Digital Library

[10]

V. Bonato, E. Marques, G.A. Constantinides, A Parallel Hardware architecture for scale and rotation invariant feature detection, IEEE Trans. Circuits Syst. Video Technol., 18 (2008) 1703-1712.

[11]

M. Boulekchour, N. Aouf, M. Richardson, Robust L∞, Convex optimisation for monocular visual odometry trajectory estimation, Robotica. (2014) 1-20.

[12]

A.A. Fathima, R. Karthik, V. Vaidehi, Image stitching with combined moment invariants and sift features, Procedia Comput. Sci., 19 (2013) 420-427.

[13]

H. Song, H. Xiao, W. He, F. Wen, K. Yuan, A fast stereovision measurement algorithm based on SIFT keypoints for mobile robot, in: Proceedings of IEEE International Conference on Mechatronics Automation, IEEE, 2013, pp. 1743-1748.

[14]

X. Wu, Q. Zhao, W. Bu, A SIFT-based contactless palmprint verification approach using iterative RANSAC and local palmprint descriptors, Pattern Recognit., 47 (2014) 3314-3326.

[15]

Q. Zhang, Y. Chen, Y. Zhang, Y. Xu, SIFT implementation and optimization for multi-core systems, in: Proceedings of IEEE International Symposium on Parallel Distributed Processing, IEEE, 2008, pp. 1-8.

[16]

S.N. Sinha, J.-M. Frahm, M. Pollefeys, Y. Genc, Feature tracking and matching in video using programmable graphics hardware, Mach. Vis. Appl., 22 (2007) 207-217.

[17]

S. Heymann, K. Müller, A. Smolic, B. Fröhlich, T. Wiegand, SIFT implementation and optimization for general-purpose GPU, in: Proceedings of the 15th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision 2007, WSCG'2007 in Co-Operation with EUROGRAPHICS, 2007, pp. 317-322.

[18]

M. Lalonde, D. Byrns, L. Gagnon, N. Teasdale, D. Laurendeau, Real-time eye blink detection with GPU-based SIFT tracking, in: Proceedings of Fourth Canadian Conference on Computer and Robot Vision (CRV '07), IEEE, 2007, pp. 481-487.

Digital Library

[19]

L. Yao, H. Feng, Y. Zhu, Z. Jiang, D. Zhao, W. Feng, An architecture of optimised SIFT feature detection for an FPGA implementation of an image matcher, in: Proceedings of IEEE International Joint Conference on Field-Programmable Technology, IEEE, 2009, pp. 30-37.

[20]

K. Mizuno, H. Noguchi, G. He, Y. Terachi, T. Kamino, H. Kawaguchi, Fast and low-memory-bandwidth architecture of sift descriptor generation with scalability on speed and accuracy for VGA video, in: Proceedings of International Joint Conference on International Conference on Field Programmable Logic and Applications, IEEE, 2010, pp. 608-611.

[21]

K. Mizuno, H. Noguchi, G. He, Y. Terachi, T. Kamino, T. Fujinaga, A low-power real-time SIFT descriptor generation engine for full-HDTV video recognition, IEICE Trans. Electron, E94-C (2011) 448-457.

[22]

L. Chang, J. Hernández-Palancar, L.E. Sucar, M. Arias-Estrada, FPGA-based detection of SIFT interest keypoints, Mach. Vis. Appl., 24 (2012) 371-392.

[23]

S. Zhong, J. Wang, L. Yan, L. Kang, Z. Cao, A real-time embedded architecture for SIFT, J. Syst. Archit., 59 (2013) 16-29.

Digital Library

[24]

F.-C. Huang, S.-Y. Huang, J.-W. Ker, Y.-C. Chen, High-performance sift hardware accelerator for real-time image feature extraction, IEEE Trans. Circuits Syst. Video Technol., 22 (2012) 340-351.

Digital Library

[25]

T. Suzuki, T. Ikenaga, SIFT-based low complexity keypoint extraction and its real-time hardware implementation for full-HD video, in: Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2012, 2012, pp. 1-6.

[26]

T. Suzuki, T. Ikenaga, Low complexity keypoint extraction based on SIFT descriptor and its hardware implementation for full-HD 60¿fps Video, IEICE Trans. Fundam. Electron. Commun. Comput. Sci., E96.A (2013) 1376-1383.

[27]

E.S. Kim, H.-J. Lee, A novel hardware design for SIFT generation with reduced memory requirement, J. Semicond. Technol. Sci., 13 (2013) 157-169.

[28]

L.-C. Chiu, T.-S. Chang, J.-Y. Chen, N.Y.-C. Chang, Fast SIFT design for real-time visual feature extraction, IEEE Trans. Image Process., 22 (2013) 3158-3167.

[29]

J. Wang, S. Zhong, L. Yan, Z. Cao, An Embedded System-on-chip architecture for real-time visual detection and matching, IEEE Trans. Circuits Syst. Video Technol., 24 (2014) 525-538.

Digital Library

[30]

J. Jiang, X. Li, G. Zhang, SIFT Hardware Implementation for real-time image feature extraction, IEEE Trans. Circuits Syst. Video Technol., 24 (2014) 1209-1220.

[31]

M. Brown, D. Lowe, invariant features from interest point groups, in: Proceedings of the British Machine Vision Conference, BMVC 2002, 2002, pp. 656-665.

[32]

J.A. Kalomiros, Optimization of a scale-invariant feature detector using scale-space scans, in: Proceedings of the IEEE 7th International Conference on Intelligent Data Acquisition and Advanced Computing Systems, IEEE, 2013, pp. 388-393.

[33]

J. Vourvoulakis, J. Lygouras, J. Kalomiros, Hardware implementation of an optimized scale-invariant feature detector for robotic applications, in: Proceedings of the IEEE International Conference on Imaging Imaging Systems and Techniques, IEEE, 2014, pp. 226-231.

[34]

R. Andraka, Survey of CORDIC algorithms for FPGA based computers, in: Proceedings of ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA, 1998, pp. 191-200.

Digital Library

[35]

K. Mikolajczyk, C. Schmid, Performance evaluation of local descriptors, IEEE Trans. Pattern Anal. Mach. Intell., 27 (2005) 1615-1630.

Digital Library

[36]

D. Scharstein, R. Szeliski, High-accuracy stereo depth maps using structured light, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003.

[37]

Yocto Project, (n.d.). https://www.yoctoproject.org/ (accessed November 3, 2015).

[38]

D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nešić, X. Wang, Pattern Recognition, Springer International Publishing, Cham, 2014.

Cited By

Tsai TWang RTung N(2024)Hardware architecture design for real-time SIFT extraction with reduced memory usageMultimedia Tools and Applications10.1007/s11042-023-15789-w83:2(6297-6317)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15789-w
Ali TBhowmik DNicol R(2023)Domain-Specific Optimisations for Image Processing on FPGAsJournal of Signal Processing Systems10.1007/s11265-023-01888-295:10(1167-1179)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1007/s11265-023-01888-2
Nandalike RSarojadevi H(2020)Multimodal image feature detection with ROI-based optimization for image registrationJournal of Real-Time Image Processing10.1007/s11554-018-0847-z17:4(1007-1013)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s11554-018-0847-z
Show More Cited By

Index Terms

Fully pipelined FPGA-based architecture for real-time SIFT extraction
1. Hardware
  1. Very large scale integration design
    1. Application-specific VLSI designs
      1. Application specific processors

Index terms have been assigned to the content through auto-classification.

Recommendations

FPGA accelerator for real-time SIFT matching with RANSAC support

A pipelined FPGA-based architecture for real-time SIFT feature matching with RANSAC support is proposed.Each feature in the current video frame is matched among the features of the previous frame that are saved to a RAM-based moving window in one clock ...
FPGA-based architecture of a real-time SIFT matcher and RANSAC algorithm for robotic vision applications

A fundamental problem in computer vision is finding correspondences between features in pairs of similar images. By comparing feature descriptors instead of pixel intensities, the matching capability is significantly increased. Keypoints extracted by ...
A real-time embedded architecture for SIFT

SIFT has shown a great success in various computer vision applications. However, its large computational complexity has been a challenge to most embedded implementations. This paper presents a low-cost embedded system based on a new architecture that ...

Comments

Information & Contributors

Information

Published In

cover image Microprocessors & Microsystems

Microprocessors & Microsystems Volume 40, Issue C

February 2016

190 pages

ISSN:0141-9331

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 February 2016

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tsai TWang RTung N(2024)Hardware architecture design for real-time SIFT extraction with reduced memory usageMultimedia Tools and Applications10.1007/s11042-023-15789-w83:2(6297-6317)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15789-w
Ali TBhowmik DNicol R(2023)Domain-Specific Optimisations for Image Processing on FPGAsJournal of Signal Processing Systems10.1007/s11265-023-01888-295:10(1167-1179)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1007/s11265-023-01888-2
Nandalike RSarojadevi H(2020)Multimodal image feature detection with ROI-based optimization for image registrationJournal of Real-Time Image Processing10.1007/s11554-018-0847-z17:4(1007-1013)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s11554-018-0847-z
Kazmi MAziz AAkhtar P(2019)An efficient and compact row buffer architecture on FPGA for real-time neighbourhood image processingJournal of Real-Time Image Processing10.1007/s11554-017-0690-716:5(1845-1858)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11554-017-0690-7
Kazmi MAziz AKundi D(2019)A Pareto-Optimal Multi-filter Architecture on FPGA for Image Processing ApplicationsCircuits, Systems, and Signal Processing10.1007/s00034-019-01083-438:10(4762-4786)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s00034-019-01083-4
Vourvoulakis JKalomiros JLygouras J(2018)FPGA-based architecture of a real-time SIFT matcher and RANSAC algorithm for robotic vision applicationsMultimedia Tools and Applications10.1007/s11042-017-5042-x77:8(9393-9415)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1007/s11042-017-5042-x
Sangeetha DDeepa P(2017)A low-cost and high-performance architecture for robust human detection using histogram of edge oriented gradientsMicroprocessors & Microsystems10.1016/j.micpro.2017.07.00953:C(106-119)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1016/j.micpro.2017.07.009
Vourvoulakis JKalomiros JLygouras J(2017)FPGA accelerator for real-time SIFT matching with RANSAC supportMicroprocessors & Microsystems10.1016/j.micpro.2016.11.01149:C(105-116)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1016/j.micpro.2016.11.011
Vourvoulakis JLygouras JKalomiros J(undefined)Acceleration of RANSAC algorithm for images with affine transformation2016 IEEE International Conference on Imaging Systems and Techniques (IST)10.1109/IST.2016.7738198(60-65)
https://dl.acm.org/doi/10.1109/IST.2016.7738198

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents