research-article

Open access

Implementing Dense Optical Flow Computation on a Heterogeneous FPGA SoC in C

Authors:

Zhilei ChaiAuthors Info & Claims

ACM Transactions on Architecture and Code Optimization (TACO), Volume 13, Issue 3

Article No.: 25, Pages 1 - 25

https://doi.org/10.1145/2948976

Published: 17 August 2016 Publication History

Abstract

High-quality optical flow computation algorithms are computationally intensive. The low computational speed of such algorithms causes difficulties for real-world applications. In this article, we propose an optimized implementation of the classical Combine-Brightness-Gradient (CBG) model on the Xilinx ZYNQ FPGA-SoC, by taking advantage of the inherent algorithmic parallelism and ZYNQ architecture. The execution time decreases to 0.82 second with a lower power consumption (1.881W). It is better than software implementation on PC (Intel i7-3520M, 2.9GHz), which costs 2.635 seconds and 35W. We use C rather than HDLs to describe the algorithm for rapid prototyping.

References

[1]

Altera. 2015. Altera SoCs: When architecture matters. Retrieved April 21, 2016 from http://www.altera.com/ devices/processor/soc-fpga/overview/proc-soc-fpga.html.

[2]

Tomer Amiaz and Nahum Kiryati. 2006. Piecewise-smooth dense optical flow via level sets. Int. J. Comput. Vis. 68, 2 (2006), 111--124.

Digital Library

[3]

Pedro Cobos Arribas and Felix Monasterio Huelin Macia. 2003. FPGA implementation of Santos-Victor optical flow algorithm for real-time image processing: An useful attempt. In Microtechnologies for the New Millennium 2003. International Society for Optics and Photonics, 23--32.

[4]

Mohammad Reza Balazadeh Bahar and Ghader Karimian. 2012. High performance implementation of the Horn and Schunck optical flow algorithm on FPGA. In Proceedings of the 2012 20th Iranian Conference on Electrical Engineering (ICEE’12). IEEE, 736--741.

[5]

Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92, 1 (2011), 1--31.

Digital Library

[6]

John L. Barron, David J. Fleet, and Steven S. Beauchemin. 1994. Performance of optical flow techniques. Int. J. Comput. Vis. 12, 1 (1994), 43--77.

Digital Library

[7]

Berkeley Design Technology, Inc. 1994. An independent evaluation of: The AutoESL AutoPilot high-level synthesis tool. Retrieved April 21, 2016 from http://www.bdti.com/MyBDTI/pubs/AutoPilot.pdf.

[8]

Michael J. Black and Paul Anandan. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63, 1 (1996), 75--104.

Digital Library

[9]

Andre R. Brodtkorb, Christopher Dyken, Trond R. Hagen, Jon M. Hjelmervik, and Olaf O. Storaasli. 2010. State-of-the-art in heterogeneous computing. Sci. Program. 18, 1 (2010), 1--33.

Digital Library

[10]

Thomas Brox, Christoph Bregler, and Jagannath Malik. 2009. Large displacement optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). IEEE, 41--48.

[11]

Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004. Springer, 25--36.

[12]

Thomas Brox and Jitendra Malik. 2011. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 3 (2011), 500--513.

Digital Library

[13]

Andres Bruhn and Joachim Weickert. 2005. Towards ultimate motion estimation: Combining highest accuracy with real-time performance. In Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), Vol. 1. IEEE, 749--755.

Digital Library

[14]

Andrés Bruhn, Joachim Weickert, Christian Feddern, Timo Kohlberger, and Christoph Schnorr. 2005b. Variational optical flow computation in real time. IEEE Trans. Image Process. 14, 5 (2005), 608--615.

Digital Library

[15]

Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. 2005a. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Int. J. Comput. Vis. 61, 3 (2005), 211--231.

Digital Library

[16]

Zhilei Chai and Jianbo Shi. 2011. Improving KLT in embedded systems by processing oversampling video sequence in real-time. In Proceedings of the 2011 International Conference on Reconfigurable Computing and FPGAs (ReConFig’11). IEEE, 297--302.

Digital Library

[17]

Haiyang Chao, Yu Gu, and Marcello Napolitano. 2014. A survey of optical flow techniques for robotics navigation applications. J. Intell. Robot. Syst. 73, 1--4 (2014), 361--372.

Digital Library

[18]

Eric S. Chung, Peter A. Milder, James C. Hoe, and Ken Mai. 2010. Single-chip heterogeneous computing: Does the future include custom logic, FPGAs, and GPGPUs? In Proceedings of the 2010 43rd Annual IEEE/ACM International Symposium on Microarchitecture. IEEE Computer Society, 225--236.

Digital Library

[19]

D. J. Evans. 1984. Parallel SOR iterative methods. Parallel Comput. 1, 1 (1984), 3--18.

Digital Library

[20]

Jeremy Fowers, Greg Brown, Patrick Cooke, and Greg Stitt. 2012. A performance and energy comparison of FPGAs, GPUs, and multicores for sliding-window applications. In Proceedings of the ACM/SIGDA International Symposium on Field Programmable Gate Arrays. ACM, 47--56.

Digital Library

[21]

Gokhan Koray Gultekin and Afsar Saranli. 2013. An FPGA based high performance optical flow hardware design for computer vision applications. Microprocess. Microsyst. 37, 3 (2013), 270--286.

[22]

Pascal Gwosdek, Andrés Bruhn, and Joachim Weickert. 2010. Variational optic flow on the sony PlayStation 3. J. Real-Time Image Process. 5, 3 (2010), 163--177.

Digital Library

[23]

Bruno Herissé, Tarek Hamel, Robert Mahony, and François-Xavier Russotto. 2012. Landing a VTOL unmanned aerial vehicle on a moving platform using optical flow. IEEE Trans. Robot. 28, 1 (2012), 77--89.

Digital Library

[24]

Dominik Honegger, Pierre Greisen, Lorenz Meier, Petri Tanskanen, and Marc Pollefeys. 2012. Real-time velocity estimation based on optical flow and disparity matching. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’12). IEEE, 5177--5182.

[25]

Berthold K. Horn and Brian G. Schunck. 1981. Determining optical flow. In Proceedings of the1981 Technical Symposium East. International Society for Optics and Photonics, 319--331.

[26]

Mateusz Komorkiewicz, Tomasz Kryjak, and Marek Gorgon. 2014. Efficient hardware implementation of the Horn-Schunck algorithm for high-resolution real-time dense optical flow sensor. Sensors 14, 2 (2014), 2860--2891.

[27]

Michael Kunz, Alexander Ostrowski, and Peter Zipf. 2014. An FPGA-optimized architecture of Horn and Schunck optical flow algorithm for real-time applications. In Proceedings of the 2014 24th International Conference on Field Programmable Logic and Applications (FPL’14). IEEE, 1--4.

[28]

Vincenzo Lippiello and Bruno Siciliano. 2012. Wall inspection control of a VTOL unmanned aerial vehicle based on a stereo optical flow. In Proceedings 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’12). IEEE, 4296--4302.

[29]

Bruce D. Lucas, Takeo Kanade, and others. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence, Vol. 81. 674--679.

Digital Library

[30]

José L. Martín, Aitzol Zuloaga, Carlos Cuadrado, Jesús Lázaro, and Unai Bidarte. 2005. Hardware implementation of optical flow constraint equation using FPGAs. Comput. Vis. Image Underst. 98, 3 (2005), 462--490.

Digital Library

[31]

Middlebury. 2015. Datasets of Middlebury. Retrieved April 21, 2016 from http://vision.middlebury.edu/ flow/data/.

[32]

Yoshiki Mizukami and Katsumi Tadamura. 2007. Optical flow computation on compute unified device architecture. In Proceedings of the 14th International Conference on Image Analysis and Processing (ICIAP’07). IEEE, 179--184.

Digital Library

[33]

Basant Kumar Mohanty and Pramod Kumar Meher. 2013. Memory-efficient high-speed convolution-based generic structure for multilevel 2-D DWT. IEEE Trans. Circuits Syst. Video Technol. 23, 2 (2013), 353--363.

Digital Library

[34]

Ruzali Rustam, Nor Hisham Hamid, and Fawnizu Azmadi Hussin. 2012. FPGA-based hardware implementation of optical flow constraint equation of Horn and Schunck. In Proceedings of the 2012 4th International Conference on Intelligent and Advanced Systems (ICIAS’12), Vol. 2. IEEE, 790--794.

[35]

Narayanan Sundaram, Thomas Brox, and Kurt Keutzer. 2010. Dense point trajectories by GPU-accelerated large displacement optical flow. In Computer Vision--ECCV 2010. Springer, 438--451.

Digital Library

[36]

Patrik Sundberg, Thomas Brox, Michael Maire, Pablo Arbeláez, and Jitendra Malik. 2011. Occlusion boundary detection and figure/ground assignment from optical flow. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 2233--2240.

Digital Library

[37]

VDMA. 2015. LogiCORE IP AXI video direct memory access v5.02.a. Retrieved April 21, 2016 from http://www.xilinx.com/support/documentation/ip_documentation/axi_vdma/v5_02_a/pg020_axi_vdma.pdf.

[38]

Vivado. 2015. Vivado Design Suite - HLx editions. Retrieved April 21, 2016 from http://www.xilinx.com/ products/design-tools/vivado/.

[39]

Li Xu, Jiaya Jia, and Yasuyuki Matsushita. 2012. Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 9 (2012), 1744--1757.

Digital Library

[40]

Henning Zimmer, Andrés Bruhn, and Joachim Weickert. 2011. Optic flow in harmony. Int. J. Comput. Vis. 93, 3 (2011), 368--388.

Digital Library

[41]

ZYNQ. 2015. Expanding the all programmable SoC portfolio. Retrieved April 21, 2016 from http://www.xilinx.com/products/silicon-devices/soc/ZYNQ/index.htm.

Cited By

Kim DLee JJung WSullivan MKim JMohror KArnold DBadia R(2023)Unity ECC: Unified Memory Protection Against Bit and Chip ErrorsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3607081(1-16)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3607081
Sun CWu XSun JSun CXu MGe Q(2023)Saliency-Induced Moving Object Detection for Robust RGB-D Vision Navigation Under Complex Dynamic EnvironmentsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.327527924:10(10716-10734)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TITS.2023.3275279
Chen WSun RChai Z(2022)An acceleration method of optical flow calculation for automotive ADAS2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00117(878-887)Online publication date: Dec-2022
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00117
Show More Cited By

Index Terms

Implementing Dense Optical Flow Computation on a Heterogeneous FPGA SoC in C
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
2. Hardware
  1. Electronic design automation
    1. High-level and register-transfer level synthesis
      1. Hardware-software codesign
  2. Integrated circuits
    1. Reconfigurable logic and FPGAs
      1. Hardware accelerators

Recommendations

Implementing FPGA-based energy-efficient dense optical flow computation with high portability in C (abstract only)
FPGA '14: Proceedings of the 2014 ACM/SIGDA international symposium on Field-programmable gate arrays

Optical flow computation is widely used in many video/image based applications such as motion detection, video compression etc. Dense optical flow field that provides more details of information is more useful in lots of applications. However, high-...
Rapid Implementation of Embedded Systems using Xilinx Zynq Platform
SEEDA-CECNSM '16: Proceedings of the SouthEast European Design Automation, Computer Engineering, Computer Networks and Social Media Conference

In any digital system design, it is crucial to achieve the lowest time-to-market possible. Indeed, that need has pushed large FPGA manufacturers to produce SoCs which will implement reprogrammable logic along with CPU and DSP cores. Especially, during ...
Implementing high-performance, low-power FPGA-based optical flow accelerators in C
ASAP '13: Proceedings of the 2013 IEEE 24th International Conference on Application-specific Systems, Architectures and Processors (ASAP)

Recent developments in High-Level Synthesis (HLS) for FPGAs are making it possible to “run” C code on FPGAs thereby making modern programming environments available to FPGA developers. In this paper, C code for a complex optical-flow algorithm is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Architecture and Code Optimization

ACM Transactions on Architecture and Code Optimization Volume 13, Issue 3

September 2016

207 pages

ISSN:1544-3566

EISSN:1544-3973

DOI:10.1145/2988523

Editor:
Koen De Bosschere
Ghent University

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2016

Accepted: 01 May 2016

Revised: 01 April 2016

Received: 01 October 2015

Published in TACO Volume 13, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National High Technology Research and Development Program of China (a.k.a 863 program)
111 Project
NSF of China
the Shanghai Natural Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
1,085
Total Downloads

Downloads (Last 12 months)133
Downloads (Last 6 weeks)20

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim DLee JJung WSullivan MKim JMohror KArnold DBadia R(2023)Unity ECC: Unified Memory Protection Against Bit and Chip ErrorsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3607081(1-16)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3607081
Sun CWu XSun JSun CXu MGe Q(2023)Saliency-Induced Moving Object Detection for Robust RGB-D Vision Navigation Under Complex Dynamic EnvironmentsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.327527924:10(10716-10734)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TITS.2023.3275279
Chen WSun RChai Z(2022)An acceleration method of optical flow calculation for automotive ADAS2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00117(878-887)Online publication date: Dec-2022
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00117
Ling YYan YHuang KChen G(2022)Ultra-Flow: An Ultra-fast and High-quality Optical Flow Accelerator with Deep Feature Matching on FPGA2022 32nd International Conference on Field-Programmable Logic and Applications (FPL)10.1109/FPL57034.2022.00017(33-39)Online publication date: Aug-2022
https://doi.org/10.1109/FPL57034.2022.00017
Chai ZLi SHe QChen MChen W(2020)FPGA-Based ROI Encoding for HEVC Video Bitrate ReductionJournal of Circuits, Systems and Computers10.1142/S021812662050182029:11(2050182)Online publication date: 5-Feb-2020
https://doi.org/10.1142/S0218126620501820
Jang SKyung C(2020)Resource-Efficient and High-Throughput VLSI Design of Global Optical Flow Method for Mobile SystemsIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2020.2984822(1-9)Online publication date: 2020
https://doi.org/10.1109/TVLSI.2020.2984822
Xu MLiu BFu PLi JHu Y(2019)Video Saliency Detection via Graph Clustering With Motion Energy and Spatiotemporal ObjectnessIEEE Transactions on Multimedia10.1109/TMM.2019.291488921:11(2790-2805)Online publication date: Dec-2019
https://doi.org/10.1109/TMM.2019.2914889
Bell SPu JHegarty JHorowitz M(2018)Compiling Algorithms for Heterogeneous SystemsSynthesis Lectures on Computer Architecture10.2200/S00816ED1V01Y201711CAC04313:1(1-105)Online publication date: 17-Jan-2018
https://doi.org/10.2200/S00816ED1V01Y201711CAC043
Chen YZou WTang YLi XXu CKomodakis N(2018)SCOM: Spatiotemporal Constrained Optimization for Salient Object DetectionIEEE Transactions on Image Processing10.1109/TIP.2018.281316527:7(3345-3357)Online publication date: Jul-2018
https://doi.org/10.1109/TIP.2018.2813165
Pu JBell SYang XSetter JRichardson SRagan-Kelley JHorowitz M(2017)Programming Heterogeneous Systems from an Image Processing DSLACM Transactions on Architecture and Code Optimization10.1145/310795314:3(1-25)Online publication date: 16-Aug-2017
https://dl.acm.org/doi/10.1145/3107953
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents