research-article

Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement

Authors:

Maria E. Angelopoulou,

Christos-Savvas Bouganis,

Peter Y. K. Cheung,

George A. ConstantinidesAuthors Info & Claims

ACM Transactions on Reconfigurable Technology and Systems (TRETS), Volume 2, Issue 4

Article No.: 22, Pages 1 - 29

https://doi.org/10.1145/1575779.1575782

Published: 01 September 2009 Publication History

Abstract

The high density image sensors of state-of-the-art imaging systems provide outputs with high spatial resolution, but require long exposure times. This limits their applicability, due to the motion blur effect. Recent technological advances have lead to adaptive image sensors that can combine several pixels together in real time to form a larger pixel. Larger pixels require shorter exposure times and produce high-frame-rate samples with reduced motion blur. This work proposes combining an FPGA with an adaptive image sensor to produce an output of high resolution both in space and time. The FPGA is responsible for the spatial resolution enhancement of the high-frame-rate samples using super-resolution (SR) techniques in real time. To achieve it, this article proposes utilizing the Iterative Back Projection (IBP) SR algorithm. The original IBP method is modified to account for the presence of noise, leading to an algorithm more robust to noise. An FPGA implementation of this algorithm is presented. The proposed architecture can serve as a general purpose real-time resolution enhancement system, and its performance is evaluated under various noise levels.

References

[1]

Angelopoulou, M. E., Bouganis, C.-S., and Cheung, P. Y. K. 2008a. Video enhancement on an adaptive image sensor. In Proceedings of the IEEE International Conference on Image Processing (ICIP). 681--684.

[2]

Angelopoulou, M. E., Bouganis, C.-S., Cheung, P. Y. K., and Constantinides, G. A. 2008b. FPGA-based real-time super-resolution on an adaptive image sensor. In Proceedings of the International Workshop on Applied Reconfigurable Computing (ARC). 125--136.

Digital Library

[3]

Arulampalam, M. S., Maskell, S., Gordon, N., and Clapp, T. 2002. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Sig. Proc. 50, 2, 174--188.

Digital Library

[4]

Baker, S. and Kanade, T. 2002. Limits on super-resolution and how to break them. IEEE Trans. Patt. Anal. Mach. Intell. 24, 9, 1167--1183.

Digital Library

[5]

Ben-Ezra, M. and Nayar, S. K. 2004. Motion-based motion deblurring. IEEE Trans. Patt. Anal. Mach. Intell. 26, 6, 689--698.

Digital Library

[6]

Bonato, V., Marques, E., and Constantinides, G. A. 2007. A floating-point extended kalman filter implementation for autonomous mobile robots. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 576--579.

[7]

Bouguet, J.-Y. 2002. Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm. Microprocessor Research Labs, Intel Corporation.

[8]

Brown, L. G. 1992. A survey of image registration techniques. ACM Comput. Surv. 24, 4, 325--376.

Digital Library

[9]

Chen, T., Catrysse, P., Gamal, A. E., and Wandell, B. 2000. How small should pixel size be? In Proceedings of the SPIE Sensors and Camera Systems for Scientific, Industrial and Digital Photography Applications. Vol. 3965. 451--459.

[10]

Constandinou, T. G., Degenaar, P., and Toumazou, C. 2006. An adaptable foveating vision chip. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS). 3566--3569.

[11]

Farrell, J., Xiao, F., and Kavusi, S. 2006. Resolution and light sensitivity tradeoff with pixel size. In Proceedings of the SPIE Electronic Imaging ’06 Conference. Vol. 6069. 211--218.

[12]

Farsiu, S., Robinson, D., Elad, M., and Milanfar, P. 2004. Advances and challenges in super-resolution. Int. J. Imag. Syst. Tech. 14, 2, 47--57.

[13]

Foveon, Inc. 2007. Foveon X3 Image Sensor. www.foveon.com.

[14]

Gamal, A. E. and Eltoukhy, H. 2005. CMOS image sensors. IEEE Circ. Devic. Mag. 21, 3, 6--20.

[15]

Irani, M. and Peleg, S. 1991. Improving resolution by image registration. Graph. Mod. Image Proc. 53, 3, 231--239.

Digital Library

[16]

Liu, Y., Bouganis, C.-S., and Cheung, P. Y. K. 2007. Efficient mapping of a Kalman filter into an FPGA using Taylor expansion. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 345--350.

[17]

Mendis, S., Kemeny, S., Gee, R., Pain, B., Staller, C., Kim, Q., and Fossum, E. 1997. CMOS active pixel image sensors for highly integrated imaging systems. IEEE J. Solid-State Circ. 32, 2, 187--197.

[18]

Park, S. C., Park, M. K., and Kang, M. G. 2003. Super-resolution image reconstruction: a technical overview. IEEE Sig. Proc. Mag. 20, 3, 21--36.

[19]

Shechtman, E., Caspi, Y., and Irani, M. 2005. Space-time super-resolution. IEEE Trans. Patt. Anal. Mach. Intell. 27, 4, 531--545.

Digital Library

[20]

Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 593--600.

[21]

Sroubek, F., Cristobal, G., and Flusser, J. 2007. A unified approach to super-resolution and multichannel blind deconvolution. IEEE Trans. Image Proc. 16, 9, 2322--2332.

Digital Library

[22]

Stark, H. and Oskoui, P. 1989. High-resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Amer. A 6, 11, 1715--1726.

[23]

Stern, A., Porat, Y., Ben-Dor, A., and Kopeika, N. S. 2001. Enhanced-resolution image restoration from a sequence of low-frequency vibrated images by use of convex projections. Appl. Opt. 40, 26, 4706--4715.

[24]

Thrun, S., Burgard, W., and Fox, D. 2005. Probabilistic Robotics. MIT Press, Cambridge, MA.

[25]

Tian, H., Fowler, B., and Gamal, A. E. 2001. Analysis of temporal noise in CMOS photodiode active pixel sensor. IEEE J. Solid-State Circ. 36, 1, 92--101.

[26]

Wan, E. A. and Merwe, R. V. D. 2000. The unscented Kalman filter for nonlinear estimation. In Proceedings of the IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC). 153--158.

[27]

Welch, G. and Bishop, G. 2006. An introduction to the Kalman filter. http://www.cs.unc.edu/welch/media/pdf/kalman_intro.pdf.

[28]

Youla, D. C. and Webb, H. 1982. Image restoration by the method of convex projections: part 1-theory. IEEE Trans. Med. Imag. MI-1, 2, 81--94.

[29]

Zitová, B. and Flusser, J. 2003. Image registration methods: A survey. Imag. Vis. Comput. 21, 11, 977--1000.

Cited By

Sun KKoch MWang ZJovanovic SRabah HSimon S(2022)An FPGA-Based Residual Recurrent Neural Network for Real-Time Video Super-ResolutionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2021.308024132:4(1739-1750)Online publication date: Apr-2022
https://doi.org/10.1109/TCSVT.2021.3080241
Georgis GLentaris GReisis D(2019)Acceleration techniques and evaluation on multi-core CPU, GPU and FPGA for image processing and super-resolutionJournal of Real-Time Image Processing10.1007/s11554-016-0619-616:4(1207-1234)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s11554-016-0619-6
Kusano HIkebe MAsai TMotomura M(2016)An FPGA-optimized architecture of anti-aliasing based super resolution for real-time HDTV to 4K- and 8K-UHD conversions2016 International Conference on ReConFigurable Computing and FPGAs (ReConFig)10.1109/ReConFig.2016.7857153(1-6)Online publication date: Nov-2016
https://doi.org/10.1109/ReConFig.2016.7857153
Show More Cited By

Index Terms

Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement
1. Hardware
  1. Integrated circuits
    1. Logic circuits
  2. Very large scale integration design
    1. Application-specific VLSI designs
      1. Application specific processors

Recommendations

An FPGA-based design for real-time Super Resolution Reconstruction
ICDSC '18: Proceedings of the 12th International Conference on Distributed Smart Cameras

Since several decades, the camera spatial resolution is gradually increasing with the CMOS technology evolution. The image sensors provide more and more pixels, generating new constraints for the suitable optics. As an alternative, promising solutions ...
Deep Residual Attention Network for Spectral Image Super-Resolution
Computer Vision – ECCV 2018 Workshops
Abstract
Spectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be ...
Face image super-resolution through locality-induced support regression

In this paper we propose a novel face image super-resolution (SR) method named Locality-induced Support Regression (LiSR). Given a low-resolution (LR) input patch, we learn a mapping function between the local support LR and high-resolution (HR) patch ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Reconfigurable Technology and Systems

ACM Transactions on Reconfigurable Technology and Systems Volume 2, Issue 4

September 2009

134 pages

ISSN:1936-7406

EISSN:1936-7414

DOI:10.1145/1575779

Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 2009

Accepted: 01 October 2008

Revised: 01 September 2008

Received: 01 May 2008

Published in TRETS Volume 2, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
504
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun KKoch MWang ZJovanovic SRabah HSimon S(2022)An FPGA-Based Residual Recurrent Neural Network for Real-Time Video Super-ResolutionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2021.308024132:4(1739-1750)Online publication date: Apr-2022
https://doi.org/10.1109/TCSVT.2021.3080241
Georgis GLentaris GReisis D(2019)Acceleration techniques and evaluation on multi-core CPU, GPU and FPGA for image processing and super-resolutionJournal of Real-Time Image Processing10.1007/s11554-016-0619-616:4(1207-1234)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s11554-016-0619-6
Kusano HIkebe MAsai TMotomura M(2016)An FPGA-optimized architecture of anti-aliasing based super resolution for real-time HDTV to 4K- and 8K-UHD conversions2016 International Conference on ReConFigurable Computing and FPGAs (ReConFig)10.1109/ReConFig.2016.7857153(1-6)Online publication date: Nov-2016
https://doi.org/10.1109/ReConFig.2016.7857153
Xu WYin SLiu LLiu ZWei S(2015)High-Performance Motion Estimation for Image Sensors with Video CompressionSensors10.3390/s15082075215:8(20752-20778)Online publication date: 21-Aug-2015
https://doi.org/10.3390/s150820752
Redlich RFigueroa MTorres SPezoa J(2015)Embedded nonuniformity correction in infrared focal plane arrays using the Constant Range algorithmInfrared Physics & Technology10.1016/j.infrared.2015.01.02669(164-173)Online publication date: Mar-2015
https://doi.org/10.1016/j.infrared.2015.01.026
Angelopoulou MBouganis C(2014)Vision-Based Egomotion Estimation on FPGA for Unmanned Aerial Vehicle NavigationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2013.229135624:6(1070-1083)Online publication date: Jun-2014
https://doi.org/10.1109/TCSVT.2013.2291356
Bengtsson TMcKelvey TYu-Hua Gu I(2013)Super-Resolution reconstruction of High Dynamic Range images with perceptual weighting of errors2013 IEEE International Conference on Acoustics, Speech and Signal Processing10.1109/ICASSP.2013.6638047(2212-2216)Online publication date: May-2013
https://doi.org/10.1109/ICASSP.2013.6638047
Pang ZDai HTan HChen D(2013)An Improved Low-Cost Adaptive Bilinear Image Interpolation AlgorithmProceedings of the 2nd International Conference on Green Communications and Networks 2012 (GCN 2012): Volume 110.1007/978-3-642-35419-9_81(691-699)Online publication date: 1-Feb-2013
https://doi.org/10.1007/978-3-642-35419-9_81
Tomasi MVanegas MBarranco FDaz JRos E(2012)Massive Parallel-Hardware Architecture for Multiscale Stereo, Optical Flow and Image-Structure ComputationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2011.216226022:2(282-294)Online publication date: 1-Feb-2012
https://dl.acm.org/doi/10.1109/TCSVT.2011.2162260
Angelopoulou MBouganis CCheung P(2011)Blur identification with assumption validation for sensor-based video reconstruction and its implementation on field programmable gate arrayIET Computers & Digital Techniques10.1049/iet-cdt.2009.00535:4(271)Online publication date: 2011
https://doi.org/10.1049/iet-cdt.2009.0053
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents