research-article

Open access

Handheld multi-frame super-resolution

Authors:

Bartlomiej Wronski,

Ignacio Garcia-Dorado,

Michael Krainin,

Chia-Kai Liang,

Peyman MilanfarAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 38, Issue 4

Article No.: 28, Pages 1 - 18

https://doi.org/10.1145/3306346.3323024

Published: 12 July 2019 Publication History

Abstract

Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to-noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google's flagship phone.

Supplementary Material

ZIP File (a28-wronski.zip)

Supplemental material

Download
18.62 MB

MP4 File (a28-wronski.mp4)

Download
197.41 MB

References

[1]

Adobe. 2012. Digital Negative (DNG) Specification. https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf.

[2]

Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. PAMI 24, 9 (2002), 1167--1183.

Digital Library

[3]

Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.

[4]

Stefanos P Belekos, Nikolaos P Galatsanos, and Aggelos K Katsaggelos. 2010. Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Processing 19, 6 (2010), 1451--1464.

Digital Library

[5]

Moshe Ben-Ezra, Assaf Zomet, and Shree K Nayar. 2005. Video super-resolution using controlled subpixel detector shifts. IEEE Trans. PAMI 27, 6 (2005), 977--987.

Digital Library

[6]

Josef Bigün, Goesta H. Granlund, and Johan Wiklund. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. PAMI 8 (1991), 775--790.

Digital Library

[7]

James F Blinn. 1982. A generalization of algebraic surface drawing. ACM TOG 1, 3 (1982), 235--256.

Digital Library

[8]

Edward Chang, Shiufun Cheung, and Davis Y Pan. 1999. Color filter array recovery using a threshold-based variable number of gradients. In Sensors, Cameras, and Applications for Digital Photography, Vol. 3650. 36--44.

[9]

CIPA. 2018. CIPA Report. www.cipa.jp/stats/report_e.html. {Online; accessed 29-Nov-2018}.

[10]

Sabrina Dammertz and Alexander Keller. 2008. Image synthesis by rank-1 lattices. In Monte Carlo and Quasi-Monte Carlo Methods 2006. 217--236.

[11]

Michael Drobot. 2014. Hybrid reconstruction anti-aliasing. In ACM SIGGRAPH Courses.

[12]

Joan Duran and Antoni Buades. 2014. Self-similarity and spectral correlation adaptive algorithm for color demosaicking. IEEE Trans. Image Processing 23, 9 (2014), 4031--4040.

[13]

Michael Elad and Arie Feuer. 1997. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Processing 6, 12 (1997), 1646--1658.

Digital Library

[14]

Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Processing 15, 1 (2006), 141--159.

Digital Library

[15]

David J Fleet and Allan D Jepson. 1990. Computation of component image velocity from local phase information. IJCV 5, 1 (1990), 77--104.

Digital Library

[16]

Flickr. 2017. Top Devices of 2017 on Flickr. https://blog.flickr.net/en/2017/12/07/top-devices-of-2017/. {Online; accessed 11-Jan-2019}.

[17]

Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008. Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Processing 17, 10 (2008), 1737--1754.

Digital Library

[18]

Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM TOG 35, 6 (2016), 191.

Digital Library

[19]

Clément Godard, Kevin Matzen, and Matt Uyttendaele. 2018. Deep burst denoising. In Proc. ECCV, Vol. 11219. 560--577.

[20]

Tomomasa Gotoh and Masatoshi Okutomi. 2004. Direct super-resolution and registration using raw CFA images. In Proc. CVPR, Vol. 2.

Digital Library

[21]

Chris Harris and Mike Stephens. 1988. A combined corner and edge detector. In Alvey Vision Conference, Vol. 15. 10--5244.

[22]

Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG 35, 6 (2016), 192.

Digital Library

[23]

Sung Hee Park and Marc Levoy. 2014. Gyro-based multi-image deconvolution for removing handshake blur. In Proc. ICCV. 3366--3373.

Digital Library

[24]

Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajkk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, et al. 2014. FlexISP: A flexible camera image processing framework. ACM TOG 33, 6 (2014), 231.

Digital Library

[25]

Carl W Helstrom. 1969. Detection and resolution of incoherent objects by a background-limited optical system. JOSA 59, 2 (1969), 164--175.

[26]

Robert Herzog, Elmar Eisemann, Karol Myszkowski, and H-P Seidel. 2010. Spatio-temporal upsampling on the GPU. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 91--98.

Digital Library

[27]

Keigo Hirakawa and Thomas W Parks. 2005. Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Processing 14, 3 (2005), 360--369.

Digital Library

[28]

Keigo Hirakawa and Thomas W Parks. 2006. Joint demosaicing and denoising. IEEE Trans. Image Processing 15, 8 (2006), 2146--2157.

Digital Library

[29]

Terence D Hunt. 2004. Image Super-Resolution Using Adaptive 2-D Gaussian Basis Function Interpolation. Technical Report. Air Force Inst of Tech Wright-Patterson AFB OH School of Engineering.

[30]

Michal Irani and Shmuel Peleg. 1991. Improving resolution by image registration. CVGIP: Graphical models and image processing 53, 3 (1991), 231--239.

Digital Library

[31]

Jorge Jimenez, Diego Gutierrez, Jason Yang, Alexander Reshetov, Pete Demoreuille, Tobias Berghoff, Cedric Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, Hugh Malan, Emil Persson, Dmitry Andreev, and Tiago Sousa. 2011. Filtering Approaches for Real-Time Anti-Aliasing. In ACM SIGGRAPH Courses.

Digital Library

[32]

Takeo Kanade and Masatoshi Okutomi. 1991. A stereo matching algorithm with an adaptive window: Theory and experiment. In Proc. IEEE ICRA. IEEE, 1088--1095.

[33]

Brian Karis. 2014. High-Quality Temporal Supersampling. In ACM SIGGRAPH Courses.

[34]

Darwin T Kuan, Alexander A Sawchuk, Timothy C Strand, and Pierre Chavel. 1985. Adaptive noise smoothing filter for images with signal-dependent noise. IEEE Trans. PAMI 2 (1985), 165--177.

Digital Library

[35]

Yeon Ju Lee and Jungho Yoon. 2010. Nonlinear image upsampling method based on radial basis function interpolation. IEEE Trans. Image Processing 19, 10 (2010), 2682--2692.

Digital Library

[36]

Brian Leung, Gwanggil Jeon, and Eric Dubois. 2011. Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking. IEEE Trans. Image Processing 20, 7 (2011), 1885--1894.

Digital Library

[37]

Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Trans. Graph. (Proc. SIGGRAPH) 37, 4 (2018), 139:1--139:13.

Digital Library

[38]

Xin Li, Bahadir Gunturk, and Lei Zhang. 2008. Image demosaicing: A systematic survey. In Visual Communications and Image Processing 2008, Vol. 6822. 68221J.

[39]

Ce Liu and Deqing Sun. 2011. A Bayesian approach to adaptive video super resolution. In Proc. CVPR. IEEE, 209--216.

Digital Library

[40]

Xiao-Ming Lu, Hari Krovi, Ranjith Nair, Saikat Guha, and Jeffrey H Shapiro. 2018. Quantum-optimal detection of one-versus-two incoherent optical sources with arbitrary separation. arXiv preprint arXiv:1802.02300 (2018).

[41]

Bruce D Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. (1981).

[42]

Rastislav Lukac and Konstantinos N Plataniotis. 2004. Normalized color-ratio modeling for CFA interpolation. IEEE Trans. Consumer Electronics 50, 2 (2004), 737--745.

Digital Library

[43]

Hugh Malan. 2012. Real-Time Global Illumination and Reflections in Dust 514. In ACM SIGGRAPH Courses.

[44]

John Marshall and E Geoffrey Walsh. 1956. Physiological tremor. Journal of neurology, neurosurgery, and psychiatry 19, 4 (1956), 260.

[45]

Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2018. Burst denoising with kernel prediction networks. In Proc. CVPR. 2502--2510.

[46]

Yusuke Monno, Daisuke Kiku, Masayuki Tanaka, and Masatoshi Okutomi. 2015. Adaptive residual interpolation for color image demosaicking. In Proc. IEEE ICIP. 3861--3865.

Digital Library

[47]

Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. 2005. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. 237--244.

Digital Library

[48]

NIH. 2018. Tremor Fact. www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Fact-Sheets/Tremor-Fact-Sheet. {Online; accessed 29-Nov-2018}.

[49]

Harry Nyquist. 1928. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers 47, 2 (1928), 617--644.

[50]

Athanasios Papoulis. 1977. Generalized sampling expansion. IEEE Trans. Circuits and Systems 24, 11 (1977), 652--654.

[51]

Cameron N Riviere, R Scott Rader, and Nitish V Thakor. 1998. Adaptive cancelling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomedical Engineering 45, 7 (1998), 839--846.

[52]

Dirk Robinson and Peyman Milanfar. 2004. Fundamental performance limits in image registration. IEEE Trans. Image Processing 13, 9 (2004), 1185--1199.

Digital Library

[53]

Dirk Robinson and Peyman Milanfar. 2006. Statistical performance analysis of super-resolution. IEEE Trans. Image Processing 15, 6 (2006), 1413--1428.

Digital Library

[54]

Yaniv Romano, John Isidoro, and Peyman Milanfar. 2017. RAISR: Rapid and accurate image super resolution. IEEE Trans. Computational Imaging 3, 1 (2017), 110--125.

[55]

Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In Proc. CVPR. 6626--6634.

[56]

Marco Salvi. 2016. An excursion in temporal super sampling. GDC2016 From the Lab Bench: Real-Time Rendering Advances from NVIDIA Research (2016).

[57]

E. A. Schäfer. 1886. On the rhythm of muscular response to volitional impulses in man. The Journal of Physiology 7, 2 (1886), 111--117.

[58]

Morteza Shahram and Peyman Milanfar. 2006. Statistical and information-theoretic analysis of resolution in imaging. IEEE Trans. Information Theory 52, 8 (2006), 3411--3437.

Digital Library

[59]

Masao Shimiziu and Masatoshi Okutomi. 2005. Sub-pixel estimation error cancellation on area-based matching. IJCV 63 (2005), 207--224. Issue 3.

Digital Library

[60]

Tiago Sousa. 2013. Graphics Gems CryENGINE3. In ACM SIGGRAPH Courses.

[61]

Molly M Sturman, David E Vaillancourt, and Daniel M Corcos. 2005. Effects of aging on the regularity of physiological tremor. Journal of Neurophysiology 93, 6 (2005), 3064--3074.

[62]

H Takeda, S Farsiu, and P Milanfar. 2006. Robust kernel regression for restoration and reconstruction of images from sparse noisy data. Proc. IEEE ICIP (2006), 1257--1260.

[63]

H Takeda, S Farsiu, and P Milanfar. 2007. Kernel regression for image processing and reconstruction. IEEE Trans. Image Processing 16, 2 (2007), 349.

Digital Library

[64]

Hanlin Tan, Xiangrong Zeng, Shiming Lai, Yu Liu, and Maojun Zhang. 2017. Joint demosaicing and denoising of noisy bayer images with ADMM. In Proc. IEEE ICIP. 2951--2955.

[65]

R.Y. Tsai and T.S. Huang. 1984. Multiframe image restoration and registration. Advance Computer Visual and Image Processing 1 (1984), 317--339.

[66]

Patrick Vandewalle, Karim Krichane, David Alleysson, and Sabine Süsstrunk. 2007. Joint demosaicing and super-resolution imaging from a set of unregistered aliased images. In Digital Photography III, Vol. 6502. International Society for Optics and Photonics, 65020A.

[67]

Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018), 64.

Digital Library

[68]

Hermann Weyl. 1910. Über die Gibbs'sche Erscheinung und verwandte Konvergenzphänomene. Rendiconti del Circolo Matematico di Palermo (1884--1940) 30, 1 (01 Dec 1910), 377--407.

[69]

Bartlomiej Wronski and Peyman Milanfar. 2018. See Better and Further with Super Res Zoom on the Pixel 3. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html.

[70]

Xiaolin Wu and Lei Zhang. 2006. Temporal color video demosaicking via motion estimation and data fusion. IEEE Trans. Circuits and Systems for Video Technology 16 (2006). Issue 2.

Digital Library

[71]

J Yen. 1956. On nonuniform sampling of bandwidth-limited signals. IRE Trans. Circuit Theory 3, 4 (1956), 251--257.

[72]

Jihun Yu and Greg Turk. 2013. Reconstructing surfaces of particle-based fluids using anisotropic kernels. ACM TOG 32, 1 (2013), 5.

Digital Library

[73]

Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color demosaicking bylocal directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging 20, 2 (2011), 023016.

Cited By

Zhang YLi TWei ZQu Y(2025)Optimization of sparse camera array arrangement using grid-based methodOptics Communications10.1016/j.optcom.2024.131137574(131137)Online publication date: Jan-2025
https://doi.org/10.1016/j.optcom.2024.131137
Liu HShao MWan YLiu YShang K(2025)SeBIR: Semantic-guided burst image restorationNeural Networks10.1016/j.neunet.2024.106834181(106834)Online publication date: Jan-2025
https://doi.org/10.1016/j.neunet.2024.106834
Afrasiabi MHosseini RAbbasfar A(2025)A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse codingSignal Processing: Image Communication10.1016/j.image.2024.117198130(117198)Online publication date: Jan-2025
https://doi.org/10.1016/j.image.2024.117198
Show More Cited By

Index Terms

Handheld multi-frame super-resolution
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Computational photography
      2. Image processing

Recommendations

High dynamic range and super-resolution from raw image bursts

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our ...
Solving a two-colour problem by applying probabilistic approach to a full-colour multi-frame image super-resolution

Reconstruction based algorithms play an important role in the multi-frame super-resolution problem. A group of images of the same scene are fused together to produce an image with higher spatial resolution, or with more visible details in the high ...
Style-Driven Image Enhancement for Entry-Level Mobile Devices
ICMVA '24: Proceedings of the 2024 7th International Conference on Machine Vision and Applications

Modern smartphones usually have automatic camera adjustment features that predetermine how images will be processed. Without an intervention from the user (e.g., manual adjustment of exposure settings, addition/removal of certain image filters), the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 38, Issue 4

August 2019

1480 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3306346

Editor:
Olga Sorkine-Hornung
ETH Zurich

Issue’s Table of Contents

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2019

Published in TOG Volume 38, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

158
Total Citations
View Citations
7,447
Total Downloads

Downloads (Last 12 months)1,160
Downloads (Last 6 weeks)184

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YLi TWei ZQu Y(2025)Optimization of sparse camera array arrangement using grid-based methodOptics Communications10.1016/j.optcom.2024.131137574(131137)Online publication date: Jan-2025
https://doi.org/10.1016/j.optcom.2024.131137
Liu HShao MWan YLiu YShang K(2025)SeBIR: Semantic-guided burst image restorationNeural Networks10.1016/j.neunet.2024.106834181(106834)Online publication date: Jan-2025
https://doi.org/10.1016/j.neunet.2024.106834
Afrasiabi MHosseini RAbbasfar A(2025)A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse codingSignal Processing: Image Communication10.1016/j.image.2024.117198130(117198)Online publication date: Jan-2025
https://doi.org/10.1016/j.image.2024.117198
Ahsanicka ASukma NMashuri NNainggolan RZahra SRamadhani D(2024)Kepastian Hukum Penyelesaian Wanprestasi Perjanjian Jual Beli Tanah di Wilayah Kabupaten Badung Provinsi BaliIndonesian Journal of Law and Justice10.47134/ijlj.v1i4.27921:4(15)Online publication date: 26-Jun-2024
https://doi.org/10.47134/ijlj.v1i4.2792
Feng Wenbin 冯Liu Zhiqiang 刘Ye Mao 叶(2024)光轴可移动圆形液晶透镜Acta Optica Sinica10.3788/AOS23183644:11(1123001)Online publication date: 2024
https://doi.org/10.3788/AOS231836
Li JLv QZhang WZhang YTan Z(2024)Burst-Enhanced Super-Resolution Network (BESR)Sensors10.3390/s2407205224:7(2052)Online publication date: 23-Mar-2024
https://doi.org/10.3390/s24072052
Kim HPark JGi Kang M(2024)Multi-frame demosaicing for the Quad Bayer CFA in the color difference domainOptics Express10.1364/OE.51006532:8(14223)Online publication date: 3-Apr-2024
https://doi.org/10.1364/OE.510065
Huang FChen YWang XWang SWu X(2024)Joint constraints of guided filtering based confidence and nonlocal sparse tensor for color polarization super-resolution imagingOptics Express10.1364/OE.50796032:2(2364)Online publication date: 9-Jan-2024
https://doi.org/10.1364/OE.507960
Zhou BLiu ZChen XZhang YFeng WYe M(2024)True-color and super-resolution imaging using an optical axis shiftable liquid crystal lensApplied Optics10.1364/AO.50389963:2(415)Online publication date: 8-Jan-2024
https://doi.org/10.1364/AO.503899
Lu YDing DPan HFu YZhang LTan FWang RChen YXue GRen JShu YLiu JTan RHe YChen J(2024)M3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile CamerasProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699371(744-756)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699371
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents