research-article

Moving gradients: a path-based method for plausible image interpolation

Authors:

Fu-Chung Huang,

Wojciech Matusik,

Ravi Ramamoorthi,

Peter BelhumeurAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 28, Issue 3

Article No.: 42, Pages 1 - 11

https://doi.org/10.1145/1531326.1531348

Published: 27 July 2009 Publication History

Abstract

We describe a method for plausible interpolation of images, with a wide range of applications like temporal up-sampling for smooth playback of lower frame rate video, smooth view interpolation, and animation of still images. The method is based on the intuitive idea, that a given pixel in the interpolated frames traces out a path in the source images. Therefore, we simply move and copy pixel gradients from the input images along this path. A key innovation is to allow arbitrary (asymmetric) transition points, where the path moves from one image to the other. This flexible transition preserves the frequency content of the originals without ghosting or blurring, and maintains temporal coherence. Perhaps most importantly, our framework makes occlusion handling particularly simple. The transition points allow for matches away from the occluded regions, at any suitable point along the path. Indeed, occlusions do not need to be handled explicitly at all in our initial graph-cut optimization. Moreover, a simple comparison of computed path lengths after the optimization, allows us to robustly identify occluded regions, and compute the most plausible interpolation in those areas. Finally, we show that significant improvements are obtained by moving gradients and using Poisson reconstruction.

Supplementary Material

JPG File (tps030_09.jpg)

Download
21.29 KB

MP4 File (tps030_09.mp4)

Download
105.38 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. on Graphics (SIGGRAPH 2004) 23, 3 (aug), 294--302.

Digital Library

[2]

Alvarez, L., Deriche, R., Papadopoulo, T., and Sanchez, J. 2007. Symmetrical dense optical flow estimation with occlusions detection. Int. Journal of Computer Vision 75, 3 (December), 371--385.

Digital Library

[3]

Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2007. A database and evaluation methodology for optical flow. In Proc. IEEE Int. Conf. Computer Vision, 1--8.

[4]

Barron, J., Fleet, D., and Beauchemin, S. 1994. Performance of optical flow techniques. Int. Journal of Computer Vision 12, 1, 43--77.

Digital Library

[5]

Belhumeur, P., and Mumford, D. 1992. A bayesian treatment of the stereo correspondence problem using half-occluded regions. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 506--512.

[6]

Black, M., and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow-fields. Computer Vision and Image Understanding 63, 1 (January), 75--104.

Digital Library

[7]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence 23, 11, 1222--1239.

Digital Library

[8]

Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. European Conf. Computer Vision, vol. 3024, 25--36.

[9]

Chen, S., and Williams, L. 1993. View interpolation for image synthesis. In Proc. SIGGRAPH 93, 279--288.

Digital Library

[10]

Fitzgibbon, A. W., Wexler, Y., and Zisserman, A. 2003. Image-based rendering using image-based priors. In Proc. IEEE Int. Conf. Computer Vision, 1176--1183.

Digital Library

[11]

Geiger, D., Ladendorf, B., and Yuille, A. 1992. Occlusions and binocular stereo. In Proc. European Conf. Computer Vision, 425--433.

Digital Library

[12]

Gonzalez, and Woods. 2002. Digital Image Processing. Prentice Hall Inc.

Digital Library

[13]

Jia, J., and Tang, C. 2008. Image stitching using structure deformation. IEEE Trans. Pattern Analysis and Machine Intelligence 30, 4, 617--631.

Digital Library

[14]

Jia, J., Sun, J., Tang, C., and Shum, H. 2006. Drag-and-drop pasting. ACM Trans. on Graphics (SIGGRAPH 2006) 25, 3 (jul), 631--637.

Digital Library

[15]

Kolmogorov, V., and Zabih, R. 2001. Computing visual correspondence with occlusions using graph cuts. Proc. IEEE Int. Conf. Computer Vision, 508--515.

[16]

Kwatra, V., Sch&#246;dl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. on Graphics (SIGGRAPH 2003) 22, 3 (July), 277--286.

Digital Library

[17]

McMillan, L., and Bishop, G. 1995. Plenoptic modeling: An image-based rendering system. In In Proc. SIGGRAPH 95, 39--46.

Digital Library

[18]

M&#233;min, E., and P&#233;rez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. Int. Journal of Computer Vision 46, 2, 129--155.

Digital Library

[19]

P&#233;rez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. on Graphics (SIGGRAPH 2003) 22, 3 (jul), 313--318.

Digital Library

[20]

Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithm. Int. Journal of Computer Vision 47, 1, 7--42.

Digital Library

[21]

Sch&#246;dl, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures. ACM Trans. on Graphics (SIGGRAPH 2000), 489--498.

Digital Library

[22]

Seitz, S., and Dyer, C. 1996. View morphing. In In Proc. SIGGRAPH 96, 21--30.

Digital Library

[23]

Sun, J., Li, Y., Kang, S., and Shum, H. 2005. Symmetric stereo matching for occlusion handling. Proc. IEEE Conf. Computer Vision and Pattern Recognition 2, 399--406.

Digital Library

[24]

Wang, H., Raskar, R., and Ahuja, N. 2004. Seamless video editing. In Int. Conf. on Pattern Recognition, vol. 3, 858--861.

Digital Library

[25]

Wang, H., Xu, N., Raskar, R., and Ahuja, N. 2005. Videoshop: A new framework for spatio-temporal video editing in gradient domain. Proc. IEEE Conf. Computer Vision and Pattern Recognition 2, 1201.

Digital Library

[26]

Wolberg, G. 1990. Digital Image Warping. IEEE Press.

Digital Library

[27]

Xiao, J., Cheng, H., Sawhney, H., Rao, C., and Isnardi, M. 2006. Bilateral filtering-based optical flow estimation with occlusion detection. In Proc. European Conf. Computer Vision, vol. 1, 211--224.

Digital Library

[28]

Zitnick, C. L., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. In Proc. IEEE Int. Conf. Computer Vision, 1308--1315.

Digital Library

Cited By

Maciel CZou Q(2024)Dynamic MRI interpolation in temporal direction using an unsupervised generative modelComputerized Medical Imaging and Graphics10.1016/j.compmedimag.2024.102435117(102435)Online publication date: Oct-2024
https://doi.org/10.1016/j.compmedimag.2024.102435
Kalluri TPathak DChandraker MTran D(2023)FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00211(2070-2081)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00211
Niklaus SHu PChen J(2023)Splatting-based Synthesis for Video Frame Interpolation2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00078(713-723)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00078
Show More Cited By

Index Terms

Moving gradients: a path-based method for plausible image interpolation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Moving gradients: a path-based method for plausible image interpolation
SIGGRAPH '09: ACM SIGGRAPH 2009 papers

We describe a method for plausible interpolation of images, with a wide range of applications like temporal up-sampling for smooth playback of lower frame rate video, smooth view interpolation, and animation of still images. The method is based on the ...
High accuracy approximation of helices by quintic curves

In this paper we present methods for approximating a helix segment by quintic Bézier curves or quintic rational Bézier curves based on the geometric Hermite interpolation technique in space. The fitting curve interpolates the curvatures as well as the ...
Shape-preserving, multiscale interpolation by univariate curvature-based cubic L₁ splines in cartesian and polar coordinates

We investigate C¹-smooth univariate curvature-based cubic L₁ interpolating splines in Cartesian and polar coordinates. The coefficients of these splines are calculated by minimizing the L₁ norm of curvature. We compare these curvature-based cubic L₁ ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 28, Issue 3

August 2009

750 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1531326

Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2009

Published in TOG Volume 28, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

113
Total Citations
View Citations
2,546
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Maciel CZou Q(2024)Dynamic MRI interpolation in temporal direction using an unsupervised generative modelComputerized Medical Imaging and Graphics10.1016/j.compmedimag.2024.102435117(102435)Online publication date: Oct-2024
https://doi.org/10.1016/j.compmedimag.2024.102435
Kalluri TPathak DChandraker MTran D(2023)FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00211(2070-2081)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00211
Niklaus SHu PChen J(2023)Splatting-based Synthesis for Video Frame Interpolation2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00078(713-723)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00078
Jarujareet UWiratchawa KPanpisut PIntharah T(2023)DeepDDM: A Compact Deep-Learning Assisted Platform for Micro-Rheological Assessment of Micro-Volume FluidsIEEE Access10.1109/ACCESS.2023.329049611(66467-66477)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3290496
Kalluri TPathak DChandraker MTran D(2023)FLAVR: flow-free architecture for fast video frame interpolationMachine Vision and Applications10.1007/s00138-023-01433-y34:5Online publication date: 11-Aug-2023
https://dl.acm.org/doi/10.1007/s00138-023-01433-y
Li XZhang BLiao JSander P(2022)Deep Sketch-Guided Cartoon Video InbetweeningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.304941928:8(2938-2952)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TVCG.2021.3049419
Geng ZLiang LDing TZharkov I(2022)RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01692(17420-17430)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.01692
Sun YZhou SCheng RTan WYan BFu L(2022)Learning Robust Image-Based Rendering on Sparse Scene Geometry via Depth Completion2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00766(7803-7813)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.00766
Argaw DKweon I(2022)Long-term Video Frame Interpolation via Feature Propagation2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00353(3533-3542)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.00353
Chen Q(2022)A Hierarchical Path-based and SSIM-motivated Method for Cartoon Image Interpolation2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA)10.1109/CVIDLICCEA56201.2022.9824260(752-755)Online publication date: 20-May-2022
https://doi.org/10.1109/CVIDLICCEA56201.2022.9824260
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents