research-article

Context-Aware Photography Learning for Smart Mobile Devices

Authors:

Yogesh Singh Rawat,

Mohan S. KankanhalliAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 12, Issue 1s

Article No.: 19, Pages 1 - 24

https://doi.org/10.1145/2808199

Published: 21 October 2015 Publication History

Abstract

In this work we have developed a photography model based on machine learning which can assist a user in capturing high quality photographs. As scene composition and camera parameters play a vital role in aesthetics of a captured image, the proposed method addresses the problem of learning photographic composition and camera parameters. Further, we observe that context is an important factor from a photography perspective, we therefore augment the learning with associated contextual information. The proposed method utilizes publicly available photographs along with social media cues and associated metainformation in photography learning. We define context features based on factors such as time, geolocation, environmental conditions and type of image, which have an impact on photography. We also propose the idea of computing the photographic composition basis, eigenrules and baserules, to support our composition learning. The proposed system can be used to provide feedback to the user regarding scene composition and camera parameters while the scene is being captured. It can also recommend position in the frame where people should stand for better composition. Moreover, it also provides camera motion guidance for pan, tilt and zoom to the user for improving scene composition.

Supplementary Material

rawat (rawat.zip)

Supplemental movie, appendix, image and software files for, Context-Aware Photography Learning for Smart Mobile Devices

Download
10.21 MB

References

[1]

R. Abdullah, M. Christie, G. Schofield, C. Lino, and P. Olivier. 2011. Advanced composition in virtual camera control. In Proceedings of the International Symposium on Smart Graphics. Springer, 13--24.

Digital Library

[2]

R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell.

Digital Library

[3]

R. Achanta and S. Susstrunk. 2010. Saliency detection using maximum symmetric surround. In Proceedings of the International Conference on Image Processing. 2653--2656.

[4]

S. Bae, A. Agarwala, and F. Durand. 2010. Computational rephotography. ACM Trans. Graphics 24:1--24:15.

Digital Library

[5]

S. Banerjee and B. L. Evans. 2007. In-camera automation of photographic composition rules. IEEE Trans. Image Process. 1807--1820.

Digital Library

[6]

W. Bares. 2006. A photographic composition assistant for intelligent virtual 3D camera systems. In Proceedings of the International Symposium on Smart Graphics. Springer, 172--183.

[7]

H. Bay, A. Ess, T. Tuytelaars, and L. V. Gool. 2008. Speeded-Up Robust Features (SURF). In Proceedings of the Conference on Computer Vision and Image Understanding. 346--359.

Digital Library

[8]

S. Bourke, K. McCarthy, and B. Smyth. 2011. The social camera: A case-study in contextual image recommendation. In Proceedings of the 16th International Conference on Intelligent User Interfaces. 13--22.

Digital Library

[9]

D. Butterfield, C. Fake, C. Henderson-Begg, and S. Mourachov. 2006. Interestingness ranking of media objects. US Patent App. 11/350,981.

[10]

C. C. Chang and C. J. Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 27:1--27:27.

Digital Library

[11]

B. Cheng, B. Ni, S. Yan, and Q. Tian. 2010. Learning to photograph. In Proceedings of the International Conference on Multimedia. ACM, 291--300.

Digital Library

[12]

N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 886--893.

Digital Library

[13]

A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. J. R. Statistical Soc. Series B (Methodological), 1--38.

[14]

Weather Forecast and Reports. 2014. (2014). Retrieved March 3, 2014 from http://www.wunderground.com.

[15]

M. Freeman. 2007. The Photographer's Eye: Composition and Design for Better Digital Photos. Focal Press.

Digital Library

[16]

H. Fu, X. Han, and Q. H. Phan. 2013. Data-driven suggestions for portrait posing. In Proceedings of the SIGGRAPH Asia Conference on Emerging Technologies. ACM, 7:1--7:3.

Digital Library

[17]

R. Gadde and K. Karlapalem. 2011. Aesthetic guideline driven photography by robots. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence. 2060--2065.

Digital Library

[18]

J. Huang, X. Yang, X. Fang, W. Lin, and R. Zhang. 2011. Integrating visual saliency and consistency for re-ranking image search results. IEEE Trans. Multimedia, 653--661.

Digital Library

[19]

R. Jacobson. 2000. The Manual of Photography: Photographic and Digital Imaging. Focal Press.

Digital Library

[20]

S. Kelby. 2006. The Digital Photography Book. Peachpit Press, Berkeley, CA.

Digital Library

[21]

J. G. Kim, H. S. Chang, J. Kim, and H. M. Kim. 2000. Efficient camera motion characterization for MPEG video indexing. In Proceedings of the IEEE International Conference on Multimedia and Expo. 1171--1174.

[22]

L. Zheng, Y. Xiaokang, L. Weiyao, Z. Hongyuan, and C. N. Xiaolin. 2014. Inferring user image-search goals under the implicit guidance of users. IEEE Trans. Circuits Syst. Video Technol.

Digital Library

[23]

D. D. Lee and H. S. Seung. 1999. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788--791.

[24]

C. Li, A. C. Loui, and T. Chen. 2010. Towards aesthetics: A photo quality assessment and photo selection system. In Proceedings of the International Conference on Multimedia. 271--280.

Digital Library

[25]

C. Liu, J. Yuen, and A. Torralba. 2011. Nonparametric scene parsing via label transfer. IEEE Trans. Pattern Anal. Mach. Intell. 2368--2382.

Digital Library

[26]

C. Lujun, Y. Hongxun, S. Xiaoshuai, and Z. Hongming. 2012. Real-time viewfinder composition assessment and recommendation to mobile photographing. In Proceedings of the Pacific-Rim Conference on Advances in Multimedia Information Processing. 707--714.

Digital Library

[27]

S. Ma, Y. Fan, and Chang W. Chen. 2014. Pose maker: A pose recommendation system for person in the landscape photographing. In Proceedings of the ACM International Conference on Multimedia. 1053--1056.

Digital Library

[28]

L. Marchesotti, F. Perronnin, D. Larlus, and G. Csurka. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In Proceedings of the IEEE International Conference on Computer Vision. 1784--1791.

Digital Library

[29]

H. Mitarai, Y. Itamiya, and A. Yoshitaka. 2013. Interactive photographic shooting assistance based on composition and saliency. In Computational Science and Its Applications, 348--363.

[30]

B. Ni, M. Xu, B. Cheng, M. Wang, S. Yan, and Q. Tian. 2013. Learning to photograph: A compositional perspective. IEEE Trans. Multimedia 1138--1151.

Digital Library

[31]

Y. S. Rawat and M. S. Kankanhalli. 2014. Context-based photography learning using crowdsourced images and social media. In Proceedings of the ACM International Conference on Multimedia, Grand Challenge. 217--220.

Digital Library

[32]

H. Su, T. Chen, C. Kao, W. Hsu, and S. Chien. 2012. Preference-aware view recommendation system for scenic photos based on bag-of-aesthetics-preserving features. IEEE Trans. Multimedia.

Digital Library

[33]

Sunrise and Sunset Calculator. 2014. http://www.timeanddate.com.

[34]

M. A. Turk and A. P. Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586--591.

[35]

P. P. Wang, W. Zhang, J. Li, and Y. Zhang. 2008. Online photography assistance by exploring geo-referenced photos on MID / UMPC. In Proceedings of the IEEE 10th Workshop on Multimedia Signal Processing. 6--10.

[36]

W. Wang, W. Lin, Y. Chen, J. Wu, J. Wang, and B. Sheng. 2014. Finding coherent motions and semantic regions in crowd scenes: A diffusion and clustering approach. In Proceedings of the European Conference on Computer Vision. 756--771.

[37]

P. Xu, H. Yao, R. Ji, X. M. Liu, and X. Sun. 2014. Where should I stand? Learning based human position recommendation for mobile photographing. Multimedia Tools Appl. 69:3--29.

Digital Library

[38]

W. Yin, T. Mei, and C. W. Chen. 2012. Crowdsourced learning to photograph via mobile devices. In Proceedings of the IEEE International Conference on Multimedia and Expo. 812--817.

Digital Library

[39]

W. Yin, T. Mei, C. W. Chen, and S. Li. 2014. Socialized mobile photography: Learning to photograph with social context via mobile devices. IEEE Trans. Multimedia. 184--200.

Cited By

Sabah SAlani MSalih B(2024)Leveraging AI Language Models for Designing Contextually Responsive Built EnvironmentsIntelligent Computing10.1007/978-3-031-62273-1_32(510-519)Online publication date: 15-Jun-2024
https://doi.org/10.1007/978-3-031-62273-1_32
Suo JZhang WGong JYuan XBrady DDai Q(2023)Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile VisionProceedings of the IEEE10.1109/JPROC.2023.3338272111:12(1607-1639)Online publication date: Dec-2023
https://doi.org/10.1109/JPROC.2023.3338272
Liu XLiu MLi JLiu SWang XLei LZuo W(2023)Beyond Image Borders: Learning Feature Extrapolation for Unbounded Image Composition2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01197(12977-12986)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01197
Show More Cited By

Index Terms

Context-Aware Photography Learning for Smart Mobile Devices
1. Information systems
  1. Information systems applications

Recommendations

Context-Based Photography Learning using Crowdsourced Images and Social Media
MM '14: Proceedings of the 22nd ACM international conference on Multimedia

This paper presents a photography model based on machine learning which utilizes crowd-sourced images along with social media cues. As scene composition and camera parameters play a vital role in aesthetics of a captured image, the proposed system ...
Real-Time Assistance in Multimedia Capture Using Social Media
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

In the last decade, we have seen significant improvement in the ease and cost of capturing multimedia content. However, the aesthetic quality of the content captured by an amateur user still needs substantial improvement. This doctoral research aims at ...
Metric calibration of a stereo rig
VSR '95: Proceedings of the IEEE Workshop on Representation of Visual Scenes

Describes a method to determine affine and metric calibration for a stereo rig. The method does not involve the use of calibration objects or special motions, but simply a single general motion of the rig with fixed parameters (i.e. camera parameters ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 12, Issue 1s

Special Issue on Smartphone-Based Interactive Technologies, Systems, and Applications and Special Issue on Extended Best Papers from ACM Multimedia 2014

October 2015

317 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2837676

Editor:
Ralf Steinmetz
Technische Universität Darmstadt, Germany

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2015

Accepted: 01 July 2015

Revised: 01 April 2015

Received: 01 January 2015

Published in TOMM Volume 12, Issue 1s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Singapore National Research Foundation under its International Research Centre @ Singapore Funding Initiative and administered by the Interactive Digital Media Programme Office

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
520
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sabah SAlani MSalih B(2024)Leveraging AI Language Models for Designing Contextually Responsive Built EnvironmentsIntelligent Computing10.1007/978-3-031-62273-1_32(510-519)Online publication date: 15-Jun-2024
https://doi.org/10.1007/978-3-031-62273-1_32
Suo JZhang WGong JYuan XBrady DDai Q(2023)Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile VisionProceedings of the IEEE10.1109/JPROC.2023.3338272111:12(1607-1639)Online publication date: Dec-2023
https://doi.org/10.1109/JPROC.2023.3338272
Liu XLiu MLi JLiu SWang XLei LZuo W(2023)Beyond Image Borders: Learning Feature Extrapolation for Unbounded Image Composition2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01197(12977-12986)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01197
Shi ZHou QSheng GKe YWang KJia Y(2023)Where2Stand: Toward a Framework for Portrait Position Recommendation in PhotographyIEEE Access10.1109/ACCESS.2023.332236311(108864-108875)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3322363
Li JShe YLiu FYu CWang XXu Y(2022)Augmented Reality Based Video Shooting Guidance for Novice UsersProceedings of the ACM on Human-Computer Interaction10.1145/35467506:MHCI(1-20)Online publication date: 20-Sep-2022
https://dl.acm.org/doi/10.1145/3546750
Yan RSun JDeng LDavis A(2022)ReCapture: AR-Guided Time-lapse PhotographyProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545641(1-14)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545641
Farhat FKamani MWang J(2022)CAPTAIN: Comprehensive Composition Assistance for Photo TakingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/346276218:1(1-24)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3462762
L. JZhai KEchevarria JFried OHanrahan PLanday J(2021)Dynamic Guidance for Decluttering Photographic CompositionsThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474755(359-371)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474755
Tan NSreedhar RVazquez CStigile JGomez MSubramani ASadalgi S(2021)An Augmented Reality Guided Capture Platform for Structured and Consistent Product Reference ImageryAdjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction10.1145/3447527.3474848(1-6)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3447527.3474848
Zhang JMiao YYu J(2021)A Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and ChallengesIEEE Access10.1109/ACCESS.2021.30830759(77164-77187)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3083075
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents