research-article

Kurator: Using The Crowd to Help Families With Personal Curation Tasks

Authors:

Mark S. Ackerman,

Walter S. LaseckiAuthors Info & Claims

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

Pages 1835 - 1849

https://doi.org/10.1145/2998181.2998358

Published: 25 February 2017 Publication History

Abstract

People capture photos, audio recordings, video, and more on a daily basis, but organizing all these digital artifacts quickly becomes a daunting task. Automated solutions struggle to help us manage this data because they cannot understand its meaning. In this paper, we introduce Kurator, a hybrid intelligence system leveraging mixed-expertise crowds to help families curate their personal digital content. Kurator produces a refined set of content via a combination of automated systems able to scale to large data sets and human crowds able to understand the data. Our results with 5 families show that Kurator can reduce the amount of effort needed to find meaningful memories within a large collection. This work also suggests that crowdsourcing can be used effectively even in domains where personal preference is key to accurately solving the task.

References

[1]

Amershi, S., Cakmak, M., Knox, W.B. and Kulesza, T., 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine,35(4), 105120.

[2]

Amershi, S., Fogarty, J. and Weld, D., 2012. Regroup: Interactive machine learning for on-demand group creation in social networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 21--30.

Digital Library

[3]

Bernard, H.R., 2011. Research methods in anthropology: Qualitative and quantitative approaches. Rowman Altamira.

[4]

Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D. and Panovich, K., 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313--322.

Digital Library

[5]

Ceroni, A., Solachidis, V., Fu, M., Kanhabua, N., Papadopoulou, O., Niederée, C. and Mezaris, V., 2015a. Investigating human behaviors in selecting personal photos to preserve memories. In 2015 IEEE International Conference on Multimedia & Expo Workshop . IEEE, 1--6.

[6]

Ceroni, A., Solachidis, V., Niederée, C., Papadopoulou, O., Kanhabua, N. and Mezaris, V., 2015b. To Keep or not to Keep: An Expectationoriented Photo Selection Method for Personal Photo Collections. In Proceedings of the 5th ACM International Conference on Multimedia Retrieval. ACM, 187--194.

Digital Library

[7]

Chan, J., Dang, S. and Dow, S.P., 2016. Improving Crowd Innovation with Expert Facilitation. In Proceedings of the 19th ACM Conference on ComputerSupported Cooperative Work & Social Computing. ACM.

Digital Library

[8]

Chang, S., Kumar, V., Gilbert, E. and Terveen, L.G., 2014. Specialization, homophily, and gender in a social curation site: findings from pinterest. In Proceedings of the 17th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 674686.

Digital Library

[9]

Chilton, L.B., Kim, J., André, P., Cordeiro, F., Landay, J.A., Weld, D.S., Dow, S.P., Miller, R.C. and Zhang, H., 2014. Frenzy: collaborative data organization for creating conference sessions. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 1255--1264.

Digital Library

[10]

Cusano, C. and Santini, S., 2014. With a little help from my friends. Multimedia tools and applications, 70(2), 1033--1048.

Digital Library

[11]

Dow, S., Kulkarni, A., Klemmer, S. and Hartmann, B., 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work. ACM, 10131022.

Digital Library

[12]

Eaton, John W., David Bateman, and Søren Hauberg, 2009. GNU Octave version 3.0.1 manual: a high-level interactive language for numerical computations. CreateSpace Independent Publishing Platform. ISBN 1441413006, http://www.gnu.org/software/octave/doc/interpreter/.

[13]

Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K. and Sandvig, C., 2015. "I Always Assumed That I Wasn't Really That Close to {Her}": Reasoning About Invisible Algorithms in News Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 153--162.

Digital Library

[14]

Fogarty, J., Tan, D., Kapoor, A. and Winder, S., 2008. CueFlik: interactive concept learning in image search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 29--38.

Digital Library

[15]

Gajos, K. and Weld, D.S., 2005. Preference elicitation for interface optimization. In Proceedings of the 18th annual ACM symposium on User interface software and technology. ACM, 173--182.

Digital Library

[16]

Guldogan, E., Kangas, J. and Gabbouj, M., 2013. Personalized representative image selection for shared photo albums. In 2013 International Conference on Computer Applications Technology. IEEE, 1--4.

[17]

Gulotta, R., Sciuto, A., Kelliher, A. and Forlizzi, J., 2015. Curatorial Agents: How Systems Shape Our Understanding of Personal and Familial Digital Information. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3453--3462.

Digital Library

[18]

Hämäläinen, A., Meinedo, H., Tjalve, M., Pellegrini, T., Trancoso, I. and Dias, M.S., 2014. Improving Speech Recognition through Automatic Selection of Age Group-Specific Acoustic Models. In Computational Processing of the Portuguese Language. Springer International Publishing, 12--23.

[19]

Hara, K., Sun, J., Moore, R., Jacobs, D. and Froehlich, J., 2014. Tohme: detecting curb ramps in google street view using crowdsourcing, computer vision, and machine learning. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 189--204.

Digital Library

[20]

Huang, T.H.K., Lasecki, W.S. and Bigham, J.P., 2015. Guardian: A Crowd-Powered Spoken Dialog System for Web APIs. In Third AAAI Conference on Human Computation and Crowdsourcing.

[21]

Jones, J. and Ackerman, M.S., 2016. Curating an Infinite Basement: Understanding How People Manage Collections of Sentimental Artifacts. In Proceedings of the 19th International Conference on Supporting Group Work. ACM.

Digital Library

[22]

Jones, J., Merritt, D., and Ackerman, M.S., 2017. KidKeeper: Design for capturing audio mementos of everyday life for parents of young children. In 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing. Accepted.

Digital Library

[23]

Kajino, H., Baba, Y. and Kashima, H., 2014. InstancePrivacy Preserving Crowdsourcing. In Second AAAI Conference on Human Computation and Crowdsourcing.

[24]

Kim, J., Cheng, J. and Bernstein, M.S., 2014. Ensemble: exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on ComputerSupported Cooperative Work & Social Computing. ACM, 745--755.

Digital Library

[25]

Kittur, A. and Kraut, R.E., 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of the 2008 ACM conference on Computer Supported Cooperative Work. ACM, 37--46.

Digital Library

[26]

Kokkalis, N., Köhn, T., Pfeiffer, C., Chornyi, D., Bernstein, M.S. and Klemmer, S.R., 2013. EmailValet: managing email overload through private, accountable crowdsourcing. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work. ACM, 1291--1300.

Digital Library

[27]

Kulkarni, A., Narula, P., Rolnitzky, D. and Kontny, N., 2014. Wish: Amplifying creative ability with expert crowds. In Second AAAI Conference on Human Computation and Crowdsourcing.

[28]

Kumar, P. and Schoenebeck, S., 2015. The modern day baby book: Enacting good mothering and stewarding privacy on facebook. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1302--1312.

Digital Library

[29]

Laput, G., Lasecki, W.S., Wiese, J., Xiao, R., Bigham, J.P. and Harrison, C., 2015. Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 19351944.

Digital Library

[30]

Lasecki, W.S., 2015. Crowd agents: interactive intelligent systems powered by the crowd. Ph.D Dissertation. University of Rochester, Rochester, NY.

[31]

Lasecki, W.S., Gordon, M., Koutra, D., Jung, M.F., Dow, S.P. and Bigham, J.P., 2014. Glance: Rapidly coding behavioral video with the crowd. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 551562.

Digital Library

[32]

Lasecki, W.S., Teevan, J. and Kamar, E., 2014. Information extraction and manipulation threats in crowd-powered systems. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 248--256.

Digital Library

[33]

Law, E., Dalton, C., Merrill, N., Young, A. and Gajos, K.Z., 2013. Curio: A Platform for Supporting MixedExpertise Crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.

[34]

Le, D., and Mower Provost, E. 2013. Emotion recognition from spontaneous speech using hidden markov models with deep belief networks. In Automatic Speech Recognition and Understanding. IEEE, 216--221.

[35]

Marshall, Catherine C., 2007. How people manage personal information over a lifetime. Personal Information Management, 57--75.

[36]

Marshall, Catherine C, Sara Bly, and Francoise BrunCottan, 2006. The long term fate of our digital belongings: Toward a service model for personal archives. In Proceedings of the Archiving Conference, 25--30.

[37]

Meinedo, H. and Trancoso, I., 2011. Age and gender detection in the I-DASH project. ACM Transactions on Speech and Language Processing (TSLP), 7(4), 13.

Digital Library

[38]

Nejdl, W. and Niederee, C., 2015. Photos to Remember, Photos to Forget. IEEE Multimedia, 22(1), 6--11.

Digital Library

[39]

Nguyen, A.T., Wallace, B.C. and Lease, M., 2015. Combining Crowd and Expert Labels using Decision Theoretic Active Learning. In Third AAAI Conference on Human Computation and Crowdsourcing.

[40]

Obrador, Pere, Rodrigo De Oliveira, and Nuria Oliver, 2010. "Supporting personal photo storytelling for social albums." In Proceedings of the International Conference on Multimedia. ACM, 561--570.

Digital Library

[41]

Oleksik, G. and Brown, L.M. 2008. Sonic gems: exploring the potential of audio recording as a form of sentimental memory capture. Proc. BCS-HCI 2008, Vol. 1, British Computer Society, 163--172.

Digital Library

[42]

Organisciak, Peter, Jaime Teevan, Susan Dumais, Robert C. Miller, and Adam Tauman Kalai. 2014. "A Crowd of Your Own: Crowdsourcing for On-Demand Personalization." In Second AAAI Conference on Human Computation and Crowdsourcing.

[43]

Rader, E. and Gray, R., 2015. Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 173--182.

Digital Library

[44]

Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W.S., Patel, J., Rahmati, N., Doshi, T., Valentine, M. and Bernstein, M.S., 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 75--85.

Digital Library

[45]

Sadjadi, S.O. and Hansen, J.H., 2013. Unsupervised speech activity detection using voicing measures and perceptual spectral flux. IEEE Signal Processing Letters, 20(3), 197--200.

[46]

Schuller, B.; Batliner, A.; Steidl, S.; and Seppi, D. 2011. Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech Communication 53(9):1062--1087.

Digital Library

[47]

Simko, J. and Bieliková, M., 2011. Games with a purpose: User generated valid metadata for personal archives. In 2011 Sixth International Workshop on Semantic Media Adaptation and Personalization. IEEE, 45--50.

Digital Library

[48]

Teevan, J., Liebling, D.J. and Lasecki, W.S., 2014. Selfsourcing personal tasks. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 2527--2532.

Digital Library

[49]

Yakel, E., 2007. Digital curation. OCLC Systems & Services: International digital library perspectives, 23(4), 335--340.

[50]

Yi, J., Jin, R., Jain, S. and Jain, A., 2013. Inferring users' preferences from crowdsourced pairwise comparisons: A matrix completion approach. In First AAAI Conference on Human Computation and Crowdsourcing.

[51]

Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D. and Horvitz, E., 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 217--226.

Digital Library

[52]

Zhao, Xuan and Lindley, Sian E., 2014. Curation Through Use: Understanding the Personal Value of Social Media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, 2431--2440.

Digital Library

Cited By

Sun KLi JZou YRadesky JBrooks CSchaub F(2024)Unfulfilled Promises of Child Safety and Privacy: Portrayals and Use of Children in Smart Home MarketingProceedings of the ACM on Human-Computer Interaction10.1145/36374228:CSCW1(1-29)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637422
Shin JRheu MHuh-Yoo JPeng W(2021)Designing Technologies to Support Parent-Child Relationships: A Review of Current Findings and Suggestions for Future DirectionsProceedings of the ACM on Human-Computer Interaction10.1145/34795855:CSCW2(1-31)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479585
Lim JLee JKim D(2021)The Effects of Feedback and Goal on the Quality of Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2021.187635537:13(1207-1219)Online publication date: 1-Feb-2021
https://doi.org/10.1080/10447318.2021.1876355
Show More Cited By

Index Terms

Kurator: Using The Crowd to Help Families With Personal Curation Tasks
1. Human-centered computing

Recommendations

Applications of collaborative annotation system in digital curation, crowdsourcing, and digital humanities

PurposeCollaboratively annotating digital texts allow users to add valued information, share ideas and create knowledge. Most importantly, annotated content can help users obtain a deeper and broader understanding of a text compared to digital content ...
Digital Preservation, Archival Science and Methodological Foundations for Digital Libraries

Digital libraries, whether commercial, public, or personal, lie at the heart of the information society. Yet, research into their long-term viability and the meaningful accessibility of their contents remains in its infancy. In general, as we have ...
Digital preservation in a box: outreach resources for digital stewardship
JCDL '12: Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries

"Digital Preservation in a Box" is a major activity of the National Digital Stewardship Alliance (NDSA) Outreach Working Group. This toolkit of digital stewardship outreach resources can be utilized by diverse communities as a gentle introduction to the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

February 2017

2556 pages

ISBN:9781450343350

DOI:10.1145/2998181

General Chairs:
Charlotte P. Lee
University of Washington
,
Steve Poltrock
Retired
,
Program Chairs:
Louise Barkhuus
Stockholm University and Cornell Tech
,
Marcos Borges
Universidade Federal do Rio de Janeiro
,
Wendy Kellogg
IBM T.J. Watson Research Center

Copyright © 2017 ACM.

© 2017 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CSCW '17

Sponsor:

SIGCHI

CSCW '17: Computer Supported Cooperative Work and Social Computing

February 25 - March 1, 2017

Oregon, Portland, USA

Acceptance Rates

CSCW '17 Paper Acceptance Rate 183 of 530 submissions, 35%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
327
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun KLi JZou YRadesky JBrooks CSchaub F(2024)Unfulfilled Promises of Child Safety and Privacy: Portrayals and Use of Children in Smart Home MarketingProceedings of the ACM on Human-Computer Interaction10.1145/36374228:CSCW1(1-29)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637422
Shin JRheu MHuh-Yoo JPeng W(2021)Designing Technologies to Support Parent-Child Relationships: A Review of Current Findings and Suggestions for Future DirectionsProceedings of the ACM on Human-Computer Interaction10.1145/34795855:CSCW2(1-31)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479585
Lim JLee JKim D(2021)The Effects of Feedback and Goal on the Quality of Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2021.187635537:13(1207-1219)Online publication date: 1-Feb-2021
https://doi.org/10.1080/10447318.2021.1876355
Chen YMonroy-Hernández AWehrman IOney SLasecki WVaish R(2020)Sifter: A Hybrid Workflow for Theme-based Video Curation at ScaleProceedings of the 2020 ACM International Conference on Interactive Media Experiences10.1145/3391614.3393657(65-73)Online publication date: 17-Jun-2020
https://dl.acm.org/doi/10.1145/3391614.3393657
Huffaker JKummerfeld JLasecki WAckerman MBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Crowdsourced Detection of Emotionally Manipulative LanguageProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376375(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376375
Chen YHerskovitz JLasecki WOney S(2020)Bashon: A Hybrid Crowd-Machine Workflow for Shell Command Synthesis2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC50065.2020.9127248(1-8)Online publication date: Aug-2020
https://doi.org/10.1109/VL/HCC50065.2020.9127248
Vitale FOdom WMcGrenere JHarrison SBardzell SNeustaedter CTatar D(2019)Keeping and Discarding Personal DataProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322300(1463-1477)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322300
Shapiro BHall R(2018)Personal Curation in a MuseumProceedings of the ACM on Human-Computer Interaction10.1145/32744272:CSCW(1-22)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1145/3274427
Foong PZhao STan FWilliams JMandryk RHancock MPerry MCox A(2018)Harvesting Caregiving KnowledgeProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173653(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173653
Ben-Sasson ABen-Sasson EJacobs KSaig EOliver NCzerwinski M(2017)Baby CROINCProceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare10.1145/3154862.3154887(110-119)Online publication date: 23-May-2017
https://dl.acm.org/doi/10.1145/3154862.3154887
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten