research-article

Appstract: on-the-fly app content semantics with better privacy

Authors:

Earlence Fernandes,

Suman NathAuthors Info & Claims

MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking

Pages 361 - 374

https://doi.org/10.1145/2973750.2973770

Published: 03 October 2016 Publication History

Abstract

Services like Google Now on Tap and Bing Snapp enable new user experiences by understanding the semantics of contents that users consume in their apps. These systems send contents of currently displayed app pages to the cloud to identify relevant entities (e.g., a movie) appearing in the current page and show information related to such entities (e.g., local theaters playing the movie). These new experiences come with privacy concerns as they can send sensitive on-screen data (bank details, medical data, etc.) to the cloud. We propose a novel approach that efficiently extracts app content semantics on the device, without exfiltrating user data. Our solution consists of two phases: an offline, user-agnostic, in-cloud phase that automatically annotates apps' UI elements with stable semantics, and a lightweight on-device phase that assigns semantics to captured app contents on the fly, by matching the annotations. With this automatic approach we annotated 100+ food, dining, and music apps, with accuracy over 80%. Our system implementation for Android and Windows Phone---Appstract---incurs minimal runtime overhead. We built eight use cases on the Appstract framework.

References

[1]

Flurry. http://www.flurry.com.

[2]

Localytics. http://www.localytics.com/.

[3]

E. Agichtein and L. Gravano. Snowball: Extracting Relations from Large Plain-text Collections. In Proc. of the 5th ACM conference on Digital libraries (DL '00), pages 85--94, 2000.

Digital Library

[4]

Android Developers. monkeyrunner. http://developer.android.com/tools/help/monkeyrunner_concepts.html.

[5]

AppBrain. Top 10 Google Play categories. http://www.appbrain.com/stats/android-market-app-categories.

[6]

S. Arzt, S. Rasthofer, C. Fritz, E. Bodden, A. Bartel, J. Klein, Y. Le Traon, D. Octeau, and P. McDaniel. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. In Proc. of PLDI '14, pages 259--269. ACM, 2014.

Digital Library

[7]

M. Asahara and Y. Matsumoto. Japanese named entity extraction with redundant morphological analysis. In Proc. NAACL, pages 8--15, 2003.

Digital Library

[8]

A. Barth, A. Datta, J. C. Mitchell, and H. Nissenbaum. Privacy and Contextual Integrity: Framework and Applications. 2013 IEEE Symposium on Security and Privacy, 0:184--198, 2006.

Digital Library

[9]

K. Benton, L. J. Camp, and V. Garg. Studying the effectiveness of android application permissions requests. In Pervasive Computing and Communications Workshops (PERCOM Workshops), 2013 IEEE International Conference on, pages 291--296, March 2013.

[10]

V. Chandramouli, A. Chakraborty, V. Navda, S. Guha, V. Padmanabhan, and R. Ramjee. Insider: Towards breaking down mobile app silos. In TRIOS Workshop held in conjunction with the SIGOPS SOSP 2015, September 2015.

[11]

M. Collins and Y. Singer. Unsupervised models for named entity classification. In Proc. of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 100--110, 1999.

[12]

M. Eirinaki and M. Vazirgiannis. Web mining for web personalization. ACM Trans. Internet Tech., 3(1):1--27, 2003.

Digital Library

[13]

W. Enck, P. Gilbert, B. gon Chun, L. P. Cox, J. Jung, P. McDaniel, and A. Sheth. TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. In Proc. of OSDI '10, pages 393--407, October 2010.

Digital Library

[14]

A. P. Felt, E. Ha, S. Egelman, A. Haney, E. Chin, and D. Wagner. Android permissions: user attention, comprehension, and behavior. In Proc. of SOUPS '12, pages 3:1--3:14. ACM, 2012.

Digital Library

[15]

L. Gomez, I. Neamtiu, T. Azim, and T. Millstein. RERAN: Timing- and Touch-sensitive Record and Replay for Android. In Proc. of ICSE '13, pages 72--81, 2013.

Digital Library

[16]

Google. Now on Tap. https://support.google.com/websearch/answer/6304517?hl=en.

[17]

Google Analytics. http://www.google.com/analytics/.

[18]

S. Hao, B. Liu, S. Nath, W. G. Halfond, and R. Govindan. PUMA: Programmable UI-Automation for Large Scale Dynamic Analysis of Mobile Apps. In Proc. of MobiSys, pages 204--217. ACM, June 2014.

Digital Library

[19]

J. Hoffart, M. A. Yosef, I. Bordino, H. Fürstenau, M. Pinkal, M. Spaniol, B. Taneva, S. Thater, and G. Weikum. Robust disambiguation of named entities in text. In Proc. of EMNLP, pages 782--792, 2011.

Digital Library

[20]

P. G. Kelley, S. Consolvo, L. F. Cranor, J. Jung, N. Sadeh, and D. Wetherall. A conundrum of permissions: Installing applications on an android smartphone. In Proc. of USEC '12, pages 68--79, 2012.

Digital Library

[21]

A. McCallum and W. Li. Early Results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-enhanced Lexicons. In Proc. of CONLL, pages 188--191, 2003.

Digital Library

[22]

D. Nadeau and S. Sekine. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3--26, Jan. 2007.

[23]

S. Nath, F. X. Lin, L. Ravindranath, and J. Padhye. SmartAds: bringing contextual ads to mobile apps. In Proc. of MobiSys, pages 111--124, 2013.

Digital Library

[24]

H. Nissenbaum. A Contextual Approach to Privacy Online. Daedalus 140, (4):32--48, Fall 2011.

[25]

M. Pasca, D. Lin, J. Bigham, A. Lifchits, and A. Jain. Organizing and Searching the World Wide Web of Facts - Step One: The One-Million Fact Extraction Challenge. In Proc. of AAAI, pages 1400--1405, 2006.

Digital Library

[26]

ProHiro. HiroMacro Auto-Touch Macro App. https://play.google.com/store/apps/details?id=com.prohiro.macro&hl=en, 2016.

[27]

Pyevolve. Machine Learning - Text feature extraction (tf-idf). http://pyevolve.sourceforge.net/wordpress/?p=1589, 2014.

[28]

L. Ratinov and D. Roth. Design challenges and misconceptions in named entity recognition. In Proc. of the 13th Conference on Computational Natural Language Learning, CoNLL '09, pages 147--155, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.

Digital Library

[29]

L. Ravindranath, S. Nath, J. Padhye, and H. Balakrishnan. Automatic and scalable fault detection for mobile applications. In Proc. of MobiSys, pages 190--203, 2014.

Digital Library

[30]

L. Ravindranath, J. Padhye, S. Agarwal, R. Mahajan, I. Obermiller, and S. Shayandeh. AppInsight: Mobile App Performance Monitoring in the Wild. In Proc. of OSDI, pages 107--120, 2012.

Digital Library

[31]

E. Riloff and R. Jones. Learning Dictionaries for Information Extraction by Multi-level Bootstrapping. In Proc. of AAAI, pages 474--479, 1999.

Digital Library

[32]

A. Ritter, S. Clark, Mausam, and O. Etzioni. Named entity recognition in tweets: An experimental study. In Proc. of the Conference on Empirical Methods in Natural Language Processing, EMNLP '11, pages 1524--1534, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics.

Digital Library

[33]

Roy Longbottom. Android NEON MP MFLOPS Benchmark. http://www.roylongbottom.org.uk/android\%20neon\%20benchmarks.htm.

[34]

S. Sarawagi. Information extraction. Found. Trends databases, 1(3):261--377, 2008.

Digital Library

[35]

J. Sommers and P. Barford. Cell vs. WiFi: On the Performance of Metro Area Mobile Connections. In Proc. of the 2012 ACM Conference on Internet Measurement Conference, IMC '12, pages 301--314, New York, NY, USA, 2012. ACM.

Digital Library

[36]

Stanford University. CoreNLP: Named Entity Recognizer. http://nlp.stanford.edu/software/CRF-NER.shtml.

[37]

Stanford University. How do you use gazettes with Stanford NER? http://nlp.stanford.edu/software/crf-faq.shtml#gazette.

[38]

Stanford University. NLP Named Entity Recognition Results on CoNLL tasks. http://nlp.stanford.edu/projects/project-ner.shtml.

[39]

S. Tata and J. M. Patel. Estimating the Selectivity of Tf-idf Based Cosine Similarity Predicates. SIGMOD Rec., 36(4):75--80, Dec. 2007.

Digital Library

[40]

VB. Microsoft beats Google to the punch: Bing for Android update does what Now on Tap will do. http://venturebeat.com/2015/08/20/microsoft-beats-google-to-the-punch-bing-for-android-update-does-what-now-on-tap-will-do, 2015.

[41]

C. Wang, K. Chakrabarti, T. Cheng, and S. Chaudhuri. Targeted disambiguation of ad-hoc, homogeneous sets of named entities. In Proc. of WWW, pages 719--728, 2012.

Digital Library

[42]

R. White. Implicit Feedback for Interactive Information Retrieval. PhD thesis, 2004. University of Glasgow.

[43]

WSJ.D. Why You Should Care About Google Now on Tap. http://blogs.wsj.com/personal-technology/2015/05/28/why-you-should-care-about-google-now-on-tap/.

[44]

N. Zemirli. WebCap: Inferring the user's interests based on a real-time implicit feedback. In Proc. of ICDIM, pages 62--67, Aug 2012.

Cited By

Malviya VTun YLeow CXynyn AShar LJiang L(2023)Fine-Grained In-Context Permission Classification for Android Apps Using Control-Flow Graph Embedding2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00056(1225-1237)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00056
Malviya VLeow CKasthuri ATun YShar LJiang L(2022)Right to Know, Right to Refuse: Towards UI Perception-Based Automated Fine-Grained Permission Controls for Android AppsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3559556(1-6)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3559556
Lee HPark JLee U(2022)A Systematic Survey on Android API Usage for Data-driven Analytics with SmartphonesACM Computing Surveys10.1145/353081455:5(1-38)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3530814
Show More Cited By

Index Terms

Appstract: on-the-fly app content semantics with better privacy
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval

Recommendations

Impacto da adoção do phonegap na interface de aplicações móveis
IHC '13: Proceedings of the 12th Brazilian Symposium on Human Factors in Computing Systems

Nowadays, there are distinct platforms for mobile applications (apps): Android, iOS, Windows Phone, Blackberry, and others. In fact, the most popular mobile apps, such as Facebook and Gmail, are available for multiple platforms. Developing and ...
Privacy concerns for mobile app download

In the mobile age, protecting users' information from privacy-invasive apps becomes increasingly critical. To precaution users against possible privacy risks, a few Android app stores prominently disclose app permission requests on app download pages. ...
Adoption of third-party libraries in mobile apps: a case study on open-source Android applications
MOBILESoft '22: Proceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems

Third-party libraries are frequently adopted in open-source Android applications (apps). These libraries are essential to the Android app development ecosystem as they often provide vital functionality that would take significant development time to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking

October 2016

532 pages

ISBN:9781450342261

DOI:10.1145/2973750

General Chairs:
Yingying Chen
Stevens Institute of Technology
,
Marco Gruteser
Rutgers University
,
Program Chairs:
Y. Charlie Hu
Purdue University, USA
,
Karthik Sundaresan
NEC Labs Princeton, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MobiCom'16

MobiCom'16: The 22nd Annual International Conference on Mobile Computing and Networking

October 3 - 7, 2016

New York, New York City

Acceptance Rates

MobiCom '16 Paper Acceptance Rate 31 of 226 submissions, 14%;

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
605
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Malviya VTun YLeow CXynyn AShar LJiang L(2023)Fine-Grained In-Context Permission Classification for Android Apps Using Control-Flow Graph Embedding2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00056(1225-1237)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00056
Malviya VLeow CKasthuri ATun YShar LJiang L(2022)Right to Know, Right to Refuse: Towards UI Perception-Based Automated Fine-Grained Permission Controls for Android AppsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3559556(1-6)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3559556
Lee HPark JLee U(2022)A Systematic Survey on Android API Usage for Data-driven Analytics with SmartphonesACM Computing Surveys10.1145/353081455:5(1-38)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3530814
Li TPopowski LMitchell TMyers BKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Screen2Vec: Semantic Embedding of GUI Screens and GUI ComponentsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445049(1-15)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445049
Ma YHu ZGu DZhou LMei QHuang GLiu X(2020)Roaming Through the Castle TunnelsACM Transactions on the Web10.1145/339505014:3(1-24)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3395050
Chattopadhyay SBanerjee A(2020)QoS-aware Automatic Web Service Composition with Multiple ObjectivesACM Transactions on the Web10.1145/338914714:3(1-38)Online publication date: 18-May-2020
https://dl.acm.org/doi/10.1145/3389147
Xiao ZLi CChen H(2020)PatternRank+NNACM Transactions on the Web10.1145/338604214:3(1-15)Online publication date: 3-May-2020
https://dl.acm.org/doi/10.1145/3386042
Wu HWang CNock RWang WYin JLu KZhu L(2020)SMINTACM Transactions on the Web10.1145/338183314:3(1-28)Online publication date: 3-May-2020
https://dl.acm.org/doi/10.1145/3381833
Chen FXia KDhabalia KHong JBrewster SFitzpatrick GCox AKostakos V(2019)MessageOnTapProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300805(1-14)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300805
Kim KRyu WPark JLee S(2019)meChat: In-Device Personal Assistant for Conversational Photo SharingIEEE Internet Computing10.1109/MIC.2018.288305923:2(23-30)Online publication date: 1-Mar-2019
https://doi.org/10.1109/MIC.2018.2883059
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Media

Figures

Other

Tables

View Table of Contents