research-article

Public Access

Evaluating Interactive Data Systems: Workloads, Metrics, and Guidelines

Authors:

Protiva Rahman,

Arnab NandiAuthors Info & Claims

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Pages 1637 - 1644

https://doi.org/10.1145/3183713.3197386

Published: 27 May 2018 Publication History

Abstract

Highly interactive query interfaces have become a popular tool for ad-hoc data analysis and exploration, posing a new kind of workload to the underlying data infrastructure. Compared with traditional systems that are optimized for throughput or batched performance, ad-hoc and interactive data exploration systems focus more on user-centric interactivity, which raises a new class of performance challenges. Further, with the advent of new interaction devices~(e.g., touch, gesture) and different query interface paradigms~(e.g., sliders), maintaining interactive performance becomes even more challenging. Thus, when building interactive data systems, there is a clear need to articulate the design space.

In this tutorial, we will describe unique characteristics of interactive workloads for a variety of user input devices and query interfaces. We will catalog popular metrics based on an extensive survey of current literature. Through two case studies, we will not only walk through previously defined metrics using real-world user traces but also highlight where these defined metrics are inadequate. Further, we will introduce some new metrics that are required to capture a complete picture of interactivity. In each case study, we also demonstrate how the behavior analyses on users' trace and performance experiments can provide guidelines to help researchers and developers design better interactive data systems.

References

[1]

Crossfilter Library. http://square.github.io/crossfilter/.

[2]

IMDb. http://www.imdb.com/.

[3]

OMDb API - The Open Movie Database. http://www.omdbapi.com/.

[4]

TPC Benchmark. http://www.tpc.org/.

[5]

UCI Repository of Machine Learning Databases. https://archive.ics.uci.edu/ml/datasets.html.

[6]

Use Multi-Touch gestures on your Mac. https://support.apple.com/en-us/HT204895.

[7]

A. Abouzied, J. Hellerstein, and A. Silberschatz. Dataplay: interactive tweaking and example-driven correction of graphical database queries. In Proceedings of the 25th annual ACM symposium on User interface software and technology, pages 207--218. ACM, 2012.

Digital Library

[8]

W. Albert and T. Tullis. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, 2013.

Digital Library

[9]

E. Bakke, D. Karger, and R. Miller. A spreadsheet-based user interface for managing plural relationships in structured data. In SIGCHI, 2011.

Digital Library

[10]

C. M. Barnum. Usability testing essentials: ready, set... test! Elsevier, 2010.

Digital Library

[11]

R. C. Basole, T. Clear, M. Hu, H. Mehrotra, and J. Stasko. Understanding interfirm relationships in business ecosystems with interactive visualization. TVCG, 2013.

Digital Library

[12]

S. Basu Roy, H. Wang, G. Das, U. Nambiar, and M. Mohania. Minimum-effort driven dynamic faceted search in structured databases. In CIKM, 2008.

Digital Library

[13]

L. Battle, R. Chang, and M. Stonebraker. Dynamic prefetching of data tiles for interactive visualization. 2016.

Digital Library

[14]

D. A. Bell, L. S. Deluca, D. J. Levinson, and R. Salem. Browser interaction for lazy loading operations, Dec. 15 2014. US Patent App. 14/570,430.

[15]

J. Bernard, N. Wilhelm, B. Krüger, T. May, T. Schreck, and J. Kohlhammer. Motionexplorer: Exploratory search in human motion capture data based on hierarchical aggregation. TVCG, 2013.

Digital Library

[16]

A. Biswas, S. Dutta, H.-W. Shen, and J. Woodring. An information-aware framework for exploring multivariate data sets. TVCG, 2013.

Digital Library

[17]

N. Cao, D. Gotz, J. Sun, and H. Qu. Dicon: Interactive visual analysis of multidimensional clusters. TVCG, 2011.

Digital Library

[18]

S.-M. Chan, L. Xiao, J. Gerth, and P. Hanrahan. Maintaining interactivity while exploring massive time series. In VAST, 2008.

[19]

S. Chaudhuri, R. Motwani, and V. Narasayya. Random sampling for histogram construction: How much is enough? In SIGMOD, 1998.

Digital Library

[20]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with ycsb. In Socc, 2010.

Digital Library

[21]

K. Dimitriadou, O. Papaemmanouil, and Y. Diao. Explore-by-example: An automatic query steering framework for interactive data exploration. In SIGMOD, 2014.

Digital Library

[22]

P. R. Doshi, E. A. Rundensteiner, and M. O. Ward. Prefetching for visual data exploration. In DASFAA. IEEE, 2003.

Digital Library

[23]

J. Faith. Targeted projection pursuit for interactive exploration of high-dimensional data sets. In IV, 2007.

Digital Library

[24]

L. Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, 2003.

[25]

J.-D. Fekete and C. Plaisant. Interactive information visualization of a million items. In INFOVIS, 2002.

Digital Library

[26]

N. Ferreira, J. Poco, H. T. Vo, J. Freire, and C. T. Silva. Visual exploration of big spatio-temporal urban data: A study of new york city taxi trips. TVCG, 2013.

Digital Library

[27]

D. Fisher, I. Popov, S. Drucker, et al. Trust me, i'm partially right: incremental visualization lets analysts explore large datasets faster. In SIGCHI, 2012.

Digital Library

[28]

P. B. Gibbons, Y. Matias, and V. Poosala. Fast incremental maintenance of approximate histograms. In VLDB, 1997.

Digital Library

[29]

N. R. Gujarathi and A. A. Shah. Parameterized computed scrolling for navigation of structured data, 2015.

[30]

D. Gunopulos, G. Kollios, V. J. Tsotras, and C. Domeniconi. Approximating multi-dimensional aggregate range queries over real attributes. In SIGMOD Record, 2000.

Digital Library

[31]

J. Heer, M. Agrawala, and W. Willett. Generalized selection via interactive query relaxation. In SIGCHI, 2008.

Digital Library

[32]

T. Igarashi and K. Hinckley. Speed-dependent automatic zooming for browsing large documents. In UIST, 2000.

Digital Library

[33]

H. Jagadish, A. Chapman, A. Elkiss, M. Jayapandian, Y. Li, A. Nandi, and C. Yu. Making database systems usable. In SIGMOD, 2007.

Digital Library

[34]

W. Javed, S. Ghani, and N. Elmqvist. Gravnav: Using a gravity model for multi-scale navigation. In Advanced Visual Interfaces, 2012.

Digital Library

[35]

L. Jiang and A. Nandi. Snaptoquery: providing interactive feedback during exploratory query specification. VLDB, 2015.

Digital Library

[36]

N. Kamat, P. Jayachandran, K. Tunga, and A. Nandi. Distributed and interactive cube exploration. In ICDE, 2014.

[37]

S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer. Wrangler: Interactive visual specification of data transformation scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3363--3372. ACM, 2011.

Digital Library

[38]

A. Kashyap, V. Hristidis, and M. Petropoulos. Facetor: cost-driven exploration of faceted query results. In CIKM, 2010.

Digital Library

[39]

A. Kashyap, V. Hristidis, M. Petropoulos, and S. Tavoulari. Effective navigation of query results based on concept hierarchies. TKDE, 2011.

Digital Library

[40]

M. Kaul, B. Yang, and C. S. Jensen. Building accurate 3d spatial networks to enable next generation intelligent transportation systems. In MDM, 2013.

Digital Library

[41]

D. A. Keim. Information visualization and visual data mining. IEEE transactions on Visualization and Computer Graphics, 2002.

Digital Library

[42]

A. Key, B. Howe, D. Perry, and C. Aragon. Vizdeck: self-organizing dashboards for visual analytics. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 681--684. ACM, 2012.

Digital Library

[43]

M. Khan, L. Xu, A. Nandi, and J. M. Hellerstein. Data tweening: incremental visualization of data transforms. Proceedings of the VLDB Endowment, 10(6):661--672, 2017.

Digital Library

[44]

R. Kosara, F. Bendix, and H. Hauser. Parallel sets: Interactive exploration and visual analysis of categorical data. TVCG, 2006.

Digital Library

[45]

S. Kullback. Information theory and statistics. Courier Corporation, 1997.

[46]

H. Lam, D. Russell, D. Tang, and T. Munzner. Session viewer: Visual exploratory analysis of web session logs. In VAST, 2007.

Digital Library

[47]

J. Lazar, J. H. Feng, and H. Hochheiser. Research methods in human-computer interaction. John Wiley & Sons, 2010.

Digital Library

[48]

Y. Li, H. Yang, and H. Jagadish. Nalix: A generic natural language search environment for xml data. TODS, 2007.

Digital Library

[49]

L. Lins, J. T. Klosowski, and C. Scheidegger. Nanocubes for real-time exploration of spatiotemporal datasets. TVCG, 2013.

Digital Library

[50]

B. Liu and H. Jagadish. A spreadsheet algebra for a direct data manipulation query interface. In ICDE, 2009.

Digital Library

[51]

Z. Liu and J. Heer. The effects of interactive latency on exploratory visual analysis. TVCG, 2014.

[52]

Z. Liu, B. Jiang, and J. Heer. imMens: Real-time Visual Querying of Big Data. EuroVis, 2013.

Digital Library

[53]

A. R. Martin and M. O. Ward. High dimensional brushing for interactive exploration of multivariate data. In Visualization, 1995.

Digital Library

[54]

P. McLachlan, T. Munzner, E. Koutsofios, and S. North. Liverac: interactive visual exploration of system management time-series data. In SIGCHI, 2008.

Digital Library

[55]

F. Moussavi. Hybrid inertial and touch sensing input device, Feb. 18 2010. US Patent App. 12/192,889.

[56]

A. Nandi, L. Jiang, and M. Mandel. Gestural query specification. VLDB, 2013.

Digital Library

[57]

N. Nourbakhsh, Y. Wang, F. Chen, and R. A. Calvo. Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks. In Proceedings of the 24th Australian Computer-Human Interaction Conference, pages 420--423. ACM, 2012.

Digital Library

[58]

C. Plaisant. The challenge of information visualization evaluation. In AVI, 2004.

Digital Library

[59]

V. Poosala, P. J. Haas, Y. E. Ioannidis, and E. J. Shekita. Improved histograms for selectivity estimation of range predicates. In SIGMOD Record, 1996.

Digital Library

[60]

J. M. Rzeszotarski and A. Kittur. Kinetica: naturalistic multi-touch data visualization. In CHI, 2014.

Digital Library

[61]

R. G. Saadé and C. A. Otrakji. First impressions last a lifetime: effect of interface type on disorientation and cognitive load. Computers in human behavior, 23(1):525--535, 2007.

[62]

A. Satyanarayan and J. Heer. Lyra: An interactive visualization design environment. In Computer Graphics Forum, 2014.

Digital Library

[63]

J. Seo and B. Shneiderman. A rank-by-feature framework for interactive exploration of multidimensional data. Information visualization, 2005.

Digital Library

[64]

B. Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In IEEE Symposium on-Visual Languages, 1996.

Digital Library

[65]

M. Singh, A. Nandi, and H. Jagadish. Skimmer: rapid scrolling of relational query results. In SIGMOD, 2012.

Digital Library

[66]

J. Sweller. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257--285, 1988.

[67]

F. Tauheed, T. Heinis, F. Schürmann, H. Markram, and A. Ailamaki. Scout: prefetching for latent structure following queries. VLDB, 2012.

Digital Library

[68]

L. F. Van Dillen, D. J. Heslenfeld, and S. L. Koole. Tuning down the emotional brain: an fmri study of the effects of cognitive load on the processing of affective images. Neuroimage, 45(4):1212--1219, 2009.

[69]

J. S. Vitter, M. Wang, and B. Iyer. Data cube approximation and histograms via wavelets. In CIKM, 1998.

Digital Library

[70]

C. Weaver. Multidimensional visual analysis using cross-filtered views. In VAST, 2008.

[71]

J. Wei, Z. Shen, N. Sundaresan, and K.-L. Ma. Visual cluster exploration of web clickstream data. In VAST, 2012.

Digital Library

[72]

L. Wilkinson. The grammar of graphics. Springer Science & Business Media, 2006.

Digital Library

[73]

W. Willett, J. Heer, and M. Agrawala. Scented widgets: Improving navigation cues with embedded visualizations. TVCG, 2007.

Digital Library

[74]

J. Woodring and H.-W. Shen. Multiscale time activity data exploration via temporal clustering visualization spreadsheet. TVCG, 2009.

Digital Library

[75]

E. Wu, L. Jiang, L. Xu, and A. Nandi. Graphical perception in animated bar charts. In arXiv.

[76]

D. Yang, Z. Guo, E. A. Rundensteiner, and M. O. Ward. Clues: a unified framework supporting interactive exploration of density-based clusters in streams. In CIKM, 2011.

Digital Library

[77]

D. Yang, E. A. Rundensteiner, and M. O. Ward. Analysis guided visual exploration of multivariate data. In Visual Analytics Science and Technology, 2007. VAST 2007. IEEE Symposium on, pages 83--90. IEEE, 2007.

Digital Library

[78]

J. Yang, A. Patro, S. Huang, N. Mehta, M. O. Ward, and E. A. Rundensteiner. Value and relation display for interactive exploration of high dimensional datasets. In INFOVIS, 2004.

Digital Library

[79]

J. Yang, W. Peng, M. O. Ward, and E. A. Rundensteiner. Interactive hierarchical dimension ordering, spacing and filtering for exploration of high dimensional datasets. In INFOVIS, 2003.

Digital Library

[80]

J. S. Yi, Y. ah Kang, and J. Stasko. Toward a deeper understanding of the role of interaction in information visualization. TVCG, 2007.

Digital Library

[81]

X. Yuan, D. Ren, Z. Wang, and C. Guo. Dimension projection matrix/tree: Interactive subspace visual exploration and analysis of high dimensional data. TVCG, 2013.

Digital Library

[82]

Z. Zhang, K. T. McDonnell, and K. Mueller. A network-based interface for the exploration of high-dimensional data spaces. In PacificVis, 2012.

Digital Library

Cited By

Wolf KWinecki DNandi A(2023)Camera-First Form FillingProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3597465.3605218(1-7)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3597465.3605218
Zegarra FCarbajal Ipenza JOmidvar-Tehrani BMoreira VAmer-Yahia SComba J(2022)Visual exploration of rating datasets and user groupsFuture Generation Computer Systems10.1016/j.future.2019.12.011105:C(547-561)Online publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.future.2019.12.011
Battle LEichmann PAngelini MCatarci TSantucci GZheng YBinnig CFekete JMoritz DMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Database Benchmarking for Supporting Real-Time Interactive Querying of Large DataProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3389732(1571-1587)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3389732
Show More Cited By

Index Terms

Evaluating Interactive Data Systems: Workloads, Metrics, and Guidelines
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Information systems
  1. Data management systems
    1. Database administration
      1. Database performance evaluation

Recommendations

Evaluating interactive data systems: Survey and case studies
Abstract
Interactive query interfaces have become a popular tool for ad hoc data analysis and exploration. Compared with traditional systems that are optimized for throughput or batched performance, these systems focus more on user-centric interactivity. ...
3D Freehand Gestural Navigation for Interactive Public Displays

Users increasingly expect more-interactive experiences with public displays for applications including learning, gaming, urban visualization, and planning. However, user interaction with applications on public displays is challenging and often doesn't ...
A Camera-Based Input Device for Large Interactive Displays

Humanýcomputer interaction using large-format displays is an active area of research that focuses on how humans can better work with computers or other machines. For this to happen, there must be an enabling technology that creates the interface between ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
546
Total Downloads

Downloads (Last 12 months)97
Downloads (Last 6 weeks)20

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wolf KWinecki DNandi A(2023)Camera-First Form FillingProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3597465.3605218(1-7)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3597465.3605218
Zegarra FCarbajal Ipenza JOmidvar-Tehrani BMoreira VAmer-Yahia SComba J(2022)Visual exploration of rating datasets and user groupsFuture Generation Computer Systems10.1016/j.future.2019.12.011105:C(547-561)Online publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.future.2019.12.011
Battle LEichmann PAngelini MCatarci TSantucci GZheng YBinnig CFekete JMoritz DMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Database Benchmarking for Supporting Real-Time Interactive Querying of Large DataProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3389732(1571-1587)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3389732
Dong LBai QKim TChen TLiu WLi CMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Marviq: Quality-Aware Geospatial Visualization of Range-Selection Queries Using MaterializationProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3389730(67-82)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3389730
Frid EGomes CJin ZBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Music Creation by ExampleProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376514(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376514
Omidvar-Tehrani BAmer-Yahia S(2020)User Group Analytics Survey and Research OpportunitiesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.291365132:10(2040-2059)Online publication date: 1-Oct-2020
https://doi.org/10.1109/TKDE.2019.2913651
Sarkhel RNandi A(2019)Deterministic routing between layout abstractions for multi-scale classification of visually rich documentsProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367471.3367508(3360-3366)Online publication date: 10-Aug-2019
https://dl.acm.org/doi/10.5555/3367471.3367508
Rahman PHebert CNandi A(2019)ICARUSProceedings of the VLDB Endowment10.14778/3275366.328497011:13(2263-2276)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.14778/3275366.3284970
Omidvar-Tehrani BAmer-Yahia SSimon EZegarra FComba JMoreira V(2019)UserDEVProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3328519.3329128(1-8)Online publication date: 5-Jul-2019
https://dl.acm.org/doi/10.1145/3328519.3329128
Rahman PNandi AFu WPan SBrdiczka OChau PCalvary G(2019)TransformerProceedings of the 24th International Conference on Intelligent User Interfaces10.1145/3301275.3302269(485-496)Online publication date: 17-Mar-2019
https://dl.acm.org/doi/10.1145/3301275.3302269
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents