Provectories: Embedding-Based Analysis of Interaction Provenance Data
Authors: 
Conny Walchshofer, Andreas Hinterreiter, Kai Xu, Holger Stitz, Marc StreitAuthors Info & Claims
Conny Walchshofer, Andreas Hinterreiter, Kai Xu, Holger Stitz, Marc StreitAuthors Info & ClaimsPages 4816 - 4831
Abstract
Understanding user behavior patterns and visual analysis strategies is a long-standing challenge. Existing approaches rely largely on time-consuming manual processes such as interviews and the analysis of observational data. While it is technically possible to capture a history of user interactions and application states, it remains difficult to extract and describe analysis strategies based on interaction provenance. In this article, we propose a novel visual approach to the meta-analysis of interaction provenance. We capture single and multiple user sessions as graphs of high-dimensional application states. Our meta-analysis is based on two different types of two-dimensional embeddings of these high-dimensional states: layouts based on (i) topology and (ii) attribute similarity. We applied these visualization approaches to synthetic and real user provenance data captured in two user studies. From our visualizations, we were able to extract patterns for data types and analytical reasoning strategies.
References
[1]
Anaconda Inc. Datashader. Accurately render even the largest data, 2016. [Online]. Available: https://datashader.org/
[2]
B. Bach, C. Shi, N. Heulot, T. Madhyastha, T. Grabowski, and P. Dragicevic, “Time curves: Folding time to visualize patterns of temporal evolution in data,” IEEE Trans. Vis. Comput. Graph., vol. 22, no. 1, pp. 559–568, Jan. 2016.
[3]
T. Blascheck, F. Beck, S. Baltes, T. Ertl, and D. Weiskopf, “Visual analysis and coding of data-rich user behavior,” in Proc. IEEE Conf. Vis. Anal. Sci. Technol., 2016, pp. 141–150.
[4]
M. Brehmer and T. Munzner, “A multi-level typology of abstract visualization tasks,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 12, pp. 2376–2385, Dec. 2013.
[5]
E. T. Brown et al., “Finding waldo: Learning about users from their Interactions,” IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 1663–1672, Dec. 2014.
[6]
E. T. Brown, S. Yarlagadda, K. A. Cook, A. Endert, and R. Chang, “ModelSpace: Visualizing the trails of data models in visual analytics systems,” in Proc. Mach. Learn. User Int. Vis. Anal. Workshop, 2018.
[7]
S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, and H. T. Vo, “VisTrails: Visualization meets data management,” in Proc. Int. Conf. Manage. Data, 2006, pp. 745–747.
[8]
H. Chung, S. Yang, N. Massjouni, C. Andrews, R. Kanna, and C. North, “VizCept: Supporting synchronous collaboration for constructing visualizations in intelligence analysis,” in Proc. IEEE Symp. Vis. Anal. Sci. Technol., 2010, pp. 107–114.
[9]
P. Cowley, L. Nowell, and J. Scholtz, “Glass box: An instrumented infrastructure for supporting human interaction with information,” in Proc. Conf. Sys. Sci., p. 296c, 2005.
[10]
Z. T. Cutler, K. Gadhave, and A. Lex, “Trrack: A library for provenance tracking in web-based visualizations,” in Proc. IEEE Vis. Conf., 2020, pp. 116–120.
[11]
W. Dou, D. H. Jeong, F. Stukes, W. Ribarsky, H. R. Lipford, and R. Chang, “Recovering reasoning process from user interactions,” IEEE Comput. Graph. Appl., vol. 29, no. 3, pp. 52–61, May/Jun. 2009.
[12]
C. Dunne, N. Henry Riche, B. Lee, R. Metoyer, and G. Robertson, “GraphTrail: Analyzing large multivariate, heterogeneous networks while supporting exploration history,” in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2012, pp. 1663–1672.
[13]
K. Eckelt et al., “Visual exploration of relationships and structure in low-dimensional embeddings,” Open Sci. Framework, Apr. 2021.
[14]
M. Feng, E. Peck, and L. Harrison, “Patterns and pace: Quantifying diverse exploration behavior with visualizations on the web,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 1, pp. 501–511, Jan. 2019.
[15]
V. A. Fock, “Konfigurationsraum und zweite quantelung,” Zeitschrift Physik, vol. 75, no. 9–10, pp. 622–647, Sep. 1932.
[16]
K. Gadhave et al., “Predicting intent behind selections in scatterplot visualizations,” Inf. Vis., vol. 20, no. 4, pp. 207–228, Oct. 2021.
[17]
M. Gleicher, “Considerations for visualizing comparison,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 1, pp. 413–423, Aug. 2018.
[18]
S. Gomez and D. Laidlaw, “Modeling task performance for a crowd of users from interaction histories,” in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2012, pp. 2465–2468.
[19]
D. Gotz and M. X. Zhou, “Characterizing Users’ visual analytic activity for insight provenance,” Inf. Vis., vol. 8, no. 1, pp. 42–55, 2009.
[20]
J. C. Gower, “A general coefficient of similarity and some of its properties,” Biometrics, vol. 27, no. 4, pp. 857–871, 1971.
[21]
S. Gratzl, A. Lex, N. Gehlenborg, N. Cosgrove, and M. Streit, “From visual exploration to storytelling and back again,” Comput. Graph. Forum, vol. 35, no. 3, pp. 491–500, 2016.
[22]
J. Heer, J. Mackinlay, C. Stolte, and M. Agrawala, “Graphical histories visualization: Supporting analysis, communication, and evaluation,” IEEE Trans. Vis. Comput. Graph., vol. 14, no. 6, pp. 1189–1196, Nov./Dec. 2008.
[23]
A. Hinterreiter, C. Steinparz, M. Schöfl, H. Stitz, and M. Streit, “Projection path explorer: Exploring visual patterns in projected decision-making paths,” ACM Trans. Int. Intell. Syst., vol. 11, no. 3–4, 2021, Art. no.
[24]
J. Hollan, E. Hutchins, and D. Kirsh, “Distributed cognition: Toward a new foundation for human-computer interaction research,” ACM Trans. Comput.-Hum. Int., vol. 7, no. 2, pp. 174–196, 2000.
[25]
M. Jacomy, T. Venturini, S. Heymann, and M. Bastian, “ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software,” PLoS One, vol. 9, no. 6, 2014, Art. no.
[26]
A. Karpathy, “tSNEJS”, 2014. [Online]. Available: https://github.com/karpathy/tsnejs/
[27]
N. Kerracher, J. Kennedy, and K. Chalmers, “A task taxonomy for temporal graph visualisation,” IEEE Trans. Vis. Comput. Graph., p. 1, 2015.
[28]
J. F. Kruiger, P. E. Rauber, R. M. Martins, A. Kerren, S. Kobourov, and A. C. Telea, “Graph layouts by t-SNE,” Comput. Graph. Forum, vol. 36, no. 3, pp. 283–294, 2017.
[29]
Z. Liu and J. Stasko, “Mental models, visual reasoning and interaction in information visualization: A top-down perspective,” IEEE Trans. Vis. Comput. Graph., vol. 16, no. 6, pp. 999–1008, Nov./Dec. 2010.
[30]
K. Madanagopal, E. D. Ragan, and P. Benjamin, “Analytic provenance in practice: The role of provenance in real-world visualization and data analysis environments,” IEEE Comput. Graph. Appl., vol. 39, no. 6, pp. 30–45, Nov. 2019.
[31]
W. C. Mankowski, P. Bogunovich, A. Shokoufandeh, and D. D. Salvucci, “Finding canonical behaviors in user protocols,” in Proc. Conf. Hum. Factors Comput. Syst., 2009, pp. 1323–1326.
[32]
M. Pohl and J. D. Haider, “Sense-making strategies for the interpretation of visualizations—bridging the gap between theory and empirical research,” Multimodal Technol. Int., vol. 1, no. 3, 2017, Art. no.
[33]
L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,” 2018,.
[34]
P. H. Nguyen, K. Xu, A. Wheat, B. L. W. Wong, S. Attfield, and B. Fields, “SensePath: Understanding the sensemaking process through analytic provenance,” IEEE Trans. Vis. Comput. Graph., vol. 22, no. 1, pp. 41–50, Jan. 2016.
[35]
C. Nobre, D. Wootton, Z. Cutler, L. Harrison, H. Pfister, and A. Lex, “reVISit: Looking under the hood of interactive visualization studies,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2021, pp. 1–13.
[36]
L. G. Nonato and M. Aupetit, “Multidimensional projection for visual analytics: Linking techniques with distortions, tasks, and layout enrichment,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 8, pp. 2650–2673, Aug. 2019.
[37]
A. Pandey, U. Syeda, and M. Borkin, “Towards identification and mitigation of task-based challenges in comparative visualization studies,” in Proc. IEEE Workshop Eval. Beyond Methodological Approaches Vis., 2020, pp. 55–64.
[38]
F. Pedregosa et al., “Scikit-learn: Machine learning in python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
[39]
People+AI Research (PAIR) Initiative, UMAP-JS, 2019. [Online]. Available: https://github.com/PAIR-code/umap-js/
[40]
G. Plique, “Graphology ForceAtlas2, 2016. [Online]. Available: https://github.com/graphology/graphology-layout-forceatlas2
[41]
M. Pohl, M. Smuc, and E. Mayr, “The user puzzle - explaining the interaction with visual analytics systems,” IEEE Trans. Vis. Comput. Graph., vol. 18, no. 12, pp. 2908–2916, Dec. 2012.
[42]
P. G. Poličar, M. Stražar, and B. Zupan, “openTSNE: A modular python library for t-SNE dimensionality reduction and embedding,” 2019, bioRxiv.
[43]
E. Ragan, A. Endert, J. Sanyal, and J. Chen, “Characterizing provenance in visualization and data analysis: An organizational framework of provenance types and purposes,” IEEE Trans. Vis. Comput. Graph., vol. 22, no. 1, pp. 31–40, Jan. 2016.
[44]
P. E. Rauber, S. G. Fadel, A. X. Falcão, and A. C. Telea, “Visualizing the hidden activity of artificial neural networks,” IEEE Trans. Vis. Comput. Graph., vol. 23, no. 1, pp. 101–110, Jan. 2017.
[45]
H. Rosling and Z. Zhang, “Health advocacy with Gapminder animated statistics,” J. Epidemiol. Glob. Health, vol. 1, no. 1, pp. 11–14, 2011.
[46]
V. Setlur, S. E. Battersby, M. Tory, R. Gossweiler, and A. X. Chang, “Eviza: A natural language interface for visual analysis,” in Proc. Symp. User Interface Softw. Technol., 2016, pp. 365–377.
[47]
H. Stitz, S. Gratzl, H. Piringer, T. Zichner, and M. Streit, “KnowledgePearls: Provenance-based visualization retrieval,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 1, pp. 120–130, Jan. 2019.
[48]
H. Stitz, S. Luger, M. Streit, and N. Gehlenborg, “AVOCADO: Visualization of Workflow–Derived data provenance for reproducible biomedical research,” Comput. Graph. Forum, vol. 35, no. 3, pp. 481–490, 2016.
[49]
S. van den Elzen, D. Holten, J. Blaas, and J. van Wijk, “Reducing snapshots to points: A visual analytics approach to dynamic network exploration,” IEEE Trans. Vis. Comput. Graph., vol. 22, no. 1, pp. 1–10, Jan. 2016.
[50]
L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach. Learn. Res., vol. 9, pp. 2579–2605, 2008.
[51]
J. Wei, Z. Shen, N. Sundaresan, and K.-L. Ma, “Visual cluster exploration of web clickstream data,” in Proc. IEEE Conf. Vis. Anal. Sci. Technol., 2012, pp. 3–12.
[52]
J. Wood, A. Kachkaev, and J. Dykes, “Design exposition with literate visualization,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 1, pp. 759–768, Jan. 2019.
[53]
K. Xu, A. Ottley, C. Walchshofer, M. Streit, R. Chang, and J. Wenskovitch, “Survey on the analysis of user interactions and visualization provenance,” Comput. Graph. Forum, vol. 39, no. 3, pp. 757–783, Jun. 2020.
[54]
Yang Chen, Jing Yang, and W. Ribarsky, “Toward effective insight management in visual analytics systems,” in Proc. IEEE Pacific Vis. Symp., 2009, pp. 49–56.
[55]
J. Zhao, M. Glueck, P. Isenberg, F. Chevalier, and A. Khan, “Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 1, pp. 340–350, Jan. 2018.
Index Terms
- Provectories: Embedding-Based Analysis of Interaction Provenance Data
Index terms have been assigned to the content through auto-classification.
Recommendations
Comparing Tangible and Multi-touch Interaction for Interactive Data Visualization Tasks
TEI '16: Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied InteractionInteractive visualization plays a key role in the analysis of large datasets. It can help users to explore data, investigate hypotheses and find patterns. The easier and more tangible the interaction, the more likely it is to enhance understanding. This ...
Automated data provenance capture in spreadsheets, with case studies
One of the most important tasks in eScience is capturing the provenance of data. While scientists frequently use off-the-shelf analysis tools to process and manipulate data, current provenance techniques such as those based on scientific workflows are ...
A study on touch & hover based interaction for zooming
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsProximity is a useful medium for interaction with high interactive digital contents. It can be used in different contexts such as for navigation through depth in 3D space in zoomable interfaces. In this paper, we propose hover-based zoom interaction as ...
Comments
Information & Contributors
Information
Published In
© 2021 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 December 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Feb 2025