Cited By
View all- Liang PYe DZhu ZWang YXia WLiang RSun G(2024)C5: toward better conversation comprehension and contextual continuity for ChatGPTJournal of Visualization10.1007/s12650-024-00980-427:4(713-730)Online publication date: 5-Apr-2024
E2Storyline: Visualizing the Relationship with Triplet Entities and Event Discovery
Authors: 
Yunchao Wang, Guodao Sun, Zihao Zhu, Tong Li, Ling Chen, and Ronghua LiangAuthors Info & Claims
Yunchao Wang, Guodao Sun, Zihao Zhu, Tong Li, Ling Chen, and Ronghua LiangAuthors Info & ClaimsArticle No.: 16, Pages 1 - 26
Abstract
The narrative progression of events, evolving into a cohesive story, relies on the entity-entity relationships. Among the plethora of visualization techniques, storyline visualization has gained significant recognition for its effectiveness in offering an overview of story trends, revealing entity relationships, and facilitating visual communication. However, existing methods for storyline visualization often fall short in accurately depicting the specific relationships between entities. In this study, we present E2Storyline, a novel approach that emphasizes simplicity and aesthetics of layout while effectively conveying entity-entity relationships to users. To achieve this, we begin by extracting entity-entity relationships from textual data and representing them as subject-predicate-object (SPO) triplets, thereby obtaining structured data. By considering three types of design requirements, we establish new optimization objectives and model the layout problem using multi-objective optimization (MOO) techniques. The aforementioned SPO triplets, together with time and event information, are incorporated into the optimization model to ensure a straightforward and easily comprehensible storyline layout. Through a qualitative user study, we determine that a pixel-based view is the most suitable method for displaying the relationships between entities. Finally, we apply E2Storyline to real-world data, including movie synopses and live text commentaries. Through comprehensive case studies, we demonstrate that E2Storyline enables users to better extract information from stories and comprehend the relationships between entities.
References
[1]
Robert Kosara and Jock Mackinlay. 2013. Storytelling: The next step for visualization. Computer 46, 5 (2013), 44–50.
[2]
Lu Qiang, Chai Bingjie, and Zhang Haibo. 2017. Storytelling by the StoryCake visualization. The Visual Computer 33, 10 (2017), 1241–1252.
[3]
Jänicke Stefan, Greta Franzini, Muhammad Faisal Cheema, and Scheuermann Gerik. 2015. On close and distant reading in digital humanities: A survey and future challenges. EuroVis (STARs) 2015 (0215), 83–103.
[4]
Panpan Xu, Yingcai Wu, Enxun Wei, Tai-Quan Peng, Shixia Liu, Jonathan J. H. Zhu, and Huamin Qu. 2013. Visual analysis of topic competition on social media. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2012–2021.
[5]
Guodao Sun, Yingcai Wu, Shixia Liu, Tai-Quan Peng, Jonathan J. H. Zhu, and Ronghua Liang. 2014. EvoRiver: Visual analysis of topic coopetition on social media. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1753–1762.
[6]
Jeffery Ansah, Lin Liu, Wei Kang, Selasie Kwashie, Jixue Li, and Jiuyong Li. 2019. A graph is worth a thousand words: Telling event stories using timeline summarization graphs. In The World Wide Web Conference. 2565–2571.
[7]
Bang Liu, Fred X. Han, Di Niu, Linglong Kong, Kunfeng Lai, and Yu Xu. 2020. Story Forest: Extracting events and telling stories from breaking news. ACM Transactions on Knowledge Discovery from Data 14, 3 (2020), 1–28.
[8]
Ayush Shrestha, Ben Miller, Ying Zhu, and Yi Zhao. 2013. Storygraph: Extracting patterns from spatio-temporal data. In Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics. 95–103.
[9]
Xi He and Ying Zhu. 2020. Integrated spatio-temporal storyline visualization with low crossover. In International Conference Information Visualisation (IV’20). IEEE, 236–241.
[10]
R. Munroe. ([n. d.]) Movie Narrative Charts. https://xkcd.com/657/. ([n. d.]).
[11]
Yuzuru Tanahashi and Kwan-Liu Ma. 2012. Design considerations for optimizing storyline visualizations. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2679–2688.
[12]
Shixia Liu, Yingcai Wu, Enxun Wei, Mengchen Liu, and Yang Liu. 2013. StoryFlow: Tracking the evolution of stories. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2436–2445.
[13]
Tan Tang, Sadia Rubab, Jiewen Lai, Weiwei Cui, Lingyun Yu, and Yingcai Wu. 2018. iStoryline: Effective convergence to hand-drawn storylines. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 769–778.
[14]
Makarand Tapaswi, Martin Bauml, and Rainer Stiefelhagen. 2014. StoryGraphs: Visualizing character interactions as a timeline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 827–834.
[15]
Zihao Zhu, Yue Shen, Sujia Zhu, Gefei Zhang, Ronghua Liang, and Guodao Sun. 2022. Towards better pattern enhancement in temporal evolving set visualization. Journal of Visualization (2022), 1–1.
[16]
Tuan Nhon Dang, Nick Pendar, and Angus Graeme Forbes. 2016. TimeArcs: Visualizing fluctuations in dynamic networks. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 61–69.
[17]
Nathalie Henry, Jean-Daniel Fekete, and Michael J. McGuffin. 2007. Nodetrix: A hybrid visualization of social networks. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1302–1309.
[18]
Daniela Oelke, Dimitrios Kokkinakis, and Daniel A. Keim. 2013. Fingerprint matrices: Uncovering the dynamics of social networks in prose literature. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 371–380.
[19]
Jian Zhao, Michael Glueck, Fanny Chevalier, Yanhong Wu, and Azam Khan. 2016. Egocentric analysis of dynamic networks with EgoLines. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 5003–5014.
[20]
Michael Ogawa and Kwan-Liu Ma. 2010. Software evolution storylines. In Proceedings of the International Symposium on Software Visualization. 35–42.
[21]
Yang Shi, Chris Bryan, Sridatt Bhamidipati, Ying Zhao, Yaoxue Zhang, and Kwan-Liu Ma. 2018. MeetingVis: Visual narratives to assist in recalling meeting context and content. IEEE Transactions on Visualization and Computer Graphics 24, 6 (2018), 1918–1929.
[22]
Qiang Lu, Xiang-yuan Zhu, Li Liu, and Shu-bo Cao. 2014. An effective demonstration for group collaboration based on storyline visualization technology. In Proceedings of the International Conference on Computer Supported Cooperative Work in Design. IEEE, 47–52.
[23]
Tan Tang, Renzhong Li, Xinke Wu, Shuhan Liu, Johannes Knittel, Steffen Koch, Thomas Ertl, Lingyun Yu, Peiran Ren, and Yingcai Wu. 2020. PlotThread: Creating expressive storyline visualizations using reinforcement learning. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 294–303.
[24]
Fabian Bolte, Mahsan Nourani, Eric D. Ragan, and Stefan Bruckner. 2020. SplitStreams: A visual metaphor for evolving hierarchies. IEEE Transactions on Visualization and Computer Graphics 27, 8 (2020), 3571–3584.
[25]
Wei Zhang, Siwei Tan, Siming Chen, Linhao Meng, Tianye Zhang, Rongchen Zhu, and Wei Chen. 2022. Visual reasoning for uncertainty in spatio-temporal events of historical figures. IEEE Transactions on Visualization and Computer Graphics (2022).
[26]
Yuzuru Tanahashi, Chien-Hsin Hsueh, and Kwan-Liu Ma. 2015. An efficient framework for generating storyline visualizations from streaming data. IEEE Transactions on Visualization and Computer Graphics 21, 6 (2015), 730–742.
[27]
Martin Gronemann, Michael Jünger, Frauke Liers, and Francesco Mambelli. 2016. Crossing minimization in storyline visualization. In Proceedings of the International Symposium on Graph Drawing and Network Visualization. 367–381.
[28]
Martin Nollenburg and Alexander Wolff. 2010. Drawing and labeling high-quality metro maps by mixed-integer programming. IEEE Transactions on Visualization and Computer Graphics 17, 5 (2010), 626–641.
[29]
Ben Jacobsen, Markus Wallinger, Stephen Kobourov, and Martin Nöllenburg. 2020. MetroSets: Visualizing sets as metro maps. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1257–1267.
[30]
Yi Guo, Shunan Guo, Zhuochen Jin, Smiti Kaul, David Gotz, and Nan Cao. 2021. A survey on visual analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics (2021).
[31]
Chao Tong, Richard Roberts, Rita Borgo, Sean Walton, Robert S. Laramee, Kodzo Wegba, Aidong Lu, Yun Wang, Huamin Qu, and Qiong Luo. 2018. Storytelling and visualization: An extended survey. Information 9, 3 (2018), 65.
[32]
Nam Wook Kim, Benjamin Bach, Hyejin Im, Sasha Schriber, Markus Gross, and Hanspeter Pfister. 2017. Visualizing nonlinear narratives with story curves. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 595–604.
[33]
Xingyu Lan, Xinyue Xu, and Nan Cao. 2021. Understanding narrative linearity for telling expressive time-oriented stories. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–13.
[34]
Sara Di Bartolomeo, Aditeya Pandey, Aristotelis Leventidis, David Saffo, Uzma Haque Syeda, Elin Carstensdottir, Magy Seif El-Nasr, Michelle A. Borkin, and Cody Dunne. 2020. Evaluating the effect of timeline shape on visualization task performance. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–12.
[35]
Catherine Plaisant, Richard Mushlin, Aaron Snyder, Jia Li, Dan Heller, and Ben Shneiderman. 2003. LifeLines: Using visualization to enhance navigation and analysis of patient records. In Proceedings of the Craft of Information Visualization. 308–312.
[36]
Katie Watson, Samuel S. Sohn, Sasha Schriber, Markus Gross, Carlos Manuel Muniz, and Mubbasir Kapadia. 2019. StoryPrint: An interactive visualization of stories. In Proceedings of the International Conference on Intelligent User Interfaces. 303–311.
[37]
Benjamin Bach, Conglei Shi, Nicolas Heulot, Tara Madhyastha, Tom Grabowski, and Pierre Dragicevic. 2015. Time curves: Folding time to visualize patterns of temporal evolution in data. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2015), 559–568.
[38]
Patrick Riehmann, Manfred Hanfler, and Bernd Froehlich. 2005. Interactive Sankey diagrams. In Proceedings of the IEEE Symposium on Information Visualization. 233–240.
[39]
Krist Wongsuphasawat and David Gotz. 2012. Exploring flow, factors, and outcomes of temporal event sequences with the outflow visualization. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2659–2668.
[40]
Krist Wongsuphasawat, John Alexis Guerra Gómez, Catherine Plaisant, Taowei David Wang, Meirav Taieb-Maimon, and Ben Shneiderman. 2011. LifeFlow: Visualizing an overview of event sequences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1747–1756.
[41]
Megan Monroe, Rongjian Lan, Hanseung Lee, Catherine Plaisant, and Ben Shneiderman. 2013. Temporal event sequence simplification. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2227–2236.
[42]
Ji Soo Yi, Niklas Elmqvist, and Seungyoon Lee. 2010. TimeMatrix: Analyzing temporal social networks using interactive matrix-based visualizations. International Journal of Human–Computer Interaction 26, 11-12 (2010), 1031–1051.
[43]
Benjamin Bach, Emmanuel Pietriga, and Jean-Daniel Fekete. 2014. Visualizing dynamic networks with matrix cubes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 877–886.
[44]
Jian Zhao, Zhicheng Liu, Mira Dontcheva, Aaron Hertzmann, and Alan Wilson. 2015. MatrixWave: Visual comparison of event sequence data. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 259–268.
[45]
Adam Perer and Fei Wang. 2014. Frequence: Interactive mining and visualization of temporal frequent event sequences. In Proceedings of the International Conference on Intelligent User Interfaces. 153–162.
[46]
Zhicheng Liu, Bernard Kerr, Mira Dontcheva, Justin Grover, Matthew Hoffman, and Alan Wilson. 2017. CoreFlow: Extracting and visualizing branching patterns from event sequences. Computer Graphics Forum 36, 3 (2017), 527–538.
[47]
Shunan Guo, Zhuochen Jin, David Gotz, Fan Du, Hongyuan Zha, and Nan Cao. 2018. Visual progression analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 417–426.
[48]
Tom Baumgartl, Markus Petzold, Marcel Wunderlich, M. Hohn, Daniel Archambault, M. Lieser, A. Dalpke, Simone Scheithauer, Michael Marschollek, and V. M. Eichel. 2020. In search of patient zero: Visual analytics of pathogen transmission pathways in hospitals. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 711–721.
[49]
Emanuel Zgraggen, Steven M. Drucker, Danyel Fisher, and Robert DeLine. 2015. Visual regular expressions for querying and exploring event sequences. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 2683–2692.
[50]
Bram C. M. Cappers and Jarke J. van Wijk. 2017. Exploring multivariate event sequences using rules, aggregations, and selections. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 532–541.
[51]
Jessica Magallanes, Tony Stone, Paul D. Morris, Suzanne Mason, Steven Wood, and Maria-Cruz Villa-Uriol. 2021. Sequen-C: A multilevel overview of temporal event sequences. IEEE Transactions on Visualization and Computer Graphics (2021).
[52]
Vanessa Peña-Araya, Tong Xue, Emmanuel Pietriga, Laurent Amsaleg, and Anastasia Bezerianos. 2022. HyperStorylines: Interactively untangling dynamic hypergraphs. Information Visualization 21, 1 (2022), 38–62.
[53]
Siming Chen, Jie Li, Gennady Andrienko, Natalia Andrienko, Yun Wang, Phong H. Nguyen, and Cagatay Turkay. 2018. Supporting story synthesis: Bridging the gap between visual analytics and storytelling. IEEE Transactions on Visualization and Computer Graphics 26, 7 (2018), 2499–2516.
[54]
Jiang Wu, Dongyu Liu, Ziyang Guo, Qingyang Xu, and Yingcai Wu. 2021. TacticFlow: Visual analytics of ever-changing tactics in racket sports. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 835–845.
[55]
Weiwei Cui, Shixia Liu, Zhuofeng Wu, and Hao Wei. 2014. How hierarchical topics evolve in large text corpora. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 2281–2290.
[56]
Samah Gad, Waqas Javed, Sohaib Ghani, Niklas Elmqvist, Tom Ewing, Keith N. Hampton, and Naren Ramakrishnan. 2015. ThemeDelta: Dynamic segmentations over temporal topic models. IEEE Transactions on Visualization and Computer Graphics 21, 5 (2015), 672–685.
[57]
Corinna Vehlow, Fabian Beck, Patrick Auwärter, and Daniel Weiskopf. 2015. Visualizing the evolution of communities in dynamic graphs. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 277–288.
[58]
Shunan Guo, Ke Xu, Rongwen Zhao, David Gotz, Hongyuan Zha, and Nan Cao. 2017. EventThread: Visual summarization and stage analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 56–65.
[59]
Sara Di Bartolomeo, Yixuan Zhang, Fangfang Sheng, and Cody Dunne. 2020. Sequence braiding: Visual overviews of temporal event sequences and attributes. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1353–1363.
[60]
Eva Mayr and Florian Windhager. 2018. Once upon a spacetime: Visual storytelling in cognitive and geotemporal information spaces. ISPRS International Journal of Geo-Information 7, 3 (2018), 96.
[61]
Renaud Blanch, Rémy Dautriche, and Gilles Bisson. 2015. Dendrogramix: A hybrid tree-matrix visualization technique to support interactive exploration of dendrograms. In 2015 IEEE Pacific Visualization Symposium (PacificVis’15). IEEE, 31–38.
[62]
Zhicheng Liu, Bernard Kerr, Mira Dontcheva, Justin Grover, Matthew Hoffman, and Alan Wilson. 2017. CoreFlow: Extracting and visualizing branching patterns from event sequences. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 527–538.
[63]
Johannes Hoffart, Fabian M. Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia. Artificial Intelligence 194 (2013), 28–61.
[64]
Dustin Arendt and Meg Pirrung. 2017. The “y” of it matters, even for storyline visualization. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology. 81–91.
[65]
Han He and Jinho D. Choi. 2021. The stem cell hypothesis: Dilemma behind multi-task learning with transformer encoders. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 5555–5577.
[66]
Longhua Qian, Guodong Zhou, Fang Kong, Qiaoming Zhu, and Peide Qian. 2008. Exploiting constituent dependencies for tree kernel-based semantic relation extraction. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling’08). 697–704.
[67]
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 402–412.
[68]
Irina Kostitsyna, Martin Nöllenburg, Valentin Polishchuk, André Schulz, and Darren Strash. 2015. On minimizing crossings in storyline visualizations. In International Symposium on Graph Drawing. Springer, 192–198.
[69]
Thomas C. van Dijk, Fabian Lipp, Peter Markfelder, and Alexander Wolff. 2017. Computing storyline visualizations with few block crossings. In Proceedings of the International Symposium on Graph Drawing and Network Visualization. 365–378.
[70]
Emilio Di Giacomo, Walter Didimo, Giuseppe Liotta, Fabrizio Montecchiani, and Alessandra Tappini. 2020. Storyline visualizations with ubiquitous actors. In International Symposium on Graph Drawing and Network Visualization. Springer, 324–332.
[71]
LLC Gurobi Optimization. 2022. Gurobi. https://www.gurobi.com/
[72]
Ali Yadollahi, Ameneh Gholipour Shahraki, and Osmar R. Zaiane. 2017. Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–33.
[73]
Kurt Koffka. 2013. Principles of Gestalt Psychology. Routledge.
[74]
Roy R. Behrens. 1998. Art, design and gestalt theory. Leonardo 31, 4 (1998), 299–303.
[75]
Matthew Brehmer and Tamara Munzner. 2013. A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2376–2385.
[76]
Cynthia Brewer. 2022. ColorBrewer. https://colorbrewer2.org/
[77]
Martin Greilich, Michael Burch, and Stephan Diehl. 2009. Visualizing the evolution of compound digraphs with TimeArcTrees. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 975–982.
[78]
Fabian Beck, Michael Burch, Stephan Diehl, and Daniel Weiskopf. 2017. A taxonomy and survey of dynamic graph visualization. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 133–159.
Index Terms
- E2Storyline: Visualizing the Relationship with Triplet Entities and Event Discovery
Recommendations
Visualizing Business Data with Generalized Treemaps
Business data is often presented using simple business graphics.These familiar visualizations are effective for providing overviews,but fall short for the presentation of large amounts of detailedinformation. Treemaps can provide such detail, but are ...
Visualizing web search results using glyphs: Design and evaluation of a flower metaphor
While the Web provides a lot of useful information to managers and decision makers in organizations for decision support, it requires a lot of time and cognitive effort for users to sift through a search result list returned by search engines to find ...
Visualizing Software Structures through Enhanced Interactive Sunburst Layout
AVI '16: Proceedings of the International Working Conference on Advanced Visual InterfacesVisualizing large software system structure in compact representations would help software architects and analysts in understanding the overall software structure accurately and efficiently. Space-filling techniques (e.g., Sunburst or Tree-map) are ...
Comments
Information & Contributors
Information
Published In
February 2024
533 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 January 2024
Online AM: 23 November 2023
Accepted: 04 November 2023
Revised: 18 October 2023
Received: 13 December 2022
Published in TIST Volume 15, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Key Research and Development Program of China
- Zhejiang Provincial Natural Science Foundation of China
- National Natural Science Foundation of China
- Fundamental Research Funds for the Provincial Universities of Zhejiang
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 247Total Downloads
- Downloads (Last 12 months)247
- Downloads (Last 6 weeks)42
Other Metrics
Citations
Cited By
View all- Liang PYe DZhu ZWang YXia WLiang RSun G(2024)C5: toward better conversation comprehension and contextual continuity for ChatGPTJournal of Visualization10.1007/s12650-024-00980-427:4(713-730)Online publication date: 5-Apr-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in