HAIChart: Human and AI Paired Visualization System

Abstract.

The growing importance of data visualization in business intelligence and data science emphasizes the need for tools that can efficiently generate meaningful visualizations from large datasets. Existing tools fall into two main categories: human-powered tools (e.g., Tableau and PowerBI), which require intensive expert involvement, and AI-powered automated tools (e.g., Draco and Table2Charts), which often fall short of guessing specific user needs.

In this paper, we aim to achieve the best of both worlds. Our key idea is to initially auto-generate a set of high-quality visualizations to minimize manual effort, then refine this process iteratively with user feedback to more closely align with their needs. To this end, we present HAIChart, a reinforcement learning-based framework designed to iteratively recommend good visualizations for a given dataset by incorporating user feedback. Specifically, we propose a Monte Carlo Graph Search-based visualization generation algorithm paired with a composite reward function to efficiently explore the visualization space and automatically generate good visualizations. We devise a visualization hints mechanism to actively incorporate user feedback, thus progressively refining the visualization generation module. We further prove that the top-$k$ visualization hints selection problem is NP-hard and design an efficient algorithm. We conduct both quantitative evaluations and user studies, showing that HAIChart significantly outperforms state-of-the-art human-powered tools ($\mathbf{21\%}$ better at Recall and $\mathbf{1.8}\times$ faster) and AI-powered automatic tools ($\mathbf{25.1\%}$ and $\mathbf{14.9\%}$ better in terms of Hit@3 and R10@30, respectively).

PVLDB Artifact Availability:
The source code, data, and/or other artifacts have been made available at https://github.com/xypkent/HAIChart.

1. Introduction

Data visualization is an effective means to uncover underlying insights (Ward et al., 2010; Qin et al., 2020; Shen et al., 2023; Ye et al., 2024). With the increasing importance of data visualization, how to help users effectively and easily create visualizations from massive datasets has attracted extensive attention from academia (Wu et al., 2021; Lee et al., 2021; Shen et al., 2022; Luo et al., 2020b, a) and industry (e.g., Tableau (tab, 2024)). Existing visualization tools can be broadly categorized into two groups based on the user effort required for creating visualizations, i.e., human-powered and AI-powered visualization tools.

Human-powered Visualization tools, including Voyager2 (Wongsuphasawat et al., 2017), DeVIL (Wu et al., 2017), and Lary (Satyanarayan and Heer, 2014), enable users to create their desired visualizations by manually specifying the data attributes, transformations, and visual encodings, as shown in Figure 1(a). This process is time-consuming and error-prone because it requires a deep understanding of the datasets and analytical tasks. Users often need to attempt multiple iterations to achieve the final visualization. Consequently, interactive visualization demands domain- and data-specific expertise, which can significantly hinder novice users from effectively engaging in visual analysis.

In response to the above challenges, recent research tries to automate the visualization process empowered by machine algorithms.

AI-powered Automatic Visualization aims to enumerate and recommend the (top-$k$) best visualizations for a given dataset based on predefined constraints (Luo et al., 2018a; Li et al., 2021; Deng et al., 2022; Luo et al., 2022b; Moritz et al., 2019) or learning-based recommendation algorithms (Qian et al., 2022; Zhou et al., 2021). For example, DeepEye (Luo et al., 2018a) recommends visualizations based on visualization rules and a ranking model, while Table2Charts (Zhou et al., 2021) generates top-$k$ visualizations using a table-to-sequence generation model. Although these tools provide a valuable starting point for data analysis by suggesting “good” visualizations, they risk misleading users with indiscriminate recommendations. The key issue is that these tools typically rely on static logical rules or deep learning models without adequately capturing user intent or feedback, hindering the tailoring of recommended visualizations to better suit users’ needs.

Figure 1. Three cases of visualization approaches

Table 1. Comparison of HAIChart with existing works (Dataset: $D$, User Operations: $U$, Visualizations: $V$)

Types	Systems	Input	Output	Multi-round?	Learned?	Recommendation Perspectives
						Domain Knowl.	User Preferences	Data Features
Human-powered	Voyager2 (Wongsuphasawat et al., 2017)	$D$, $U$	$V$	✗	✗	✓	✗	✗
	DeVIL (Wu et al., 2017)	$D$, $U$	$V$	✗	✗	✓	✓	✗
	Lary (Satyanarayan and Heer, 2014)	$D$, $U$	$V$	✗	✗	✓	✓	✗
AI-powered	DeepEye (Luo et al., 2018a)	$D$	$V$	✗	✓	✓	✗	✓
	VizML (Hu et al., 2019)	$D$	$V$	✗	✓	✗	✗	✓
	Draco-Learn (Moritz et al., 2019)	$D$	$V$	✗	✓	✓	✓	✓
	Data2Vis (Dibia and Demiralp, 2019)	$D$	$V$	✗	✓	✗	✗	✓
	KG4VIS (Li et al., 2021)	$D$, Knowl. Graph	$V$	✗	✓	✓	✗	✗
	Table2Charts (Zhou et al., 2021)	$D$	$V$	✗	✓	✗	✗	✓
	VizGRank (Gao et al., 2021)	$D$	$V$	✗	✗	✓	✗	✓
	PVisRec (Qian et al., 2022)	$D$	$V$	✗	✓	✗	✓	✓
	LLM4Vis (Wang et al., 2023)	$D$	$V$	✗	✓	✗	✗	✓
	PI2 (Chen and Wu, 2022)	$D$, SQL Queries	$V$, Interface	✓	✗	✓	✓	✗
Human-AI Paired	HAIChart	$D$, User Feedback	$V$, Hints	✓	✓	✓	✓	✓

Human and AI Paired Visualization. To mitigate the limitations of the above methods and strike a balance between human-powered and AI-powered visualization, we present Human and AI Paired Visualization, as shown in Figure 1(c). Our key idea is to initially recommend a set of high-quality visualizations, and then continue to refine the recommended visualizations by providing users with visualization hints for further tuning. This approach not only simplifies the process for novice users seeking to discover effective visualizations but also empowers users to actively participate in and guide the AI-powered visualization process through the human-centered feedback, as shown in Table 1.

Challenges. There are three challenges in our problem. (C1) How can we effectively and efficiently explore the visualization search space to recommend visualizations that are both high-quality and relevant? (C2) How can we evaluate the “goodness” of a generated visualization comprehensively? (C3) How can we understand and effectively integrate user feedback to guide the system toward visualizations that align with users’ requirements?

Our Proposal: HAIChart. To address these challenges, we propose HAIChart to automatically and progressively recommend a series of visualizations that adapt to evolving user preferences over time. This process is characterized by the system’s ability to learn from user interactions and feedback, iteratively refining the recommended visualizations. The objective is to converge towards a final set of visualizations that closely align with the user’s preferences, within the constraints of the available data and visualization search space. This iterative optimization is essentially a series of decisions based on user intent, similar to the key idea in reinforcement learning.

We adopt the reinforcement learning framework to implement HAIChart. We treat a visualization as a sequence of operations, such as chart types and $x$/$y$-axes configurations. Thus, the “state” is the current sequence of incomplete visualization, the “action” refers to selecting the subsequent visualization operation within that state, and the “environment” is the visualization system itself. The environment evaluates the quality of the visualizations and provides rewards designed to encourage the agent to improve the visualizations it generates. An agent is tasked with the generation and recommendation of visualizations.

We utilize the Monte Carlo Graph Search (MCGS) algorithm to build this agent, enabling it to efficiently navigate the extensive search space to discover optimal visualizations (addressing C1). To more accurately evaluate the visualizations, we design a composite reward function that takes into account the data features, visualization domain knowledge, and user preferences to ensure the quality of the visualizations (addressing C2). Moreover, we devise a visualization hints selection algorithm for recommending useful hints (e.g., “explore why flights are delayed”) to assist the user in data exploration. This approach can integrate user feedback for better visualization results (addressing C3).

Contributions. We make the following notable contributions.

(1) Problem Statement. We formally define the problem of human and AI paired visualization (Section 2).

(2) HAIChart. We propose a reinforcement learning-based system, HAIChart, to integrate human insights and AI capabilities for better visualizations. HAIChart initially generates a set of promising visualizations for users and iteratively improves the alignment of recommended visualizations with user preferences by selecting a set of visualization hints for interaction (Sections 3 & 4).

(3) Learning-to-Rate Visualization. We design a composite reward function to rate the quality of visualizations, which is used to accurately guide the visualization generation process (Section 5).

(4) Visualization Hints Selection. We design visualization hints as the proxy to collect user feedback, prove the selection of the top-$k$ visualization hints is an NP-hard problem, and design an efficient algorithm to select the top-$k$ hints (Section 6).

(5) Experiments. We conduct both quantitative evaluations and user studies to demonstrate that HAIChart outperforms human-powered tools in terms of both accuracy ($21\%$ better at Recall) and efficiency in creating visualizations ($1.8\times$ faster) and is better than AI-powered systems in effectiveness by 25.1% and 14.9% better in terms of Hit@3 and R10@30, respectively (Section 7).

2. Problem and Solution Overview

Figure 2. An Overview of HAIChart

2.1. Problem

Given a relational table $D$, let $\mathcal{V}$ be all valid visualizations, and $\mathbf{V}^{+}$ be a set of “good” visualizations, both being associated with $D$. In practice, this set of visualizations $\mathbf{V}^{+}$ is often manually selected by the users, such as the visualizations in a dashboard.

Visualization Selection. The problem of visualization selection is to select a set $\mathbf{V}$ of visualizations from all valid visualizations $\mathcal{V}$, which maximizes $\mathbf{V}\cap\mathbf{V}^{+}$ (i.e., maximizes the number of good visualizations being selected) and minimizes $\mathbf{V}\setminus\mathbf{V}^{+}$ (i.e., minimize the number of bad visualizations being selected).

When being operated manually, visualization selection is time-consuming, and its effectiveness depends on the user’s expertise in data visualization and domain knowledge of specific datasets.

To reduce human effort, visualization recommendation algorithms have been studied. These algorithms are mostly powered by machine learning methods. However, these recommendation algorithms are mainly one round.

One-Round Visualization Recommendation. The problem of one-round visualization recommendation is to automatically suggest a set $\mathbf{V}^{S}$ of visualizations and asks the user to select a subset $\mathbf{V}\subseteq\mathbf{V}^{S}$.

In practice, a one-round visualization recommendation may not be sufficient to support the user’s diversified intent, e.g., selecting all charts for a dashboard. Therefore, we propose to study a new problem, which does multiple rounds of interactions with users to collectively select all required good visualizations.

Multi-Round Visualization Recommendation. The problem of multi-round visualization recommendation is to recommend $n$ rounds of visualizations to users. In each round, we can suggest $\mathbf{V}_{i}^{S}$ visualizations (optionally with some natural language hints), and the user can select a subset $\mathbf{V}_{i}\subseteq\mathbf{V}_{i}^{S}$ visualizations in the $i$-th round. After $n$ rounds, the user will select $\mathbf{V}=\bigcup_{i=1}^{n}\mathbf{V}_{i}$ visualizations.

2.2. Solution Overview

HAIChart Overview. As shown in Figure 2, HAIChart consists of two primary components: the offline part is learning-to-rate visualization and the online part for automatically recommending good visualizations and visualization hints for user interaction.

Offline: Learning-to-Rate Visualization. As shown in Figure 2(a), the offline part is responsible for understanding and evaluating visualization quality. To achieve this, we incorporate visualization rule of thumb, data features, and user preferences to learn a composite reward function.

Online: Multi-Round Visualizations and Hints Recommendation. As shown in Figure 2(b), HAIChart leverages an agent powered by Monte Carlo Graph Search to recommend a set of promising visualizations along with visualization hints to guide further exploratory actions. Specifically, this agent initially traverses the visualization search space effectively and efficiently, generates possible visualizations, estimates their quality based on the well-trained composite reward function, and finally returns a ranked list of high-quality visualizations. In addition, the system computes visualization hints, derived from the current states of the Monte Carlo Graph. The user can browse the recommended visualizations and hints. Once the user selects a hint, the system will further explore the visualization space and recommend visualizations guided by the selected hint.

3. Reinforcement Learning for Multi- Round Visualization Recommendation

3.1. Preliminary

Inspired by previous works (Moritz et al., 2019; Luo et al., 2018a, 2022b), we adopt a visualization query to represent all possible visualizations discussed in this paper.

Visualization Query. Given a relational table $D$ with the scheme $R(A_{1},...,A_{m})$, a visualization query $Q$ is composed of a sequence of visualization operations (i.e., actions) $\mathcal{A}(\textsf{mark}{,}~{}\textsf{encoding}{,}~{}\textsf{transformation}{)}$. Specifically:

• mark represents the type of visualization charts including bar, line, pie, and scatter.

• encoding denotes the encoding of data attributes including $x$/$y$-axes, aggregate functions, and color schemes.

• transformation defines data transformations: filtering, binning, grouping, sorting, and top-$k$ operations.

Applying the visualization query $Q$ to the table $D$ results in the corresponding visualization $V$, i.e., $V=Q(D)$. Therefore, any visualization $V$ for a table $D$ can be represented by a combination of data attributes and cell values in $D$, along with visualization operations (i.e., actions) in $\mathcal{A}$.

Figure 3. A visualization query

Example 3.1 (Visualization Query).

Figure 3 shows an illustrative query ($Q$) applied to the flight delays dataset ($D$). The query visualizes a bar chart with the $x$-axis encoding cities (i.e.,  City column) and the $y$-axis showing the average delay (i.e., AVG(Delay)). The bar chart ($Q(D)$) shows the trend in average delays, showing that LA (i.e., Los Angeles) has the shortest average delays, while MSP (i.e., Minneapolis) has the longest average delays.

Undoubtedly, the search space for visualizations is very large, growing exponentially with the increase in number of columns and combinations of visualization operations.

To effectively navigate this huge search space, we introduce the concept of visualization query graph to represent all possible visualizations for a given dataset, defined as follows:

Definition 3.2 (Visualization Query Graph).

Given a table $D$, the visualization query graph $\mathcal{G(V,E)}$ is defined as a directed acyclic graph. Specifically, each node $v\in\mathcal{V}$ represents a visualization operation, and each directed edge $e\in\mathcal{E}$ indicates a transition from node $v_{i}$ to $v_{j}$. The weight of an edge $(v_{i},v_{j})$, denoted by $w_{ij}$, indicates the effectiveness of transitioning from operation $v_{i}$ to $v_{j}$. A path from the root node to an end node within this graph represents a sequence of visualization operations, which together form a candidate visualization query.

Example 3.3 (Visualization Query Graph).

As illustrated in Figure 4, each layer of the graph corresponds to the possible values that a visualization action can take. For example, the chart type (mark) could be bar, line, scatter, pie, and etc. Taking the blue path as an example, it represents a visualization query – “mark Bar encoding x City y AVG(Delay) transform group City”. After executing this query over the dataset, it will create a bar chart.

Figure 4. A visualization query graph

3.2. HAIChart Details

Given a dataset for visualization, users typically select visualization types, data columns, and transformation operations based on their preliminary understanding. If the results do not meet their analysis requirements, they adjust their operations to optimize the output. This visualization process is a sequence of decision-making based on user intent, similar to Reinforcement Learning (RL), where feedback from the visualization system (i.e., environment) helps optimize strategies to achieve satisfactory outcomes.

Therefore, we model the problem of visualization generation and recommendation as a Markov Decision Process (MDP) (Bellman, 1957) and implement it using an RL framework.

The key RL components of HAIChart are as follows:

State. To apply the RL framework to our problem, it is crucial to accurately define the state, which serves as input to the agent and aids in making decisions during visualization generation. We transform the visualization generation problem into a process that starts from an initial state and proceeds through a series of visualization operation/action decisions to reach a target state. Each state corresponds to a visualization query consisting of multiple visualization clauses, as described in Section 3.1.

Specifically, visualization queries can be classified into two types:

(1) Partial Visualization Queries. These queries need to be further extended to form a complete query and reflect historical decision paths in Monte Carlo Graph. For example, an incomplete query might have specified the type of chart (e.g., bar chart) but not yet selected a specific data field for the $x$- or $y$-axis.

(2) Complete Visualization Queries. These queries do not require additional extensions and can have the visualization quality evaluated by our reward function. After that, the system updates the nodes and edges of the graph based on the evaluation results.

Action. An action is an operation that an agent can perform based on its current state, determining the next step in a visualization query sequence. As discussed in Section 3.1, the visualization query can be categorized into three main types of actions: mark, encoding, and transformation. Therefore, for a given dataset $D$, the available action space $\mathcal{A}$ is fixed.

Agent. The agent combines the Monte Carlo Graph Search (MCGS) algorithm and the Upper Confidence Bound (UCB) (Auer et al., 2002) algorithm to generate visualizations and make decisions. The MCGS performs well in navigating complex and uncertain search spaces, while the UCB algorithm effectively balances the trade-off between exploiting known optimal solutions and exploring new possibilities, thereby optimizing the agent’s decision-making process (see Section  4).

Reward. The reward function evaluates the quality of the visualizations created by the MCGS algorithm and updates the visualization query graph accordingly. This guides the agent to generate higher-quality visualizations. The reward function considers three key factors: data features, visualization domain knowledge, and user preferences to accurately assess visualization quality (see Section 5).

Environment. The environment in HAIChart first uses a pruning algorithm to efficiently generate valid visualization queries during the creation process. Second, it evaluates the quality of valid queries and returns a reward to guide the training process.

Figure 5. An example of MCGS-based visualization generation

Input: Dataset $D$;

Output: Top-$k$ Visualization Results $R=\{V_{1},...,V_{k}\}$;

1 $node\leftarrow$ Initialize($D$); $\mathcal{G}\leftarrow\{node\}$; $R\leftarrow\{\}$;

2for each iteration do

3       Set a visualization query $Q=\text{``''}$;

4       $node=root$;

5      while $Q$ is invalid do

6             if node is fully expanded then

7                   $node\leftarrow$ select($\mathcal{G}$, $node$);

8                   Add $node$ to $Q$;

9                   if $Q$ is valid then

10                         break;

11                  

12             else

13                   $node\leftarrow$ expand($\mathcal{G}$, $node$);

14                  

15                  $Q\leftarrow$ simulation($\mathcal{G}$, $node$);

16                  

17            

18      $score\leftarrow$ reward($Q$);

19       BackPropagation($score$, $Q$, $\mathcal{G}$);

20       $V\leftarrow$ $Q(D)$; // Corresponding visualization

21       add $V$ to $R$;

22

return $R$;

Algorithm 1 MCGS-based Visualization Generation

4. Visualization Generation

4.1. Monte Carlo Graph Search-based Visualization Generation

In Section 3.1, we introduced the visualization query graph, where each visualization query is represented by a path within the graph. All possible paths within the query graph constitute the search space for visualizations. We can use search algorithms to explore the graph and find high-quality visualizations. However, as the dataset size increases, the search space expands exponentially, making it challenging to efficiently find good visualization results.

Traditional graph search methods like Depth-First Search (DFS) and Breadth-First Search (BFS) perform well in structured search spaces but often fall short in large and complex spaces because they cannot dynamically adjust the search strategy based on the information already found. To address these limitations, we propose a Monte Carlo Graph Search-based algorithm to navigate the vast search space efficiently.

Key Idea. Our key idea is to dynamically accumulate and utilize shared information from nodes throughout the search process. Initially, shared information is limited, but as the search progresses, the accumulated information from explored nodes becomes increasingly significant, providing greater support for decision-making. This allows the algorithm to gradually reduce its reliance on random simulations, leveraging accumulated knowledge to guide the search and enhance efficiency (Leurent and Maillard, 2020).

We first overview our Monte Carlo Graph Search-based visualization generation algorithm. As shown in Algorithm 1, the algorithm begins by taking a dataset $D$ as input, aiming to generate the top-$k$ visualization results $R$. In the initial phase, the algorithm initializes a root node based on the dataset $D$, which serves as the starting node of the graph $\mathcal{G}$ (Line 1).

In each iteration, the algorithm starts by initializing a visualization query $Q$ (Line 3) and then starts exploring from the root node. During the node selection phase, each choice is added to the visualization query $Q$ (Line 7-8) until an unexpanded node is encountered, followed by an expansion of this node (Line 12). When the visualization query $Q$ is valid, the algorithm utilizes a reward function to calculate the score of the query (Line 14), then performs backpropagation to update the information on the graph $\mathcal{G}$ (Line 15). Finally, the query is transformed into the corresponding visualization results and stored in $R$ (Lines 16-17). After $k$ visualizations are selected, the algorithm returns this set $R$.

Next, we will detail the procedure of Algorithm 1 through Figure 5. The algorithm mainly includes the following four steps:

Selection. In the selection phase (Line 7 in Algo. 1), the algorithm starts from the root node and recursively selects the optimal child nodes until it reaches a node that has not yet been fully expanded. As shown in Figure 5, the blue nodes represent the nodes selected in this phase, while the green nodes represent the currently selected nodes that have not been fully expanded. To effectively utilize the feedback information provided by the reward function and balance exploration and exploitation, we adopt the Upper Confidence Bound (UCB) algorithm. The key idea of this algorithm is to select the child node with the highest confidence upper bound at each iteration. The specific selection strategy of the UCB algorithm can be represented by the following formula:

(1) $$UCB=\begin{matrix}\underbrace{\bar{X}_{i}}\\ \textit{exploitation}\end{matrix}+\begin{matrix}\underbrace{c\sqrt{{2\ln n}/{n_{i}}}}\\ \textit{exploration}\end{matrix}$$

where $\bar{X}_{i}$ is the average reward of child node $i$, $n$ is the number of visits to the current node, $n_{i}$ is the number of visits to child node $i$, and $c$ is a constant used to balance exploration and exploitation.

Expansion. During the expansion phase (Line 12 in Algo. 1), the algorithm selects the next valid action based on the current state (i.e., an incomplete visualization query). Specifically, it removes those low-quality visualizations that are syntactically incorrect or violate visualization rules (see Section 4.2). For example, when processing node $v_{2}$ (in Figure 5), the algorithm identifies two high-benefit candidates and randomly selects one for further expansion.

Simulation. During the simulation phase (Line 13 in Algo. 1), the algorithm starts at the current node (e.g., node $v_{3}$ shown in Figure 5) and performs simulation actions. First, it checks whether the current visualization query is valid. If the query is invalid, the algorithm randomly explores the next node according to the visualization grammar rule until a valid query is constructed. Once a valid query is formed, the learning-to-rate visualization part assigns a reward.

Backpropagation. This phase (Line 15 in Algo. 1) propagates the simulation results through the visualization query graph. As shown in Figure 5, after the simulation phase, the score obtained by node $v_{3}$ is propagated along its path. Each node and edge along the path updates its reward value and visit count based on the simulation outcomes. Our learning-to-rate visualization module evaluates the visualization quality and updates the node information along the search path, guiding the algorithm to make more precise decisions in subsequent explorations by feeding the results of each simulation back into the visualization query graph.

Terminal Condition. The above steps are repeated until the maximum number of iterations is reached.

Figure 6. Comparison of tree and graph structures

Comparison of MCGS and MCTS. The Monte Carlo method solves complex problems through random sampling but requires simulation steps for each new node, impacting performance based on node count (Robert et al., 1999). Traditional MCTS methods rely on extensive, random, and time-consuming simulations to determine the next step (Shi et al., 2021). Unlike traditional MCTS, our MCGS algorithm explores a graph structure instead of a tree structure, allowing for more effective information sharing between nodes and reducing the number of nodes, thereby increasing search efficiency (Tot et al., 2022).

As shown in Figure 6, visualization queries $Q_{1}$ and $Q_{2}$ differ only in chart type; other clauses are identical. In a tree structure (Figure 6(b)), this information cannot be shared due to the strict parent-child hierarchy, leading to redundancy. In contrast, the graph structure (Figure 6(a)) allows node information sharing across different queries, reducing redundancy and improving search efficiency.

More concretely, we compare the number of nodes between MCGS and MCTS. Consider 4 chart types, a dataset with $m$ columns, encoding that considers both the $x$- and $y$-axes, and data transformations that include grouping and 4 aggregate operations. In MCTS, each path is independent, resulting in a total node count of $N_{\text{tree}}=4\times m^{3}\times 4=16m^{3}$. In MCGS, using the graph structure to share nodes, the total node count is $N_{\text{graph}}=4+3\times m+4=3m+8$. The reduction factor is $R=\frac{N_{\text{tree}}}{N_{\text{graph}}}=\frac{16m^{3}}{3m+8}$. As $m$ increases, the benefit of using MCGS becomes more significant, reducing the number of nodes compared to MCTS.

In summary, by sharing nodes and reducing redundant calculations through a graph structure, MCGS significantly reduces the number of nodes and simulations, enabling more efficient search (Czech et al., 2021; Shen et al., 2018; Saffidine et al., 2012).

4.2. Optimization Techniques

Rule-based Pruning. In visualization generation process, selecting and expanding nodes are crucial. However, the traditional UCB algorithm may lead to inappropriate node selections due to the lack of visualization-specific knowledge. To alleviate these issues, we introduce a rule-based pruning algorithm.

The key aspect of this algorithm is integrating domain knowledge, such as data transformation rules and visualization rules, to ensure the generated visualizations are syntactically correct. For example, after completing a GROUP BY operation, if an aggregate function needs to be chosen for the $y$-axis, and the $y$-axis field is categorical, then functions such as SUM, AVG, and NONE become inapplicable. Our algorithm aims to exclude these inappropriate options to enhance the efficiency and accuracy of the search.

Furthermore, We present a function $L(S,A)$ that takes the current state $S$ and the candidate operation set $A$ as inputs, and outputs a set of operations meeting specific constraints, formalizing the pruning process based on domain-specific knowledge for visualizations.

Adaptive Random Exploration Strategy. During the visualization generation process, each selected visualization operation (e.g., chart type) significantly affects the final visualization. Although the UCB algorithm is designed to make the best choice based on the current state, it may tend to over-explore high-scoring branches as the number of simulations accumulates, overlooking other possibilities. Moreover, the UCB algorithm might favor sub-optimal or inefficient visualization actions, especially when the initial attempts are inaccurately evaluated by the reward function (Silver et al., 2017; Czech et al., 2021).

To address these issues, we propose an adaptive random exploration strategy. This strategy adjusts the random selection probability based on the number of already selected clauses, facilitating the transition from exploitation to exploration. This approach increases search space coverage and diversity while avoiding local optima. It mirrors user behavior in constructing visualization queries, with extensive exploration at the initial stage and more targeted selection based on earlier trials at later stages.

The optimized formula for the selection strategy is as follows:

(2) $$a_{i}=\begin{cases}random(L(S_{i},A_{i})),&P_{\text{exploration}}\\ \arg\max_{b\in L(S_{i},A_{i})}\left[Q_{i}(b)+c\sqrt{\frac{2\ln t}{N_{i}(b)}}\right],&P_{\text{exploitation}}\end{cases}$$

In our proposed decision-making strategy, the final choice $a_{i}$ for each node $i$ is based on a series of parameters and functions. The set of legal clauses available for node $i$ is represented by $L(S_{i},A_{i})$. The average reward for each clause $b$ is estimated by $Q_{i}(b)$, while $N_{i}(b)$ tracks the number of times clause $b$ has been visited. The total number of visits to the current node is denoted by $t$, and the constant $c$ is used to balance exploration in the process.

Furthermore, the balance between exploration and exploitation in the strategy is controlled by two main probability parameters: $P_{\text{exploration}}$ and $P_{\text{exploitation}}$. The exploration probability, $P_{\text{exploration}}$, is defined as $p_{n}\alpha^{n}$, where $p_{n}$ is the initial probability of making a random choice and $\alpha$ is a constant less than 1 that controls the decay rate of the random selection probability. On the other hand, the exploitation probability, $P_{\text{exploitation}}$, is defined as $1-p_{n}\alpha^{n}$. As the number of clauses $n$ selected in node $i$ increases, the probability of random selection gradually decreases, while the probability of selection based on the UCB algorithm correspondingly increases.

5. Learning-to-Rate Visualization

Unlike easily simulated environments for games like Atari (Schrittwieser et al., 2020) or Go (Silver et al., 2016), where clear reward and punishment rules exist, visualization evaluation lacks such clarity and can be biased if based on a single criterion.

To address this, we propose a composite reward function that incorporates visualization best practices, data features, and user preferences to comprehensively evaluate visualization quality.

5.1. Composite Reward Function

The composite reward function incorporates rules of thumb for visualizations, data features, and user preferences to evaluate the quality of visualization results. The formula for the Composite Reward Function (CRF) is as follows:

(3) $$CRF=S_{k}\times(\beta S_{d}+(1-\beta)S_{u})$$

Where, $S_{k}$ represents the assessment result from visualization domain knowledge, $S_{d}$ indicates the score based on data features, and $S_{u}$ denotes the user preference score. When evaluating a visualization result, we first conduct a preliminary evaluation based on visualization domain knowledge. If the result aligns with the domain knowledge, $S_{k}$ is assigned a value of 1; otherwise, it is 0. Clearly, when $S_{k}=0$ (i.e., the visualization result does not align with domain knowledge), the reward value $CRF$ becomes 0. Conversely, when $S_{k}=1$, the calculation of $CRF$ depends on the weighted combination of $S_{d}$ and $S_{u}$, where the weight coefficient $\beta$ is introduced to balance the importance between data characteristic scoring and user preference scoring, set based on experience.

5.2. Learn Composite Reward Function

We leverage well-established, human-annotated visualization corpora (Hu et al., 2019; Qian et al., 2022) and domain knowledge from the visualization community to learn the composite reward function.

Leveraging Domain Knowledge. We employ a rule-based method to ensure that the generated visualizations align with best practices and accurately reflect data features. Specifically, this method leverages the data selection, transformation, and visualization rules used by DeepEye (Luo et al., 2018a).

For example, the rules for pie charts include: (1) Data selection rules: the $x$-axis should represent categorical data, while the $y$-axis should represent numerical data. (2) Data transformation rules: pie charts are unsuitable for data aggregated with the AVG operation, as they are primarily used to show proportions. (3) Visualization rules: the $y$-axis values should not include negatives, and there should be at least two distinct $x$-axis values to convey meaningful information. Using this approach, we design 15 rules for different types of visualizations and use these rules as strict constraints to filter out low-quality visualizations.

Capturing Data Features. Our model is trained on a dataset annotated by real users (Luo et al., 2018a), containing 285,236 visualizations and their scores from 42 different domains. For each visualization, we analyze key features, including the data types of the $x$- and $y$-axes, the number of rows, extremes (maximum and minimum values), value diversity (number of different values and the ratio of unique values), and the correlation between the data of the $x$- and $y$-axes and the type of chart. We extract 14 core features for model training.

We use LambdaMART (Wu et al., 2008) to train our scoring model, which maps complex relationships between features and scores by building decision trees. Each tree evaluates features, such as the type of data on the $x$-axis, then examines other features, and finally outputs a score. This method enables the model to understand the deep connections between feature combinations and scores. When a new visualization is input, the model computes its features and uses the trained LambdaMART to predict a score.

Learning Common User Preferences. Capturing user preferences for visualization is essential in visualization recommendation systems. Although it is ideal to consider individual preferences in online recommendations, this poses a challenge due to the cold start problem – new users often lack sufficient interaction history, especially in visualization scenarios. In addition, collecting real-time interaction data and fine-tuning models online for each user can be resource-intensive and time-consuming.

To address these challenges, our goal is to first learn common user preferences offline and then make online adjustments to the MCGS algorithm based on feedback from visualization hints, which we will discuss in Section 6. This approach strikes a balance between recommending high-quality visualizations and fine-tuning the recommendations to better suit individual users.

To achieve this, our system utilizes the Generative Adversarial Network (GAN) (Goodfellow et al., 2020), specifically the IRecGAN (Bai et al., 2019), to learn common user preferences using a real-world visualization corpus. This corpus includes user interaction logs collected from the Plotly community (Qian et al., 2022). The dataset comprises historical interaction records of 17,469 users, with an average of 5.41 datasets and 1.85 visualizations per user, and each dataset contains an average of 24.39 attributes. For effective learning of user preferences across datasets and visualizations, we extracted 606 visualization configurations from the original dataset, inspired by (Qian et al., 2022). Each configuration includes dataset-independent design choices such as chart types, colors, and sizes, enabling the model to learn preferences without direct dataset associations and to identify patterns from design choices. By aggregating data, user, and visualization features across different datasets, we generated 15,531 training samples to train the IRecGAN model (Bai et al., 2019).

In the first round of visualization recommendations, the evaluations obtained from the trained IRecGAN model serve as a crucial indicator of general user preferences. In the interaction phase (e.g., the second round), HAIChart refines and optimizes the recommendation strategy based on the visualization hints selected by users, as detailed in Section 6.

6. Visualization Hints Selection

While the MCGS-based visualization generation algorithm can recommend high-quality visualizations, it might not always align with individual preferences. Conversely, interactive tools like Voyager2 (Wongsuphasawat et al., 2017) offer flexible exploration options but can overwhelm users with their complexity. These tools require users to sift through dense control panels and numerous visualization options, much like tackling a challenging “fill-in-the-blank” puzzle.

To address these issues, we introduce a visualization hints module. Visualization hints represent the high-level visualization intent derived by our system, as shown in Figure 7. This module simplifies the complex decision-making process of visualization creation into straightforward, user-friendly selections, akin to “multiple-choice questions”. This method collects user intent, allowing our MCGS algorithm to be fine-tuned online to better suit individual users.

Definition 6.1 (Visualization Hint).

A visualization hint $h$ corresponds to a visualization operation or action, such as selecting data fields, applying aggregate operations, and choosing chart types. It is expressed in easily understandable natural language. Each hint $h$ is associated with a set of visualizations $\mathbb{V}=\{v_{1},v_{2},\ldots,v_{n}\}$,which are ranked based on the values of the composite reward function to prioritize high-value visualizations.

Figure 7. Visualization refinement with visualization hints

As shown in Figure 7, consider a data field-based hint: “Explore $\langle$data field name$\rangle$ over categories or time”, this hint essentially defines a partial visualization query relevant to the data field. Therefore, the user can browse these hints and select a desired hint for visualization. Next, HAIChart will generate a set of relevant visualizations based on the selected hint through the MCGS process. This approach not only simplifies the steps for the user to define a visualization query but also speeds up the process of gaining data insights by guiding the user through meaningful hints to quickly explore and understand the data.

Our goal is to select top-$k$ visualization hints that not only cover different aspects but also ensure high-quality visualizations, all while maintaining an appropriate number of visualizations.

Definition 6.2 (Top-$k$ Visualization Hints Selection).

Given a set of hints $\mathbb{H}=\{h_{1},h_{2},\ldots,h_{n}\}$, where each hint $h_{i}$ is associated with a set of visualizations $\mathbb{V}_{i}$, and each visualization $v\in\mathbb{V}_{i}$ has an associated reward value $r_{v}$, the goal is to select a subset $\mathbb{H^{\prime}}\subseteq\mathbb{H}$ consisting of $k$ hints. The selection must maximize the total reward value of the subset $\mathbb{H^{\prime}}$, while ensuring the total count of visualizations in $\mathbb{H^{\prime}}$ does not exceed a predefined budget $B$. The optimization problem can be formulated as follows:

(4) $$\text{Maximize}\quad F(\mathbb{H^{\prime}})=\sum_{h_{i}\in\mathbb{H^{\prime}}}\sum_{v\in\mathbb{V}_{i}}r_{v}$$

(5) $$\text{Subject to}\quad\sum_{h_{i}\in\mathbb{H^{\prime}}}|\mathbb{V}_{i}|\leq B\quad\text{and}\quad|\mathbb{H^{\prime}}|=k$$

Where $|\mathbb{V}_{i}|$ represents the number of visualizations associated with hint $h_{i}$, $F(\mathbb{H^{\prime}})$ is the total reward value of the selected subset $\mathbb{H^{\prime}}$, $|\mathbb{H^{\prime}}|$ is the number of selected hints, and $B$ is the upper limit on the budget for the number of visualizations.

However, the problem of selecting hints is NP-hard because it is equivalent to a known NP-hard problem – the Budgeted Maximum Coverage problem (Khuller et al., 1999). To address this, we propose an efficient algorithm to select the top-$k$ visualization hints. To achieve effective hint selection, the first step is to construct a comprehensive set of hints and evaluate the benefit of each hint. The purpose of this initial phase is to ensure that in the subsequent selection process, the best hint can be selected from this evaluated set, thereby maximizing the reward value. In this way, the algorithm mainly consists of two steps: candidate hints generation and top-$k$ hints selection.

Input: Set of hints $\mathbb{H}=\{h_{1},h_{2},\ldots,h_{n}\}$, $B$, $k$;

Output: Selected top-k set of hints $\mathbb{H}^{\prime}$;

1 $\mathbb{H}^{\prime}\leftarrow\emptyset$; $totalCost\leftarrow 0$;

2 $\mathbb{H}_{v}\leftarrow\{h\in\mathbb{H}\mid|h|\leq B\}$; // 1. Filter valid hints

3

4$\mathbb{H}_{v}\leftarrow\text{SortByScore}(\mathbb{H}_{v})$; // 2. Sort hints by score

5 // 3. Selection of top-k hints

6 for each $h_{i}$ in $\mathbb{H}_{v}$ do

7       if $|\mathbb{H}^{\prime}|<k$ and $totalCost+|h_{i}|\leq B$ then

8             $\mathbb{H}^{\prime}.\text{append}(h_{i})$;

9             $totalCost\leftarrow totalCost+|h_{i}|$;

10      if $|\mathbb{H}^{\prime}|=k$ then

11             break;

12      

return $\mathbb{H}^{\prime}$;

Algorithm 2 Top-$k$ Visualization Hints Selection

Candidate Visualization Hints Generation. We identify high-value nodes (i.e., visualization operations) from our visualization query graph to pick the candidate hints. When computing the reward for each hint, we introduce a decay coefficient $\delta$ to adjust the scoring weight of each visualization, considering that the same visualization appearing in different hints may reduce its uniqueness. The decay coefficient $\delta$, is defined as: $\delta=\log\left(\frac{N_{\text{total}}}{N_{\text{viz}}}\right)$, where $N_{\text{total}}$ is the total number of hints and $N_{\text{viz}}$ is the number of hints containing the specific visualization. This coefficient reflects the frequency of visualizations, with higher decay coefficients for more frequent visualizations, thereby reducing their overall scores. After multiplying this score by the reward value of the visualization, its final value is determined. Therefore, after collecting all the hints, we calculate the frequency of the appearance of each visualization and combine it with the decay coefficients to compute its score.

Top-$k$ Visualization Hints Selection. Our goal is to select the best subset of hints that maximizes the total reward within a pre-defined budget $B$. Algorithm 2 shows the pseudo-code. It first selects all hints with a cost not exceeding the budget $B$, forming a candidate hint set (Line 2). Then, it sorts valid hints based on the corresponding average visualization score computed by the composite reward function (Line 3). The algorithm continues to go through this sorted set, picking hints to add to the final selected set until one of two conditions is met: the number of selected hints reaches the pre-defined $k$, or adding more hints would cause the total cost to exceed the budget $B$ (Line 5-10). In this way, the algorithm prioritizes high-scoring hints while maintaining the budget constraint, thus maximizing the total reward value under the given budget.

User Feedback for Refinement. HAIChart leverages user-selected visualization hints to guide the node exploration strategy during the MCGS process. When users select a specific hint, they direct the search graph to expand in a certain direction. For example, as shown in Figure 7, if a user clicks on the hint “Explore delay over categories or time”, the system focuses on applying the data field “delay” to the $y$-axis. During the search process, the system freezes other nodes and explores only those related to “delay”, ensuring the generated results are relevant to target field. Thus, this method can align the search process with user preferences and effectively prunes the search space, enhancing the efficiency and accuracy of MCGS-based visualization recommendations.

7. Experiments

Table 2. Statistics of the experimental datasets (Vis.: Charts)

Datasets	#-Tables	#-Vis.	Avg(#-Vis.)	Avg(#-Rows)	Avg(#-Col.)	Max(#-Col.)
VizML	79,475	162,905	2	2,817.8	3.3	25
KaggleBench	8	252	31.5	32,585.9	9.1	15

Figure 8. Statistics of the VizML dataset

Table 3. Details in KaggleBench dataset

	Datasets	#-Rows	#-Columns	#-Vis.
D1	StudentPerformance	1,000	8	34
D2	AirplaneCrashes	5,191	6	22
D3	VideoGame	6,825	15	48
D4	GooglePlayStore	9,659	11	33
D5	AvocadosPrice	18,249	7	20
D6	SuicideRates	27,820	9	25
D7	Zomato	51,717	11	49
D8	GunViolence	140,226	6	21

7.1. Experiment Settings

Datasets. Table 2 shows two real-world datasets for experiments.

(1) VizML (Hu et al., 2019), derived from the Plotly community, features around 120,000 dataset-visualization pairs created by real users. This dataset was refined by removing entries missing table or chart data and cleaning up invalid characters. The dataset has four types of charts, namely bar, pie, line, and scatter visualizations, culminating in 79,475 valid dataset-visualization pairs. These were randomly divided into training, validation, and testing sets in a 7:1:2 ratio, allocating 55,632 pairs for training, 7,947 for validation, and 15,896 for testing. The statistical information of the dataset, shown in Figure 8, indicates a broad coverage, including various tables and visualizations used by users across different domains and tasks.

(2) KaggleBench (Gao et al., 2021) is a public benchmark designed to evaluate the effectiveness of visualization recommendation. Its data mainly come from numerous data competitions and the corresponding visualizations provided by the Kaggle platform. This dataset was refined by filtering out low-quality datasets, removing rows with missing data, and fixing invalid characters. Finally, we obtained 8 datasets for evaluation, as shown in Table 3.

We only use the VizML dataset to train or configure all methods. We used the above two datasets to test different methods.

Table 4. Effectiveness the first round of visualization recommendations

D	Tasks	Metrics	The State-of-the-Art Methods						Our Methods
			Data2Vis (Dibia and Demiralp, 2019)	VizGRank (Gao et al., 2021)	DeepEye (Luo et al., 2018a)	PVisRec (Qian et al., 2022)	VizML (Hu et al., 2019)	LLM4Vis (Wang et al., 2023)	MCTS	HAIChart-	HAIChart
VizML	Data Queries	Hit@1	47.5%	57.6%	52.4%	52.3%	-	-	78.3%	79.7%	79.3%
		Hit@3	51.3%	67.2%	67.6%	58.7%	-	-	88.2%	91.3%	91.9%
	Design Choices	Hit@1	41.7%	34.9%	34.1%	28.9%	28.7%	47.9%	42.4%	50.6%	48.7%
		Hit@3	43.7%	42.9%	40.7%	51.3%	-	-	77.1%	81.8%	81.5%
	Overall	Hit@1	24.3%	25.6%	25.7%	21.8%	-	-	33.1%	37.9%	36.9%
		Hit@3	26.9%	30.1%	33.9%	42.3%	-	-	64.7%	68.4%	67.4%
KaggleBench	Data Queries	P@10	41.2%	58.7%	62.5%	42.5%	-	-	52.2%	60.0%	63.8%
		R10@30	25.0%	50.0%	48.7%	67.5%	-	-	73.6%	80.1%	83.7%
	Design Choices	P@10	88.7%	87.5%	93.7%	91.9%	Hit@2:78.3%	Hit@2:87.6%	93.8%	96.3%	96.3%
		R10@30	95.0%	81.3%	95.0%	85.0%	-	-	92.5%	96.2%	96.2%
	Overall	P@10	28.7%	43.7%	48.7%	36.7%	-	-	45.4%	51.3%	55.0%
		R10@30	13.8%	41.3%	33.7%	60.0%	-	-	63.8%	72.5%	74.9%

Methods. We evaluate the following methods.

(I) AI-powered Visualization Methods:

(1) Data2Vis (Dibia and Demiralp, 2019) transforms datasets into visualization queries using a sequence-to-sequence model.

(2) VizGRank (Gao et al., 2021) achieves visualization recommendation by modeling the relationships between visualizations as graphs and using graph-based ranking algorithms.

(3) DeepEye (Luo et al., 2018a) combines data features with domain knowledge to recommendation top-$k$ good visualizations.

(4) PVisRec (Qian et al., 2022) recommends a set of visualizations by learning from user preferences.

(5) VizML (Hu et al., 2019) uses deep learning models for visualization recommendations. It focuses on five specific types of tasks, excluding data querying. Our evaluation will assess VizML’s effectiveness in recommending design choices.

(6) LLM4Vis (Wang et al., 2023) uses in-context learning to interact with ChatGPT for recommending visualizations. Like VizML, it excludes data querying tasks. Thus, our comparison focuses on design choices, evaluated using the same Hit@$k$ metric as LLM4Vis.

(II) Human-powered Visualization Methods:

(7) Voyager2 (Wongsuphasawat et al., 2017) is an interactive system that allows users to create and explore visualizations through click-based interactions.

(III) Human and AI paired Visualization Methods:

(8) HAIChart (ours) is the full implementation based on MCGS and the composite reward function, as described in this paper.

(9) HAIChart- (ours) differs from HAIChart in the composite reward function, where the capturing data features model is trained on the VizML corpus.

(10) LLM4Vis+ (ours) is an improved version of LLM4Vis (Wang et al., 2023). We enhanced LLM4Vis to support data queries and top-$k$ visualizations.

(11) MCTS-based Baseline (ours) follows the same implementation as HAIChart, except it uses the MCTS-based method for recommending visualizations.

Metrics. Following previous studies evaluating automatic visualization systems (Zhou et al., 2021; Dibia and Demiralp, 2019; Hu et al., 2019; Luo et al., 2023), we employ Hit@$k$, P@$k$, and Rt@$k$ as evaluation metrics. Given that creating a visualization involves data queries, design choices, and final integration, our evaluation is divided into three tasks. The metrics are defined as follows:

(1) Hit@$k$: This metric evaluates whether the ground truth appears in the top-$k$ results. We apply this to the VizML dataset with $k$ set to 3, as each user typically creates about 2 visualizations on average.

(2) P@$k$: This metric measures how many ground truths are in the top-$k$ results. For the KaggleBench dataset, we set $k$ to 10, considering that each dataset typically contains over 20 visualizations.

(3) R$t$@$k$: This metric assesses how many of the top-$t$ ground truths are covered in the top-$k$ results. We use R10@30 for the KaggleBench dataset, analyzing how many of the top 10 ground truths are included in the first 30 results returned by the model.

Experimental Environment. Experiments were performed on an Ubuntu 22.04 Server LTS with dual Intel Xeon 8383C CPUs, 512GB RAM, and eight NVIDIA RTX 4090 GPUs.

7.2. Experimental Results

7.2.1. Exp-1: Effectiveness of the First-round of Recommendations.

This experiment evaluates the effectiveness of the first-round visualization recommendations by HAIChart. We tested all methods on the VizML and KaggleBench datasets. For VizML, with an average of 2.05 visualizations per dataset, we used the Hit@$k$ metric. For KaggleBench, with an average of 31.5 visualizations per dataset, we used P@$k$ and Rt@$k$ metrics.

Table 4 shows the results, and we have the following findings.

(1) Overall, our methods (HAIChart and HAIChart-) significantly outperform all state-of-the-art methods across all metrics, showing the effectiveness of our framework. The performance gap between HAIChart- and HAIChart is primarily due to the capturing data features module: HAIChart- is trained on the VizML training set, while HAIChart is trained on a more diversified corpus (Luo et al., 2018a).

(2) On the VizML dataset, HAIChart achieves 36.9% Hit@1 and 67.4% Hit@3 on the overall task, surpassing the competitive methods DeepEye and PVisRec by 11.2% and 25.1%, respectively. On the KaggleBench dataset, HAIChart achieves 55% P@10 and 74.9% R10@30 on the overall task, outperforming DeepEye and PVisRec by 6.3% and 14.9%. These results demonstrate HAIChart’s effectiveness in leveraging MCGS algorithms with a composite reward function to find high-quality visualizations in a large search space.

(3) DeepEye performs well on the P@10 metric using a rule-based approach, while PVisRec excels on the Rt@30 metric with personalized recommendations. By blending visualization rules, user preferences, and data features, HAIChart achieves strong results in both the P@10 and Rt@30 metrics.

Table 5. Performance vs. #-Iterations on KaggleBench

Tasks	Metrics	DeepEye	Voyager2			LLM4Vis+ (ours)			HAIChart (ours)
		Iter. 1	Iter. 1	Iter. 2	Iter. 3	Iter. 1	Iter. 2	Iter. 3	Iter. 1	Iter. 2	Iter. 3
Data Queries	P@10	62.5%	45.0%	55.1%	58.0%	47.2%	65.0%	74.3%	63.8%	69.5%	79.2%
Design Choices	P@10	93.7%	78.7%	96.3%	97.4%	75.6%	91.5%	97.2%	96.3%	97.6%	99.3%
Overall	P@10	48.7%	40.0%	44.9%	45.7%	41.0%	55.6%	65.3%	55.0%	58.2%	68.8%

Table 6. Effectiveness of Hints Selection

Dataset	Metrics	Round 1	Round 2	Round 3
KaggleBench	Hit@1	64.7%	65.2%	69.1%
	Hit@3	79.4%	82.2%	85.7%
	Hit@5	88.2%	89.6%	92.1%

Figure 9. Qualitative analysis on user study

Figure 10. Quantitative analysis on user study

7.2.2. Exp-2: Effectiveness of Multi-round Recommendations.

The goals of this experiment are to: (1) verify if HAIChart enhances the effectiveness of its visualization recommendations after multiple rounds of user interaction, and (2) validate whether the visualization hints effectively assist users and improve their efficiency in data exploration. To this end, we design a user study with 17 participants, comparing HAIChart against state-of-the-art methods: DeepEye (Luo et al., 2018a), Voyager2 (Wongsuphasawat et al., 2017), and LLM4Vis+.

Participant. We invited 17 participants (7 females, 10 males, aged 21-33) to our study, comprising 12 experts and 5 non-experts. The experts included 6 Ph.D. candidates, 4 master’s students, and 2 undergraduates majoring in computer science, all with varying levels of data analysis experience. Most experts had used at least one data visualization tool (e.g., Tableau) for over a year. The non-experts, from non-technical backgrounds, had basic experience in data processing, primarily using Excel for simple tasks.

Task. Our study had participants use HAIChart, DeepEye, LLM4Vis+, and Voyager2 to engage with 8 KaggleBench datasets (Table 3). Each dataset included specific analysis tasks, such as examining student subject scores and correlating performance with background factors in the StudentsPerformance dataset. Additionally, participants conducted open-ended explorations to assess the tools’ adaptability and efficiency in unstructured tasks.

Procedure. (1) Preparation: Initially, we introduced participants to the study’s context and the datasets. We also showed them how to use HAIChart, DeepEye, LLM4Vis+, and Voyager2 through examples, allowing time for exploration. This step ensured they were familiar with both systems’ features, datasets, and tasks before starting the experiment. (2) Experimentation: During the experiment, participants were encouraged to share their thoughts and questions as they used the systems. We logged their interactions, tracking time spent, interactions count, selected hints or fields, and the visualizations generated during each session. After each round of interaction, HAIChart provided new visualization recommendations along with the top-$9$ hints to guide users to the next step of exploration. (3) User Feedback: After the experiment, participants evaluated HAIChart, LLM4Vis+, and Voyager2 based on ease of learning, ease of use, quality of results, helpfulness in exploration, and response speed. These criteria, chosen to assess user experience and system performance from multiple perspectives (Li et al., 2024), were rated on a five-point Likert scale (Allen and Seaman, 2007) from 1 (Strongly Disagree) to 5 (Strongly Agree). We also conducted interviews to gather participants’ feedback, encouraging comments on any aspect of the tools used, and recorded these comments for analysis.

User Study Result. We conduct quantitative and qualitative analyses.

(1) As shown in Table 5, HAIChart improves visualization recommendations through user feedback. After two rounds of interaction, P@10 increased from 55% to 68.8%, outperforming LLM4Vis+ and Voyager2 by 3.5% and 23.1%, respectively. HAIChart’s visualization hints guide users efficiently, unlike Voyager2’s manual exploration. LLM4Vis+ requires time-consuming natural language queries and domain knowledge. DeepEye, which achieved 48.7% in the first round, lacks multi-turn recommendation support and does not improve with additional interactions.

(2) In addition, as shown in Figure 10, HAIChart outperforms LLM4Vis+ and Voyager2 with fewer interactions and less time, being 1.8 and 2.2 times faster, respectively. The average recall is 83.7%, which is 21% and 24.8% higher than LLM4Vis+ and Voyager2. This indicates that HAIChart meets user analysis needs with lower interaction costs. Finally, user feedback (Figure 10) indicates that HAIChart is easy to learn and use, helping users explore and achieve high-quality visualization results more effectively.

(3) Conversations with participants provided insightful feedback on HAIChart’s practicality in data exploration. Participants widely acknowledged that HAIChart simplifies the data exploration task and enhances efficiency by providing helpful hints. For example, Participant P1 noted, “Voyager2’s lack of beginner-friendliness complicates choosing data fields for analysis. In contrast, HAIChart streamlines exploration with its visualization hints, making decisions on data fields and operations easier.” In addition, Participant P5 pointed out a limitation with LLM4Vis+: “Natural language interaction simplifies creating initial visualizations, but fine-tuning is challenging. LLMs often misunderstand my intent, requiring repeated adjustments that fail to achieve the desired results. Hints or a control panel are more intuitive and convenient.” This suggests that natural language ambiguity may cause ‘communication’ issues with LLMs during data exploration. Intuitive control panels or exploration guidance like hints would enhance the user experience.

Nevertheless, user feedback also reveals the limitations of HAIChart. For example, Participant P10 suggested, “HAIChart’s hint-clicking interaction suits beginners, but adding a control panel for custom field selection would increase flexibility.” Furthermore, Participant P15 pointed out, “HAIChart supports only common chart types. Future versions should include more complex types like maps and Sankey diagrams to meet extensive analysis needs.”

7.2.3. Exp-3: Effectiveness of Visualization Hints Selection.

This experiment analyzes interaction logs about HAIChart from 17 users in a multi-round recommendation experiment (Exp-2) to validate the effectiveness of the hint selection using the Hit@$k$ metric.

The results in Table 6 show that Hit@1 is 64.7% in the first round of recommendations, demonstrating the system’s ability to accurately predict the hint most interesting to the user. As the number of interaction rounds increases, the recommendation accuracy improves further. By the third round, the Hit@3 value increased to 85.7%. These results demonstrate the system’s effectiveness in prioritizing the information required by the user and the effectiveness of the hint selection in guiding the user’s decision-making process.

7.2.4. Exp-4: Ablation Study of the MCGS Optimization Techniques

We proposed several MCGS optimization techniques, including Rule-based Pruning and an Adaptive Random Exploration Strategy. We conducted ablation studies using the VizML and KaggleBench datasets to evaluate their impact on overall performance.

The results in Table 7 show that removing any optimization technique from either dataset leads to a decline in performance, confirming their importance. Specifically, in the VizML dataset, removing the Adaptive Random Exploration Strategy reduces the improvement from Hit@1 to Hit@3 from 30.5% to 8.7%, which demonstrates that this strategy enhances result diversity and avoids local optima, thereby improving performance.

7.2.5. Exp-5: Ablation Study of the Composite Reward Function

Our composite reward function consists of three components: data features, visualization domain knowledge, and user preferences. We conducted ablation studies using the VizML and KaggleBench datasets to evaluate their impact on overall performance.

Table 7 shows the results with the following observations.

Overall, removing any component decreased the performance of HAIChart. Performance drops were more significant when removing domain knowledge or user preferences, indicating that these aspects are crucial for aligning visualizations with user needs. In summary, integrating data features, visualization knowledge, and user preferences in the composite reward function enhances the effectiveness of visualization recommendations.

Table 7. Ablation studies on HAIChart (overall performance)

Methods		VizML		KaggleBench
		Hit@1	Hit@3	P@10	R10@30
HAIChart		36.9%	67.4%	55.0%	74.9%
Opt. Tech.	w/o Rule-based Pruning	34.6%	65.3%	37.4%	40.0%
	w/o Adapt. Random Exploration	33.4%	42.1%	45.7%	65.0%
Composite Reward Func.	w/o Domain Knowl.	26.3%	60.2%	31.0%	33.8%
	w/o User Preferences	30.8%	64.2%	40.7%	54.9%
	w/o Data Features	34.2%	64.1%	36.0%	68.7%

7.2.6. Exp-6: Sensitivity Analysis of the Parameters

We conducted an experimental analysis on key parameters of HAIChart: (1) For the MCGS, we evaluated the impact of the number of iterations $n$ and the balance parameter $c$ in the UCB algorithm on the quality of recommendation results. Typically, $c$ is set to $\sqrt{2}$ (Gray et al., 2023). Therefore, in this experiment, we varied $c$ from 0.5 to 2.5 and $n$ from 25 to 125 iterations. (2) For top-$k$ visualization hints selection, we evaluated the impact of different $k$ values (ranging from 3 to 15) and budgets $B$ (ranging from 10 to 50) on the selection of hints. We also tested how displaying only the top-$l$ visualization results for each hint (where $l$ ranges from 1 to 10) affects the overall performance.

Figure 11 and 12 show the results with the following insights.

(1) Different values of the balance parameter $c$ affect the quality of recommendations, but overall, HAIChart performs well and is stable. We find that a value of 1.5 for $c$ achieves a good balance between exploration and exploitation, leading to better results.

(2) Increasing the number of iterations typically enhances performance by exploring more high-value visualizations. However, increasing iterations from 100 to 125 shows minimal improvement while extending search time. Therefore, setting a suitable number of iterations balances accuracy and efficiency.

Figure 11. Parameter sensitivity analysis for MCGS

Figure 12. Parameter sensitivity analysis for Hints Selection

(3) Figure 12(a) shows that when each hint displays the top-$l$ visualizations, increasing $l$ improves the hit rate, reflecting the effectiveness of our visualization ranking. A lower $l$ (e.g., 1) performs poorly in capturing user interests. When $l$ is 8 or more, the hit rate stabilizes above 90% with no further significant improvement. Thus, $l$ should be chosen within an appropriate range to meet user needs.

(4) For the top-$k$ visualization hints selection, Figure 12(b) shows that increasing $k$ and $B$ values improves recommendation quality. However, when $k$ reaches 9 and $B$ reaches 40, performance improvements slow down. This indicates that selecting appropriate $k$ and $B$ values can balance cost and effectiveness.

Figure 13. Efficiency on KaggleBench datasets

Figure 14. A Running Example of HAIChart

7.2.7. Exp-7: The Efficiency of HAIChart.

We compared the efficiency of HAIChart and existing methods using the KaggleBench dataset, which includes eight tables (D1 to D8) of varying sizes.

Figure 13 shows the results with the following observations:

(1) HAIChart generates visualizations in 1 to 5 seconds, with an average of 2.4 seconds, and hint generation takes only 1 to 3 milliseconds, validating its capability for rapid visualization exploration.

(2) HAIChart is, on average, 1.8 times faster than MCTS, especially for datasets with many columns. For example, with the D3 dataset (15 columns), HAIChart’s graph-based structure outperforms MCTS’s tree-based structure by 2.9 times.

Overall, HAIChart is efficient enough for data visualization.

7.3. System Demonstration

We implemented the HAIChart, a web-based interface designed to facilitate visualization exploration, as shown in Figure 14. In this interface, we demonstrate two main scenarios: (1) First-Round Visualization Recommendations, and (2) Multi-Round Visualization Recommendations based on Hints.

Datasets. We utilize the Flight Delay Statistics dataset, which comprises records including the date, carrier, destination city (destcity), departure delay (depdelay), arrival delay (arrdelay), and the number of passengers. This dataset allows us to analyze various factors contributing to flight delays.

First-Round Visualization Recommendations. Initially, users upload their dataset. HAIChart then displays the data fields of the table, allowing users to explore further by clicking on fields of interest. HAIChart also logs users’ historical actions, making it easier to retrieve previous analyses, as shown in Figure 14- ❶.

After uploading the dataset, HAIChart employs a Monte Carlo Graph Search algorithm guided by a composite reward function to automatically recommend a series of high-quality visualizations. As illustrated in Figure 14- ❷, the initial recommendations include a scatter plot revealing the relationship between flight arrival delays (arrdelay) and departure delays (depdelay), and a bar plot showing the relationship between carriers and passengers. These visualizations help users quickly grasp key information within the dataset, providing an intuitive analytical perspective on various factors contributing to flight delays.

To enhance the user experience, HAIChart supports interactive operations on visualizations, such as zooming, switching chart types, and viewing underlying data (Figure 14- ❷). These features enrich the visualization editing experience and enable thorough exploration and analysis of the data, uncovering potential insights.

Multi-Round Visualization Recommendations based on Hints. Existing interactive visualization tools, such as Voyager2 (Wongsuphasawat et al., 2017), require users to iteratively refine visualizations through complex control panels. In contrast, HAIChart simplifies data exploration with visualization hints. This feature recommends hints, enabling users to bypass cumbersome decisions and explore data via a multiple-choice format, lowering barriers to data exploration.

For instance, users can explore analysis directions by browsing hints recommended by HAIChart after viewing initial visualizations. As shown in Figure 14- ❸, users interested in “Compare arrdelay to different categories” can select this hint. HAIChart then optimizes recommendations based on “arrdelay” and records user preferences, as demonstrated in Figure 14- ❹. Using the Monte Carlo Graph Search algorithm and constrained search techniques, HAIChart provides high-quality visualizations related to “arrdelay,” such as a stacked bar chart comparing “arrdelay” across different destination cities and a line chart showing “arrdelay” trends over time (Figure 14- ❺). HAIChart continuously refines recommendations based on user preferences and data features, offering an intuitive interface for data exploration.

8. Related Work

Human-powered Visualization tools such as Tableau, Voyager, and Polaris (tab, 2024; exc, 2024; Wongsuphasawat et al., 2015; Stolte et al., 2008; Wu et al., 2017; Satyanarayan and Heer, 2014) allow users to select or adjust data sources, chart types, and data transformation operations for data visualization. For example, Tableau (tab, 2024) offers a code-free interface for creating visualizations through click-and-drag actions. However, these tools are highly dependent on user skills, presenting challenges such as steep learning curves.

AI-powered Visualization. AI-powered automatic visualization (Luo et al., 2018a; Hu et al., 2019; Li et al., 2021; Zhou et al., 2021; Gao et al., 2021; Qian et al., 2022; Luo et al., 2022a, 2018b, 2020a; Qin et al., 2018; Chai et al., 2023) uses algorithms to automatically generate and recommend meaningful visualizations for datasets. For example, DeepEye (Luo et al., 2018a) uses machine learning to recommend good visualizations based on data features and domain knowledge. PVisRec (Qian et al., 2022) recommends personalized visualizations based on the user’s past interactions. However, its performance declines with new users or datasets lacking historical information.

Large Language Models for Visualization (LLM4VIS). LLM4VIS leverages large language models to transform natural language queries into data visualizations (Luo et al., 2022b, 2021c, 2021b, 2021a; Wang et al., 2022; Tian et al., 2023; Wang et al., 2023; Tang et al., 2022). For example, ChartGPT (Tian et al., 2023) employs LLMs to generate visualizations from natural language queries, while LLM4Vis (Wang et al., 2023) uses few-shot learning to suggest visualization types and explain them in text. These methods rely on users providing clear query descriptions and may struggle with accurately modifying visualizations through natural language. Our method complements LLM4VIS by providing visualization hints and easy-to-adjust features, reducing user input and making the visualization creation process more efficient.

Reinforcement Learning is a learning paradigm characterized by trial-and-error explorations, where an agent receives feedback from its environment, which is commonly applied in sequence generation tasks, such as item recommendation (Wang et al., 2021). Recent research has also adopted reinforcement learning for visualization tasks (Zhou et al., 2021; Chen and Wu, 2022; Wu et al., 2020; Deng et al., 2022). For example, PI2 (Chen and Wu, 2022; Tao et al., 2022) employs MCTS to generate interactive UI widgets from SQL logs, assisting developers in understanding necessary queries for analysis tasks. Unlike PI2, which requires users to provide SQL queries, HAIChart automatically recommends good visualizations for datasets. While PI2 allows users to refine query results through UI widgets, HAIChart introduces visualization hints to guide data exploration. Both systems aim to facilitate data exploration but differ in their application scenarios.

9. Conclusion

We introduce HAIChart, which pairs human insight with AI capabilities to enhance visualization quality progressively and iteratively through user feedback. HAIChart utilizes a Monte Carlo Graph Search-based visualization algorithm for automatically recommending high-quality visualizations. It is also equipped with a visualization hints mechanism to actively incorporate the user feedback and thus fine-tune the visualization generation algorithm iteratively. HAIChart has been validated for its effectiveness in creating high-quality visualizations that align with user preferences. Future research could explore integrating LLMs into our framework to generate and evaluate visualizations, potentially enhancing the generalizability and robustness for diverse applications.