Are LLMs Naturally Good at
Synthetic Tabular Data Generation?

By a composite dataset, we mean one that is best modeled as a mixture of distributions rather than a single distribution for the full table. The traditional approach to fine-tuning pioneered by methods such as GReaT work only if we assume that the table can be modeled by a single distribution rather than a mixture. For instance, in a geolocation dataset, fine-tuning with GReaT works for one US state at a time.

In Fig. 1, we aim to generate synthetic location data made up of (latitude, longitude) tuples for US states. Fig. 1 (a) depicts sampled real locations for the state of Delaware (DE) chosen for its irregular shape. In Fig. 1 (b), we depict the results from GReaT which shows a very admirable 1% violation rate. Following their approach, we fine-tune the Distill-GPT2 LLM [35] by supplying tuples of (latitude, longitude, state) in a randomized order.

However, when we extend this method to all 51 US states and territories, we can see that the generated data for Delaware drops significantly in quality, with violations growing to 23% after fine-tuning (Fig. 1 (c)). Generated data for other states suffer from similar violations. With our fine-tuning approach (PAFT) presented later (Fig. 1 (d) we are able to support the generation of synthetic data for all 51 states with minimal violations.

Observation 1: Modeling one class can be done in an order-agnostic manner but modeling a multi-class dataset requires paying careful attention to feature ordering.

Observation 2:. Within a composite, multi-class, dataset, there is variance in difficulty of modeling specific classes (e.g., New Mexico is easier to model for LLMs than Delaware or Alaska). See Fig. 5 and Fig. 10 in Appendix.

Observation 3: While any order can achieve satisfactory results for unmixed data, it becomes increasingly crucial to employ a better order when the dataset becomes progressively more composite. We conduct a simple experiment based on a simulated table with four attributes and corresponding six column-wise dependencies (see Appendix C.1). Figure 3 shows the relationship between different order permutations. In this simulated table, there are a total of six dependencies between columns (to be made precise using the notion of FDs later); see Appendix C.1. The evaluation measure captures the extent to which the generated data preserves the functional dependencies present in the original data, so that 6 means the generated data is faithful to the original schema complexity and 0 means the data has maximum violations. From Figure 3, we observe that as the schema complexity increases (progressively adding more classes from left to right), it becomes increasingly important to establish a precise sequence in order to provide higher quality generated data. Thus, with randomly guessed order permutations, auto-regressive generation with LLMs are severely limited in their ability when it comes to simulating composite datasets with complex functional dependencies.

A recently presented paper, TabSyn [43], showcased remarkable advancements in joint-distribution learning generative models, surpassing previous models of similar lineage, in terms of distributional correlation measures and machine learning efficiency. However, as we will show, methods like TabSyn fail to capture real-world dependencies, e.g., the relationship between protocol and port number in network traffic data [42], or the geographical boundaries of US states as described earlier. Constraints in tabular data can be inferred empirically from conditional distributions in the data and thus methods such as LLMs are particularly powerful because of their autoregressive nature.

Observation 4: While joint distribution learning models (CTGAN [40], CorpulaGAN [40], TabSyn [43]) and conditional distribution models (GReaT and ours) exhibit comparable performance in modeling joint distributions, LLM-based fine-tuning demonstrates greater advantages for capturing conditional distributions, functional relationships, and domain constraints (see Figure 5 and Table 2).

In summary, feature ordering can be both a nuisance and a gift. It is a nuisance because it demands additional constraints to be modeled. It can be a gift because it suggests ways to sequentially generate data even by autoregressive LLMs. Our proposed approach (PAFT ) aims to achieve an optimal permutation order for fine-tuning LLMs.

While training $p_{\theta}(\cdot)$, it is usually assumed that all records $\mathbf{x}_{i}\in\mathcal{D}$ are independent. Generating new data samples $\hat{\mathbf{x}}_{i}\in\hat{\mathcal{D}}$ can be done in various ways (e.g., see [41, 40, 45]) which aim to directly estimate the joint distribution $\mathbb{P}(\mathcal{A})$ or, as is done here in PAFT , where $\mathbb{P}(\mathcal{A})$ is estimated by an autoregressive LLM based generative process, as a product of multiple conditional densities governed by the input ordering.

Autoregressive LLM models are pre-trained to maximize the likelihood of target token $x_{ij}\in\mathcal{D}$, conditioned upon the autoregressive context $\mathbf{x}_{(i,1:j-1)}\in\mathcal{D}$ where $\mathcal{D}$ is the training corpus comprising a large amount of textual data (in the pre-training context). Eq. 1 defines the general training criterion of LLM training using the self-supervised next-token prediction task with ‘w’ denoting the context length.

The generation of a single instance (i.e., database record) $\mathbf{x}_{i}\in\mathcal{D}$ is given by Eq. 2:

Specifically, each database record is generated as a product of conditional distributions.

Input Encoding. To support the processing of our records $\mathbf{x}_{i}\in\mathcal{D}$ by a pre-trained LLM, we adopt the following encoding:

In Eq. 3, $c_{j}$ represents the attribute name of the $j^{th}$ database column while $x_{i,j}$ represents the actual value of the $j^{th}$ column for the $i^{th}$ record. Further, we can assume we have a mechanism to obtain a feature order permutation $\mathbf{k}$ to govern the order of the attributes in $\mathbf{t}_{i}$, such that $\mathbf{t}_{i}(\mathbf{k})=[t_{i,k_{1}},t_{i,k_{2}},..,t_{i,k_{m}}]$ (where $i\in\{1,..,n\},k_{j}\in\{1,..,m\}$.), represents the same record but with the attribute order governed by the permutation $\mathbf{k}$, This definition admits the random feature order as a special case in which $\mathbf{k}$ is a random permutation.

Since we consider autoregressive LLM-based generative models, employing the chain rule to sequentially produce each column of a table record $\mathbf{t}_{i}$, we can view each generation step as approximating the joint distribution of the table columns as a product of conditional distributions (i.e., $\mathbb{P}(t_{i,1},...,t_{i,m})\simeq\prod_{j=1}^{m}\mathbb{P}(t_{i,j}|t_{i,1},..,t_{i,{j-1}})$). However, as the number of columns increases and the relationships between columns get more conditional, the likelihood of encountering training and sampling bias due to class imbalance also rises [40]. To minimize such adverse effects, we can consider injecting knowledge of the pre-existing functional relationships among columns, to govern the autoregressive generation process. To infer such functional relationships, we leverage a learned dependency graph derived from functional dependency (FD) relations which enables us to effectively determine the appropriate training and sampling sequence. This, in turn, allows us to alleviate potential biases during training by establishing a generation curriculum leading to improved estimation accuracy of the joint distribution $\mathbb{P}(\textbf{t}_{i})$ in auto-regressive prediction $\mathbb{P}(t_{i,k_{1}},...,t_{i,{k_{m}}})\simeq\prod_{j=1}^{m}\mathbb{P}(t_{i,k_{j}}|t_{i,k_{1}},..,t_{i,k_{j-1}})$, where the ordering $t_{i,k_{1}}\dots,t_{i,k_{m}}$ is obtained by a feature order permutation function $\mathbf{k}=\phi(\mathcal{D},\mathcal{S})$. We detail the requisite background and design of $\phi(\mathcal{D},\mathcal{S})$ in sections 3.2 and  3.3.

A functional dependency (FD) is a relationship $R$ in schema $S$ that exists when a subset of attributes $A\subset S$ uniquely determines another subset $B\subset S$ of attributes. We succinctly represent an FD as $R:A\rightarrow B$ which specifies that $B$ is functionally dependent on $A$. FDs can be record-level or schema-level.

We leverage FD discovery techniques to govern the order of the autoregressive data generation process in PAFT . A large body of research from the database literature on FD discovery [26, 44, 27] can be leveraged in PAFT , including methods that account for noisy FDs [44]. In this work, we focus on leveraging schema-level FDs discovered using a state-of-the-art FD discovery algorithm [27] to govern the autoregressive data generation process.

FD Distillation. The result of traditional FD discovery, yields complex (i.e., multi-attribute) functional dependencies between columns which are ambiguous to resolve in an autoregressive generation setting. Hence we undertake an intermediate FD distillation step to simplify multi-attribute functional dependencies into multiple single attribute FDs as detailed below.

We first construct a dependency graph model $G(\mathcal{V},\mathcal{E})$ where $\mathcal{V}$ represents the set of vertices with each $v\in\mathcal{V}$ representing an attribute in $\mathcal{S}$ and $e_{ij}\in\mathcal{E}$ representing an edge relation from attribute $v_{i}$ to attribute $v_{j}$. Further, we consider two types of edges in $\mathcal{E}$, specifically each $e_{ij}$ may be a type-1 edge or a type-2 edge (defined next). The two edge types (i.e., type-1, type-2) in $\mathcal{E}$ are derived from three classes of FDs, as shown in Fig. 4(a). Subsequently, we proceed to examine each individual case: 1) Single attribute left-hand side (LHS) and single attribute right-hand side (RHS) 2) Multi-attribute LHS and single-attribute RHS. 3) single or multi-attribute LHS and multi-attribute RHS. We shall use the example table in Fig.  6 to define each FD case.

[Type-1 Edge]. Let us consider an example of FD Case 1, wherein the column State functionally determines column Bird. In such a case, we enforce that the value for the attribute State be generated prior to the value for the attribute Bird. Accordingly, a forward directed edge from State to Bird is created in the column dependency graph $\mathcal{G}$. We term such forward directed edges as type-1 edges in $\mathcal{G}$. [Type-2 Edge]. The other type of edge in $\mathcal{G}$, arises when we encounter an FD with a multi-attribute LHS and a single attribute RHS (i.e., FD Case 2). As per FD Case 2, the values of multiple columns in the LHS would collectively decide the value of the column on the RHS. As an example in Fig. 4(a), the tuple of columns Latitude, Longitude functionally determines State. For such FDs, two backward edges are added in the column dependency graph $\mathcal{G}$, connecting State to both Latitude and Longitude. We term such backward directed edges as type-2 edges in $\mathcal{G}$.

FD Case 3 relationships are ones where the RHS has multiple attributes and the LHS could have single or multiple-attributes. Such relationships do not directly result in an edge in our dependency graph $\mathcal{G}$. Instead, as classically done in FD literature [27], we subject such FD relationships to an intermediate decomposition step. Specifically, the multi-attribute RHS of FD Case 3 relationships is decomposed into multiple single-attribute RHS dependencies each comprising the original LHS. Further, in each of these new decomposed relationships, if the LHS is single-attribute, it is treated as a FD Case 1 relationship (i.e., a directed edge from $LHS$ to $RHS$ is added to $\mathcal{G}$), else it is handled as an FD Case 2 relationship, wherein for each attribute in the multi-attribute LHS, a backward dependency edge from the single-attribute RHS is added to the dependency graph $\mathcal{G}$. Therefore, for every column in the right-hand side (RHS), we employ either type-1 or type-2 edge construction, depending on the value of its left-hand side (LHS). Full procedure detailed in Appendix A.2 Algorithm 1 .

Until this point, our construction of the graph $\mathcal{G}$ has only been limited to considering pair-wise relationships between columns in $\mathcal{D}$. Graph properties like functional dependency transitivity, require us to obtain a total ordering on the nodes $v\in\mathcal{V}$ of $\mathcal{G}(\mathcal{V},\mathcal{E})$ that is deterministic in nature for effective auto-regressive LLM training. This implies that in order to obtain a feature order permutation ($\mathbf{k}$) from the derived functional dependency relationships (Sec. 3.2), a computation must be performed on the entire dependency graph $G(\mathcal{V},\mathcal{E})$.

We define this task of obtaining a total feature order $\mathbf{k}$ from $\mathcal{G}(\mathcal{V},\mathcal{E})$ as an optimization step which seeks to produce $\mathbf{k}$ while minimizing the number of violated relationships in $\mathcal{G}(\mathcal{V},\mathcal{E})$. Appendix A.2 Algorithm 2 outlines this procedure.

Consider the trivial case of having an empty FD graph $\mathcal{G}$ i.e., $|\mathcal{E}|=\emptyset$. If the columns of a table are not functionally dependent on each other, then the order of generation is not important and $\mathbf{k}$ can be some arbitrary permutation of the columns in $\mathcal{S}$. For all other cases, our total feature ordering algorithm operates in three phases, as shown in Fig. 4(b). [Phase 1: Condensation]. It is apparent that if a graph is not a directed acyclic graph (DAG), there is no optimal solution to the total feature ordering problem. In other words, there must be FDs that cannot be satisfied in the resulting total order permutation $\mathbf{k}$. In such cases, we compute the strongly connected components (SCC), and condense them into super nodes, thus transforming the original graph into a DAG. [Phase 2: Ordering]. An application of a topological sort onto the DAG from Phase 1 will result in a total feature ordering with all SCCs in $\mathcal{G}$ compressed into super nodes. [Phase 3: Expansion of SCC]. Once the topological sort is conducted in Phase 2, we finally expand the SCC super nodes (via. arbitrary ordering) such that although the intra-SCC ordering of the nodes within the SCC is arbitrary, their ordering relative to non-SCC nodes is maintained.

After the optimized feature order permutation is obtained (via Appendix A.2 Algorithm 2), we fine-tune the LLM with the textually encoded table record $\mathbf{t}_{i}$ such that the auto-regressive generation process is governed by the optimal feature order permutation $\mathbf{k}=\phi(\mathcal{D},S)$. Specifically, we generate the table governed by order $\mathbf{k}$ as defined in Eq. 4.

We employ the Low-Rank Adaptation (LoRA) fine-tuning strategy [16]. To generate synthetic rows, we first sample the initial token $p(t_{i,k_{1}})$ from the marginal distribution of variable $k_{1}$ in actual training data, and then use Eq. 4 to sequentially sample subsequent tokens $p(t_{i,k_{j}})$, where $j\in{2,...,m}$.

Dataset. We evaluated the efficiency of PAFT through experiments on six real datasets commonly used in synthetic table generation studies like GReaT [3] CTGAN [40]. These are: Beijing [7], US-locations [13], California Housing [24], Adult Income [2], Seattle [34], and Travel [37]. Separately we also generate a set of four simulated datasets (as described in Algorithm 3 Appendix C.1). Details of dataset statistics are in the Appendix C.1.

Baseline. In benchmarking suite, we have baselines that consist of current deep learning approaches for synthetic data generation (CTGAN [40], CopulaGAN [40], TabSyn [43]), and the most advanced LLM fine-tuning synthetic table generator GReaT [3]. To guarantee an equitable comparison, we employ the Distill-GReaT model for both LLM techniques in all tests. Details in Appendix C.2.

Figure 5 and 10 illustrates that various classes within a complicated dataset may present distinct challenges when it comes to modeling, respectively in terms of visualization and statistics. Both baseline and PAFT can provide satisfactory performance in less complex classes. As the level of complexity rises, the importance of finding the most optimal permutation becomes crucial, and hence we find that PAFT , which is aware of the functional dependencies and hence able to better capture semantic structure in the data, outperforms the GReaT baseline which is agnostic of functional dependencies. The variation may arise due to statistical factors or semantic factors in the actual world, which are examined in a detailed case study (Appendix B.4).

We may infer the degree of dependency retention of data’s intrinsic characteristics as another lens through which to evaluate the quality of the data generated by PAFT and baselines. In line with this, we conducted rule checks that were derived from real-world scenarios, and subsequently evaluated the generated data from all DL and LLM baselines, and our PAFT model. Table 2 displays the rate of violation in the generated data. From the table, we observe that PAFT adheres to data’s characteristics more faithfully (i.e., significantly fewer rule violations) than baseline methods, by learning together with functional dependencies.

We next assesses effectiveness of the generated (i.e., synthetic) data by measuring how well discriminative models trained on synthetic data perform on their target discrimination task versus when trained on real data. As shown in Table 3, PAFT  is best or second best in over 80% of (dataset, method) combinations. Other research questions are discussed in the appendix, as indicated in Table 3.