PCF-GAN: generating sequential data via the characteristic function of measures on the path space

Let $X$ be an ${\mathbb{R}}^{d}$-valued random variable with the law $\mu=\mathbb{P}\circ X^{-1}$. The characteristic function of $X$, denoted as $\Phi_{X}:\mathbb{R}^{d}\rightarrow\mathbb{C}$, maps each $\lambda\in\mathbb{R}^{d}$ to the expectation of its complex unitary transform: $\Phi_{X}:\lambda\longmapsto\mathbb{E}_{X\sim\mu}\left[e^{i\langle\lambda,X\rangle}\right]$. Here $U_{\lambda}:\mathbb{R}^{d}\rightarrow\mathbb{C},x\mapsto e^{i\langle\lambda,x\rangle}$ is the solution to the linear controlled differential equation:

where $\mathbf{0}$ is the zero vector in ${\mathbb{R}}^{d}$ and $\langle\cdot,\cdot\rangle$ is the Euclidean inner product on $\mathbb{R}^{d}$.

References [11, 16] studied the squared characteristic function distance (CFD) between two $\mathbb{R}^{d}$-valued random variables $X$ and $Y$ with respect to another probability distribution $\boldsymbol{\Lambda}$ on ${\mathbb{R}}^{d}$:

It is proved in [25, 1] that if the support of $\Lambda$ is ${\mathbb{R}}^{d}$, then ${\rm CFD}_{\boldsymbol{\Lambda}}$ is a distance metric, so that $\text{CFD}^{2}_{\boldsymbol{\Lambda}}(X,Y)=0$ if and only if $X$ and $Y$ have the same distribution. This justifies the usage of $\text{CFD}^{2}_{\boldsymbol{\Lambda}}$ as a discriminator for GAN training to learn finite-dimensional random variables from data.

Let ${\rm BV}\left([0,T];{\mathbb{R}}^{d}\right)$ be the space of ${\mathbb{R}}^{d}$-valued paths of bounded variation over $[0,T]$. Consider

For a discrete time series $x=(t_{i},x_{i})_{i=0}^{N}$, where $0=t_{0}<t_{1}<\cdots<t_{N}=T$ and $x_{i}\in\mathbb{R}^{d}$ ($i\in\{0,\cdots,N\}$), we can embed it into some ${\bf x}\in\mathcal{X}$ whose evaluation at $(t_{i})_{i=1}^{N}$ coincides with $x$. This is well suited for sequence-valued data in the high-frequency limit with finer time-discretisation and is often robust in practice ([27, 26]). Such embeddings are not unique. In this work, we adopt the linear interpolation for embedding, following [23, 18, 32].

Let $\mathbb{C}^{m\times m}:=\left\{m\times m\text{ complex matrices}\right\}$, $I_{m}$ be the identity matrix, and $*$ be conjugate transpose. Write $U(m)$ and $\mathfrak{u}(m)$ for the Lie group of $m\times m$ unitary matrices and its Lie algebra, resp.:

By a slight abuse of notations, $\mathcal{U}_{M}(\mathbf{x})$ is also called the unitary feature of ${\bf x}$ under $M$. Unitary feature is a special case of the Cartan/path development, for which one may consider paths taking values in any Lie group $G$. We take only $G=U(m)$ here; $m\neq d$ in general ([6, 30]).

Motivated by the universality and characteristic property of unitary features ([7], see Section A.3), we constructed a unitary layer which transforms any $d$-dimensional time series $x=(x_{0},\cdots,x_{N})$ to the unitary feature of its piecewise linear interpolation ${\bf X}$. It is a special case of the path development layer [26], when Lie algebra is chosen as $\mathfrak{u}(m)$. In fact, the explicit formula holds: $\mathcal{U}_{M}({\bf X})=\prod_{i=1}^{N+1}\exp\left(M(\Delta x_{i})\right)$, where $\Delta x_{i}:=x_{i}-x_{i-1}$ and $\exp$ is the matrix exponential.

The unitary feature of a path $\mathbf{x}\in\mathcal{X}$ plays a role similar to that played by $e^{i\langle x,\lambda\rangle}$ to an ${\mathbb{R}}^{d}$-valued random variable. Thus, for a random path $\mathbf{X}$, the expected unitary feature can be viewed as the characteristic function for measures on the path space ([7]).

In the above, $\mathcal{U}_{M}(\mathbf{x})\in U(m)$ is the unitary feature of the path $\mathbf{x}$ under $M$. See Definition 2.1.

Similarly to the characteristic function of ${\mathbb{R}}^{d}$-valued random variables, the PCF always exists. Moreover, we have the following important result, whose proof is presented in Appendix A.

We now introduce a novel and natural distance metric, which measures the discrepancy between distributions on the path space via comparing their PCFs. Throughout, $d_{\rm HS}$ denotes the metric associated with the Hilbert–Schmidt norm $\|\bullet\|_{\rm HS}$ on $\mathbb{C}^{m\times m}$:

We shall not distinguish between ${\mathcal{M}}$ and $\mathbb{P}_{\mathcal{M}}$ for simplicity.

PCFD exhibits several mathematical properties, which provide the theoretical justification for its efficacy as the discriminator on the space of measures on the path space, leading to empirical performance boost. First, PCFD has the characteristic property.

Furthermore, ${\rm PCFD}_{\mathcal{M}}$ has a simple uniform upper bound for any fixed $m\in\mathbb{N}$:

Under mild conditions, ${\rm PCFD}$ is a.e. differentiable with respect to a continuous parameter, thus ensuring the feasibility of gradient descent in training.

Similarly to metrics on measures over $\mathbb{R}^{d}$ (cf. [2, 24]), we construct a metric based on PCFD, denoted as $\widetilde{\rm PCFD}$, on the space $\mathcal{P}(\mathcal{X})$ of Borel probability measures over the path space, and we prove that it metrises the weak-star topology on $\mathcal{P}(\mathcal{X})$. Throughout, $\stackrel{{\scriptstyle\text{d}}}{{\rightarrow}}$ denotes the convergence in law.

The formal statement and proof can be found in Lemma B.2 and Theorem B.8 in the Appendix.

Similar to [40] for ${\mathbb{R}}^{d}$, we prove that PCFD can be interpreted as an MMD with a specific kernel $\kappa$ (see Appendix B.3). Example B.12 illustrates that the PCFD has the superior test power for hypothesis testing on stochastic processes compared with CF distance on the flattened time series.

Now, we shall illustrate how to compute the PCFD on the path space.

Let $\bar{\mathbf{X}}:=\{\mathbf{x}^{i}\}_{i=1}^{n}$ and $\bar{\mathbf{Y}}:=\{\mathbf{y}^{i}\}_{i=1}^{n^{\prime}}$ be i.i.d. drawn respectively from $\mathcal{X}$-valued random variables $\mathbf{X}$ and $\mathbf{Y}$. First, for any linear map $M\in\mathfrak{u}(m)^{d}$, the empirical estimator of $\mathbf{\Phi}_{\mathbf{X}}(M)$ is the average of unitary features of all observations $\bar{\mathbf{X}}=\{\mathbf{x}_{i}\}_{i=1}^{n}$, i.e., $\mathbf{\Phi}_{\bar{\mathbf{X}}}(M)=\frac{1}{n}\sum_{i=1}^{n}\mathcal{U}_{M}(\mathbf{x}_{i})$. We then parameterise the $\mathfrak{u}(m)^{d}$-valued random variable $\mathcal{M}$ via the empirical measure $\mathcal{M}_{\theta_{M}}$, i.e., $\mathcal{M}_{\theta_{M}}=\sum_{i=1}^{k}\delta_{M_{i}}$, where $\theta_{M}:=\left\{M_{i}\right\}_{i=1}^{k}\in\mathfrak{u}(m)^{d\times k}$ are the trainable model parameters. Finally, define the corresponding empirical path characteristic function distance (EPCFD) as

Our approach to approximating $\mathcal{M}$ via the empirical distribution differs from that in [25], where $\mathcal{M}$ is parameterised by mixture of Gaussian distributions. In §4.1 and §5, it is shown that, by optimising the empirical distribution, a moderately sized $k$ is sufficient for achieving superior performance, in contrast to a larger sample size required by [25].

In this subsection, we apply the EPCFD to GAN training for time series generation as the discriminator. We train the generator to minimise the EPCFD between true and synthetic data distribution, whereas the empirical distribution of $\mathcal{M}$ characterised by $\theta_{M}\in\mathfrak{u}(m)^{d\times k}$ is optimised by maximising EPCFD.

By an abuse of notation, let $\mathcal{X}:={\mathbb{R}}^{d\times n_{T}}$ ($\mathcal{Z}:={\mathbb{R}}^{e\times n_{T}}$, resp.) denote the data (noise, resp.) space, composed of $\mathbb{R}^{d}$ ($\mathbb{R}^{e}$, resp.) time series of length $n_{T}$. As discussed in §2.2, $\mathcal{X}$ and $\mathcal{Z}$ can be viewed as path spaces via linear interpolation. Like the standard GANs, our model is comprised of a generator $G_{\theta_{g}}:\mathcal{Z}\rightarrow{\mathbb{R}}^{d\times n_{T}}$ and the discriminator $\text{EPCFD}_{\theta_{M}}:\mathbb{P}(\mathcal{X})\times\mathbb{P}(\mathcal{X})\rightarrow\mathbb{R}^{+}$, where $\theta_{M}\in\mathfrak{u}(m)^{k\times d}$ is the model parameter of the discriminator, which fully characterises the empirical measure of $\mathcal{M}$. The pre-specified noise random variable $\mathbf{Z}=(Z_{t_{i}})_{i=0}^{n_{T}-1}$ is the discretised Brownian motion on $[0,1]$ with time mesh $\frac{1}{n_{T}}$. The induced distribution of the fake data is given by $G_{\theta_{g}}(\mathbf{Z})$. Hence, the min-max objective of our basic version PCF-GAN is

We apply mini-batch gradient descent to optimise the model parameters of the generator and discriminator in an alternative manner. In particular, to compute gradients of the discriminator parameter $\theta_{M}$, we use the efficient backpropagation algorithm through time introduced in [26], which effectively leverages the Lie group-valued outputs and the recurrence structure of the unitary feature. The initialisation of $\theta_{M}$ for the optimisation is outlined in the Section B.4.1.

In order to effectively learn the distribution of high-dimensional or complex time series, using solely the EPCF loss as the GAN discriminator fails to be the best approach, due to the computational limitations imposed by the sample size $k$ and the order $m$ of EPCFD. To overcome this issue, we adopt the approach [41, 25], and train a generator that matches the distribution of the embedding of time series via the auto-encoder structure. Figure 3 illustrates the mechanics of our model.

To proceed, let us first recall the generator $G_{\theta_{g}}:\mathcal{Z}\rightarrow\mathcal{X}$ and introduce the embedding layer $F_{\theta_{f}}$, which maps $\mathcal{X}$ to $\mathcal{Z}$ (the noise space). Here $\theta_{f}$ is the model parameters of the embedding layer and will be learned from data. To this end, it is natural to optimize the model parameters $\theta_{g}$ of the generator by minimising the generative loss $L_{\text{generator}}$, which is the EPCFD distance of the embedding between true distribution $\mathbf{X}$ and synthetic distribution $G_{\theta_{g}}(\mathbf{Z})$; in formula,

Encoder$(F_{\theta_{f}})$-decoder$(G_{\theta_{g}})$ structure: The motivation to consider the auto-encoder structure is based on the observation that the embedding might be degenerated when optimizing $L_{\text{generator}}$. For example, no matter whether true and synthetic distribution agrees or not, $F_{\theta_{f}}$ could be simply a constant function to achieve the perfect generator loss 0. Such a degeneracy can be prohibited if $F_{\theta_{f}}$ is injective. In heuristic terms, the “good" embedding should capture essential information about real time series of $\mathbf{X}$ and allows the reconstruction of time series $\mathbf{X}$ from its embedding $F_{\theta_{f}}(\mathbf{X})$. This motivates us to train the embedding $F_{\theta_{f}}$ such that $F_{\theta_{f}}\circ G_{\theta_{g}}$ is close to the identity map. If this condition is satisfied, it implies that $F_{\theta_{f}}$ and $G_{\theta_{g}}$ are pseudo-inverses of each other, thereby ensuring the desired injectivity. In this way, $F_{\theta_{f}}$ and $G_{\theta_{g}}$ serve as the encoder and decoder of raw data, respectively.

To impose the injectivity of $F_{\theta_{f}}$, we consider two additional loss functions for training $\theta_{f}$ as follows:

Reconstruction loss $L_{\text{recovery}}$: It is defined as the $l^{2}$ samplewise distance between the original and reconstructed noise by $F_{\theta_{f}}\circ G_{\theta_{g}}$, i.e., $L_{\text{recovery}}=\mathbb{E}[|Z-F_{\theta_{f}}(G_{\theta_{g}}(\mathbf{Z}))|^{2}]$. Note that $L_{\text{recovery}}=0$ implies that $F_{\theta_{f}}(G_{\theta_{g}}(\mathbf{z}))=\mathbf{z}$, for any sample $\mathbf{z}$ in the support of $\mathbf{Z}$ almost surely.

Regularization loss $L_{\text{regularization}}$: It is proposed to match the distribution of the original noise variable $\mathbf{Z}$ and embedding of true distribution $\mathbf{X}$. It is motivated by the observation that if the perfect generator $G_{\theta}(\mathbf{Z})=\mathbf{X}$ and $F_{\theta_{f}}\circ G_{\theta_{g}}$ is the identity map, then $\mathbf{Z}=F_{\theta_{f}}(\mathbf{X})$. Specifically,

where we distinguish $\theta_{M}^{\prime}$ from $\theta_{M}$ in $L_{\text{generator}}$. The regularization loss effectively stabilises the training and resolves the mode collapse [41] due to the lack of infectivity of the embedding.

Training the embedding parameters $\theta_{f}$: The embedding layer $F_{\theta_{f}}$ aims to not only discriminate the real and fake data distributions as a critic, but also preserve injectivity. Hence we optimise the embedding parameter $\theta_{f}$ by the following hybrid loss function:

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters that balance the three losses.

Training the EPCFD parameters $(\theta_{M},\theta_{M}^{\prime})$: Note that $L_{\text{generator}}$ and $L_{\text{regularization}}$ have trainable parameters of EPCFD, i.e., $\theta_{M}$ and $\theta_{M}^{\prime}$. Similar to the basic PCF-GAN, we optimize $\theta_{M}$ and $\theta_{M}^{\prime}$ by maximising the EPCFD to improve the discriminative power.

By doing so, we enhance the discriminative power of $\text{EPCFD}_{\theta_{M}}$ and $\text{EPCFD}_{\theta_{M}^{\prime}}$. Consequently, this facilitates the training of the generator such that the embedding of the true data aligns with both the noise distribution and the reconstructed noise distribution.

Differentiability of EPCFD with respect to parameters of the embedding layer and generators are guaranteed by Theorem 3.6, as long as $F_{\theta_{f}}\circ G_{\theta_{g}}$ satisfies the Lipschitz condition thereof. Let us also stress on two key advantages of our proposed PCF-GAN. First, it possesses the ability to generate synthetic time series with reconstruction functionality, thanks to the auto-encoder structure in PCF-GAN. Second, by virtue of the uniform boundedness of PCFD shown in Lemma 3.5, our PCF-GAN does not require any additional gradient constraints of the embedding layer and EPCFD parameters, in contrast to other MMD-based GANs and Wasserstein-GAN. It helps with the training efficiency and alleviates the vanishing gradient problem in training sequential networks like RNNs.

We provide the pseudo-code for the proposed PCF-GAN in Algorithm 1.