An Interpretable Model for Real-time Tracking of Economic Indicators Using Social Media Data

We use the following definitions of interpretability: An interpretable ML model is defined as “the extraction of relevant knowledge from a machine-learning model concerning relationships either contained in data or learned by the model” [29]. Relevancy is defined if the model provides insights for a specific problem and for a particular audience. Recent works point out that interpretations of model outcomes must be based on features that are meaningful to the practitioners, so they can be analyzed and put into practice [28]. We show how our model embeds the notion of interpretability in the feature space while conceptualizing the model and thus provide inferences that are meaningful to the practitioners. This contrasts with the explain-ability that is sought as a post hoc step especially in deep learning systems [6, 20]. At times, post hoc analysis either has uncovered model interpretations that a practitioner knows to be incorrect [43] or are not tailored to the domain experts, as they do not take prior domain knowledge into account [28].

We focus on two indicators: consumer confidence index or ICS, one of the most widely cited economic surveys in the United States, and unemployment insurance claims data (UI), another important indicator describing the health of the economy. Accurately measuring consumer confidence is pivotal to governments, businesses, media, and individuals, as it describes both current economic conditions and captures consumer’s hope for the near future [12] and influences whether the economy expands or contracts, as consumer spending drives 70% of the GDP [11]. The initial claims data, along with other employment data, are closely watched by market analysts, as they determine the strength of the labor market. Both economic indicators have been studied in conjunction with correlating them to signal in SM data.

Different types of SM data (from social networking applications like Twitter, Reddit, Facebook, etc.) and Google search history data have been used to construct official indicators of consumer confidence. Researchers have mostly used the sentiment of various phrases related to job(s) or economy or mentions of purchase intentions in search engine or SM posts. There are a number of advantages in studying economic indicators using SM data. First, these datasets are collected continuously and provide frequent sensing of public perceptions than administrative and polling data. Such timely data might offer market analysts with almost real-time information and assist in when economic decisions must be made needed prior to the availability of official indicators. Second, SM are available at a lower (or no) cost compared to traditional polling, which often cost tens or hundreds of thousands of dollars. Third, analyzing SM offers a unique opportunity to glean signals from personal conversations and web searches, which might assist in capturing additional signs/parameters about the population [1].

The general approach to understanding SM data involves extracting quantitative metrics, mostly sentiments, from its content. If the goal is to demonstrate that SM can supplement traditional surveys, then metrics from SM is added to the statistical models used to construct survey indices and their net gain is studied. For example, models that include information from SM have been shown to produce more accurate estimates of election outcomes than than models that include only traditional poll data [48]. If the goal is to show that social media can potentially be used to substitute or supplement surveys/polls, then longitudinal correlations between two data sources are studied [30]. Our work explores the latter idea.

Researchers have extracted sentiments and content features from Twitter, Facebook, news, and Google search queries regarding purchase history, and so on, and used them to model ICS with mixed results [15, 30, 33, 35, 46]. Though some studies highlight a good correlation prior to 2012, a comprehensive study states that the relation disappeared when more recent data were included in the analysis [10].

Most existing studies use a single measure (mostly a sentiment feature) and study correlations to forecast CCI values. Most of these studies are based on the sentiment of few keywords, such as keywords like jobs, job on Twitter. These analyses utilize methods that look for correlations over a short period of time or with coarser granularity of data, e.g., monthly. Some works [1] have employed n-grams models to analyze the textual content in social media and used these to learn a pattern of relationships between SM text and survey responses for prediction purposes. Some works [35] are autoregressive in nature, with some components that factor SM data. However these models have a linearity assumption that fails to learn the complex relationship between high-dimensional SM data and economic indicators. Additionally, existing works have failed to assess the temporal stability of their estimates over longer periods of time.

Researchers have used Reddit data for a variety of applications such as understanding online user behavior [2], (dis)information propagation, health informatics [4], and so on. In the context of studying public opinion on economic issues, researchers have recently started using it [38] using GP regression to estimate CCI.

Gaussian Processes have been studied extensively in geostatistics (as kriging), meterology, and so on, as they provide high flexibility in modeling as well as a principled way of learning the hyper-parameters [41].

An additive GP with kernels that fully decompose additively is given aswhere \(\bf x\) is the vector in D-dimensional space and \({k_{i}}_{i=1}^D\) is a sub-kernel. When each of these subkernels operate on single dimensions, these are considered as Generalized Additive Models (GAMs). Works in the area of additive GPs assert that decomposing a function into additive components provides better extrapolatory power [17] away from the training instances. The problem with GAM GPs is that they only model first-order interactions, which is an unrealistic assumption in real problem settings. An extension of this is to consider higher-order interactions [17]; however, the number of combinations grows exponentially. Thus, learning the structure of kernel is extremely challenging. Researchers have resorted to learning the structure of the kernels via enumeration methods (where sub-kernels consider every combination of feature interactions up to a degree) [17], search methods (where possible decompositions are traversed by a search algorithm) [40], and projection-pursuit (where a projected-additive GP is learned by iteratively optimizing projection directions from regression residuals) [13]. While search methods are burdened by combinatorial search, random projection based methods impart no interpretability into the model.

Recent works [18, 23, 44] have explored additive GP where kernels operate on a subset of dimensions, which are obtained either randomly or via an informed search in model space. Compared to our proposed model, these works solely focus on Bayesian optimization and use a heuristic search strategy while still working with smaller-dimensional spaces (few tens).

A GP is a stochastic process, indexed by \({\bf x} \in \mathbb {R}^d\) (ignoring the time index \(t\) for simplicity), and is completely specified by its mean \(m({\bf x})\) and its covariance/kernel function \(k({\bf x},{\bf x^{\prime }})\), as shown in \(f({\bf x}) \sim GP(m({\bf x}),k({\bf x},{\bf x^{\prime }}))\). The mean is usually assumed to be 0 (\(m({\bf x}) = 0\)), and covariance between any two evaluations of \(f({\bf x})\) is \(k({\bf x},{\bf x}^{\prime }) = \mathbb {E}[(f({\bf x})- m({\bf x}))(f({\bf x}^{\prime }) - m({\bf x}^{\prime }))]\), where \(m({\bf x}) = \mathbb {E}[f({\bf x})]\). The covariance function defines the notion of nearness or similarity in GPs and thus encodes different types of nonlinear relationships between the covariates and targets. Gaussian Processes belong to the class of Bayesian non-parametric models, where no assumptions are made on the functional form of the relationships between covariates and targets, and, thus, these methods are known to learn highly non-linear boundaries. For details of GP for machine learning, the reader is directed to Reference [41].

At the core of our proposed model lies the GP regression (GPR), which is stated as \(y \sim \mathcal {N}(f({\bf x}), \sigma ^2_n)\). Given training data, \(\mathcal {D} = \lbrace {\bf x}_i,y_i\rbrace _{i=1}^N\) and a GP prior on \(f()\), the posterior distribution of \(y_*\) (for an unseen input vector, \({\bf x}_*\)) is a Gaussian distribution, with the following mean and variance that follows from the conditional and marginal properties of the multivariate Gaussian distribution:

Here, \({{\bf y}} = [y_1,y_2,\ldots ]^\top\), and \(K\) is a matrix that contains the covariance function evaluation on each pair of training data, i.e., \(K[i,j] = k({{\bf x}}_i,{{\bf x}}_j)\), \({{\bf k}}\) is a vector of the kernel computation between each training data and the test point, i.e., \({{\bf k}}[i] = k({{\bf x}_*},{{\bf x}}_i)\), \(k_* = k({{\bf x}_*},{{\bf x}_*})\), and \(I\) is an identity matrix. The marginal likelihood of the training dataset for GPR is given bywhere \(X\) is the \((N \times d)\) data matrix.

We use the Matern 3/2 class of covariance functions given bywhere \(r = \Vert {\bf x} - {\bf x}^{\prime }\Vert _2\), \(\ell\) denotes the characteristic length scale and \(\sigma ^2_f\) denotes the signal variance associated with the covariance function. This covariance function is a product of an exponential and a polynomial of order 1 and is suited for learning non-smooth behavior, such as those exhibited by financial time series [19, 26]. We also empirically determined that Matern 3/2 provided better results than other kernel choices, like squared-exponential.

The above formulation assumes that a single, global characteristic length scale \(l\) is sufficient to capture the associations in all feature dimensions, which is a restrictive condition. To overcome this, a kernel that employs feature-specific characteristic length scales for each of the input dimension is used. Such a kernel is called an automatic relevance determination (ARD) kernel and has been used as feature selection in econometric data [19].

Uncertainty in GP: GPR provides a measure of uncertainty along with the predictions via the variance in Equation (3). The uncertainty is a measure of trust provided by our model and might assist in algorithmic decision making by telling where (time points) and how much to trust the individual predictions.

We propose a novel Group AGP model, which hierarchically decomposes the underlying function as a sum of GPs, each operating on a group, and, further, models each group as an interaction of effects. We show how such a hierarchical decomposition exploits clustering to learn the structure of data. This additive structure imparts the Group AGPs framework increased flexibility and interpretability.

The Group AGP model decomposes the latent function value to be learned, \(f({\bf x})\), for a given input D-dimensional \({\bf x}\) using the following additive form:where \(f_{g_i}({\bf x}^{(g_i)})\) is the group-specific latent function, which are taken to be independent stochastic processes, and then the covariance function of \(f(x)\) will be of the form:where \(k_{g_i}({{\bf x}}^{(g_i)}, {{\bf x}}^{(g_i)^{\prime }})\) is a kernel function operating on each \({{\bf x}}^{(g_i)}\) (subset of features). This kernel function is hierarchically modelled as the following product form:where \(k_{g_ij}({{\bf x}}^{(g_{ij})}, {{\bf x}}^{(g_{ij})^{\prime }})\) is the kernel function for the \(j{\rm th}\) group. Thus \(f(\bf x)\) is composed of \(c\) additive groups, and each additive group hierarchically models the interactions within itself. We refer to each \(f_{gi}({\bf x^{(gi)}})\)’s as the additive groups and grouping of different dimensions into these groups as the decomposition of the function.

Example. To predict the consumer confidence index from SM, the additive groups could capture the sentiment and the content extracted from clusters that contain important topics of discussion affecting consumer confidence like growth of economy or jobs. In Group AGP formalism, each cluster is modeled as an interaction of content and sentiment effects and the number of clusters correspond to the number of additive components (denoted by \(c\) in Equation (7)).

A pictorial depiction of our model for two additive groups is shown in Figure 1. Here, the kernels \(k_{11}\) and \(k_{12}\) can correspond to the content and sentiment values for a cluster about economy, and \(k_{21}\) and \(k_{22}\) can correspond to the content and sentiment values for a cluster about jobs. Then, \(f_{1}\) shows a 1D function drawn from a GP, whose kernel is defined as a product of \(k_{11}\) and \(k_{12}\). \(f_{2}\) shows a 1D function drawn from a GP, whose kernel is defined as a product of \(k_{21}\) and \(k_{22}\). Next, \(f\) is shown as an addition of two functions \(f_{1}\) and \(f_{2}\). The number of additive components has an important bearing in our model as it embeds interpretability along the feature space in modeling phase, which is preferred than post hoc attempts at explaining the model behavior to practitioners [28]. Our proposed decomposition into additive structures is beneficial in domains, where the dimensionality of the feature space is large, as in the case of natural language processing.

The posterior distribution for a test input, \({\bf x}_*\), is obtained using Equations (2) and (3), with the key difference being the use of the group additive covariance function given in Equations (7) and (8). In Group AGP, the kernel matrix is the sum of kernel functions of individual additive components. Additive decomposition has been shown to better learn the structure of the underlying data compared to local kernels (such as squared exponential kernel) and allows better generalization capability away from the training samples [17]. Thus, our methodology is tailored to work with small training data regime, which are frequently encountered in many real-world problems, where one has to learn from limited labeled data, such as expensive surveys, and learning is guided by temporal and contextual constraints.

The hyper-parameters of the Group AGP model consist of the length-scale (\(\ell\)) parameter for the individual Matern kernel function for each effect, i.e., \(k_{g_{ij}}({{\bf x}}^{(g_{ij})}, {{\bf x}}^{(g_{ij})^{\prime }})\), and one signal-variance (\(\sigma ^2_f\)) parameter for each of the \(c\) groups. We fix the signal-variance parameter for all but one Matern kernel function within a group to 1. Thus, there will be \((cl + c)\) kernel hyper-parameters and one additional likelihood noise, \(\sigma _n^2\). The hyper-parameters are estimated by maximizing the marginalized log-likelihood of the training data (see Equation (4)) using conjugate gradient descent.

In the previous section, we have described how to use Group AGPs to estimate the value of a given target for a given kernel structure based on \(c\) (number of additive groups). The space of models is large, and choosing the right number of \(c\) requires a method to compare models. We specify a Group AGP model with c clusters as \(M_{c}\) and perform model selection on \(c\).¹

Metrics for model selection: Rather than choosing basing our model evaluation on a single criteria, for example, model complexity, our discovery of the best model is guided by an interplay of three criteria—it should explain the observed data, reduce the error on the test set, and provide the best interpretability score. To provide a great fit on the observed data, we want a model that maximizes the log-likelihood of the training data and penalizes the model’s complexity. For the second criteria, we measure the error on unseen test point via mean log loss, which takes the absolute difference between the prediction and the target, while also incorporating the variance associated with the prediction. For the third criteria, we provide a novel scoring method called the interpretabilty score (IS) that quantifies the ease of interpretation of what different additive components encode.

Now, we explain how the three criteria for model selection are defined.

Thus, the interplay of these three factors helps us choose the best Group AGP model. All our model’s evaluation and comparative analysis is done with the selection model.

Here, we go into more detail about our model’s interpretability. Through interpretability we want to answer the following two questions and also describe how our model answers them.

The above subsections describe how content embeddings, linguistic, and sentiment features are extracted corresponding to each post timestamped at \(t\), say, \(p_t\). We point out that the three feature spaces for \(p_t\) are stacked together as a single vector, \({\bf x}_{p_{t}} = embed_{p_{t}}, ling_{p_{t}}, sent_{p_{t}}.\)

Next, we create a collection of posts spanning a time period of \(T\) (order of years), denoted by \(\mathcal {P}_T = {p_1, p_2,\ldots , p_t}\), where \(t \in T\), and perform clustering on their embeddings. Though any clustering algorithm could be employed, we used \(k\)-means clustering with \(c\) clusters. This results in each post in \(\mathcal {P}_T\) to be associated with a cluster ID along with its feature space. Now the feature space representation for each post, \(p_t\), in \(\mathcal {P}_T\) is given as \({\bf x}_{p_{t}} = {embed_{p_{t}}, ling_{p_{t}}, sent_{p_{t}}, c_i}\), where \(c_i\) denotes the \(i{\rm th}\) cluster.

We will shortly describe how \(T\) and \(c\) are set. Our rationale behind clustering is twofold: (1) It helps to unravel the different topics existing in the SM data. (2) It delineates the sentiment and psychological features associated with the posts by the topics. We later show how these clusters correspond to building interpretability into our model.

Using GT, we are interested in the normalized search volume of goods and services that are traditionally studied in relation to consumer spending in the U.S. Following works by References [7, 34, 35, 45, 50], we queried the GT website using search queries that match the categories defined in the U.S. Bureau of Economic Analysis (BEA). Some examples of BEA categories to search queries include the following: durable goods – computer and electronics; non durable goods – food and drink retailers; and services – health insurance.

Corresponding to each category, the GT website was queried to extract longitudinal data detailing the normalized volume of queries on the Google search. As noted under Results, data were extracted at weekly or monthly levels to match the temporal granularity of the corresponding target to be predicted.

In this section, we detail how Group AGP model is evaluated. Both the covariates and targets are time series data. We denote the time series of covariates as \({\bf X}_t \equiv {\bf x}_1,{\bf x}_2,\ldots ,{\bf x}_t\), where each \({\bf x}_t \in \mathbb {R}^d\); \(d\) refers to the number of features extracted from SM data.

The target time series is denoted as \({\bf y}_t \equiv y_1,y_2,\ldots ,y_t\), where each \(y_i \in \mathbb {R}\). We denote a time series starting at index \(t_{1+1}\) and ending at index \(t_2\) as \({\bf X}_{t_1:t_2}\).

Algorithm details: Ours is a regression task, where the longitudinal covariates are \({\bf X}_t\) and the targets are \({\bf y}_t\). To make prediction at a given timestep \(t\), \(\hat{y}_{t}\), we train our Group AGPs model on \({\bf X}^{train}\) and \({\bf y}^{train}\). The training data are the historical data derived from fixed length window of size \(w\). The learning in our method is accomplished using data residing in window \(w\), which is advantageous as our estimates are governed by only recent relationship between covariates and targets that is encoded within the window.

We introduce the notion of a prediction step (indicated by \(\Delta\)), which encodes how far ahead can our model reliably predict. Thus, to predict at time point \(t\), rather than training until time point \(t-1\), our training data are lagged further by \(\Delta\) timesteps, ranging from \({t-\Delta -w ; {t-\Delta }}\).

The training data are used to estimate the kernel hyper-parameters of our Group AGP model, denoted as \(\theta _t\). At inference, the training data along with the input at \(t\) are used to estimate the target, denoted as \(\hat{y}_t\), which is a Gaussian distribution with mean \(\bar{y}_t\) and variance \(\sigma ^2_t\). We use the variance, \(\sigma ^2_t\), as the model uncertainty at \(t\). To make predictions at \(t+1\), the training window is shifted to the right by 1, while maintaining its size as \(w\). The method to produce estimates for the entire time series is described as the routine Monitor in Algorithm 1.

The MonitoringWithMissingTargets routine in Algorithm 1 describes how our model is used to evaluate reducing the frequency of survey data, i.e., some of the \(y_{t}\) values are not available or missing. To fill in the missing value at \(t^{\prime }\), we use the historical data until \(t^{\prime }-1\) for training the Group AGP model, and predictions are derived at \(t^{\prime }\). The mean of the predictions \(\bar{y}_{t^{\prime }}\) is now our estimates at \(t^{\prime }\). This procedure is repeated for all the unavailable target values. More details about this experimentation are given in Section 5.

In this section, we describe the two types of targets used in this work, consumer confidence index and the unemployment insurance claims data, as follows.

Consumer Confidence Data: The ICS collected by the Institute for Social Research at the from University of Michigan is an important survey that measures U.S. consumer confidence. Researchers have extensively studied it in social sciences, as well as in prior works on SM forecasting. The survey consists of responses aggregated to a single index of five questions based on an individual’s personal finances, their outlook for the economy in the near future, and their recent buying decisions. A nationally representative sample of approximately 500 respondents is drawn, and the poll is administered via telephone interviews. Although this indicator is released monthly, the daily time series is also available. We perform our analysis at both daily and monthly granularity. While the monthly ICS data are available from 1978, the daily ICS data are only available from January 2008 to May 2017. It is important to note that there are error bounds corresponding to target data, too. With monthly ICS, the 95% confidence interval is +/\(-\) 3.29%, while the confidence interval with daily data is not available to us.

Unemployment claims revised: In Reference [1], features are extracted from Twitter data between 2011 and 2013 related to job losses, and it is correlated with the revised claims of unemployment insurance (UI). The researchers argue that it is important to focus on understanding these UI claims, as they state the health of job flows, which is of importance to economists, market participants, and policymakers. Additionally, since the UI claims data are available at high frequency, the researchers claim that it is a “good” indicator to test the performance of social media derived signal. They describe a linear model to predict the unemployment signal, with independent variable as the employment situation.

As a measure of our model performance and its comparison with existing works, we employ the following metrics.

We conducted experiments to assess three important characteristics of our model, described as follows:

Google Trends has been studied in the past to predict UI [8, 50]. The idea behind the use of GT is that it is natural for an individual to look for jobs or search for unemployment claims on search engines once they are unemployed. We have used the same GT categories as past research [8, 50], i.e., “jobs” and “welfare and unemployment.” Thus, the GT tool is queried with these categories from 07/2011 to 07/2016, aligning with the UI claims target. Both data are at weekly granularity. The GT tool returns the normalized values of search queries belonging to the two categories, which translates to two features. Since there is no rich or high-dimensional feature space to be explored, our Group AGP model reduces to a simple additive model with two components, each modeling a single feature with a Matern kernel. Thus, no model selection is explored in this case.

Here, we show results on an important evaluation using Reddit data for ICS for reducing the frequency of ICS survey. Reddit is chosen for this analysis as it is available at fine daily granularity. Table 6 shows that Reddit data can be used to reduce the cost of surveys by estimating accurate responses in periods when no survey was undertaken. The step in the Table 6 refers to the frequency of conducting the survey: A step of 2 means that a survey can be done every second month; a step of 3 means every third month, so on. As we decrease the frequency of the surveys, the estimates during the off-month period decrease at a steady rate.

Figure 16 shows the performance of our model when survey data for alternate month was made unavailable. We see how our predictions closely track the targets, and it is an encouraging result showing that Reddit data can be used to supplement surveys. In Figure 17, we take a snapshot the time series (July–January 2014), so that we focus on how our predictions and their associated uncertainties vary when surveys are run every second, third, and fourth month, respectively, for the duration of our experimental data. While the estimates are close to the target for some of the months, they are off for others.

Comparing the left- to the rightmost diagram in Figure 17, we notice that as more time elapses between surveys, the quality of the estimates suffer. This is because to make prediction at time \(t\), the model relies on a window of past 2 years of data, where the missing survey points have been replaced by the model estimates. When the time elapsed between the surveys is large, the predictions rely mostly on model estimates. However, once the next survey is available, the model can again reliably predict the subsequent target values.

While previous works have use Twitter for estimate ICS, we use Reddit data. There is an important difference in communicative dynamics between the two platforms: Twitter is more appropriate to study social connections and diffusion or virality of information [39], whereas Reddit provides a discussion platform through which individuals can discuss their plans of buying commodities or houses, job losses or gains, and general chatter about the economy. Reddit provides moderated and dedicated communities in the form of subreddits for discussion, which are not present on Twitter. Specific subreddits have been successfully studied for understanding opinions related to important niche topics, like mental illness, sexual violence, and so on [22, 24, 27, 31, 47]. Since the ICS is ultimately intended to understand human behavior, we explored Reddit as a publicly available data source that is captures public optimism or pessimism in jobs and economic affairs.

Both Reddit and GT data capture similar but also disparate information about individuals’ perception toward consumer sentiment. Compared to Reddit data, GT data are consolidated from web search queries. For example, an individual is likely to announce a new job or economy-related sentiment on Reddit but would do a web search for specific purchase intentions. While Twitter/Reddit data are public, raw individual Google searches are not publicly available, and only aggregated normalized frequencies are available through the GT tool, thereby making certain case-studies or uses of the data difficult or impossible.

Being non-parametric in nature, the Group AGP model incurs a \(\mathcal {O}(T^3)\) computational cost to obtain a single prediction, where \(T\) is the number of the training instances. The primary driver for this cost is the inversion of the \((T \times T)\) kernel matrix in Equations (2) and (3). This is clearly more expensive than other linear models explored in this article (typically \(O(1)\)); however, this is the tradeoff for better performance.

Since the hierarchical decomposition in Group AGP does not change the number of training instances \(T\) used to obtain a single prediction, a major cost of complexity is still dominated by \(\mathcal {O}(T^3)\). The model does introduce a number of hyper-parameters (length-scales and variances), but these scale linearly with number of components, \(c\). Since \(c \ll T\), the hierarchical decomposition of Group AGP does not significantly increase its complexity, which is still in the order of \(\mathcal {O}(T^3)\). Additionally, our model’s complexity does not increase with time, as we use a sliding window with fixed-length \(T\). The estimation of the model hyper-parameters is an iterative optimization procedure, in which each step has complexity (\(\mathcal {O}(T^3)\)). However, the number of steps needed for convergence is data dependent.

Despite their advantages, sensing with SM data has some pitfalls. Some of the most important limitations include the lack of representativeness of SM users compared to the general population and the lack of demographic information present in social media that could help rectify the bias. When researchers use social media data, they either use a data dump (e.g., Reddit) or may be provided a sample of the data (e.g., Twitter). In both cases, studies based on these datasets suffer from sampling bias. GT data do not reveal demographic information about individuals who generated the data, thus hindering the use of these data for society wide studies. While the Reddit and GT data used in this study are anonymized, care should be exercised to preserve the privacy of users if disparate user-generated SM data are employed [3]. Keeping these limitations in mind, we suggest that Reddit and GT data can be used to supplement traditional survey responses to consumer confidence and labor flows, especially for the near future, but may not completely replace these surveys.