Targeted Marketing PDF
Targeted Marketing PDF
a r t i c l e i n f o a b s t r a c t
Article history: We propose a data mining (DM) approach to predict the success of telemarketing calls for selling bank long-term
Received 1 November 2013 deposits. A Portuguese retail bank was addressed, with data collected from 2008 to 2013, thus including the ef-
Received in revised form 28 February 2014 fects of the recent financial crisis. We analyzed a large set of 150 features related with bank client, product and
Accepted 4 March 2014
social-economic attributes. A semi-automatic feature selection was explored in the modeling phase, performed
Available online 13 March 2014
with the data prior to July 2012 and that allowed to select a reduced set of 22 features. We also compared four
Keywords:
DM models: logistic regression, decision trees (DTs), neural network (NN) and support vector machine. Using
Bank deposits two metrics, area of the receiver operating characteristic curve (AUC) and area of the LIFT cumulative curve
Telemarketing (ALIFT), the four models were tested on an evaluation set, using the most recent data (after July 2012) and a
Savings rolling window scheme. The NN presented the best results (AUC = 0.8 and ALIFT = 0.7), allowing to reach
Classification 79% of the subscribers by selecting the half better classified clients. Also, two knowledge extraction methods, a
Neural networks sensitivity analysis and a DT, were applied to the NN model and revealed several key attributes (e.g., Euribor
Variable selection rate, direction of the call and bank agent experience). Such knowledge extraction confirmed the obtained
model as credible and valuable for telemarketing campaign managers.
© 2014 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.dss.2014.03.001
0167-9236/© 2014 Elsevier B.V. All rights reserved.
S. Moro et al. / Decision Support Systems 62 (2014) 22–31 23
results in Refs. [6,8], comparable NN and SVM performances were ob- analyzes the obtained results; finally, conclusions are drawn in
tained in Ref. [5], while DT outperformed NN and SVM in Ref. [24]. Section 4.
These differences in performance emphasize the impact of the problem
context and provide a strong reason to test several techniques when ad- 2. Materials and methods
dressing a problem before choosing one of them [9].
DSS and BI have been applied to banking in numerous domains, such 2.1. Bank telemarketing data
as credit pricing [25].
However, the research is rather scarce in terms of the specific area of This research focus on targeting through telemarketing phone calls
banking client targeting. For instance, Ref. [17] described the potential to sell long-term deposits. Within a campaign, the human agents exe-
usefulness of DM techniques in marketing within Hong-Kong banking cute phone calls to a list of clients to sell the deposit (outbound) or, if
sector but no actual data-driven model was tested. The research of meanwhile the client calls the contact-center for any other reason, he
Ref. [19] identified clients for targeting at a major bank using pseudo- is asked to subscribe the deposit (inbound). Thus, the result is a binary
social networks based on relations (money transfers between stake- unsuccessful or successful contact.
holders). Their approach offers an interesting alternative to traditional This study considers real data collected from a Portuguese retail
usage of business characteristics for modeling. bank, from May 2008 to June 2013, in a total of 52,944 phone contacts.
In previous work [23], we have explored data-driven models for The dataset is unbalanced, as only 6557 (12.38%) records are related
modeling bank telemarketing success. Yet, we only achieved good with successes. For evaluation purposes, a time ordered split was initial-
models when using attributes that are only known on call execution, ly performed, where the records were divided into training (four years)
such as call duration. Thus, while providing interesting information for and test data (one year). The training data is used for feature and model
campaign managers, such models cannot be used for prediction. In selection and includes all contacts executed up to June 2012, in a total of
what is more closely related with our approach, Ref. [15] analyzed 51,651 examples. The test data is used for measuring the prediction ca-
how a mass media (e.g., radio and television) marketing campaign pabilities of the selected data-driven model, including the most recent
could affect the buying of a new bank product. The data was collected 1293 contacts, from July 2012 to June 2013.
from an Iran bank, with a total of 22,427 customers related with a six Each record included the output target, the contact outcome ({“fail-
month period, from January to July of 2006, when the mass media cam- ure”, “success”}), and candidate input features. These include telemar-
paign was conducted. It was assumed that all customers who bought keting attributes (e.g., call direction), product details (e.g., interest rate
the product (7%) were influenced by the marketing campaign. Historical offered) and client information (e.g., age). These records were enriched
data allowed the extraction of a total of 85 input attributes related with with social and economic influence features (e.g., unemployment varia-
recency, frequency and monetary features and the age of the client. A bi- tion rate), by gathering external data from the central bank of the Portu-
nary classification task was modeled using a SVM algorithm that was guese Republic statistical web site1. The merging of the two data sources
fed with 26 attributes (after a feature selection step), using 2/3 random- led to a large set of potentially useful features, with a total of 150 attri-
ly selected customers for training and 1/3 for testing. The classification butes, which are scrutinized in Section 2.4.
accuracy achieved was 81% and through a Lift analysis [3], such model
could select 79% of the positive responders with just 40% of the cus-
tomers. While these results are interesting, a robust validation was 2.2. Data mining models
not conducted. Only one holdout run (train/test split) was considered.
Also, such random split does not reflect the temporal dimension that a In this work, we test four binary classification DM models, as implement-
real prediction system would have to follow, i.e., using past patterns to ed in the rminer package of the R tool [5]: logistic regression (LR), decision
fit the model in order to issue predictions for future client contacts. trees (DTs), neural network (NN) and support vector machine (SVM).
In this paper, we propose a personal and intelligent DSS that can au- The LR is a popular choice (e.g., in credit scoring) that operates a
tomatically predict the result of a phone call to sell long term deposits smooth nonlinear logistic transformation over a multiple regression
by using a DM approach. Such DSS is valuable to assist managers in pri- model and allows the estimation of class probabilities [33]:pðcjxk Þ ¼
oritizing and selecting the next customers to be contacted during bank
marketing campaigns. For instance, by using a Lift analysis that analyzes 1 ,where p(c|x) denotes the probability of class c given
M
1þ exp w0 þ∑i¼1 wi xk;i
the probability of success and leaves to managers only the decision on
how many customers to contact. As a consequence, the time and costs the k-th input example xk = (xk,1, …, xk,M) with M features and wi de-
of such campaigns would be reduced. Also, by performing fewer and notes a weight factor, adjusted by the learning algorithm. Due to the ad-
more effective phone calls, client stress and intrusiveness would be di- ditive linear combination of its independent variables (x), the model is
minished. The main contributions of this work are: easy to interpret. Yet, the model is quite rigid and cannot model ade-
quately complex nonlinear relationships.
• We focus on feature engineering, which is a key aspect in DM [10], and
The DT is a branching structure that represents a set of rules,
propose generic social and economic indicators in addition to the
distinguishing values in a hierarchical form [2]. This representation
more commonly used bank client and product attributes, in a total
can be translated into a set of IF–THEN rules, which are easy to under-
of 150 analyzed features. In the modeling phase, a semi-automated
stand by humans.
process (based on business knowledge and a forward method)
The multilayer perceptron is the most popular NN architecture [14].
allowed to reduce the original set to 22 relevant features that are
We adopt a multilayer perceptron with one hidden layer of H hidden
used by the DM models.
nodes and one output node. The H hyperparameter sets the model
• We analyze a recent and large dataset (52,944 records) from a Portu-
learning complexity. A NN with a value of H = 0 is equivalent to
guese bank. The data were collected from 2008 to 2013, thus including
the LR model, while a high H value allows the NN to learn complex
the effects of the global financial crisis that peaked in 2008.
nonlinear relationships. For a given input xk the state of the i-th
• We compare four DM models (LR, DT, NN and SVM) using a realistic
neuron (si) is computed by: si ¼ f wi;0 þ ∑ j∈P i wi; j s j Þ ,where Pi
rolling window evaluation and two classification metrics. We also
represents the set of nodes reaching node i; f is the logistic function;
show how the best model (NN) could benefit the bank telemarketing
wi,j denotes the weight of the connection between nodes j and i; and
business.
s1 = xk,1,, …, sM = xk,M. Given that the logistic function is used, the
The paper is organized as follows: Section 2 presents the bank data
and DM approach; Section 3 describes the experiments conducted and 1
http://www.bportugal.pt/EstatisticasWeb/Default.aspx?Lang=en-GB
24 S. Moro et al. / Decision Support Systems 62 (2014) 22–31
output node automatically produces a probability estimate (ϵ[0,1]). Given that the training data includes a large number of contacts
The NN final solution is dependent of the choice of starting weights. (51,651), we adopt the popular and fast holdout method (with R dis-
As suggested in Ref. [13], to solve this issue, the rminer package tinct runs) for feature and model selection purposes. Under this holdout
uses an ensemble of Nr different trained networks and outputs the scheme, the training data is further divided into training and validation
average of the individual predictions [13]. sets by using a random split with 2/3 and 1/3 of the contacts, respective-
The SVM classifier [4] transforms the input x ∈ ℜM space into a ly. The results are aggregated by the average of the R runs and a Mann–
high m-dimensional feature space by using a nonlinear mapping Whitney non-parametric test is used to check statistical significance at
that depends on a kernel. Then, the SVM finds the best linear sepa- the 95% confidence level.
rating hyperplane, related to a set of support vector points, in the In real environment, the DSS should be regularly updated as new
feature space. The rminer package adopts the popular Gaussian ker- contact data becomes available. Moreover, client propensity to
nel [13], which presents less parameters than other kernels (e.g., subscribe a bank product may evolve through time (e.g., changes in
polynomial): K(x, x′) = exp(−γ||x − x′||2), γ N 0. The probabilistic the economic environment). Hence, for achieving a robust predictive
SVM output is given by Ref. [35]: f(xi) = ∑ m j = 1yjαjK(xj, xi) + b evaluation we adopt the more realistic fixed-size (of length W) rolling
and p(i) = 1/(1 + exp(Af(xi) + B)), where m is the number of sup- window evaluation scheme that performs several model updates and
port vectors, yi ∈ {−1, 1} is the output for a binary classification, b discards oldest data [18]. Under this scheme, a training window of W
and αj are the coefficients of the model, and A and B are determined consecutive contacts is used to fit the model and then we perform pre-
by solving a regularized maximum likelihood problem. dictions related with the next K contacts. Next, we update (i.e., slide) the
Before fitting the NN and SVM models, the input data is first stan- training window by replacing the oldest K contacts with K newest
dardized to a zero mean and one standard deviation [13]. For DT, contacts (related with the previously predicted contacts but now we as-
rminer adopts the default parameters of the rpart R package, which im- sume that the outcome result is known), in order to perform new K pre-
plements the popular CART algorithm [2] For the LR and NN learning, dictions, and so on. For a test set of length L, a total number of model
rminer uses the efficient BFGS algorithm [22], from the family of updates (i.e., trainings) are U = L/K. Fig. 1 exemplifies the rolling win-
quasi-Newton methods, while SVM is trained using the sequential min- dow evaluation procedure.
imal optimization (SMO) [26]. The learning capabilities of NN and SVM
are affected by the choice of their hyperparameters (H for NN; γ and C, a
complex penalty parameter, for SVM). For setting these values, rminer 2.4. Feature selection
uses grid search and heuristics [5].
Complex DM models, such as NN and SVM, often achieve accurate The large number (150) of potential useful features demanded a
predictive performances. Yet, the increased complexity of NN and stricter choice of relevant attributes. Feature selection is often a key
SVM makes the final data-driven model difficult to be understood by DM step, since it is useful to discard irrelevant inputs, leading to simpler
humans. To open these black-box models, there are two interesting pos- data-driven models that are easier to interpret and that tend to provide
sibilities, rule extraction and sensitivity analysis. Rule extraction often better predictive performances [12]. In Ref. [34], it is argued that while
involves the use of a white-box method (e.g., decision tree) to learn automatic methods can be useful, the best way is to perform a manual
the black-box responses [29]. The sensitivity analysis procedure works feature selection by using problem domain knowledge, i.e., by having
by analyzing the responses of a model when a given input is varied a clear understanding of what the attributes actually mean. In this
through its domain [7]. By analyzing the sensitivity responses, it is pos- work, we use a semi-automatic approach for feature selection based
sible to measure input relevance and average impact of a particular on two steps that are described below.
input in the model. The former can be shown visually using an input im- In the first step, business intuitive knowledge was used to define a
portance bar plot and the latter by plotting the Variable Effect Charac- set of fourteen questions, which represent certain hypotheses that are
teristic (VEC) curve. Opening the black-box allows in explaining how tested. Each question (or factor of analysis) is defined in terms of a
the model makes the decisions and improves the acceptance of predic- group of related attributes selected from the original set of 150 features
tion models by the domain experts, as shown in Ref. [20]. by a bank campaign manager (domain expert). For instance, the ques-
tion about the gender influence (male/female) includes the three
2.3. Evaluation features, related with the gender of the banking agent, client and
client–agent difference (0 \ if same sex; 1 \ else). Table 1 exhibits
A class can be assigned from a probabilistic outcome by assigning a the analyzed factors and the number of attributes related with each
threshold D, such that event c is true if p(c|xk) N D. The receiver operat- factor, covering a total of 69 features (reduction of 46%).
ing characteristic (ROC) curve shows the performance of a two class In the second step, an automated selection approach is adopted,
classifier across the range of possible threshold (D) values, plotting based an adapted forward selection method [12]. Given that standard
one minus the specificity (x-axis) versus the sensitivity (y-axis) [11]. forward selection is dependent on the sequence of features used and
The overall accuracy is given by the area under the curve (AUC = that the features related with a factor of analysis are highly related,
∫10ROCdD), measuring the degree of discrimination that can be obtained we first apply a simple wrapper selection method that works with a
from a given model. AUC is a popular classification metric [21] that pre-
time
sents advantages of being independent of the class frequency or specific
false positive/negative costs. The ideal method should present an AUC of training data test data
1.0, while an AUC of 0.5 denotes a random classifier. W K
In the domain of marketing, the Lift analysis is popular for accessing model training test
the quality of targeting models [3]. Usually, the population is divided updates 1 set set
into deciles, under a decreasing order of their predictive probability K W K
for success. A useful Lift cumulative curve is obtained by plotting the 2 training test
set set
population samples (ordered by the deciles, x-axis) versus the cumula-
... ... ...
tive percentage of real responses captured (y-axis). Similarly to the AUC K W K
metric, the ideal method should present an area under the LIFT (ALIFT) U training test
cumulative curve close to 1.0. A high ALIFT confirms that the predictive set set
model concentrates responders in the top deciles, while an ALIFT of 0.5
corresponds to the performance of a random baseline. Fig. 1. Schematic of the adopted rolling window evaluation procedure.
S. Moro et al. / Decision Support Systems 62 (2014) 22–31 25
Table 2
Final set of selected attributes.
1: Interest rate nat.avg.rate National monthly average of deposit interest rate 0.781
suited.rate Most suited rate to the client according to bank criteria
dif.best.rate.avg Difference between best rate offered and the national average
2: Gender ag.sex Sex of the agent (male/female) that made (outbound) or answered (inbound) the call 0.793
3: agent experience ag.generic If generic agent, i.e. temporary hired, with less experience (yes/no) 0.799
ag.created Number of days since the agent was created
5: Client–bank relationship cli.house.loan If the client has a house loan contract (yes/no) 0.805
cli.affluent If the client is an affluent client (yes/no)
cli.indiv.credit If the client has an individual credit contract (yes/no)
cli.salary.account If the client has a salary account (yes/no)
7: Phone call context call.dir Call direction (inbound/outbound) 0.809
call.nr.schedules Number of previously scheduled calls during the same campaign
call.prev.durations Duration of previously scheduled calls (in s)
8: Date and time call.month Month in which the call is made 0.810
9: Bank profiling indicators cli.sec.group Security group bank classification 0.927
cli.agreggate If the client has aggregated products and services
cli.profile Generic client profile, considering assets and risk
10: Social and economic indicators emp.var.rate Employment variation rate, with a quarterly frequency 0.929
cons.price.idx Monthly average consumer price index
cons.conf.idx Monthly average consumer confidence index
euribor3m Daily three month Euribor rate
nr.employed Quarterly average of the total number of employed citizens
lesser features (around a factor of 7) when compared with the full fea- the domain user can select the best TPR and FPR trade-off that serves
ture approach. Also, it is also much faster (around a factor of 5) when its needs. The NN ROC curve is related with the highest area (AUC)
compared with the simple forward selection. and outperforms all other methods within most (75%) of the FPR
range (e.g., NN is the best method for FPR within [0.00,0.10],
3.2. Predictive knowledge and potential impact [0.26,0.85] and [0.94,1.00]).
Focusing on the case studied of bank telemarketing, it is difficult to
The best model from previous section (NN fed with 22 features from financially quantify costs, since long term deposits have different
Table 2, with H = 6 and Nr = 7) was tested for its predictive capabilities amounts, interest rates and subscription periods. Moreover, human
under a more realistic and robust evaluation scheme. Such scheme is agents are hired to accept inbound phone calls, as well as sell other
based on a rolling window evaluation (Section 2.3) over the test data, non-deposit products. In addition, it is difficult to estimate intrusiveness
with L = 1293 contacts from the most recent year. Taking into account of an outbound call (e.g., due to a stressful conversation). Nevertheless,
the computational effort required, the rolling window parameters were we highlight that current bank context favors more sensitive models:
fixed to the reasonable values of W = 20000 (window size) and K = 10 communication costs are contracted in bundle packages, keeping costs
(predictions made each model update), which corresponds to U = 130 low; and more importantly, the 2008 financial crisis strongly increased
model updates (trainings and evaluations). We note that a sensitivity the pressure for Portuguese banks to increase long term deposits.
analysis was executed over W, where other W configurations were test- Hence, for this particular bank it is better to produce more successful
ed (e.g., 19,000 and 21,000) leading to very similar results. For compar- sells even if this involves loosing some effort in contacting non-buyers.
ison purposes, we also tested LR, DT and SVM (as set in Section 3.1). Under such context, NN is the advised modeling technique, producing
The results of all U = 130 updates are summarized in Table 5. While the best TPR and FPR trade-off within most of the sensitive range. For
a trained model only predicts K = 10 contact outcomes (in each up- the range FPR within [0.26,0.85], the NN gets a high TPR value (ranging
date), the AUC and ALIFT metrics were computing using the full set of from 0.75 to 0.97). The NN TPR mean difference under the FPR range
predictions and desired values. Similarly to the modeling phase, the [0.45,0.62] is 2 pp when compared with SVM and 9 pp when compared
best results are given by the NN model and for both metrics, with im- with DT. For demonstrative purposes, the right of Fig. 2 shows the con-
provements of: 2.7 pp for SVM, 3.7 pp for DT and 7.9 pp for LR, in fusion matrix related with the NN model and for D = 0.5.
terms of AUC; and 1.6 pp for SVM, 2.1 pp for DT and 4.6 pp for LR, in The left of Fig. 3 plots the Lift cumulative curves for the predictions
terms of ALIFT. Interestingly, while DT was the worse performing tech- using the four models, while the right of Fig. 3 shows examples of cumu-
nique in the modeling phase, prediction tests revealed it as the third lative lift response values for the best three models (NN, SVM and DT)
best model, outperforming LR and justifying the need for technique and several sample size configurations (e.g., 10% and 50%). Under the
comparison in every stage of the decision making process [9]. cumulative lift analysis, the NN model is the best model within a large
The left of Fig. 2 plots the ROC curves for the four models tested. A portion (77%) of the sample size range. In effect, NN outperforms the
good model should offer the best compromise between a desirable SVM model for the sample size ranges of [0.06;0.24] and [0.27;0.99],
high true positive rate (TPR) and low false positive rate (FPR). The for- presenting an average difference of 2 pp within the range [0.27:0.9].
mer goal corresponds to a sensitive model, while the latter is related Also, NN is better than DT for the sample size ranges of [0,0.22],
with a more specific model. The advantage of the ROC curve is that
Table 4
Comparison of feature selection methods for the modeling phase using NN model (bold
Table 3
denotes the best AUC).
Comparison of DM models for the modeling phase (bold denotes the best value).
Method #Features Time elapsed (in s) AUC metric
Metric LR DT SVM γe ¼ 2−7:8 ; C ¼ 3 NN He ¼ 6; N r ¼ 7
no selection 150 3223 0.832
AUC 0.900 0.833 0.891 0.929a forward selection 7 97,975 0.896
ALIFT 0.849 0.756 0.844 0.878a proposed 22 18,651a 0.929
a a
Statistically significant under a pairwise comparison with SVM, LR and DT. Includes interview with domain expert (5400 s) for Table 1 definition.
S. Moro et al. / Decision Support Systems 62 (2014) 22–31 27
Table 5 data (51,651 oldest contacts). Fig. 4 exhibits the respective input impor-
Comparison of models for the rolling window phase (bold denotes the best value). tance bar plot (the attribute names are described in more detail in
Metric LR DT SVM NN Table 2). A DT was also applied to the output responses of the NN
model that was fitted with all training data. We set the DT complexity
AUC 0.715 0.757 0.767 0.794
ALIFT 0.626 0.651 0.656 0.672 parameter to 0.001, which allowed us to fit a DT as a low error, obtaining
a mean absolute error of 0.03 when predicting the NN responses. A large
tree was obtained and to simplify the analysis, Fig. 5 presents the ob-
[0.33,0.40], [0.46,0.9] and [0.96,1]. The largest NN difference when com- tained decision rules up to six decision levels. An example of an extract-
pared with DT is achieved for the sample size range of [0.46,0.9], ed rule is: if the number of employed is equal or higher than 5088
reaching up to 8 pp. Since for this particular bank and context the pres- thousand and duration of previously scheduled calls is less than
sure is set towards getting more successful sells (as previously ex- 13 min and the call is not made in March, April, October or December,
plained), this is an important sample size range. Currently, the bank and the call is inbound then the probability of success is 0.62. In Fig. 5,
uses a standard process that does not filter clients, thus involving a call- decision rules that are aligned with the sensitivity analysis are shown
ing to all clients in the database. Nevertheless, in the future there can be in bold and are discussed in the next paragraphs.
changes in the bank client selection policy. For instance, one might An interesting result shown by Fig. 4 is that the three month Euribor
imagine the scenario where telemarketing manager is asked to reduce rate (euribor3m), computed by the European Central Bank (ECB) and
the number of contacts by half (maximum of the bank's current inten- published by Thomson Reuters, i.e., a publicly available and widely
tions). As shown in Fig. 3, without the data-driven model conceived, used index, was considered the most relevant attribute, with a relative
telemarketing would reach expectedly just 50% of the possible sub- importance around 17%. Next comes the direction of the phone call (in-
scribers, while with the NN model proposed here would allow to bound versus outbound, 15%), followed by the number of days since the
reach around 79% of the responses, thus benefiting from an increase of agent login was created, which is an indicator of agent experience in the
29 percentage points of successful contacts. This result attests the utility bank contact center, although not necessarily on the deposit campaign,
of such model, which allows campaign managers to increase efficiency since each agent can deal with different types of service (e.g., phone
through cost reduction (less calls made) and still reaching a large por- banking). The difference between the best possible rate for the product
tion of successful contacts. being offered and the national average rate is the fourth most relevant
When comparing the best proposed model NN in terms of modeling input attribute. This feature was expected to be one of the top attributes,
versus rolling window phases, there is a decrease in performance, with a since it stands for what the client will receive for subscribing the bank
reduction in AUC from 0.929 to 0.794 and ALIFT from 0.878 to 0.672. deposit when compared to the competition. Along with the Euribor
However, such reduction was expected since in the modeling phase rate, these two attributes are the ones from the top five which are not
the feature selection was tuned based on validation set errors, while specifically related to call context, so they will be analyzed together fur-
the best model was then fixed (i.e., 22 inputs and H = 6) and tested ther ahead. Last in the top five attributes comes the duration of previous
on completely new unseen and more recent data. Moreover, the obtain- calls that needed to be rescheduled to obtain a final answer by the client.
ed AUC and ALIFT values are much better than the random baseline of It is also interesting to notice that the top ten attributes found by the
50%. sensitivity analysis (Fig. 4) are also used by the extracted decision
tree, as shown in Fig. 5.
3.3. Explanatory knowledge Concerning the sensitivity analysis input ranking, one may also take
into consideration the relevance of the sixth and eighth most relevant
In this section, we show how explanatory knowledge can be extract- attributes, both related to social quarterly indicators of employment,
ed by using a sensitivity analysis and rule extraction techniques the number of employees and the employment variation rate, which re-
(Section 2.2) to open the data-driven model. Using the Importance func- veal that these social indicators play a role in success contact modeling.
tion of the rminer package, we applied the Data-based Sensitivity Anal- While client attributes are specific of an individual, they were consid-
ysis (DSA) algorithm, which is capable of measuring the global influence ered less relevant, with six of them in the bottom of the input bar plot
of an input, including its iterations with other attributes [7]. The DSA al- (Fig. 4). This does not necessarily mean that these types of attributes
gorithm was executed on the selected NN model, fitted with all training have general few impact on modeling contact success. In this particular
1.0
D=0.5
0.8
NN model
(ROC point for D=0.5):
0.6
Predicted
TPR
DT
SV
LR
0.0
baseline
Fig. 2. ROC curves for the four models (left) and example confusion matrix for NN and D = 0.5 (right).
28 S. Moro et al. / Decision Support Systems 62 (2014) 22–31
1.0
Examples of DT, SVM and NN
0.8
response values:
Sample
size DT SVM NN
0.6
Responses
5% 8.0% 9.0% 10.4%
10% 15.3% 17.2% 19.2%
20% 35.1% 33.6% 35.6%
0.4
30% 52.2% 50.2% 51.3%
40% 67.7% 65.9% 67.7%
50% 77.2% 76.1% 78.9%
60% 80.0% 84.1% 86.4%
0.2
baseline
Fig. 3. Lift cumulative curves for the four models (left) and examples of NN and DT cumulative lift response values (right).
case, the profiling indicators used were defined by the bank and the ob- behavior is explained by a more recent research [30], which revealed
tained results suggest that probably these indicators are not adequate that while prior to 2008 a weak positive relation could be observed be-
for our problem of targeting deposits. tween offered rate for deposits and savings rate, after 2008, with the fi-
The sensitivity analysis results can also be visualized using a VEC nancial crisis, that relation reversed, turning clients more prone to
curve, which allows understanding the global influence of an attribute savings while the Euribor constantly decreased. This apparent contra-
in the predicted outcome by plotting the attribute range of values versus diction might be due to clients' perception of a real economic recession
the average sensitivity responses [7]. We analyzed the top five most rel- and social depression. Consumers might feel an increased need to con-
evant attributes, with the corresponding VEC curves being plotted in the sider saving for the future as opposed to immediate gratification coming
left (Euribor and product offered interest rates) and right (remaining from spending money in purchasing desired products or services. This
top 5 relevant attributes) of Fig. 6. observation emphasizes the inclusion of this kind of information on
When considering the Euribor rate, one might think that a lower similar DM projects. Concerning the difference between best product
Euribor would result in a decline in savings rate since most European rate offered and national average, Fig. 6 confirms our expectation that
banks align their deposit interest rate offers with ECB indexes, particu- an increase in this attribute does increase the probability for subscribing
larly with the three month Euribor [27]. Still, the right of Fig. 6 reveals a deposit. Still, once the difference reaches 0.73%, the influence on the
the opposite, with a lower Euribor corresponding to a higher probability probability of subscription is highly reduced, which means that an inter-
for deposit subscription, and the same probability decreasing along with est rate slightly above the competition seems to be enough to make the
the increase of the three month Euribor. A similar effect is visible in a de- difference on the result. It is also interesting to note that the extracted
cision node of the extracted DT (Fig. 5), where the probability of success DT reveals a positive effect of the rate difference with a successful con-
decreases by 10 pp when the Euribor rate is higher than 0.73. This tact (Fig. 5).
The right of Fig. 6 shows the influence of the second, third and fifth
most relevant attributes. Regarding call direction, we validate that cli-
ents contacted through inbound are keener to subscribe the deposit. A
similar effect is measured by the extracted DT, where an inbound call in-
euribor3m creases the probability of success by 25 pp (Fig. 5). Inbound is associated
call.dir
ag.created with less intrusiveness given that the client has called the bank and thus
dif.best.rate.avg he/she is more receptive for a sell. Another expected outcome is related
call.prev.durations
nr.employed with agent experience, where the knowledge extraction results show
call.month that it has a significant impact on a successful contact. Quite interesting-
emp.var.rate ly, a few days of experience are enough to produce a strong impact,
call.nr.schedules
nat.avg.rate given that under the VEC analysis with just six days the average proba-
cons.conf.idx bility of success is above 50% (Fig. 6) and the extracted DT increases the
cons.price.idx
suited.rate
probability of successful sell by 9 pp when the experience is higher or
ag.sex equal than 3.3 days (Fig. 5). Regarding the duration of previously sched-
cli.profile uled calls, it happens often that the client does not decide on the first call
ag.generic
cli.sec.group on whether to subscribe or not the deposit, asking to be called again,
cli.house.loan thus rescheduling another call. In those cases (63.8% for the whole
cli.salary.account dataset), a contact develops through more than one phone call. The sen-
cli.aggregate
cli.indiv.credit sitivity analysis (Fig. 6) shows that more time already spent on past calls
cli.affluent within the same campaign increases probability of success. Similarly,
the extracted DT confirms a positive effect of the duration of previous
0 5 10 15 20 calls. For instance, when the duration is higher or equal than 13 min
(left node at the second level of Fig. 5), then the associated global prob-
ability of success is 0.3, while the value decreases to 0.05 (25 pp differ-
Fig. 4. Relative importance of each input attribute for the NN model (in %). ence) if this duration condition is false.
S. Moro et al. / Decision Support Systems 62 (2014) 22–31 29
It is interesting to note that some explanatory variables are uncon- In this study, we propose a personal and intelligent DSS that uses a
trolled by the commercial bank (e.g., three month Euribor rate) while data mining (DM) approach for the selection of bank telemarketing cli-
others are partially controlled, i.e., can be influenced by bank managers' ents. We analyzed a recent and large Portuguese bank dataset, collected
decisions (e.g., difference between best offered and national average from 2008 to 2013, with a total of 52,944 records. The goal was to model
rates, which also depends on competitors' decisions), and other vari- the success of subscribing a long-term deposit using attributes that
ables can be fully controlled (e.g., direction of call, if outbound; agent ex- were known before the telemarketing call was executed. A particular
perience — ag.created; duration of previously scheduled calls). Given emphasis was given on feature engineering, as we considered an initial
these characteristics, telemarketing managers can act directly over set of 150 input attributes, including the commonly used bank client
some variables, while analyzing expectations influenced by uncon- and product features and also newly proposed social and economic in-
trolled variables. For instance, managers can increase campaign invest- dicators. During the modeling phase, and using a semi-automated fea-
ment (e.g., by assigning more agents) when the expected return is high, ture selection procedure, we selected a reduced set of 22 relevant
while postponing or reducing marketing campaigns when a lower suc- features. Also, four DM models were compared: logistic regression
cess is globally predicted. (LR), decision trees (DTs), neural networks (NNs) and support vector
machines (SVMs). These models were compared using two metrics,
area of the receiver operating characteristic curve (AUC) and area of
4. Conclusions the LIFT cumulative curve (ALIFT), both at the modeling and rolling win-
dow evaluation phases. For both metrics and phases, the best results
Within the banking industry, optimizing targeting for telemarketing were obtained by the NN, which resulted in an AUC of 0.80 and ALIFT
is a key issue, under a growing pressure to increase profits and reduce of 0.67 during the rolling window evaluation. Such AUC corresponds
costs. The recent 2008 financial crisis dramatically changed the business to a very good discrimination. Moreover, the proposed model has im-
of European banks. In particular, Portuguese banks were pressured to pact in the banking domain. For instance, the cumulative LIFT analysis
increase capital requirements (e.g., by capturing more long term de- reveals that 79% of the successful sells can be achieved when contacting
posits). Under this context, the use of a decision support system (DSS) only half of the clients, which translates in an improvement of 29 per-
based on a data-driven model to predict the result of a telemarketing centage points when compared with the current bank practice, which
phone call to sell long term deposits is a valuable tool to support client simply contacts all clients. By selecting only the most likely buyers, the
selection decisions of bank campaign managers. proposed DSS creates value for the bank telemarketing managers in
0.5
0.63
euribor3m (in %)
25
y (probability for subscribing the deposit)
y (probability for subscribing the deposit)
dif.best.rate.avg (in %) 21
0.6
17
12
0.4
1.37
0.5
4
0.3
2.1 call.dir
0.4
ag.created (days)
2.84 2.78 3.81 call.prev.durations (min)
1.76 Inbound
0.2
0.73
0.3
0.2
4.31
−1.32 75
5.04 0 Outbound
−2.35 0
0.1
Fig. 6. VEC curves showing the influence of the first and fourth (left) and second, third and fifth (right) most relevant attributes.
30 S. Moro et al. / Decision Support Systems 62 (2014) 22–31
terms of campaign efficiency improvement (e.g., reducing client intru- [14] S.S. Haykin, Neural networks and Learning Machines, Prentice Hall, 2009.
[15] Sadaf Hossein Javaheri, Mohammad Mehdi Sepehri, Babak Teimourpour, Response
siveness and contact costs). modeling in direct marketing: a data mining based approach for target selection,
Two knowledge extraction techniques were also applied to the pro- in: Yanchang Zhao, Yonghua Cen (Eds.), Data Mining Applications with R, Elsevier,
posed model: a sensitivity analysis, which ranked the input attributes 2014, pp. 153–178, (chapter 6).
[16] Philip Kotler, Kevin Lane Keller, Framework for Marketing Management, 5th edition
and showed the average effect of the most relevant features in the NN re- Pearson, 2012.
sponses; and a decision tree, which learned the NN responses with a low [17] Kin-Nam Lau, Haily Chow, Connie Liu, A database approach to cross selling in the
error and allowed the extraction of decision rules that are easy to inter- banking industry: practices, strategies and challenges, Journal of Database Market-
ing and Customer Strategy Management 11 (3) (2004) 216–234.
pret. As an interesting outcome, the three month Euribor rate was con- [18] William Leigh, Russell Purvis, James M. Ragusa, Forecasting the nyse composite
sidered the most relevant attribute by the sensitivity analysis, followed index with technical analysis, pattern recognizer, neural network, and genetic algo-
by the direction call (outbound or inbound), the bank agent experience, rithm: a case study in romantic decision support, Decision Support Systems 32 (4)
(2002) 361–377.
difference between the best possible rate for the product being offered
[19] David Martens, Foster Provost, Pseudo-social network targeting from consumer
and the national average rate, and the duration of previous calls that transaction data, NYU Working Papers Series, , CeDER-11-05, 2011.
needed to be rescheduled to obtain a final answer by the client. Several [20] David Martens, Foster Provost, Explaining data-driven document classifications, MIS
of the extracted decision rules were aligned with the sensitivity analysis Quarterly 38 (1) (2014) 73–99.
[21] David Martens, Jan Vanthienen, Wouter Verbeke, Bart Baesens, Performance of clas-
results and make use of the top ten attributes ranked by the sensitivity sification models from a user perspective, Decision Support Systems 51 (4) (2011)
analysis. The obtained results are credible for the banking domain and 782–793.
provide valuable knowledge for the telemarketing campaign manager. [22] M. Moller, A scaled conjugate gradient algorithm for fast supervised learning, Neural
Networks 6 (4) (1993) 525–533.
For instance, we confirm the result of Ref. [30], which claims that the fi- [23] Sérgio Moro, Raul Laureano, Paulo Cortez, Enhancing bank direct marketing through
nancial crisis changed the way the Euribor affects savings rate, turning data mining, Proceedings of the Forty-First International Conference of the
clients more likely to perform savings while Euribor decreased. More- European Marketing Academy, European Marketing Academy, 2012, pp. 1–8.
[24] David L. Olson, Dursun Delen, Yanyan Meng, Comparative analysis of data mining
over, inbound calls and an increase in other highly relevant attributes methods for bankruptcy prediction, Decision Support Systems 52 (2) (2012)
(i.e., difference in best possible rate, agent experience or duration of pre- 464–473.
vious calls) enhance the probability for a successful deposit sell. [25] Robert Phillips, Optimizing prices for consumer credit, Journal and Review Pricing
Management 12 (2013) 360–377.
In future work, we intend to address the prediction of other telemar- [26] John Platt, Sequential minimal optimization: a fast algorithm for training support
keting relevant variables, such as the duration of the call (which highly vector machines, Technical Report MSR-TR-98-14, , Microsoft Research, 1998.
affects the probability of a successful contact [23]) or the amount that is [27] Gerard O. Reilly, Information in financial market indicators: an overview, Quarterly
Bulletin Articles 4 (2005) 133–141 (Central Bank of Ireland).
deposited in the bank. Additionally, the dataset may provide history
[28] Roland T. Rust, Christine Moorman, Gaurav Bhalla, Rethinking marketing, Harvard
telemarketing behavior for cases when clients have previously been Business Review 1 (2010) 1–8.
contacted. Such information could be used to enrich the dataset (e.g., [29] R. Setiono, Techniques for extracting classification and regression rules from artifi-
computing recency, frequency and monetary features) and possibly cial neural networks, in: D. Fogel, C. Robinson (Eds.), Computational Intelligence:
The Experts Speak, IEEE Press, 2003, pp. 99–114.
provide new valuable knowledge to improve model accuracy. Also it [30] P. Stinglhamber, Ch. Van Nieuwenhuyze, M.D. Zachary, The impact of low interest
would be interesting to consider the possibility of splitting the sample rates on household financial behaviour, Econometric Reviews 2 (2011) 77–91
according to two sub-periods of time within the range of 2008–2012, (National Bank of Belgium).
[31] Fabrice Talla Nobibon, Roel Leus, Frits CR Spieksma, Optimization models for
which would allow us to analyze impact of hard-hit recession versus targeted offers in direct marketing: exact and heuristic algorithms, European Jour-
slow recovery. nal of Operational Research 210 (3) (2011) 670–683.
[32] Efraim Turban, Ramesh Sharda, Dursun Delen, Decision Support and Business Intel-
ligence Systems, 9th edition Pearson, 2011.
Acknowledgments [33] W. Venables, B. Ripley, Modern Applied Statistics with S, 4th edition Springer, 2003.
[34] Ian H. Witten, Eibe Frank, Data Mining: Practical Machine Learning Tools and Tech-
niques, 2nd edition Morgan Kaufmann, 2005.
We would like to thank the anonymous reviewers for their helpful
[35] T.F. Wu, C.J. Lin, R.C. Weng, Probability estimates for multi-class classification by
suggestions. pairwise coupling, Journal of Machine Learning Research 5 (2004) 975–1005.
References
Sérgio Moro holds a 5 year degree diploma in Computer En-
[1] David Arnott, Graham Pervan, Eight key issues for the decision support systems dis- gineering from Instituto Superior Técnico (Technical Univer-
cipline, Decision Support Systems 44 (3) (2008) 657–672. sity of Lisbon), and a MSc in Management Information
[2] Leo Breiman, Jerome H. Friedman, Richard A. Olshen, Charles J. Stone, Classification Systems from ISCTE — University Institute of Lisbon, where
and Regression Trees, Wadsworth & Brooks, Monterey, CA, 1984. he is also a PhD researcher in the fields of business intelli-
[3] David S. Coppock, Why lift? Data modeling and mining, Information Management gence and decision support systems. He currently works in
(2002) 5329-1 2002, (Online; accessed 19-July-2013). the Business Intelligence & Analytics team of a Portuguese
[4] C. Cortes, V. Vapnik, Support vector networks, Machine Learning 20 (3) (1995) bank, where he is responsible for projects such as GRC (Gov-
273–297. ernance, Risk and Compliance), and ALM (Asset–Liability
[5] Paulo Cortez, Data mining with neural networks and support vector machines using Management). Previously he was responsible for software
the r/rminer tool, Advances in Data Mining. Applications and Theoretical Aspects, development for the bank's contact center.
6171, Springer, 2010, pp. 572–583.
[6] Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, José Reis, Modeling
wine preferences by data mining from physicochemical properties, Decision Sup-
port Systems 47 (4) (2009) 547–553.
[7] Paulo Cortez, Mark J. Embrechts, Using sensitivity analysis and visualization tech- Paulo Cortez is an Associate Professor at the Department of
niques to open black box data mining models, Information Sciences 225 (2013) 1–17. Information Systems at University of Minho and Coordinator
[8] Dursun Delen, A comparative analysis of machine learning techniques for student of the Information Systems and Technologies R&D group of
retention management, Decision Support Systems 49 (4) (2010) 498–506. ALGORITMI Research Centre. He completed his PhD (2002)
[9] Dursun Delen, Ramesh Sharda, Prajeeb Kumar, Movie forecast guru: a Web-based in Computer Science and Habilitation (2013) in Information
DSS for Hollywood managers, Decision Support Systems 43 (4) (2007) 1151–1170. Systems and Technologies at the same university. His re-
[10] Pedro Domingos, A few useful things to know about machine learning, Communica- search interests include: business intelligence, data mining,
tions of the ACM 55 (10) (2012) 78–87. neural networks, evolutionary computation and forecasting.
[11] Tom Fawcett, An introduction to roc analysis, Pattern Recognition Letters 27 (8) Currently, he is associate editor of the journals Expert Sys-
(2006) 861–874. tems and Neural Processing Letters. He has published more
[12] Isabelle Guyon, André Elisseeff, An introduction to variable and feature selection, than 92 indexed (ISI or Scopus) papers. His research has
Journal of Machine Learning Research 3 (2003) 1157–1182. appeared in Journal of Heuristics, Decision Support Systems,
[13] T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data Min- Information Sciences and others (see http://www3.dsi.
ing, Inference, and Prediction, 2nd edition Springer-Verlag, NY, USA, 2008. uminho.pt/pcortez).
S. Moro et al. / Decision Support Systems 62 (2014) 22–31 31