See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220933956

A model-driven framework for ETL process development

Conference Paper · October 2011

DOI: 10.1145/2064676.2064685 · Source: DBLP

CITATIONS READS

59 2,462

4 authors:

Zineb El Akkaoui Esteban Zimanyi

Institut National des Postes et Télécommunications Université Libre de Bruxelles
9 PUBLICATIONS 195 CITATIONS 244 PUBLICATIONS 3,358 CITATIONS

SEE PROFILE SEE PROFILE

Jose-Norberto Mazón Juan Trujillo

University of Alicante University of Alicante
135 PUBLICATIONS 1,883 CITATIONS 298 PUBLICATIONS 4,755 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

educational data mining View project

BPMN4ETL project View project

All content following this page was uploaded by Juan Trujillo on 22 May 2014.

The user has requested enhancement of the downloaded file.

 A Model-Driven Framework for ETL Process Development

Zineb El Akkaoui, Esteban Zimányi Jose-Norberto Mazón, Juan Trujillo

Dept. of Computer and Decision Engineering Dep. of Software and Computing Systems
Université Libre de Bruxelles, Belgium University of Alicante, Spain
zelakkao,ezimanyi@ulb.ac.be jnmazon,jtrujillo@dlsi.ua.es

ABSTRACT ically heterogeneous and distributed, and ETL processes

ETL processes are the backbone component of a data ware- should therefore cope with these aspects and ensure the ex-
house, since they supply the data warehouse with the neces- pected data quality required by the data warehouse. There-
sary integrated and reconciled data from heterogeneous and fore, the ETL process development is complex, prone-to-fail,
distributed data sources. However, the ETL process devel- and time consuming [8, 15, 18]. Actually, ETL process de-
opment, and particularly its design phase, is still perceived velopment constitutes the most costly part of a data ware-
as a time-consuming task. This is mainly due to the fact that house project, in both time and resources, as stated by a
ETL processes are typically designed by considering a spe- recent Forester Research report1 . Even more, this report ar-
cific technology from the very beginning of the development gues that the complexity of integration solutions continues
process. Thus, it is difficult to share and reuse methodolo- to grow, with higher-quality data and more-robust metadata
gies and best practices among projects implemented with and auditability requirements.
different technologies. To the best of our knowledge, no at-
tempt has been yet dedicated to harmonize the ETL process One of the main causes of this increased complexity is that,
development by proposing a common and integrated devel- in practice, ETL processes are still designed by considering
opment strategy. To overcome this drawback, in this paper, a specific vendor tool from the very beginning of the devel-
a framework for model-driven development of ETL processes opment process, which raises multiple drawbacks. On the
is introduced. The benefit of our framework is twofold: (i) one hand, these tools have a steep learning curve, due to
using vendor-independent models for a unified design of ETL the important effort required to assimilate the manipulated
processes, based on the expressive and well-known standard platform philosophy, e.g., the underlying languages with a
for modeling business processes, the Business Process Mod- wide field of functionality assortments, nonstandard ETL
eling Notation (BPMN), and (ii) automatically transforming development scenarios, complex wizards, etc. On the other
these models into the required vendor-specific code to exe- hand, the design of an ETL process using a vendor-specific
cute the ETL process into a concrete platform. tool is usually performed through a Graphical User Interface
(GUI). Thus, the generated design tends to be proprietary
1. INTRODUCTION since it can only be executed within the tool, and no avail-
A data warehouse is a “collection of integrated, subject- able mechanisms allow to seamlessly generate its executable
oriented databases designated to support the decision mak- code for other platforms.
ing process” [3]. By integrating different data sources, a data
warehouse allows to provide a complete and accurate view First attempts to improve this situation consider a con-
on an organization’s operational data, synthesized into a set ceptual modeling stage for developing ETL processes in a
of strategic indicators, or measures, which can be analyzed vendor-independent manner [7, 13, 16]. These proposals
according to axes of interest, or dimensions. Therefore, a improve the development of ETL processes, since they al-
data warehouse enables the production of fast and efficient low documenting the ETL processes and supporting the de-
business reports from operational data sources. signer tasks. However, they lack effective mechanisms for
automatically generating vendor-specific code to execute the
The cornerstone component that supplies the data ware- ETL process into a concrete platform.
houses with all the necessary data is the ETL (Extraction/
Transformation/ Load) process. It is worth mentioning that To overcome this problem, the present work proposes a Model-
the data sources that populate a data warehouse are typ- Driven Development (MDD) framework for ETL processes.
This framework aims at covering the overall ETL develop-
ment process, including the automatic generation of vendor-
specific code for several platforms. Fig. 1 shows a con-
ceptual overview of our approach and how it is integrated
with existing ETL platforms. Vendor-independent models of
ETL processes are defined using BPMN4ETL, a platform-
independent design model of ETL processes based on the
Business Process Model Notation (BPMN). From this model
DOLAP’11, October 28, 2011, Glasgow, Scotland, UK. 1
http://www.forrester.com
 MDD adapting them to their requirements. Another inter-
Layers
esting advantage, is the ability to predefine dedicated
Vendor- BPMN4ETL
Independent ETL subprocesses for a frequent usage through the
Design
company, regardless to the deployed technologies. For
T
OMB C# example, an ETL subprocess can be the computation
C# Editor &
Vendors
patterns for common strategic indicators or cleansing
OMB Plus Compiler SSIS patterns suitable to the company data.
OWB Designer
Vendor- Design Center
Specific APIs
APIs • Interoperability: using a common design, ETL process
APIs
models with distinct paradigms can be combined in
APIs
m order to prompt inter-communication or data exchange
yste
ros oft S between ETL processes.
m Mic
Text-based yste
S
cle
Ora
Graph-based
As a second contribution, our model-driven process provides
a set of model-to-text transformations to be automatically
Modeling executed in order to obtain code for different ETL tools from
Type models. These transformations induce a set of benefits:
Figure 1: MDD approach for ETL process development
• Transformations patterns: transformations are orga-
nized into technology-independent templates. There-
can be derived a vendor-specific implementation tailored to fore, extending our framework by adding new target
several technologies, e.g., Oracle Warehouse Builder (OWB) ETL tools, can be easily performed by obtaining the
or SQL Server Integration Services (SSIS). corresponding code for each template. Since this step
can be semi-automated, programming details are reused
For creating and managing ETL processes, in addition to the and high expertise is not more necessary.
traditional graphical languages, these technologies generally
provide programming capabilities by a fourth-generation lan- • Validation: the user has the possibility to examine and
guage (4GL). This 4GL can be either a procedural language ameliorate the generated code before its execution.
(e.g. Oracle Metabase or OMB) or a procedural language
• Performance: since not all the integration issues are
(e.g. C#). Transformations between a vendor-independent
best solved by the same tool, the designer should be
model and this vendor-specific code are formally established
able to chose the best implementation for each issue.
by using model-to-text transformations2 , an OMG standard
Each module can then be implemented in a particular
for transformations from models to text.
tool. Moreover, modularity allows sharing the work
among several engines, and the correct combination of
The present framework relies on our previous work as fol-
the whole results may increase execution performance
lows. First, the BPMN4ETL metamodel for designing ETL
using the validation asset explained below.
processes described in [2] has been improved to be used as
the vendor-independent part of our framework. Second, the • Code quality improvement: the framework should en-
model-driven approach has been customized from a generic able code enhancement by using the best practices and
data warehouse approach [5] into a concrete and appropriate lesson learned when designing ETL processes.
implementation for the ETL component. Hence, our frame-
work is motivated by the high-expressiveness of the BPMN
graphical language allowing to design any kind of ETL pro- The remainder of this paper is organized as follows. After
cesses, as well as by the performance of the MDD technolo- briefly reviewing the related literature in Section 2, Section 3
gies which mainly enhance the automatic implementation thoroughly describes the proposed MDD-based framework.
into any executable code. More precisely, the contributions In Section 4, we propose a vendor-independent pattern for
of the present work are as follows. developing the model-to-text transformations. Then, we
show how this transformation pattern can be customized
Our first contribution is a vendor-independent metamodel and implemented for the Oracle platform in Section 5. Sec-
for designing ETL processes that allows designers to focus tion 6 concludes the paper and discusses future work.
on the ETL semantics instead of wasting efforts in imple-
mentation issues. By using our metamodel the following 2. RELATED WORK
advantages can be highlighted: Existing approaches for developing ETL process highlight
two main axes: (i) designing ETL processes independently
of a specific vendor, and then (ii) implementing into an ex-
• Reusability: the same ETL process design can be eas-
ecutable code tailored to a certain technology.
ily reused independently of the specific ETL technol-
ogy. This constitutes a strong benefit since it avoids
On the one hand, several authors [6, 17] argue that ETL pro-
losing time in developing the same process multiple
cesses can be adequately designed by a workflow language.
times. For example, departments of the same com-
Furthermore, other approaches [9, 10] recommend the use
pany can easily exchange designed ETL processes by
of ontologies for describing data sources and targets, thus
2
http://www.omg.org/spec/MOFM2T/1.0/ providing the necessary information for solving an adequate
ETL process design by inference. Other approaches as [11] Vendor-Indepedent Vendor- Specific
apply graph operation rules in order to guide the ETL de- Design Implementation

sign steps, also by using ontologies. Unfortunately, building

Layer M3
such ontologies is a tedious task since it requires a high cor- MOF EBNF
rectness and a blow-by-blow description of the data stores.
Therefore, we advocate the use of a rich workflow language
that provides various enhancements for the designer work Tool 4GL
Behavior
without requiring the definition of any ontology. Behavioral
Packages
Grammar

On the other hand, the implementation of the ETL design Expression

has been discussed from many points of view. For example, Packages

Layer M2
Tool 4GL
an attempt has been concerned about the optimization of Expression
the physical ETL design through a set of algorithms [14]. Grammar
Resource
Also, a UML-based physical modeling of the ETL processes Packages
was introduced by [4]. This approach formalizes the data
storage logical structure, and the ETL hardware and soft-
ware configurations. Although both works deal with inter- M2T Transformation
esting adjacent issues about the implementation topic, they
mainly miss to produce a code for executing ETL processes.

Layer M1
Paradoxically, an exclusive ETL programming approach us- ETL Process ETL Process
M2T Application
Model Tool Code
ing the Python language has been claimed by [12]. Yet,
this approach omits to supply an independent-vendor de-
sign which decreases the reusability and easy-of-use of the

System
Vendor-Independent Vendor-Specific
provided framework. System System

Finally, other works propose to take into account both ETL

development axes: designing and code generation. For ex- Figure 2: MDD layers for the ETL development framework
ample, a conceptual metamodel for designing ETL processes
based on BPMN, and an approach for generating its cor-
responding BPEL code is described in [2]. Also, some au- The Model Instance layer (M0) is a representation of the
thors propose a Model-Driven Architecture approach for de- real-world system where the ETL process design and im-
signing the ETL processes by means of the UML Activity plementation are intended to perform. This may be repre-
Diagram, and then obtaining a representation of the ETL sented, respectively, by a vendor-independent graphical in-
process through specific-vendor metamodels defined manu- terface and by a vendor-specific ETL engine. At the Model
ally, [5] . However, both works lack in a wider support for layer (M1), both the ETL Process Model is designed and the
more than one concrete implementation platform. ETL process code is derived by applying a set of transforma-
tions, thus moving from the design to the implementation.
Hence, the current work constitutes an extension of these The Meta-Model layer (M2) consists of the BPMN4ETL
two previous proposals [2, 5]. It refines the metamodel in [2] metamodel that defines ETL patterns at the design phase,
and disseminates the code generation into any ETL pro- and a 4GL grammar at the implementation phase. Finally,
gramming language. Further, it improves the transforma- the Meta-Meta-Model level (M3), corresponds to the MOF
tion proposed in [5] by directly obtaining code correspond- meta-metamodel at the design phase, while it corresponds to
ing to the target platform, bypassing the laborious task of the Backus Naur Form (BNF) at the implementation phase.
defining an intermediate representation of the target tool.
In the following, we focus on the two M1 and M2 layers
since they describe the ETL semantics. We describe both
3. MODEL-DRIVEN FRAMEWORK layers by means of the small data warehouse example shown
As already said, the present work addresses the ETL process in Fig. 3. In this example, two types of data sources feed
development using a Model-Driven Development (MDD) ap- the data warehouse with the necessary data (see Fig. 3): a
proach. In this section, we concretely show how this ap- database and a flat file. These sources are based on the
proach allows to organize the various components of this relational and the recordset paradigms, respectively, for ta-
framework in order to efficiently perform the design and im- bles and files, as defined in the Common Warehouse Meta-
plementation phases of the ETL process development. model4 . The database contains the Product and Category ta-
bles, where each product belongs to a particular category. A
3.1 MDD-Based Framework similar information is provided by the Product Category flat
MDD is an approach to software development where exten- file source, with the difference that this information about
sive models are created before source code is written. As products and their categories is combined within a same
shown in Fig. 2, the MDD approach defines four main layers schema. The target data warehouse contains a simple fact
(see Meta-Object Facilities (MOF)3 ): the Model Instance table Category Fact and its product hierarchy, as displayed
layer (M0), the Model layer (M1), the Meta-Model layer in Fig. 3b. The product hierarchy is composed of both the
(M2), and the Meta-Meta-Model layer (M3). Category Dimension and the Product Dimension tables.

3 4
http://www.omg.org/spec/MOF/2.0/ http://www.omg.org/spec/CWM/1.1/
 Also, the data from the file should be filtered so as to drop
Product Category Product_Category
null Product ID records, since there is no way to validate
Product_ID Category_ID P_C_ID this constraint within a file, in contrast with a table. Then,
Product_Name Category_Name Product_ID
both flows from the database and the file can be merged
UPrice Product_Name
Category_ID UPrice
in order to compute the target measures for the fact table,
the AvgPrice. Finally, a key lookup is included to check the
referential integrity between the processed data and the old
(a) data in the Category Fact table, before its loading.

It is worth mentioning that each task can be configured by

Category_Dimension Product_Dimension Category_Fact customizing its behavior in order to fulfill the design objec-
Category_SK Product_SK Product_SK tive. For example, the extraction and loading tasks hold
Category_Name Product_Name AvgPrice many configurations, such as the loading strategies and the
UPrice UPrice automatic data type mapping. Also, the transformations
Category_SK
are composed of various computations, branching and merg-
(b) ing tasks, and each of these categories can be customized
through a set of parameters, like conditions, computations
Figure 3: (a) Data source schemas; (b) Data warehouse tar- and input/output cardinalities. For simplicity, such config-
get schema. urations are not displayed in Fig. 4.

3.2.2 BPMN4ETL Metamodel (M2)

3.2 Vendor-Independent Design DataEvent InputSet 0..* Expression
This section describes the vendor-independent design side
of the framework shown in Fig. 2. This part comprises the name name name
0..* 0..1
creation of the ETL model based on the set of constructs eventHandling type oclBody
provided by the BPMN4ETL metamodel. 0..* successors() 0..*

0..1 0..1
3.2.1 BPMN4ETL Model (M1) DataSubProcess 0..1 DataTask 0..1 OutputSet
Product Category Product_Category 0..* name
DB Input DB Input File Input 0..* taskType
name type
Conversion 1..*
DConversion 1
Loop DataProcess
P_C_Join Null_Category
Join Filter Condition name

P_C_Merge
Merge Figure 5: Excerpt of the data process package

Avg_Price As we have said, the example ETL process in Fig. 4 is de-

Aggregation
signed according to the BPMN4ETL metamodel, shown in
SK_Lookup Fig. 5. The metamodel is mainly based on BPMN, a de facto
Key Lookup standard notation for modeling business processes. BPMN
provides a common and understandable framework for de-
Category_Fact scribing business processes independently of the used tools.
DB Output BPMN allows users to express various pertinent features
about processes, such as events allowing the process to in-
Figure 4: ETL process model for the Category Fact load teract with the external world and container s enabling to
structure the business process and tracing its boundaries.
The ETL model that provides the example data warehouse
with the necessary data from the data sources is depicted in Our BPMN-based metamodel is structured in several pack-
Fig. 4. This process is mainly composed of tasks of multi- ages. The root package is the ControlProcess, which captures
ple types: starting from the extractions (e.g., DB Input and the whole ETL process populating a data warehouse through
File Input), passing through the transformations (e.g., Fil- multiple DataProcesses. Fig. 5 shows an excerpt of the data
ter, Join, ...) and ending by the loading (e.g., DB Output) process package, a core part of ETL processes.
into the data warehouse. In the example, the data from the
Category table requires the creation of a new field using the Typically, a DataProcess depicts the complete data flow from
variable derivation VDerivation task. Then, the product and a set of data sources to a target data warehouse. It in-
category data can be joined together. On the other hand, ex- volves multiple DataTasks belonging to various taskTypes,
tracting data from a file generates fields of a character data such as a join, filter, merge, variableDerivation, etc. Also, the
type. Hence, a typical behavior when dealing with files con- DataEvent class enables the user to custom the exception
sists of converting these character types into their original handling for each data task. Moreover, the subsequence of
data type, depicted as a DConversion task in our example. data tasks sharing the same objective or similar character-
istics may be organized through subprocesses, represented BOUND TO TABLE ’PRODUCT’

by the DataSubProcess class. The flows incoming from and OMBALTER MAPPING ’CATEGORY FACT MAP’ \
ADD TABLE OPERATOR ’CATEGORY’ \
outgoing to a data task are respectively represented by the BOUND TO TABLE ’CATEGORY’
InputSet and OutputSet classes. The Expression class cap-
OMBALTER MAPPING ’CATEGORY FACT MAP’ \
tures the expressions applied by a data task on its input ADD FLAT FILE OPERATOR ’PRODUCT CATEGORY’ \
BOUND TO FLAT FILE ’ /ETLTOOL/PRODUCT CATEGORY’
sets in order to produce its output set.
#TRANSFORMATIONS
#JOIN
Additional packages of our metamodel are Condition, Com- OMBALTER MAPPING ’CATEGORY FACT MAP’ \
ADD JOINER OPERATOR ’ P C JOIN ’
putation and Query, representing different types of expres- SET PROPERTIES ( JOIN CONDITION )
VALUES ( ’ INGRP1.CATEGORY ID = INGRP2.CATEGORY ID ’ )
sions. Other packages capture fundamental aspects about
#CONVERSION
data storage in ETL processes. The Resource package rep- OMBALTER MAPPING ’CATEGORY FACT MAP’ \
resents the data resources from/to which data is extract- ADD EXPRESSION OPERATOR ’CONVERSION’ \
ALTER ATTRIBUTE ’ P C ID ’ OF GROUP
ed/loaded, and the Intermediate Data Store (IDS) package, ’ INGRP1 ’ OF OPERATOR ’CONVERSION’ \
SET PROPERTIES ( DATATYPE,PRECISION,SCALE )
represents the stagging area for storing non-persistent data VALUES ( INTEGER,30,0 ) \
...
participating in the ETL process.
#FILTER
OMBALTER MAPPING ’CATEGORY FACT MAP’ \

3.3 Vendor-Specific Implementation ADD FILTER OPERATOR ’NULL CATEGORY’ \

SET PROPERTIES ( FILTER CONDITION ) \
VALUES ( ’PRODUCT CATEGORY.CODE CATEGORY \
In this section, we highlight the implementation components I S NULL ’ )
of the framework, i.e., the ETL process code and the 4GL #MERGE
grammar. We use Oracle as example target system. OMBALTER MAPPING ’CATEGORY FACT MAP’ \
ADD SET OPERATION OPERATOR ’MERGE’ \
SET PROPERTIES (SET OPERATION) \
VALUES ( ’ UNION ’ )
3.3.1 Code (M1) #AGGREGATION
The Oracle MetaBase (OMB) scripting language for Ora- OMBALTER MAPPING ’CATEGORY FACT MAP’ \
ADD AGGREGATOR OPERATOR ’AVGPRICE’ \
cle Warehouse Builder (OWB) can be assimilated to SQL SET PROPERTIES ( GROUP BY CLAUSE, HAVING CLAUSE) \
VALUES ( ’ INGRP1.CATEGORY CODE’ , \
language for the Oracle database. A snippet of the OMB ’ INGRP.CATEGORY ID<>NULL ’ )
code corresponding to the Category Fact load process can ...

be seen in Listing 1. Creating a data process (MAPPING) in #ADD CONNECTIONS BETWEEN TASKS
OMBALTER MAPPING ’CATEGORY FACT MAP’ \
OMB requires first to create a project and to define a mod- ADD CONNECTION FROM GROUP ’OUTGRP1’ \
OF OPERATOR ’ JOIN ’ TO GROUP ’ INGRP1 ’ \
ule where to locate the mapping. Also, LOCATIONs for the OF OPERATOR ’MERGE’ BY NAME
resources used by the mapping should be established. Then, ...

the mapping is defined with its data tasks (OPERATORs),

and matched to each other using connections.
3.3.2 Programming Grammar (M2)
In our example, the merge task is created using the ADD Grammars are formalisms for defining (programming) lan-
SET OPERATION OPERATOR command. The task proper- guages in the same way that metamodels represent the lan-
ties are configured using the SET VALUES command, e.g., guage of models. As shown in Fig. 2, in our framework we
the type property of the merge task has the value a UNION. used the Extended Backus-Naur Form (EBNF) for defining
Then, this task is linked to the join task using the ADD the OMB and C# grammars. Thus, the OMB grammar de-
CONNECTION command. Connecting operators in OMB is fines the syntax structure for valid OMB statements. For in-
done by connecting each attribute of the operators inputs stance, in order to generate the merge task statement shown
and outputs GROUPs, or by connecting all the input or in Listing 1, the part of grammar used during the production
output attributes simultaneously. For instance, the merge is shown in Listing 2.
operator has two input connections incoming from the join
and filter operators, and one output connection ongoing to Listing 2: OMB Grammar Snippet
the aggregation operator. alterMappingCommand = OMBALTER MAPPING ”mappingName ”
”alterMapDetailClause ”
mappingName = ”QUOTED STRING”
Note finally that additional code is needed in order to ex- a l t e r M a p D e t a i l C l a u s e = ADD ” a d d O p e r a t o r C l a u s e ”
a d d O p e r a t o r C l a u s e = ” o p e r a t o r T y p e ” OPERATOR ” o p e r a t o r N a m e ”
ecute the mappings, such as the creation of the hosting [ SET ” s e t P r o p e r t i e s C l a u s e ” ]
operatorType = ”UNQUOTED STRING”
project, the source connections, and the deployment code. operatorName = ”QUOTED STRING”
This code can be created semi-automatically since it is com- s e t P r o p e r t i e s C l a u s e = PROPERTIES ” p r o p e r t y K e y L i s t ” VALUES
”propertyValueList ”
mon among all data processes and independent from the propertyKeyList = ”( ” ”propertyKey ” { ” , ”
”propertyKey ” } ”) ”
ETL model to implement. propertyValueList = ”( ” ”propertyValue ” { ” , ”
”propertyValue ” } ”) ”
propertyKey = ”UNQUOTED STRING”
Listing 1: OMB code for our Category Fact load propertyValue = ( ”QUOTED STRING” | ”INTEGER LITERAL ” |
”FLOATING POINT LITERAL ” )

OMBCREATE ORACLE MODULE ’ STAGING TABLES ’

OMBCREATE LOCATION ’MY LOCATION’

SET PROPERTIES ( TYPE, VERSION,
DESCRIPTION, BUSINESS NAME )
4. TRANSFORMATIONS
VALUES ( ’ORACLE DATABASE’ , ’ 1 0 . 2 ’ , Transformations are used for automatically generating the
’ this is a location ’ , ’ location ’)
ETL code from an input ETL model. In this section, we
OMBCREATE MAPPING ’CATEGORY FACT MAP’
define the transformations as matching statements between
#DATA INPUT TASKS
OMBALTER MAPPING ’CATEGORY FACT MAP’ \
the metamodel and the grammar at the M2 layer, in order
ADD TABLE OPERATOR ’PRODUCT’ \ to be executed at the M1 layer. At the same time, we show
how the abstract patterns for such transformations provide Eclipse is a software development environment comprising
useful guidance when developing code generators for new an extensible plug-in system. It provides the Eclipse Model-
ETL tools. Our approach uses OMG’s model-to-text (M2T) ing Framework (EMF), which comprises a number of model-
standard for expressing the transformations. driven development capabilities like modeling, inter-model
transformations, and code generation. More precisely, EMF
4.1 M2T Transformations allows the definition of models and metamodels by means of
M2T organizes transformations in a set of templates, respon- the Ecore tools. In practice, metamodels are created using
sible for generating code for the input model elements. Each the Ecore meta-metamodel, which is the implementation of
template contains the code portions corresponding to a sin- MOF in Eclipse. From these metamodels it is possible to
gle metamodel concept. This code may be static or dynamic. generate the genmodel, which is a set of Java classes that
Static code is replicated literally during the execution. Dy- represent each metamodel concept. The genmodel is useful
namic code corresponds to OCL expressions specified using for building dedicated tools, such as editors, to create and
the metamodel concepts. These expressions are computed manage the models corresponding to our metamodel.
on the input model during execution. In practice, the in-
put model elements are sequentially iterated, and for each As shown in Fig. 7, the BPMN4ETL metamodel has been
element, the corresponding template code is generated and implemented by Ecore. Also, in order to guarantee the
appended to an output file. correctness of BPMN4ETL models created by users, the
metamodel implementation contains two types of validation.
Thus, the transformations between the BPMN4ETL meta- Firstly, it exploits EMF default diagnostic methods for eval-
model and the OMB grammar are performed by means of uating the syntactic conformance of the created models with
templates. However, the template code depends on target their metamodel. Secondly, we have build specific validation
ETL tool. For this reason, we show next how an abstract rules in order to guarantee the semantics correctness. For
level for the transformation templates can be useful, since it example, some validation rules that we have implemented for
specifies the required templates, and describes their common a data task element are: no InputSet for data input tasks,
processing order across ETL tools. no OutputSet data output tasks, and at least one connec-
tion for each data task (see Fig. 7). Therefore, based on
4.2 Transformation Patterns this metamodel implementation, Eclipse enables to build an
Since ETL code for different software tools follows a com- editor for creating valid ETL models.
mon strategy, we predefined patterns of abstract templates,
such as the one shown in Fig. 6. These patterns provide Further, EMF provides model-to-text transformations sup-
to the developer a better view of the typical order in tem- port through Acceleo, the Eclipse tool for programming code
plate building. Then, the code can be semi-automatically generators that follows OMG’s model-to-text specification.
generated from the tool. Similarly to the object-oriented In Acceleo, each set of related transformations is called a
programming paradigm, each abstract template can be as- module. Each module may contain one or several transfor-
similated to an abstract class, which can be implemented mation templates.
by customizing its code. Finally, these patterns may be
Listing 3: Concrete templates code snippet for OMB
adapted to the potential dissimilarities between program-
ming languages, as it will be discussed in Section 5. [ template p u b l i c addDProcess ( d p r o c e s s : DataProcess ) ]
#C r e a t i o n o f t h e c o r r e s p o n d i n g mapping t o t h e
#d a t a p r o c e s s
OMBCREATE MAPPING ’ [ d p r o c e s s . n a m e . t o S t r i n g ( ) / ] ’ \
The abstract pattern highlighted in Fig. 6 shows the cre- [ f o r ( t : DataTask | d p r o c e s s . d a t a T a s k s ) ]
[ t.addTask ( ) / ]
ation of new code from an ETL model. Other patterns ad- [/ for ]
dress maintaining and managing this code. This pattern [/ template ]

algorithm depicts the following steps: It starts by creating [ t e m p l a t e p u b l i c addDTask ( t : DataTask ) ]

#### Add t h e t a s k [ t . n a m e / ] t o t h e mapping ####
the main control process template addNewCP and then iter- OMBALTER MAPPING ’ [ t . d a t a P r o c e s s . n a m e / ] ’
[ i f ( t.taskType.toString () = ’ DataInput ’ ) ]
atively creating its components. A main component is the [ t.addDITask ( ) / ]
data process. Thus, for each existing data process, a new [/ i f ]
[ i f ( t.taskType.toString () = ’ VariableDerivation ’ ) ]
empty template is created addNewDP with all connection [ t.addVDTask ( ) / ]
[/ i f ]
templates corresponding to the involved resources addCon- [ i f ( t.taskType.toString () = ’ Join ’ ) ]
[ t.addJoTask ( ) / ]
nection. Note that this template is applied for each resource [/ i f ]
related to the data process depicted with the foreach loop. [ i f ( t.taskType.toString ()
[ t.addDOTask ( ) / ]
= ’ DataOutput ’ ) ]

The data tasks templates addDTask are thereupon obtained [/ i f ]

[ i f ( t.taskType.toString () = ’ Merge ’ ) ]
according to their types, and they are linked to each other [ t.addMeTask ( ) / ]
[/ i f ]
matchDTasks in the same order of the input model. Simi- ...
larly, other control tasks templates of the control processes [/ template ]
...
are created and matched. Finally, subprocess, loop, and [ template public convertCondition ( cc : Condition ) ]
[ i f cc.cCondition.oclIsUndefined ()]
event features are added to the process components. [ cc.sCondition.convertSCondition ( ) / ] [ / i f ]
[ i f not ( c c . c C o n d i t i o n . o c l I s U n d e f i n e d ( ) ) ]
[ cc.cCondition.convertCondition ()/]
5. APPLYING TRANSFORMATIONS [ cc.sCondition.convertSCondition ( ) / ] [ / i f ]
[/ template ]
This section describes the Eclipse-based implementation of
our framework, specifically, the code generation. It empha-
sizes an excerpt of the transformation code that should be Listing 3 shows an excerpt of the templates for transforming
applied to the ETL model in Fig 4 in order to get the cor- a data process, some data tasks and a conditional expression
responding OMB code. into OMB code. Further, each abstract template includes
 foreach r : Resource
addKLTask useConnection

ConvertCond.

addVGTask addSeqTask

addCProcess addNewCP addRandTask

foreach dp : DataProcess <<inherit>>

addGFTask ConvertCond.
addDProcess addNewDP

<<Use>> foreach ct : ControlTask foreach r : Resource

addOFTask ConvertCond.
addCTask addConnection
<<Use>> foreach r : Resource
foreach dt : DataTask
matchCTasks useConnection
addDTask addDITask

ConvertQuery
addSubprocess matchDTasks
addVDTask ConvertComput.

addJTask ConvertCond.

addDOTask useLocation

ConvertQuery

Template relationships Use: a template means replace by the used template with creation of the current template
Template execution order Inherent: from a template means to replace without creating the current template

Figure 6: Abstract templates pattern for ETL process creation

the necessary code to be generated by the transformation. Fig. 4 can be designed differently. Both tasks can be com-
For example, the abstract template addDProcess depicted in bined and mapped to the cube operator in the OWB, instead
Fig. 6 is implemented by the Acceleo template addDProcess, of a direct mapping to a lookup and a table operators such
an excerpt of which is defined in Listing 3 and can be seen as those proposed by our approach.
in the Acceleo editor in Fig. 7.
6. CONCLUSIONS AND FUTURE WORK
Concretely, the template addDProcess creates a new data Even though data warehouses are used since the early 1990s,
process and calls the templates for creating all the entailed the design and modeling of ETL processes is still accom-
data tasks. Indeed, depending on the type of the operator plished in a vendor-dependent way by using tools that al-
taskType, a specific template is applied, e.g. addJoTask and low to specify them according to the schemas represented in
addMeTask which aims, respectively, at adding a join and a concrete platforms. Therefore, their design, implementation
merge tasks. The final template displays a recursive tem- and even more their maintenance must be done depending
plate which is responsible of transforming a combined con- on the target platform.
dition into the expression language of OMB, i.e., SQL code.
The combined condition is represented in our metamodel in In this paper, we have provided, to the best of our knowl-
a recursive tree structure. With regards to this structure, edge, the first modeling approach for the specification of
the transformation template calls the simple condition tem- ETL processes in a vendor-independent way and the auto-
plate convertCondition in the left part of the generated code, matic generation of its corresponding code in commercial
and recall the combined condition template for its right part. platforms. This is done thanks to the expressiveness of our
BPMN-based metamodel and to the model-driven develop-
When running the resulted OMB code generated by the ment capabilities, provided by Eclipse and Acceleo, in code
transformations of our example and then opening the OWB generation.
Design Center, it is possible to verify the conformance of the
resulted process with the initial ETL model. Our approach considers Oracle and Microsoft as represen-
tative target implementation and execution tools, although
It is worth mentioning that some custom templates, not only Oracle’s solution has been described in this paper for
mentioned in the abstract template pattern, may be added space reasons. Moreover, it offers some new techniques in
during the implementation. These templates differ among order to guide developers in building other code generators
ETL tools, thus they need to be specified for each target for new platforms.
tool. Further, the generated code is closely linked to how
the transformations are programmed. In fact, the trans- Currently, our framework covers the design and implemen-
formation algorithm through the abstract template pattern tation phases of the ETL process development. One future
viewed in Section 4 is the most direct way to create the ETL work expects to extend this framework in order to handle
code. However, a more optimized code can be generated by the whole ETL development life-cycle, i.e., by involving the
using specific ’macros’ provided by tools. For instance, the analysis phase as well. This work should enhance the quality
tasks of type Lookup and DB Input in the example process in of the generated code by our approach by means of: i) en-
 Figure 7: Acceleo implementation

hancing the transformation implementation using some ex- conceptual design of ETL processes for both
isting knowledge-based techniques, and ii) by building the structured and semi-structured data. International
necessary metrics for proposing the ’best’ implementation Journal on Semantic Web and Information Systems,
related to a certain ETL process. 3(4):1–24, 2007.
[11] D. Skoutas, A. Simitsis, and T. Sellis. Ontology-driven
conceptual design of ETL processes using graph
7. REFERENCES transformations. In Journal on Data Semantics XIII,
[1] J. Bézivin. On the unification power of models. number 5530 in LNCS, pages 122–149. Springer, 2009.
Software and System Modeling, 4(2):171–188, 2005.
[12] C. Thomsen and T. Pedersen. pygrametl: A powerful
[2] Z. El Akkaoui and E. Zimányi. Defining ETL programming framework for extract-transform-load
worfklows using BPMN and BPEL. In DOLAP’09, programmers. In DOLAP’09, pages 49–56. ACM
pages 41–48. ACM Press. Press.
[3] W. Inmon. Building the Data Warehouse. Wiley, 2002. [13] J. Trujillo and S. Luján-Mora. A UML based approach
[4] S. Luján-Mora and J. Trujillo. Physical modeling of for modeling ETL processes in data warehouses. In
data warehouses using UML. In DOLAP’04, pages ER’03, LNCS 2813, pages 307–320, 2003. Springer.
48–57, 2005. ACM Press. [14] V. Tziovara, P. Vassiliadis, and A. Simitsis. Deciding
[5] J. Mazón and J. Trujillo. An MDA approach for the the physical implementation of ETL workflows. In
development of data warehouses. Decision Support DOLAP’07, pages 49–56, 2007. ACM Press.
Systems, 45(1):41–58, 2008. [15] P. Vassiliadis, A. Simitsis, and E. Baikous. A
[6] A. Simitsis. Mapping conceptual to logical models for taxonomy of ETL activities. In DOLAP’09, pages
ETL processes. In DOLAP’05, pages 67–76, 2005. 25–32. ACM Press.
ACM Press. [16] P. Vassiliadis, A. Simitsis, P. Georgantas,
[7] A. Simitsis and P. Vassiliadis. A methodology for the M. Terrovitis, and S. Skiadopoulos. A generic and
conceptual modeling of ETL processes. In CAiSE’03 customizable framework for the design of ETL
Workshops, pages 305–316, 2003. CEUR Workshop scenarios. Information Systems, 30(7):492–525, 2005.
Proceedings. [17] P. Vassiliadis, A. Simitsis, and S. Skiadopoulos.
[8] A. Simitsis and P. Vassiliadis. A method for the Conceptual modeling for ETL processes. In
mapping of conceptual designs to logical blueprints for DOLAP’02, pages 14–21, 2002. ACM Press.
ETL processes. Decision Support Systems, [18] L. Wyatt, B. Caufield, and D. Pol. Principles for an
45(1):22–40, 2008. ETL benchmark. In TPCTC’09, LNCS 5895, pages
[9] D. Skoutas and A. Simitsis. Designing ETL processes 183–198, 2009. Springer.
using semantic web technologies. In DOLAP’06, pages
67–74, 2005. ACM Press.
[10] D. Skoutas and A. Simitsis. Ontology-based

View publication stats