Utilizing MS SQL Server Tools (DTS and SSIS)

for Data Exchange between PI and RDBMS

White Paper

Page: 1
Introduction..................................................................................................3
Presented Scenarios..........................................................................................................................................................3
Scenario 1 - PI Data Export to Flat File(s) via DTS.....................................4
Introduction to DTS..........................................................................................................................................................4
DTS Packages.......................................................................................................................................................4
DTS Tasks.............................................................................................................................................................4
DTS Transformations...........................................................................................................................................4
DTS Package Workflow.......................................................................................................................................4
DTS Connections..................................................................................................................................................4
Scenario 1 - Configuration Steps......................................................................................................................................5
Create New DTS Package............................................................................................................................................5
Create first DTS Connection using PI OLEDB............................................................................................................6
Set PI OLEDB Defer Execution property to True:.......................................................................................................7
Create a second DTS connection using the Text File Provider....................................................................................8
Specify DTS Transformation........................................................................................................................................9
Package Execution......................................................................................................................................................12
Scenario 2 - PI Data Export to Flat File(s) Using SSIS..............................13
Introduction to SSIS.......................................................................................................................................................13
Integration Services Packages............................................................................................................................13
SSIS Connections...............................................................................................................................................13
Control Flow Elements.......................................................................................................................................13
Data Flow Elements...........................................................................................................................................13
SSIS Variables, Event Handlers and Log Providers...........................................................................................13
Scenario 2 - Configuration Steps....................................................................................................................................14
Create a new Integration Services Project in Visual Studio 2005..............................................................................14
Create the Data Flow..................................................................................................................................................15
Specify Source Connection........................................................................................................................................16
Create the Destination connection..............................................................................................................................18
Execute the package...................................................................................................................................................19
Scenario 3 - E-Mail Data Exported from PI via DTS..................................20
Scenario 3 - Configuration Steps....................................................................................................................................20
Scenario 4 – Data Import from MS ISA Server to PI..................................22
Scenario 4 - Configuration Steps....................................................................................................................................22
Create Data Flow and Configure Connections...........................................................................................................22
Transformation to Unicode.........................................................................................................................................23
Execution....................................................................................................................................................................24
Conclusion......................................................................................................................................................................24

Page: 2
Introduction
This document focuses on copying and transforming PI data to and from OLE DB compliant data-sources
utilizing tools from the Microsoft SQL Server’s data integration environment. Today, nearly every database vendor
offers tools that facilitate data import/migration to their data stores. The well-known databases like SQL Server
(Microsoft), Oracle, DB2 (IBM), Sybase - all have incorporated migration tools allowing users to easily bring data into
their systems. One of the most popular Relational Database Management Systems (RDBMS) is the Microsoft SQL
Server. Moreover, with the recent release of this product - SQL Server version 2005, Microsoft came with an RDBMS
that can be utilized by nearly everybody - home users as well as big enterprises. In this White Paper we will therefore
concentrate on scenarios that employ the OLE DB provider for the PI Server – PI OLEDB, which can be configured and
used within the aforementioned Microsoft’s SQL Server RDBMS environment. We will present the scenarios in the
form of a number of commented examples that, as we do believe, can provide useful hints applicable in today’s
ubiquitous heterogeneous environments. In addition, and in compliance with today’s trend to minimize the end-user’s
own coding, we will focus only on those of the variety of possible solutions, which do not require any programming
knowledge. Such an approach is usually more generic and the resulting application often better maintainable and
extendable.

Presented Scenarios
Even though the PI System is not a genuine relational database, thanks to the PI OLEDB data provider, one can
transform and ‘publish’ PI data in the form of relational objects (i.e. catalogs, tables and views). PI OLEDB can thus be
considered a ‘gateway’ through which the PI Server’s data can be accessed from within any environment that
‘understands' OLE DB and SQL language. MS SQL Server 2000 Data Transformation Services (DTS) or SQL Server
2005 Integration Services (SSIS) both do; and both leverage the relational database standards to such an extent that
developing sophisticated data migration/ transformation tasks requires only limited or even no coding effort.

The following scenarios will be presented:

Scenario 1 - PI Data Export to Flat File(s) via DTS

Scenario 2 - PI Data Export to Flat File(s) via SSIS

Scenario 3 - E-Mail Data Exported from PI via DTS

Scenario 4 - Data Import from MS ISA Server to PI via SSIS

Page: 3
Scenario 1 - PI Data Export to Flat File(s) via DTS

Introduction to DTS
Data Transformation Services (DTS) is part of Microsoft SQL Server 2000. DTS is a set of graphical tools and
programmable objects that let you extract, transform, and consolidate data from disparate data sources into single or
multiple destinations. DTS is based on OLE DB standards and, as such, can connect to any OLE DB compliant data
source. As DTS can also work with text files, exporting and transforming of PI data is thus only matter of configuration
and specification of an appropriate SQL query. Before we start, a short overview of the used terminology:

DTS Packages
DTS package is an organized collection of connections, DTS tasks, DTS transformations, and workflow constraints.

DTS Tasks
DTS task is a discrete set of functionality, executed as a single step in a DTS package. DTS task can for example be:
 Data import and export
 Data transformation
 Database objects copying
 Sending and receiving messages to and from other users’ packages
 Executing a set of Transact-SQL statements or Microsoft ActiveX scripts against a data source

DTS Transformations
DTS data transformation task is one or more functions or operations applied against a piece of data before the data
arrives at the destination. The source data is not changed.

DTS Package Workflow

You can define the sequence of execution steps in a package – a DTS workflow.

DTS Connections
DTS is based on OLE DB architecture that allows copying and transforming data from a variety of sources;
for example:
 SQL Server and Oracle, using OLE DB providers,
 ODBC sources, using the Microsoft OLE DB Provider for ODBC
 Access 2000, Excel 2000, Microsoft Visual FoxPro, dBase, Paradox
 Text files, using the built-in DTS flat file provider
 Microsoft Exchange Server, Microsoft Active Directory and other non-relational data sources.
 Other data sources provided by third-party vendors, such as PI OLEDB

Note: For more information, please refer to SQL Server 2000 BooksOnLine, chapter "DTS Overview" or,
for more complex info about SQL Server and DTS, open the following link:
http://msdn.microsoft.com/sql/2000/default.aspx

Page: 4
Scenario 1 - Configuration Steps
The PI OLEDB data provider exposes nearly a complete collection of PI databases and can therefore be used within
DTS to access PI archive data, PI point database, PI batch database, PI module database, etc. In this example, we will
export the SELECTed PI archive data to a flat file.
The first step is to establish the corresponding connections – a connection to the PI Server through PI OLEDB, and a
flat file connection (by simply providing the file path and name). The second step is to specify the SQL query and any
data transformation - i.e. we can optionally modify the result-set of an SQL query (that retrieves the data from PI
through PI OLEDB) so that the final structure in the flat file ‘better fits’ our requirements. The transformation is
optional, and if not specified, the file structure will reflect the result-set obtained by executing the SELECT statement.
The data will thus be stored into the file ‘as SELECTed’.
In the following paragraphs we describe step-by-step the aforementioned DTS related tasks. The configuration
environment will be MS SQL Server 2000 Enterprise Manager:

Create New DTS Package

Figure 1. Microsoft SQL Server 2000 Enterprise Manager, Create New DTS Package

Page: 5
Create first DTS Connection using PI OLEDB

Figure 2. DTS Connection Object using PI OLEDB.

While in DTS Package design mode, select the “Other Connection” item from the Connection menu and point to the
“PI OLEDB Provider” (if installed, is listed among the existing OLE DB providers on a machine). The Properties
button brings you to the window where you specify the PI Server (in the “Data Source” edit box) as well as the
authentication credentials (PI username and password).

Page: 6
Set PI OLEDB Defer Execution property to True:

Figure 3. Setting PI OLEDB Property “Defer Execution” to True

The PI OLEDB specific property “Defer Execution” is needed to workaround the SQL Server behavior which
automatically executes a query on individual tables without a WHERE clause (see the Note below).

Note: To retrieve the metadata, DTS opens cursors on all tables involved in the given transformation.
Because PI OLEDB just emulates the server-side cursor on the client side, executing a query without
WHERE clause would copy all the data from the individual table to the client, even if they are finally
not retrieved. To avoid this very expensive operation, always set the Defer Execution initialization
property to true.

Page: 7
Create a second DTS connection using the Text File Provider

Figure 4. DTS Connection for Destination (OLE DB Provider for Text Files)

Repeat the step depicted on Figure 2, just select the OLE DB provider for text files “Text File (Destination)”.
The Properties button now allows for setting of the “Text File (Destination)” specific properties like column and row
delimiters etc. This will influence the data format in the destination text file.

Note: Make sure that the specified directory exists. The file itself is created by the DTS during the package
execution, but the directory structure must be there; the text file path is not checked during DTS
package configuration.

Page: 8
Specify DTS Transformation
Go to the Task menu (or Task toolbar), click on the “Transform Data Task” and continue first pointing to the source
and second to the destination DTS objects (connections). You will get an arrow (representing the “Data Transformation
Task”) that connects the two DTS objects together:

Figure 5. DTS Transformation between Source and Destination Objects

Double click the arrow and specify the relevant properties:

a) In the Source Tab, specify the following PI OLEDB SQL query as:

SELECT tag, time, value, status FROM piarchive..picomp2 WHERE

time > '*-1h' AND tag = 'SINUSOID'

Page: 9
Figure 6. SQL Query

Note: Be aware that when you specify the Table/View as source for the DTS Transformation Task (see the
radio-button in Figure 6.), DTS executes a SELECT statement without any WHERE condition
(against the specified table). This step is potentially very expensive, especially with tables from the
piarchive catalog.

b) Selecting the Destination Tab allows to specify the structure of the data in the destination Text File. In this very
simple case, the Text File structure will correspond to the result-set of the aforementioned query.

Page: 10
Figure 7. Destination File Data Structure

Note: The destination text file structure reflects the results of the source SQL query, which means that
when you change the source SQL statement, it is recommended you also re-generate the
destination file structure by opening the “Transform Data Task Properties” dialog, then select the
Destination Tab and click “Define Columns” -> “Populate From Source” -> “Execute” buttons.

Page: 11
c) The Transformations Tab specifies the actual transformations. In our simple example, DTS derives the source ->
destination column relation automatically and assigns a default Transformation type, which is “Copy Column”.
In DTS, there are several Transformation types at your disposal. More about individual transformation types you can
find in the DTS help. At this place we just shortly mention one useful transformation – “Data Time String”:
Press the New button (in the Transformation Tab) and continue selecting the “Data Time String” transf. type; you get
the following dialog where you can specify the desired format for the time column:

Figure 8. Date Time String Transformation

d) The Lookups tab allows utilizing the “Lookup Queries”.

e) The Options tab enables for better logging (exception handling) and additional data format specification (e.g. using
some other-than-default row delimiters).

Package Execution

Go to Package menu and click Execute and verify the results.

Note: After successful DTS package execution, it may happen that there are trailing zeroes in the sub-
second part of the Timestamp column. To avoid it, cast to string:
SELECT CAST(Time AS String)...
See the PI OLEDB manual for more about casting.
Finally, the package execution can be automated using the SQL Server Agent so that you can execute it on a periodical
basis.

Page: 12
Scenario 2 - PI Data Export to Flat File(s) Using SSIS
The principles and goals of this scenario will resemble the DTS Scenario 1. We will show the configuration steps in
the MS SQL Server 2005 environment, will utilize the .Net Provider for OLE DB (source connection) and send data
again to a flat file (destination connection). All other steps will be very similar.

Introduction to SSIS
SQL Server 2005 Integration Services (SSIS) are part of the recently released Microsoft SQL Server 2005. SSIS is
the DTS successor for building ‘high performance data integration solutions’, utilizing the Extraction, Transformation
and Load (ETL) packages as its basic building blocks. SSIS introduces many new features like the fully programmable
object model, enhanced debugging etc. Moreover, SSIS integrates with the “Visual Studio 2005” environment, which
brings the SSIS packages development onto a ‘entirely new level’. As in Scenario 1, we start with a brief summary of
the terminology used:

Integration Services Packages

A package (also called the ETL package) is an organized collection of connections, control flow elements, data flow
elements, event handlers, variables and configurations. The ETL package is the basic unit of work.

SSIS Connections
They connect to different types of data sources. The following list depicts just a couple of supported data sources:
 Native OLE DB providers– e.g. SQL Server, Oracle, PI OLEDB, Analysis Services, Data Mining Services,
OLAP, DTS Packages, etc
 .Net Providers for OLE DB – wrap the native OLE DB providers and ‘open the managed world’ for them.
 .Net Providers – SQL Server, Oracle, ODBC
 Excel
 Flat Files
 FTP, HTTP (File Transfer Protocol, Hyper Text)
 MSMQ (Message queue)
 ODBC (Open DataBase Connectivity)
 SMTP (Simple Mail Transfer Protocol)

Control Flow Elements

A package consists of a number of control flow elements and, optionally one or more data flows. There are three
different types of control flow elements:
 containers define the structure of packages
 tasks implement the functionality
 precedence constraints connect the executables, containers, and tasks into an ordered control flow

Data Flow Elements

There are three data flow element types: sources, transformations, and destinations. These element types allow for
building the data flows in an ETL package.

SSIS Variables, Event Handlers and Log Providers

SSIS variables can be used in expressions to dynamically update column values and property expressions or to control
the execution. Event handlers handle the run-time events (e.g. errors). Log providers support package logging.

Note: Have a look at the SSIS detailed help in the MS SQL Server BooksOnline or open the following link:
http://www.microsoft.com/sql/default.mspx

Page: 13
Scenario 2 - Configuration Steps
In the following paragraphs we will describe step-by-step scenario 2 related tasks. As we have already mentioned,
the configuration environment is MS Visual Studio 2005:

Create a new Integration Services Project in Visual Studio 2005.

Figure 9. New SSIS project in VS 2005

Page: 14
Create the Data Flow

Select the Data Flow pane, drag the “OLE DB Source” item and the “Flat File Destination” item to the design area of
the Data Flow pane. The aforementioned SSIS items (OLE DB and Flat File), each can be found in the corresponding
list of the Toolbox pane (Data Flow Sources and Data Flow Destinations). Finally, connect the source with the
destination.

Figure 10. Data Flow Pane with source and Destination SSIS Items

Page: 15
Specify Source Connection

Double click the OLE DB Source box in the Data Flow pane and click the New button. The connection manager dialog
reveals (see the picture below). Select the “Native OLE DB\PI OLEDB Provider” (from the topmost drop-down-list)
and provide the "Server or file name" (PI Server Name) and the "User name" and "Password" (check the "Allow saving
password" check box):

Figure 11. SSIS Connection Manager

Hit the Data Links button and select the All tab; again (as depicted on Figure 3 in scenario 1, set the “Defer Execution
Property” to True.

Page: 16
Set the “Data access mode” to “SQL Command” and specify the following SQL query as:

SELECT tag, time, CAST(value AS Float32) value_float32, status FROM

piarchive..picomp2 WHERE time > '*-1h' AND tag = 'SINUSOID'

Figure 12. PI OLEDB query.

Note: There is no equivalent for the Variant (COM) data type in the “.NET” world. As some of the PI
OLEDB tables do use Variants columns, the SSIS converts them to strings. In cases where the
Variant subtype is known, we highly recommend to use the CAST function. The query used in the
example demonstrates that.

Page: 17
Create the Destination connection

Double click the “Flat File Destination” box in Data Flow pane and click the New button. The “Flat File” connection
manager will show up (see the picture below). Select "Delimited format", set the "Connection Manager Name" and fill
in the “File name”. The flat file columns and their data types will be automatically derived from the PI OLEDB SQL
query.

Figure 13. SSIS Flat File Connection Manager

By select Mappings in the “Flat File Destination Editor” you can see the default column mappings that have been
created. This default mapping can be modified.

Page: 18
Execute the package.

The Data Flow Task is now ready for execution. Select “Start Debugging” either by clicking on the toolbar icon or in
“Debug" menu. The package should run without errors:

Figure 14. SSIS Data flow Package Execution

Should any runtime error appear, the corresponding SSIS item becomes red and VS 2005 provides you with a complete
error description.

Finally, you can see the ‘SELECTed' data (during debugging) by specifying a data viewer. Double click the arrow and
specify how you would like to see the rows. The Grid viewer for example looks e.g. as follows:

Figure 15. SSIS Grid Viewer

Page: 19
Scenario 3 - E-Mail Data Exported from PI via DTS
In this example, we will configure a DTS task that automates sending reports (results of SELECT queries
executed against the PI OLEDB provider) to e-mail recipients. This task is again fully achievable by configuring and
connecting a couple of DTS tasks. The resulting DTS package can then be scheduled on a periodical basis (via the SQL
Server Agent) to send reports to appropriate recipients.
As in scenarios 1or 2, we will describe step-by-step what DTS tasks need to be employed. The configuration
environment will be the MS SQL Server 2000 Enterprise Manager, nevertheless the same can be achieved using SSIS
and VS 2005.

Scenario 3 - Configuration Steps

The “Send Mail Task” in DTS can be provided with an attachment. Because our scenarios (1 and 2) export the
SELECTed PI data in the ASCII file form, we can very easily extend them so that we just drag the “Send Mail Task”
into the DTS pane and assign the extracted file as the e-mail attachment:

Figure 16. Sending e-mails with Attached PI Data

Page: 20
The “Send_Mail” task is easily configurable as it just gets the attachment file and the e-mail address:

Figure 17. e-Mail Attachment specification

The dashed green arrow in Figure 16 specifies the workflow. I.e. - data has to be extracted and consequently e-mailed.
The send e-mail task can further be made dependent on the success/failure of the previous steps (data extraction to the
file).

Page: 21
Scenario 4 – Data Import from MS ISA Server to PI
The Microsoft Internet Security and Acceleration Server - ISA 2004 Server can be configured to create logs of
network activity of the computer where it runs. These logs can either be used for reporting purposes or for
troubleshooting network issues. You have the option to configure what and where this information will be stored: into a
file, into an ODBC compliant data source, or into the Microsoft Data Engine (MSDE). The last one is the default
storage. As these log entries are time stamped, we can transfer the ISA log info to PI utilizing PI OLEDB and SQL
Server 2005 Integration Services. The ISA log data, stored in corresponding PI tags, can later be exploited for any
analysis using PI client tools.

Scenario 4 - Configuration Steps

Create Data Flow and Configure Connections

For configuring the task in SSIS, we need a new project for it. Refer to Scenario 2, start with the first two steps
(create a new SSIS project and drag the two “OLE DB Data Flow” items into the pane). The data flow direction
(compared to Scenario 2) are reverted. Data will go from a foreign data source (in our case - SQL Server) to PI. We
again start with the connections; the following picture shows how to configure the SSIS Connection object for SQL
Server:

Figure 18. Connection to SQL Server Instance that Stores ISA Logs

The source SQL query can look as follows:

Page: 22
SELECT 'Martin-VisitedURL' as Tag, logTime as Time, CAST(uri as varchar(512)) as
Value from WebProxyLog WHERE clientIP = 3232235555 WHERE logTime > GETDATE()-1

Note: The WebProxyLog table stores rows sent from the MS ISA Server 2004. Please refer to ISA Server
help or inspect the following links: http://www.microsoft.com/isaserver/default.mspx or
www.isaserver.org

Fill the above stated SQL query into the “SQL command text” edit box in the “OLE DB Source editor” dialog (see
Figure 12). The query gets data from the ‘uri’ column that stores URLs opened by a specific IP address. Because we
plan to copy the data to the PI tag “Martin-VisitedURL”, we use such an alias in the SELECT column list. Obviously
such tag named “Martin-VisitedURL” must exist in the PI Server. We want to run the query once a day, therefore the
WHERE clause limits the amount of SELECTed rows using GETDATE()-1; in PI terminology this translates into ‘*-
1d’.

The second connection is a link to PI (see Picture 12 in scenario 2). In PI (PI OLEDB), we need to create a view that
CASTs the value column of the picomp2 table to WString. Use e.g. the PI OLEDB tester (an application that is installed
along with the PI OLE DB provider) and run the following query:

CREATE VIEW piarchive.picomp_x as SELECT tag, time, CAST(value AS WString)

value_wstring FROM piarchive.. picomp2

Transformation to Unicode

We cannot create direct mappings between the SELECTed columns from the SQL Server table and the above PI view
because SSIS mandates an exact data type match between the mapped columns. We therefore need to employ another
SSIS object – the “Data Conversion” item. This way we convert all input string columns to Unicode:

Page: 23
Figure 19. Data Conversion SSIS Item

Double clicking the “Data Conversion” box you can assign the desired data type to the appropriate columns. In our case
we changed the data type of the input columns “Tag” and “Value” to “Unicode string [DT_WSTR]”.

Execution

Press ‘F5’ to run the package. As described in Scenario 2, MS Visual Studio 2005 gives you plenty of possibilities how
to debug and tune the package. After successful execution, the data SELECTed from SQL Server should end up in PI
tags. Moreover, we recommend taking a look at SQL Server’s task-scheduling possibilities (via SQL Server Agent).
SQL Server 2005 environment offers various options how to automate the package execution, and the SQL Server
BooksOnline is an immense source of useful information.

Note: Scheduling the SSIS package execution through the aforementioned SQL Server Agent, we get a
scenario that closely resembles the RDBMSPI interface operation. Nevertheless, such scenario can
be recommended only for batch-like bulk-data transfer, because the presented concept lacks
features necessary for the reliable, continuous service-like operation. The RDBMSPI interface has
sophisticated error handling, deals with ODBC link disconnections, exploits the PI data buffering,
implements a number of various retrieval strategies etc. and therefore remains OSIsoft’s premier tool
for interfacing relational databases.

Conclusion
We have shown just a small fraction of the capabilities of both integration environments – DTS as well as SSIS. The
core strength of both products is the orientation on data related tasks that can be accomplished by configuring objects
which honor the IT standards like OLE DB, SQL, etc. Moreover, SSIS integrates with VS 2005 – a development
environment, until now used solely by programmers. In combination with SSIS, the VS 2005 environment now gets
opened to ordinary users. Users, who know what they need and, until now, couldn’t achieve it without a programmer…

Even though the aforementioned examples present just simple use-cases, SSIS shows how Microsoft envisions tools for
the data integration; We therefore recommend: keep experimenting - as we are sure, you will find other useful scenarios
for the export/import of your PI data.

Page: 24