BizTalk Technical Document

BizTalk Technical Document

Prepared for

NALCO
Thursday, 1 December 2011
Version 1.0
Prepared by

MPS Partners
Contributors
Ashish Talati

Prepared for NALCO

The information contained in this document represents the current view of MPS Partners, LLC. on the issues discussed as of the date of publication. Because
MPS Partners must respond to changing market conditions, it should not be interpreted to be a commitment on the part of MPS Partners, and MPS Partners
cannot guarantee the accuracy of any information presented after the date of publication.
MPS PARTNERS MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be
reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or
otherwise), or for any purpose, without the express written permission of MPS Partners, LLC..
MPS Partners may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document.
Except as expressly provided in any written license agreement from MPS Partners, the furnishing of this document does not give you any license to these
patents, trademarks, copyrights, or other intellectual property.
The descriptions of other companies products in this document, if any, are provided only as a convenience to you. Any such references should not be considered
an endorsement or support by MPS Partners. MPS Partners cannot guarantee their accuracy, and the products may change over time. Also, the descriptions are
intended as brief highlights to aid understanding, rather than as thorough coverage. For authoritative descriptions of these products, please consult their
respective manufacturers.
2010 MPS Partners, LLC.. All rights reserved. Any use or distribution of these materials without express authorization of MPS Partners is strictly prohibited.
MPS Partners and Windows are either registered trademarks of MPS Partners, LLC. in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Page ii
BizTalk Technical Document

Revision and Signoff Sheet

Change Record
Date

Author

Version

Change reference

10/18/201
1

Ashish Talati

1.0

Created

11/30/201
1

Ashish Talati

1.1

Completed Cooling Controller

Reviewers
Name

Version
approved

Position

Date

Page
BizTalk Technical Document

Table of Contents
1

Environment.....................................................................................1

1.1

Overview............................................................................................................ 1

1.2

Installation Summary......................................................................................... 1

Implementation................................................................................2

2.1

Overview............................................................................................................ 2

2.2

TFS..................................................................................................................... 3

2.3

Cooling Controller Process.................................................................................3

2.4

Schemas Cooling Controller............................................................................ 4

2.4.1 Cooling Schema.......................................................................................... 4
2.4.1 Cooling Property Schema............................................................................6
2.4.1 Cooling Alarm Schema................................................................................7
2.4.1 Cooling Apache Schema............................................................................. 7
2.4.1 Record Schema........................................................................................... 8

2.5

Scenario Components...................................................................................... 10
2.5.1 Cooling Data Receipt................................................................................ 10
2.5.2 Helper Components.................................................................................. 13
2.5.3 Cooling Processor..................................................................................... 14
2.5.4 Testability.................................................................................................. 16

3
3.1

Monitoring and Exception Handling..................................................19

Tracking & Message Monitoring.......................................................................19
3.1.1 BizTalk....................................................................................................... 19

Page
BizTalk Technical Document

Environment

1.1 Overview
The NALCO BizTalk proof of concept did not plan for performance testing as a
success criteria, so a highly available environment was not created. Two
environments were created for the POC:

Development Workstation: Used by developers to create and test the

solution.

Developer Integration Server: Solution was deployed to this server for testing
purposes. This environment is typically were multiple developers code is
integrated and tested before moving on to a QA process.

The development workstation had the following software installed:

Windows 7

IIS 7

SQL Server 2008 R2

Visual Studio 2010

BizTalk Server 2010

The development integration server had the following software installed:

Windows 2008 R2 Enterprise

IIS 7

SQL Server 2008 R2

Visual Studio 2010

BizTalk Server 2010

1.2 Installation Summary

Because each of the environments were installed with single server envionrments,
the installations were very basic and followed the installation guides from Microsoft.
The following guides were used:
Development Workstation:
Installing BizTalk Server 2010 on Windows 7 and Windows Vista.docx
Development Integration Server:
Installing BizTalk Server 2010 on Windows Server 2008 R2 and 2008.docx
Page 1
BizTalk Technical Document

Implementation

2.1 Overview
The following is the high level process when a Boiler or Cooling controller Data file
( either in email format or an XML Format ) is received:

File Receipt:
o

File is Archived

Data is parsed and Converted to Schema

Optionally if file is compressed, uncompress it

Component adds context information into message Header

Mapper component: transforms file to canonical

Schema Validator

Validation & Enrichment

o

Business validation

Business Process
o

Cooling Data Process

Cooling Alarms Process

Boiler Data, Event, Alarms Process

Web for All Data and Configuration Process

Monitoring
o

Monitoring Orchestration Performance

This document will now describe each component of the scenario in a little more
detail to show how this was implemented using BizTalk Server.

Page 2
BizTalk Technical Document

2.2 TFS
All source code are stored in TFS @ $/3DT Web/WPS Biztalk Projects
A separate BizTalk application and solution are created for each Parser. Every
solution is following same naming convention based on type of artifacts for BizTalk.
The default naming convention is that the name of a folder containing a BizTalk
Project is identical to projects output DLL name ( without extension ). For example,
if BizTalk project resides in Cooling.Controller.Orchestrations folder, then that project
must output a DLL named Cooling.Controller.Orchestrations.dll
Typical example of project naming structure is
Orchestration = Project.Orchestraions
Pipelines = Project.Pipelines
Pipeline Components = Project.PipelineComponents
Helper Components = Project.Components
Schemas = Project.Schema ( or Project.Generated.Schemas )
Maps/Transformaton = Project.Transform
Unit Tests = Project.Tests

2.3 Cooling Controller Process

High Level Process

Cooling Alarms, Cooling Data, Cooling Apache email messages are picked up
by BizTalk Receive Location

Receive Pipeline component

o

Parses email Header and store that information into context properties

Parses email body and using BizTalk flat file disassembler component
will convert Flat file into XML ( for apache and data ). For alarm parser
XML file is created with appropriate email body as received
Page 3
BizTalk Technical Document

Filters out SPAM message ( by looking into email header information as

well as content in email body )

Invoke Archive Process to archive data based on business

requirement

Few Business processes are done in pipeline component ( Message

logging, saving email Subject Line, inserting blacklisted alarms )

Pipeline component will drop either apache, alarm or data XML message to
BizTalk MessageBox

Alarms Data

Alarms Orchestration/Workflow will pickup the message and perform

business processes ( Parse Alarms - .net SP Call ) and saving debug
data ( Debug2 )

For Noflow, Interlock or NDM alarms, it will call Alarms recurring

Workflow

Email Data
o

Workflow will find out ControllerId based on controller header data and
insert data into ParameterValuesN or ParameterValuesT for each
section data received for controller

Apache Data
o

Workflow will execute business process by calling parseApache stored

procedure

2.4 Schemas Cooling Controller

Before we define the scenarios itself, it makes sense to look at each of the message
types being used in the scenario.

2.4.1 Cooling Schema

The Cooling Data schema is a flat file schema created using BizTalk Flat File
Schema wizard feature. Each section within the file are created as a separate
schema and all those schemas are then imported into main CoolingFF Schema.

Page 4
BizTalk Technical Document

Since not all the groups can come in data file, Choice Group option with max
Occurs of 0 is set for all sections.

The following diagram is a representation of this schema:

Page 5
BizTalk Technical Document

The schema was modified to relax the way input data was received. This was done
to allow looser checking of data in the pipeline ( for e.g. not always all data section
are received in input message )

Cooling Property Schema

The Cooling property schema was created to store the most common data field in
message context. When message is received in BizTalk, it contains the actual data
as well as the message context. The message context is same concept as Message
Header, SOAP Header or WCF Header as in a container to store various properties
that can be used throught BizTalk Process.
There are two types of message context properties. Property fields are context
properties used by BizTalk Messaging enging mainly for message routing and some
evaluations within BizTalk Orchestrations. One limitation is they can be upto 255
characters.
It provides centralized mechanism for pulling key pieces of information that is then
easily accessible to BizTalk Server components that are handling the message as it
passes through BizTalk Server.

Page 6
BizTalk Technical Document

Email header related data are used throughout the process to subscribe and/or
route message along the process.

The way to identify if its property schema is by looking into its Properties windows
with a Schema Type as Property

Cooling Alarm Schema

The Cooling Alarm schema will transform Cooling Alarm data from email body in Flat
file to message schema

Cooling Apache Schema

The Cooling Apache schema will transform Cooling Apache data from email body in
Flat file to message schema

Page 7
BizTalk Technical Document

Record Schema

The data that are received for Cooling controller has multiple sections and each
section has multiple rows. For every single rows that is received in data, we need
to update Parameter Table for all parameters for that row. To improve the
performance instead of calling stored procedure to update parameter values for
every single column in a row, the whole row is send to SQL server as Composite
Operation. Following is the sample of DATA2 Section received with multiple rows
with specific parameter values for that timeframe.
[DATA2]|
02/23/2011 04:15
-999.00
0.00 0.00
128
0

104.22 49.84
0.00 0.00
0.00 0.00
128
0

1151.34
0.00 0.00
128
0
0
0

0.00 -532.00
1.06
-14.51 11.16 94.42 0.00
0
0
128
0
|

0.01
1
128

107.91
1
0

02/23/2011 04:30
-999.00
0.00 0.00
128
0

105.99 50.80
0.00 0.00
0.00 0.00
128
0

1149.05
0.00 0.00
128
0
0
0

0.00 -532.00
1.06
-14.44 11.17 95.27 0.00
0
0
128
0
|

0.01
1
128

109.49
1
1

02/23/2011 04:45
-999.00
0.00 0.00
128
0

105.54 50.84
0.00 0.00
0.00 0.00
128
0

1181.72
0.00 0.00
128
0
0
0

0.00 -532.00
1.00
-14.28 11.14 95.34 0.00
0
0
128
0
|

0.01
1
128

109.51
1
2

02/23/2011 05:00
-999.00
0.00 0.00
128
0

105.84 51.24
0.00 0.00
0.00 0.00
128
0

1190.01
0.00 0.00
128
0
0
0

0.00 -532.00
1.00
-14.28 11.14 95.44 0.00
0
0
128
0
|

0.01
1
128

110.25
1
2

[END_DATA2]|

To process and perform Composite operation on a single row, for every single
section a new Schema is created for simpler mapping process. Looping and parsing
single record from a source data is done within BizTalk Orchestration using xPath.

Page 8
BizTalk Technical Document

Page 9
BizTalk Technical Document

Scenario Components
1

Cooling Data Receipt

The purpose of the cooling receipt process is to archive the message that is
received ( with an ability to disable at runtime ) and bring the cooling data into
BizTalk and transform it to a format that can be used to process the message. This
process is represented by the BizTalk receive pipeline as shown below:
Cooling Receive Pipeline Component

The following sections describe each portion of the process.

Page 10
BizTalk Technical Document

File Archive

The file archive component was implemented as a BizTalk custom pipeline

component. BizTalk tracing is an alternate way to achieve that as well, but since we
already have an existing process to archive the data to a file the same approach is
taken here. This gives us an exact replica of the file before any processing
commences. Below is a screenshot of the configuration of that pipeline component:

Disassemble Component

BizTalk Disassembler pipeline Component is implement to

Single Component to process Cooling water Data, Alarms and Apache Data

Parse the incoming email message ( as *.eml ) and create a strongly typed
.Net Object

Page 11
BizTalk Technical Document

Determine validity of message and filter out Spam

Business Processing like verifying blacklisted alarms, Verify PI data archiving

Promote Context Properties

Archive Data

Disassemble Flat file email Body data and create XML representation using
BizTalk Standard Flat file disassembler.

Page 12
BizTalk Technical Document

Helper Components

Helper Components are .Net Class Library that are used throughout the Business
Processing. They are used from within Pipeline Components or from within
Orchestrations.

EmailObject : A strongly typed .net Object to represent email Object

ParseEmailObject : A .net Class to parse email message ( *.eml ) and create a

email Object identifying business specific data like email subject, soldTo,
username, message type

CoolingDynamicTransform : A .net class ( serializable ) used from within

BizTalk Orchestration to perform Dynamic Transformation using XSLT.

Page 13
BizTalk Technical Document

3
1

Cooling Processor
Composite Update

The out of box WCF-SQL adapter enables adapter clients to perform composite operations on
the SQL Server database. A composite operation can include:

Insert, Update, and Delete operations. A Select operation is not supported as part of
a composite operation.

Stored procedures executed as operations.

A single composite operation can have any number of these operations, in any order.
Composite operation pattern is implemented to improve performance and send multiple
calls to Stored procedure as a single composite operation.

The following orchestration represents this process:

Data Transformation and Translation

BizTalk mapper is used to create maps, which are used for translating and
transforming xml messages. Depending on the type and complexity of
transformation, following different transformation features in within BizTalk Mapper
are used

Using Inline XSL

Inline XSLT is used in the map below to transform many to one relationship

Page 14
BizTalk Technical Document

Using BizTalk Functoids

Usage of Inbuilt BizTalk Functoids to perform and check if source element is
null or empty and based on the conditional apply the mapping to destination
schema

Using XSL
BizTalk mapper by default uses XSL to perform transformation and
translation. If out of the box functoid provided by BizTalk doesnt satisfy
business requirement, they can be extended with using custom XSL.

Page 15
BizTalk Technical Document

Testability

All BizTalk Artifacts had unit Testing enabled and depending on type of BizTalk
Artifact Object mocking was performed if needed.

Schema Testing

BizTalk Projects were enable to allow Schema Testing

Page 16
BizTalk Technical Document

Once the project is enabled, testing can be performed using TestTools provided by
Microsoft BizTalk Assemblies.

Maps Testing

Maps can be tested as Schema, alternatively BizUnit ( open source ) was used to
perform BizTalk Map testing as well as validating the output data via XPath

Page 17
BizTalk Technical Document

PipelineComponent Testing

Pipelinetesting unit tests are written to use Winterdom Tools

Page 18
BizTalk Technical Document

Monitoring and Exception Handling

Tracking & Message Monitoring

1

BizTalk

BizTalk has its own tracking and debugging capabilities built into the administration
tool. We can use these abilities to get information about the processes being run as
well as view message contents and processes. Please see screenshots below:

We can also use the administration tool to debug the orchestration processes or
view a process that occurred in the past.

Page 19
BizTalk Technical Document