Data Store :

Business scenario1:

When ever user creates an invoices in source system (ariba )We are getting the invoices from sender
system and CPI will do transformation and then we are sending it to target system for payment this
process will continue ..

For any reason Say target system is down .. or if there is any issue with cpi transformation(while doing
groovy script or message mapping )so this cpi interface will be failed .. what ever data we are getting
from source system will be lost inorder to avoid this situation we are storing the data in Data store

Everyday thousands of invoices are being sent to target system(oracle) ..some upgrade is happening at
target system at that time .. if there is no option to retrigger from source system again to target
system we need to store the payload in Data store

I can store the Data and read the data from data store

In a day there are 30 invoices have failed because of so and so reason

We can create an iflow and place an timer and fetch the 30 invoices from data store and reprocess the
invoices to target system .. no dependecy with source system .. based on schedule it will read the
data from data store and send to oracle or target system

Business scenario2:
Say I have one HR system where we maintain employee details and I also have Manager system where
it has Manager Information of an employee ..

Now HR system will be sending the data to SFTP server as employee .csv Manager system wil lalso be
sendign the data to SFTP server as Manager.csv

Now I will create One iflow/interface in CPI to write/store the employee .csv data to Data store

I will create another iflow/interface in CPI to write/store the Manager .csv data to Data store

In my third interface I will read the data from both the iflows and Merge the data and do the
transfromations and send it to Oracle which is my target system ?

Data store: data base lo table lantidi its like a data base table ------entries ID -> you can give dynamic
value or contast value (you can store employe id in property and provide it here also if you dont give it
will generate an unique value )

Difference between get and select – single message we can read is get and select is multiple messages
/entries ->

Get = message header+body select is only = Message body

I am taking this two data stores and create an iflow and

Data store : is nothing but a table in database to store data writing the data into data store and
reading the data from data store using entry ID

One data store can have Multiple Messages with Multipe entry IDs
  Get
 Select
 Write
 Delete
 Once you write the data into data store it will be shown in Monitor->Manage stores -
>Datastores
 Once you create a Queue using Jms adaptor at receiver side it will be shown in Monitor -
>Manage stores ->Message queues
 In order to create Number ranges you have to Go to Monitor ->Manage stores -> Number
Ranges
 When you create variable using write variables it will be shown in Monitor ->Manage stores-
>Variable

Once you create a variable and store the value in it you don’t have any option in CPI to Get the variable
value you need to use Content Modifier and select the type as Global Variable ..

Select – This step selects entries from a transient data store and creates a bulk message containing the
data store entries.
Write – This step performs a Write operation on the transient data store
Get – This step gets a specific entry from the transient data store.
Delete – This step deletes an entry from a transient data store. (Give it a try) Leave us a comment, if you
face any issue.

Visiblity drop down :

 Integration flow
 Global

Select Multiple records from data source using select operation

Entity ID can be static or dynamic ..use content Modifier to get the value from source xml and save it in
property and that property can be provided in Enitity ID

Something like this${property.ProductID}

Once you successfully read the data from data store it will be deleted if you check the check box delete
on completion

Say you are getting multiple products from source you can use splitter before data so that we will split
one large message into individual messages

Say in your xml you have 30 products with product IDs.. we can save/write them into single data store
with different product IDs by providing Entity ID as dynamic you can read multiple entries or records
by using select operation

If you send same message to the data store using write operation it will fail unless you check overwrite
existing message check box
 --------------------------------------------------------------------------------------------------------------------------------

CPI Data Stores

There are many scenarios where message content needs to be stored either as a whole, parts of, as an
error response or specific configuration data. Data Stores offer users with a means to achieve this
temporary storage in CPI. There are two types of Data Stores - Local and Global. Local Data Stores are
restricted to the iFlows they are created in while Global Data Stores can be accessed by various iFlows.
Use Cases

There are three common use cases for data stores:

Store & Pick Up Scenario
 This scenario globally stores the message for later use by another iFlow.
 It can be seen as a push pull pattern. The message received from a sender is stored in the data store. A
receiver seeking information actively polls (reads) the message in another message processing run
(iFlow).

Asynchronous Decoupling Scenario

 This scenario decouples the sender & receiver systems.

 The message is received by the sender is stored and independently received by another iFlow to forward
to the receiver.

Asynchronous Decoupling with Error Scenario

 This scenario decouples sender & receiver systems when an error occurs.
 If the message is successful, it is passed to the receiver within the same iFlow.
 If an error occurs, the error messaged is stored in Data Store to be accessed by another iFlow for
processing.
 In this scenario by using Data Stores SAP CPI users can create a generic error handling approach for
iFlows.

Main Operations of Data Stores

In Data Store Operations, there are four operations available that caters for in memory processing and
enables real time access to data.

WRITE: It stores the messages temporarily in the data store. If you use a Write operation, you can write
more than one message to a data store from an iFlow.

SELECT: You can select more than one message from the bundle of messages that are already available
in the data stores and send it to the iFlow for further message processing.

GET: It can fetch a specific message from the data store and can get messages based off the reference id
from the data store to the iflow for further message processing.

DELETE: This operation can delete messages individually or through a list of reference IDs.
Visbility: within integration flow or global

After storing the data in data store if the Retention threshold is given as 2 then till 2 days it will be in
waiting status so that other flow will pick the data it will wait.. after 2 days The status will be overdue
Writing the data into data store:

Reading the data from data store:

Timer settings:
What is the scope of property vs variable vs data store

------------------------------------------------------------------------------------------------------------------------------------------

I WANT TO Entry ID TO BE Dynamic :

To create an Variable called Product ID and we are reading the Property in write variable ..
Data store read :

You need to check the checkbox Delete on Completion

Select Data store :

In Data store write : to achieve Entity ID dynamically we have followed the below process

 Content Modifier we have created an Property Product! ID with xpath expression //productID
and we have read that property in Write data store .
 We have created a Global variable called productID and read the property into it

In Data Store Read :

 Using Content modifier we have created the property and stored the variable value
 That property is given in the data Store get Entity ID

Suppose I have two integration flows ( IF1,IF2), where IF2 is connected to IF1 via process direct .

Now properties in IF1 cannot be accessed in IF2 because the scope of property is with the iflow (IF1), but
where as headers created in the IF1 can be accessed in IF2 because it as global scope.

--------------------------------------------------------------------------------------------------------------------------------------
If you store values in headers or properties in your iflow, those values will be gone when the iflow
finishes. If you need to store values longer than that, you can store them in a variable using the Write
Variables step. In other words, they're a way to persist data.

A global variable can be accessed by any iflow, but a local variable can only be accessed by the iflow that
wrote it.

You can see all the variables, you've stored, in the Operations view => Manage Stores => Variables.

Variables are similar to Data Stores, really. But variables store scalar values (i.e. one number, one
timestamp etc.) whereas Data Stores contain complete payloads (e.g. an XML or JSON document).

Please note that while there's a Write Variables step, there's no Read Variables step. To fetch the value
of a variable into a property or header, you use a Content Modifer with the type set to either Local
Variable or Global Variable.

Headers are important aspect of Message which will get propagated to reciever system or target
system.If developers use the headers across their integration flows and don’t like them to be sent out of
Cloud Integration, it's recommended to use properties since those doesnt gets propagated to receiver
system.

This field refers different type of values can be assigned to this headers.

1. Constant: Any constant value that will be carried across integration process.
2. Expression:Values can be called using camel simple expressions like ${header.<HEADER_NAME>}
,${property.<PROPERTY_NAME>.Refer the blog to know more about Camel Simple Expressions.
3. Global Variable: We can use Write Variables step type to create a variable at a certain point
within the message processing sequence to be used across Integration Flows of cloud
integration tenant.To consume the Global variable ( with the scope of tenant ) we can set type
as Global Variable and set it to header or property in Content Modifier.
4. Header: Pass value from other header to newly created header.
5. Local Variable:We can use Write Variables step type to create a variable at a certain point within
the message processing sequence within same integration flow.To consume the Local variable
( with the scope of integration flow ) we can set type as Local Variable and set it to header or
property in Content Modifier.

6. Number Range:It helps to insert unique sequence numbers as part of the inbound or outbound
messages.To consume Number Ranges we can set type as Number Ranges and set it to header
or property in Content Modifier.
 7. Property:source value depending on the type either header name,a property name, a variable
name or any other type.
8. XPath: Set the value of XPath to header.Refer to know different XPath functions supported.

3. Exchange Property:

Properties are important aspect of Message which will not get propagated to reciever system or target
system.Boundary condition for properties is integration flow.Rest of the configuration remains same as
header.

4.Message Body

Body contains actual message or content.There are 2 ways you can set the body value either a constant
or using camel simple expressions.If body is kept empty that the body is not changed.

In SAP Cloud Integration (CPI), variables have two types of scope: local and global:
 Local scope: Variables are only accessible and visible to one IFlow.
 Global scope: Variables can be read and modified by any IFlow.
Here are some things to know about variables in CPI:
 Creating variables
To create a variable, you can add a "Write Variables" shape to your IFlow and use a camel expression to
store values.
 Using global variables
You can use global variables in other flows by calling them in the content modifier.

Scope/Visibility of Variables

Variables have Local scope by default. This means that they are visible and accessible to only one IFlow.
But what if we wanted to read this value, or possibly even modify this value from another IFlow? In that
case, we will have to use a Global variable.

In SAP Cloud Platform Integration (CPI), the scope of a property is limited to the integration flow
(IFlow). This means that property parameters are not passed on to the receiver
 The decision to use a header or property depends on the purpose. For example, if information needs to
be sent to the receiver system, use the header. If the information is internal to the IFlow, use the
property.

Here are some examples of how to use properties and headers in CPI:
 Fetch and store a product ID: Use a property to fetch a product ID from an incoming payload and store it
for later use in the IFlow.
 Set headers: Use the setHeader function to set the headers for the receiver and context.
 Read headers: Use the getHeader function to read headers from a Postman call.
 Read properties: Use the getProperty function to read properties created in the IFlow.
 Access HTTP header values: Enter the HTTP header parameter names in the Allowed Header(s) field in
the Runtime Configuration settings

The scope of the element declared in the Header is beyond the scope of the IFlow, The scope of the
element decalred in the Property is only within the IFlow i.e. the property parameters are not handed
over to the receiver

A Content Modifer is one of the most important functions provided by SAP.

A content modifier shall help you to modify the incoming message or payload, by changing it's
content that are involved in Message Processing (i.e. Message Header, Message Exhange Property, or
Message Body).

In the Message Header, and Message Exchange Property, you can declare Header Element, and
Property Elements, which can or cannot be included in the outgoing Message to the next step, as
shown in the screenshots below:

Header:
Property:

Question: What is the difference between Header, and Property?

The scope of the element declared in the Header is beyond the scope of the IFlow,

The scope of the element decalred in the Property is only within the IFlow i.e. the property
parameters are not handed over to the receiver.

In the Message Body, you can do the following:

1. Call the Header Elements, like:

${header.elementname}

2. Call the Property Elements, like:

${property.elementname}

3. Call the body of the previous Content Modifier, like:

${in.body}
As shown in the screenshot below, I have called the Header, and body, and since, the received expects
the output in xml, the tags have been maintained like wise.

OUTPUT from this Content Modifier is:

Now, I have used one more content modifier to explain you the functionalities even better, so, follow
the series of images to see how it works.

Here, I have inserted a Groovy Script after every Content Modifier because I wanted to see the payload
after the IFlow is deployed. I shall share the script's content in the Part 7 of this series.

In the Content Modifier 2, below is the Header, Property, and Message Body:

As you can see in the screenshot above, I have used the code ${in.body} to call the content of the
previous i.e. Content Modifier 1.

Now, the output to this, after you have "Saved as version", and "Deployed" is below.

The ${in.body} holds the entire input payload which receives from the sender/previous Content modifier
step. You can directly pass the entire payload to the receiver system or you can manipulate the data
according to the receiver structure.

Properties and headers are only available at runtime in that particular iflow. Unlike variables and data
stores, there's no local or global scope. Also, when the iflow's executed, they're gone; there's no
persistence.

-----------------------------------------------------------------------------------------------------------------------------

Q: What is the different between local variable and global variable?

A: Local variable can be access via same iFlow only. Global variable can be access via differet iFlows.

Q: How to read local variable and global variable?

A: Use Content Modifier read to either header or property.

Q: How to write Variable?

A: In iFlow, use 'Write Variables' step, take value from header/property/Xpath/expression.
Q: At iFlow first run, variable not created yet but need to use some initial/default value for processing,
how to handle such chicken-and-egg situation?
A: Use Content Modifier read variable and set default value.

Q: Beside create, how to update variable?

A: Just use 'Write Variables', if existed will just update/replace.

Q: Is it possible local and global variable having same name?

A: Yes. Since the scope is different between local and global.

Q: How to do delta synchronization via timestamp?

A: Use variable to remember last processed timestamp, so that next scheduled run will resume from last
process timestamp onward.

Q: What need to consider when design delta synchronization via timestamp?

A: (1)Data should be sorted by timestamp.
(2) Timestamp should be unique (E.g. only date without time might not work).
(3) The right date field should use for delta synchronization.
(4) Only update last processed timestamp at last step if all processing success.
(5) Timer/scheduler interval.

Q: What if I need to revert back to earlier past timestamp?

A: Build in same iFlow a manualRun/adhocRun flag to set manual timestamp, override value in variable.

Q: Should I use global variable or local variable?

A: Use global if other iFlow need to access same variable. Global can behave like local, but not the other
way round.

Q: Can I use SAP CPI manage variable page to write variable?

A: No, as of now, no way to create/update variable manually,but via IFlow only.

Q: What other way to manage read write of global variable?

A: Build generic iflow and use Postman to read write global variable.

Q: What ways can be use to delete variable?

A: Manual delete via 'Manage Variables' page.

Q: What other potential use of variable?

A: Access same variable value in different branches of Multicast (because property will not work).
Behavior of CPI's message header/property in a multicast

SAP CPI - Use SELECT Operation To Fetch All Records From Datastore

Hi All,

There might be requirement to store the records temporarily in Datastore and use it later till all entries
are consumed. For this article I have created two Iflows one is to store payload using WRITE operation
and another Iflow is to fetch all the records from Datastore using SELECT operation in group of two.
Hope this will be usefully to other who trying to achieve the same.

IFlow 1 - WRITE operation to store data. I am using ID value from incoming payload to generate entry
IDs in Datastore.
IFlow 2 - I am using loop process call to fetch all the entries from datastore in group of 2. To avoid
passing last empty records to target system I am using router. This will suppress processing of empty
record.

Below is the used condition for the loop

So my loop will end whenever select step will give me empty entry or max number of iterations are
reached. I am fetching records in group of 2 using SELECT operation.

Below is the list of entries in my datastore - DS_Books

Iflow 2 executed succesffuly, below are the monitoring page screenshots
I have received mail with datastore records -
Please share your thoughts if there are any other ways to achieve this.

Learning :-

 With GET operation this is not possible, GET is used for fetching single record at a time.
 This looks for the entry ID to fetch the data

 If we have selected save header values during WRITE operation, this can be fetched with GET
operation

 You can download datastore record to check header

  With SELECT we can group the records and can be used with loop call

If you want to retrigger the failed messages from CPI

Primarly Messages can be failed at three layers

Extract: if source system is down that can be very well managed ..

Transformation: what about the case is when we have received the message and we are doing some
transformation it is failing because of the transformation

Load:

You can use Jms adaptor to build the scenario to retrigger the failed message where you can hold the
message in the queue and you can reprocess in case of any failure but this adaptor and design comes
with some level of constraint because it going to store the messages in the queue and you have to
manage the queue ie., queue management and what about the case if the customer they are not in
mood to use jms adaptor