ODI Document
ODI Document
ODI Document
(DOCUMENTATION)
By M. Ambreen U. Jahnavi Mallika P. Sashank P. Praneeth B. Manoj kumar
Table of Contents I. Introduction To ODI 1. Repository Management 1.1 Master Repository Creation 2. Topology Manager 2.1 Connecting to Master Repository 2.2 Work Repository Creation 2.3 Creating Data Servers 2.4 Creating Schemas a) physical Schema b) Logical Schema 2.5 Creating Agent a) physical Agent b) Logical Agent 3. Designer 3.1 Connecting to Designer 3.2 Creating ODI Models a)for Data Base source b)for Flat File source 3.3 Assigning key column to the target table 3.4 Creating Projects a) Using Wizard b)Manually 3.5 Creating a Folder. 3.6 Creating Interfaces 3.7 Creating Packages 3.8 Knowledge Modules a) Importing Knowledge Module 3.9 Creating Scenarios 3.10 Creating Procedures 3.11 Creating Functions 3.12 Creating Sequences 3.13 ODI Variables 3.14 Markers 3.15 Memos 3.16 ODI Tools 3.17 Scheduling 4. Security 4.1 Profile Creation & Deletion 4.2 User Creation & Deletion 4.3 Assigning & Removing a Profile to & from a user 4.4 Authorization 4.5 Password Policy 4.6 Exporting & Importing an Object 4.7 Exporting & Importing the Topology/Security Settings 5. Operator 5.1 Sessions 5.2 Schedules 5.3 Purging 5.4 Scenarios
I. Introduction To ODI
Oracle Data Integrator provides a new declarative design approach to defining data transformation and integration processes, resulting in faster and simpler development and maintenance. Based on a unique E-LT architecture
Technical Architecture
Oracle Data Integrator is built around a modular repository architecture and accessed in client/server or thin client mode by components that are developed entirely in Java: the graphical interface modules and execution agents.
The Repository is a comprehensive, open and shared meta data repository. It stores the meta data for the accessed applications involved in the transformation and integration processing, the developed project versions and all of the information required for their use (planning, scheduling and execution reports). Topology Manager is the graphical interface module used to manage the data describing the information system's physical and logical architecture. The site, machine, and data server descriptions will enable Oracle Data Integrator to execute the same interfaces in different environments. Designer is a tool used to design data integrity checks and to build transformations. In particular, it incorporates functions for: Automatic reverse-engineering of existing applications or databases Graphical development and maintenance of transformation and integration interfaces Visualization of data flows in the interfaces Automatic documentation generation Customization of the generated code Security Manager is the tool for managing users and permissions in Oracle Data Integrator. Operator is the production management and monitoring module. It is designed for IT production operators. The Agent is used to execute the inter-application interfaces developed with Designer. It integrates the Scheduler to execute the interfaces according to a predefined schedule. Meta data Navigator is a thin client, browser based interface to access the Oracle Data Integrator Repository, browse, search and analyze all its contents, and perform advanced data lineage tracing, cross referencing, impact analysis, etc. You may even use this tool to manage all of your production processes.
1. REPOSITORY MANAGEMENT
The Repository is a comprehensive, open and shared metadata repository. It stores the meta data for the accessed applications involved in the transformation and integration processing, the developed project versions and all of the information required for their use (planning, scheduling and execution reports). Master Repository Creation Creating the master repository includes the creation of tables and automatic importing of definitions for the different technologies. Navigation : Start->programs-> Oracle->ODI->Repository Management->Master Repository Creation. Give the Database URL, Driver, User, Password and Technology as shown :
The screen shot of the script which automatically runs after clicking OK
shown below :
After Successful completion of the script we get the following dialog box. Click OK
2.TOPOLOGY MANAGER
2.1 Connecting to Master Repository through Topology Manager Navigation : Start -> Programs -> Oracle -> ODI -> Topology Manager. Give the following details :
After clicking ok, we get the below screen. Again click ok to connect to Topology
2.2Work Repository Creation Click on the Repository tab which is located at the bottom. Right click on the work repository node and select insert repository. The following wizard opens, fill in the appropriate details like Data Server name, user,password and JDBC details.
Click Apply and then say OK. 2.3 Creating Data Servers : Go to the Physical Architecture and click on technology to expand, then choose the oracle technology Right click on it and click on Insert Data Server.
A window named data server appears, fill in the definition and JDBC Tab which includes the data server name ,user ,password etc
2.4 Creating Schemas Clicking on OK on the above data server window , physical schema screen will be opened. Enter the details like schema name, work schema name etc.
Assign Logical Schema to corresponding physical schema: Go to Logical architecture : Click on technology to expand, click on oracle node to expand. We see the logical schemas that are created. Here we need to assign this logical schema to the corresponding physical schema created in the physical architecture.
2.5 Creating Agent Used to execute the interfaces developed with Designer. It integrates the scheduler to execute the interfaces according to a predefined schedule
Creating Physical Agent Go to Physical Architecture and right click on Agents node and click the insert agent. Give the details like name of the agent, port number and host. Here the Host is the ip address of the system one is connected to.
Creating Logical Agent 1. Go to the logical architecture, right click on the agents node and click insert agent. The agent creation window appears. 2. Give the logical agent name and assign the physical agent to it.
3. Designer Module
3.1 Connecting to Designer Go to Start -> Programs -> Oracle -> ODI -> Designer Click the first icon beside the login name of the following window
Click on test button, then click on OK. We get the below screen, again click on OK
3.2 Creating ODI Models Creating ODI Model for a Data Base source 1.Go to the Models tab at the bottom. 2.Click on the first button on the top for the creation of a Model folder an enter model folder name 3.Right click on this folder and select insert model
or one can directly create the model with out creating a model folder by clicking on the fourth button.
4.The following screen appears, fill in the appropriate details like name of the model, Technology, Logical schema in the Definition tab.
4.Click on the Reverse tab, select context as Global and select the type of objects to be reverse engineered. 5.Click on Selective Reverse tab and check the button Objects to Reverse. We get a list of all the tables present in the database. By default all the tables were selected. To deselect them click on the button at the right most corner.
6.Select the source table from which we need to transfer the data and click on Reverse button. 7.Similarly create another model, for the target table considering the logical schema of the target database. (Note : The structure for the target table has to be created in the target database before creating the model).
Creating ODI Model for a Flat File source To create a new model for the flat file target datastore, perform the following steps: 1.Click the Models tab in the left panel of the ODI Designer. Click the Insert Model Folder icon. On the screen that appears set the name to FILES. 2.Right-click the Files model folder and select Insert Model. 3.On the screen that appears, enter the values provided in the following table. Click the Reverse tab (don't click the Reverse button). Select Global from the Context drop-down list. Click OK.
Value
FILE_DEMO_SRC
4. Click apply and ok. Creating a New ODI Source Datastore To create a new ODI source datastore, perform the following steps: 1. Create a Model folder and name it as Files. 2. Add a model to this folder and select file as the technology and global as the context. 3. Expand the Files folder, right-click the newly created Files model, and then select Insert Datastore. On the screen that appears, set Name to <<name>> and select flat file that is created in specified path ie, 384_datastore.txt(D:\OraHome_1\oracledi\demo\file\384_datastore.txt) next to Resource Name. Click Open.
4. Click the Files tab. Set the File Format to Delimeterd, and then click the Columns tab.
5. On the Columns tab, click the Reverse button. The following screen appears. Click Yes, and then click OK. The Column Setup Wizard appears.
6. After the columns are marked, click each column data and set the Name and Datatype for each column. For the names and data types of each column, refer to the following table. Click OK when you have finished. Column C1 C2 C3 C4 C5 7.Click OK. Name empid ename salary deptno Creation Date Data Type numeric string numeric numeric date
3.3 Assigning key column to the target table Note: If primary key already exists then skip this step. 1.Expand the target model and target table. 2.Right click Constraints and click Insert Key. 3.Give a name for the constraint in the Description and select the columns, we consider as the key columns for the target table in the Columns tab. 4.Click on Apply and say OK. 3.4 Creating Projects 1.File->Wizard Give Name 2.Import Knowledge Modules from D:\OraHome_1\oracledi\impexp path. Import -CKM Oracle -IKM Oracle Incremental Update -LKM Oracle to Oracle(@source is DB,@target is DB) -RKM Oracle
Choose required Knowledge modules for your project and click next. In the next window select interface and click ok. Your project will be appeared in the projects tree structure. 3.5 Creating a Folder. Right click on a particular Project and click on Insert Folder as shown in the following screen shot.
click apply and ok. Creating Interfaces 1.Right click on the Interfaces button and select Insert Interface 2.Give a name for the interface in the Definition tab and click on Diagram tab. A window appears asking to drag and drop the Source and Target Models, Click OK. 3.Click on the Models tab at the bottom and drag and drop the source table on the left side of the diagram i.e., source panel. 4.Similarly drag and drop the target table on the right side of the diagram i.e., target panel.
5.A window appears asking for automatic mapping. If the source and target tables have similar structures, then click on OK.
We can check the knowledge modules assigned to the interface, by clicking on Flow and Controls tabs.
Click Apply and then click on Execute to execute the interface. We can check the status of the interface from the Operator module.
Go to Start -> Programs -> Oracle -> ODI -> Operator or we can directly navigate from Designer using the button on the top
Creating Packages 1.Expand the Project and right click on Packages and click Insert Package. 2.Give a name for the package in the Definition tab and click on Diagram tab. 3.Drag and drop the interfaces and connect the interfaces using the arrows on the top which indicate the success and failure as shown:
3.8 Knowledge Modules KMs contain the knowledge required by Oracle Data Integrator to perform a specific set of tasks against a specific technology or set of technologies. Different types of Knowledge Modules Oracle Data Integrator' Open Connectors use 6 different types of Knowledge Modules: 1) RKM (Reverse Knowledge Modules) Used to perform a customized reverse-engineering of data models for a specific technology. a)RKM Oracle : Used to perform reverse-engineering from oracle data models. 2) LKM (Loading Knowledge Modules) Used to extract and load data from the source to staging area. a)LKM File to Oracle (EXTERNAL TABLE) Source - File Staging - Oracle b)LKM File to Oracle (SQLLDR) Source - File Staging Oracle
c)LKM File to SQL Source - File Staging Oracle d)LKM Oracle to Oracle (DBLINK) Source - Oracle Staging - Oracle e)LKM SQL to Oracle Source - Oracle Staging Oracle f)LKM SQL to SQL Source - Any Staging Any 3)JKM (Journalizing Knowledge Modules) Used to create a journal of data modifications (insert, update and delete) of the source databases to keep track of the changes. Oracle Data Integrator supports two journalizing modes: Simple Journalizing tracks changes in individual data stores in a model. JKM: JKM Oracle Simple Consistent Set Journalizing tracks changes to a group of the model's data stores. JKM: JKM Oracle Consistent Ex on JKM : Required setups : a)set up the CDC parameters: 1. In the Models tree in the Designer Navigator, select the model that you want to journalize. 2. Double-click this model to edit it. 3. In the Journalizing tab, select the journalizing mode you want to use: Consistent Set or Simple. 4. Select the Journalizing Knowledge Module (JKM) you want to use for this model. Only Knowledge Modules suitable for the data model's technology and journalizing mode, and that have been previously imported into at least one of your projects will appear in the list. 5. Set the Options for this KM. See the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data Integrator for more information about this KM and its options. 6)Click Apply and ok. b)Add or remove datastores for the CDC: 1. Right-click the model, sub-model or datastore that you want to add to/remove from the CDC in the Model tree in the Designer Navigator.
2. Right-click then select Changed Data Capture > Add to CDC or Changed Data Capture > Remove from CDC to add to the CDC or remove from the CDC the selected datastore, or all datastores in the selected model/sub-model. c)Set the datastores order (consistent set journalizing only): 1. In the Models tree in the Designer Navigator, select the model journalized in consistent set mode. 2. Double-click this model to edit it. 3. Go to the Journalized Tables tab. 4. If the datastores are not currently in any particular order, click the Reorganize button. This feature suggests an order for the journalized datastores based on the foreign keys defined in the model. Review the order suggested and edit the datastores order if needed. 5. Select a datastore from the list, then use the Up and Down buttons to move it within the list. You can also directly edit the Order value for this datastore. 6. Repeat the previous step until the datastores are ordered correctly. 7. From the File menu, select Save All. d)Add or remove subscribers: 1. In the Models tree in the Designer Navigator, select the journalized data model if using Consistent Set Journalizing or select a data model or an individual datastore if using Simple Journalizing. 2. Right-click, then select Changed Data Capture > Subscriber > Subscribe. A window appears which lets you select your subscribers. 3. Type a Subscriber name, then click the Add Subscriber button. Repeat the operation for each subscriber you want to add. 4. Click OK. 5. In the Execution window, select the execution parameters: Select the Context into which the subscribed must be registered. Select the Logical Agent that will run the journalizing tasks. 6. Click OK. 7. The Session Started Window appears. 8. Click OK. e)To start or drop the journals: 1. In the Models tree in the Designer Navigator, select the journalized data model if using Consistent Set Journalizing or select a data model or an individual datastore if using Simple Journalizing. 2. Right-click, then select Changed Data Capture > Start Journal if you want to start the journals, or Changed Data Capture > Drop Journal if you want to stop them. 3. In the Execution window, select the execution parameters: Select the Context into which the journals must be started or dropped. Select the Logical Agent that will run the journalizing tasks. 4. Click OK. 5. The Session Started Window appears. 6. Click OK.
IKM (Integration Knowledge Modules) Used to integrate (load) data from staging area to the target tables. IKM Oracle Incremental Update To load the entire data from source to target Staging - Oracle Target - Oracle IKM Oracle Slowly Changing Dimension To apply scd types (type1, type2, type3) Staging - Oracle Target - Oracle IKM SQL to File Append Staging - Oracle Target - File IKM SQL to SQL Append Staging - Any Target - Any CKM (Check Knowledge Modules) Used to check that constraints on the sources and targets are not violated. CKM Oracle CKM SQL SKM (Service Knowledge Modules) Used to generate the code required for creating data services. SKM Oracle SKM SQL
Importing a KM To import a Knowledge Module into Oracle Data Integrator: Select the project into which you want to import the KM Right-click on the project, and select Import > Import Knowledge Module Specify the Import Directory , and select one or more Knowledge Module Files to Import from this directory Select Duplication as the Import Mode , then click OK The Knowledge Modules are imported into the work repository, and become usable within the project.
3.9 Creating Scenarios Go To appropriate interface or package for which you want to create the scenario and then right click on the object (interface or package) and click on Generate Scenario
3.10 Procedures A Procedure is a set of commands that can be executed by an agent. These commands concern all technologies accessible by Oracle Data Integrator (OS, JDBC, JMS commands, etc). A Procedure is a reusable component that allows you to group actions that do not fit in the Interface framework. Procedures should be considered only when what you need to do cant be achieved in an interface. In this case, rather than writing an external program or script, you would include the code in Oracle Data Integrator and execute it from your packages. Procedures require you to develop all your code manually, as opposed to interfaces. A procedure is composed of command lines, possibly mixing different languages. Every command line may contain two commands that can be executed on a source and on a target. The command lines are executed sequentially. Some command lines may be skipped if they are controlled by an option. These options parameterize whether or not a command line should be executed as well as the code of the commands.
Creating a New Procedure To create a new procedure: 1. In Designer Navigator select the Procedures node in the folder under the project where you want to create the procedure. 2. Right-click and select New Procedure. 3. On the Definition tab fill in the procedure Name.
4. Check the Multi Connections if you want the procedure to manage more than one connection at a time. Multi Connections: It is useful to choose a Multi connection procedure if you wish to use data that is retrieved by a command sent on a connection (the source connection, indicated on the Source tab) in a command sent to another connection (the target connection, indicated on the Target tab). This data will pass though the execution agent. If you access one connection at a time (which enables you to access different connections, but only one at a time) leave the Multi Connections box unchecked. 5. Select the Target Technology and if the Multi Connections box is checked also select the Source Technology. Each new Procedure line will be based on this technology. You can also leave these fields empty and specify the technologies in each procedure command. 6.Click on details tab and click add symbol, enter target command. If Multi connection option checked then we need to enter commands for source and target .
7.Click apply and ok. A new procedure is created, and appears in the Procedures list in the tree under your Project. Calling an existing procedure 1. In Designer Navigator select the Procedures node in the folder under the project where you want to create the procedure. 2. Right-click and select New Procedure. 3. On the Definition tab fill in the procedure Name. 4. Click on details tab and click add symbol, enter the name and choose source technolgy as oracle. 5. Write the plsql code to call the procedure in command Ex: begin empbh1(7844); end; 6.Click apply and ok.
Using Procedures A procedure can be used in the following ways: Executing the Procedure directly in Designer Navigator for testing its execution. Using a Procedure in a Package along with interfaces and other development artifacts for building a data integration work flow. Generating a Scenario for a Procedure for launching only this procedure in a run-time environment. Executing the Procedure To run a procedure: 1. In the Project view of the Designer Navigator, select the procedure you want to execute. 2. Right-click and select Execute. 3. In the Execution dialog, set the execution parameters. 4. Click OK. 5. The Session Started Window appears. 6. Click OK.
Using a Procedure in a Package Procedures can be used as package steps. Note that if you use a procedure in a package step, the procedure is not a copy of the procedure you created but a link to it. If this procedure is modified outside of the package, the package using the procedure will be changed, too. If you dont want to use the option values set on the Options tab of the Procedure, set the new options values directly in the Options tab of the Procedure step. Generating a Scenario for a Procedure It is possible to generate a scenario to run a procedure in production environment, or to schedule its execution without having to create a package using this procedure. The generated scenario will be a scenario with a single step running this procedure. 3.11 Creating functions 1. In Designer Navigator select the User Functions node in a project or the Global User Functions node in the Others view. 2. Right-click and select New User Function. The User Function Editor opens. 3. Fill in the following fields: Name: Name of the user function, for example NullValue Group: Group of the user function. If you type a group name that does not exist, a new group will be created with this group name when the function is saved. Syntax: Syntax of the user function that will appear in the expression Editor; The arguments of the function must be specified in this syntax, for example: NullValue($(variable), $(default)) 4. Click Apply and ok.
To create an implementation: 1. In Designer Navigator double-click the User Function for which you want to create the implementation. The User Function Editor opens. 2. In the Implementations tab of the User Function Editor, click Add Implementation. The Implementation dialog opens. 3. In the Implementation syntax field, type the code of the implementation, for example nvl($(variable), $(default)) 4. Check the boxes for the implementation's Linked technologies 5. Check Automatically include new technologies if you want the new technologies to use this syntax. 6. Click OK. 7. From the File menu, click Save. To change an implementation: 1. In the Implementations tab of the User Function Editor, select an implementation, then click Edit. 2. In the Implementations tab of the user function, select an implementation, then click Edit Implementation. The Implementation dialog opens. 3. Change the Implementation syntax and the Linked technologies of this implementation 4. Check Automatically include new technologies if you want the new technologies to use this syntax. 5. Click OK. 6. Click Apply and ok.
Calling an existing Database functions from odi is similar to calling existing procedures 3.12 Creating Sequences
A sequence is a value that increments automatically when used. A sequence can be created as a global sequence or in a project . Global sequences are common to all projects, whereas project sequences are only available in the project where they are defined. Creating Sequences To create a standard sequence: 1. Click the Sequences node in the project or on the node Global Sequences. 2. Right click and select Insert Sequence.
3.Enter the sequence Name, then select Standard Sequence. 4.Enter the initial Position and the Increment. 5.Click ok. The sequence appears in the tree view. Using Sequences: The sequences can be used in all Oracle Data Integrator expressions, such as in: Mappings, Filters, Joins, Constraints. A sequence can be used in all statements with the following syntax: #<SEQUENCE_NAME>_NEXTVAL - the sequence value is incremented then substituted directly into the text of the expression. A sequence is resolved by an agent, it can therefore be incremented only when the agent itself processes one record. The sequence value will be incremented only once, even if the SQL statement
processes 10,000 rows, because the agent does not process each records, but just sends the command to the database engine. To make sure that a sequence is updated for each row inserted into a table, each row must be processed by the Agent. To make this happen, follow the steps below: 1. Make the mapping containing the sequence be executed on the target. 2. Set the mapping to be active for inserts only. Updates are not supported for sequences. 3. If you are using an "incremental update" IKM, you should make sure that the update key in use does not contain a column populated with the sequence. For example, if the sequence is used to load the primary key for a datastore, you should use an alternate key as the update key for the interface. 4. If using Oracle Data Integrator sequences with bind syntax (:<SEQUENCE_NAME>_NEXTVAL), you must configure the data flow such that the IKM transfers all the data through the agent. You can verify this by checking the generated integration step in Operator . It should have separate INSERT and SELECT commands executed on different connections, rather than a single SELECT...INSERT statement. To Increment sequence value for each row: 1. Right Click on 'IKM Oracle Increment Update' Knowledge module and click on duplicate.
2. Copy of IKM Oracle Incremental Update will be added to your KMS. Double click on it the following window appears, In that goto details tab. 3. Select 'Insert new rows' command and double click on that command.
Command on Source: insert into <%=snpRef.getTable("L","TARG_NAME","A")%> ( <%=snpRef.getColList("", "[COL_NAME]", ",\n\t", "", "((INS AND (NOT TRG)) AND REW)")%> <%=snpRef.getColList(",", "[COL_NAME]", ",\n\t", "", "((INS AND TRG) AND REW)")%>
) values ( <%=snpRef.getColList("",":"+ "[COL_NAME]", ",\n\t", "", "((INS AND (NOT TRG)) AND REW)")%> <%=snpRef.getColList(",", "[EXPRESSION]", ",\n\t", "", "((INS AND TRG) AND REW)") %>) Command on Target: select <%=snpRef.getColList("", "[COL_NAME]", ",\n\t", "", "((INS AND (NOT TRG)) AND REW)")%> <%=snpRef.getColList(",", "[EXPRESSION]", ",\n\t", "", "((INS AND TRG) AND REW)")%> from <%=snpRef.getTable("L","INT_NAME","W")%> where IND_UPDATE = 'I' 4. Click on apply . 5. Click Ok. Now the sequence value increments for each row.
3. Specify the following variable parameters: NAME : Name of the variable, in the form it will be used. This name should not contain characters that could be interpreted as word separators (blanks, etc.) by the technologies the variable will be used on. Variable names are case-sensitive. That is, "YEAR" and "year" are considered to be two different variables. The variable name is limited to a length of 400 characters. DATATYPE :Type of variable Alphanumeric : (255 characters) Date : (This format is a Java date format that matches your machine's local parameters. Numeric : (Maximum 10 digits) Text : (Unlimited length) ACTION : This parameter shows the length of time the value of a variable is kept for Non-persistent: The value of the variable is kept in memory for a whole session. Latest value: Oracle Data Integrator stores in its repository the latest value held by the variable. 4. If you want the variable's value to be set by a query then Select the Refreshing tab. a. Select the logical Schema where the command will be executed, then edit the command text in the language of the schema's technology. b. Click Testing query on the DBMS to check the syntax of your expression. c. Click Refresh to test the variable by executing the query immediatel variable action is set to Historize or Latest Value, you can view the returned value on the History tab of the Variable editor. d. Click Apply and ok
ASSIGNING THE VALUE OF A VARIABLE: A Set Variable step of type Assign sets the current value of a variable. In Oracle Data Integrator you can assign a value to a variable in the following ways: 1. Retrieving Variable value from a SQL select statement ie, When creating your variable, define a SQL statement to retrieve its value. 2. Explicitly setting the value in a package 3. Incrementing The Value :Incrementing only applies to variables defined with a numeric data type. Drag and drop your numeric variable into the package and select the Set Variable and Assign options in the Properties panel as well as the desired increment. Note that the increment value can be positive or negative.
3.14 Markers
Flags are defined using markers. These markers are organized into groups, and can be applied to most objects in a project or a models. Typical marker groups are: The development cycle (development, test, production) Priorities (low, medium, urgent, critical) Progress (10%, 20%, etc) Global and Project Markers Markers are defined in a project or in the Other view (Global Markers). The project markers can be used only on objects of the project, and global markers can be used in all models of the repository. Flagging Objects To flag an object with an icon marker:
1. In Designer Navigator, select an object in the Projects or Models accordion. 2. Right-click and select Add Marker, then select the marker group and the marker you want to set. The marker icon appears in the tree. The marked object also appears under the marker's node. You can thus see all objects having a certain marker. If you click in the tree an icon marker belonging to an auto-incremented marker group, you switch the marker to the next one in the marker group, and the icon changes accordingly. To flag an object with string, numeric and date markers: 1. In Designer Navigator, double-click the object in the Projects or Models accordion. 2. In the object editor, select the Markers tab. 3. Click Insert a Marker. 4. In the new line, select the Group and Marker. You may also set the Value. If the marker has an associated icon, it appears in the tree.
3.15 Memos
A memo is an unlimited amount of text attached to virtually any object, visible on its Memo tab. When an object has a memo attached, the memo icon appears next to it. To edit an object's memo: 1. Right-click the object. 2. Select Edit Memo. 3. The Object editor opens, and the Memo tab is selected. Hiding Markers and Memos You can temporarily hide all markers and memo flags from the tree views, to improve readability. To hide all markers and memo flags: Deselect the Display Markers and Memo Flags option in the Designer Navigator toolbar menu. This preference is stored on a per-machine basis.
3.16
ODI Tools
Oracle Data Integrator Tools (also called Oracle Data Integrator Commands) are commands provided for performing specific tasks at run-time. The tools may be added while working with Designer. They may be used into Packages, or into Procedures Commands or Knowledge Modules Commands. To use an Oracle Data Integrator Tool into a Package: 1)Open the Package you want to modify. 2)Click the Display Toolbox button in the menu bar of the Package window. 3)In the Toolbox , Select the icon of the Tool you want to use and click into the package window. A step with the tool icon appears in the package. 4)Click the Selection button in the menu bar, then double click this step. 5)A Properties window for this tool appears. Set the value of the parameters in this properties window. 6)Drag and drop other objects like interfaces, procedures ,etc and link them in a sequence 7)Click Apply .
1)Metadata
During a personalized reverse, it is possible to load temporary tables: SNP_REV_TABLE: Table of the datastores SNP_REV_COL: Table of the columns SNP_REV_KEY: Table of the unique keys SNP_REV_KEY_COL: Table of the key columns SNP_REV_JOIN: Table of the foreign keys SNP_REV_JOIN_COL: Table of the foreign keys columns SNP_REV_COND: Table of the conditions (check constraints) with meta-data through a specific processing, from a Knowledge Module or from Oracle Data Integrator interfaces. The processing can be summarized in three major steps: Purge the temporary tables of the reverse engineering Personalized processing to feed these reverse engineering tables from the meta-data Execution of the integration of the meta-data in the Repository
OdiReverseGetMetaData This command allows to reverse-engineer the given model in the temporary reverse tables. This command is preceded by OdiReverseResetTable, and is followed by OdiReverseSetMetaData. Required Properties: -MODEL=<model_id> OdiReverseResetTable The command OdiReverseResetTable can purge these temporary tables for a given model. Required Properties: -MODEL=<model_id> OdiReverseSetMetaData The command OdiReverseSetMetaData is used to integrate data from these temporary tables into the Repository for a given data model. Required Properties: -MODEL=<model_id>
OdiExportAllScen Exports a group of scenarios from the connected repository to XML export files named SCEN_<scenario name><scenario version>.xml . This command reproduces the behavior of the export feature available in Designer and Operator. Required Properties: -TODIR=<Dir> OdiExportObject Exports an object from the connected repository to an XML export file. This command reproduces the behavior of the export feature available in the user interface. Required Properties: -CLASS_NAME=<class_name> ( Class of the object to export. The list of classes : interface,package..etc) -I_OBJECT=<I_Object> (Object identifier. It is the Internal Identifier that appears in the Version tab of the object edit window.) OdiExportScen Exports automatically in an XML format a scenario contained in the current work repository. Required Properties: -SCEN_NAME= Name of the scenario to be exported. -SCEN_VERSION=Version of the scenario to be exported. -FILE_NAME=Name of the file export.
OdiExportWork Exports the work repository to a directory or a ZIP export file. Required Properties: -TODIR=Target directory for the export. OdiGenerateAllScen Generates automatically a set of scenarios from components contained in a folder or a project, filtered by markers. Required Properties: -PROJECT=Project containing the components to generate scenarios for.
OdiImportObject Imports the contents of an XML export file to a repository. This command reproduces the behavior of the import feature available from the user interface. The object is imported into the connected master/work repository or into the <workRepositoryName> work repository attached to the connected master repository. Required Properties: -FILE_NAME=Name of the XML export file to import. OdiImportScen Imports automatically a scenario to the current work repository from an XML export file. Required Properties: -FILE_NAME= Name of the file export. 3)Utilities OdiKillAgent When an Agent is listening on a TCP/IP port, as a listener agent, scheduler agent or web agent, it is possible to stop it with the OdiKillAgent command. OdiPingAgent Performs a connection test on an agent. If the agent is not started, this command generates an error. Required Properties: -PHYSICAL_AGENT_NAME= Name of the physical agent to test. OdiPurgeLog Allows the purge of the execution logs. If -FROMDATE is omitted, the purge is done starting with the oldest session. If -TODATE is omitted, the purge is done up to the current date If both parameters are omitted, the whole log is purged.
OdiReinitializeSeq This command is used to reinitialize an Oracle Data Integrator Sequence. Required Properties: -SEQ_NAME=Name of the sequence to reinitialize. It must be prefixed with "GLOBAL." for a global sequence, or by <project code>. for a project sequence. -CONTEXT=Context in which the sequence must be reinitialized. -STD_POS= Position to which the sequence must be reinitialized. OdiStartScen Starts a scenario. The optional parameter AGENT_CODE is used to dedicate this scenario to another agent than the current agent. The parameter SYNC_MODE can start a scenario in synchronous or asynchronous mode.
Required Properties: -SCEN_NAME=Name of the scenario to start -SCEN_VERSION=Version of the scenario to start. If the version specified is -1 , the last version of the scenario is executed. OdiUpdateAgentSchedule Forces a agent to recalculate its schedule of tasks. This has the same effect as manually telling the agent to update its schedule in Topology Manager. Required Properties: -AGENT_NAME=The name of the physical agent to update. 4)Internet OdiReadMail Can be used to read emails as well as their attachments from a POP or IMAP internet mail service. Required Properties: -MAILHOST=IP address of the POP or IMAP mail server -USER= Valid mail server account. -PASS= Password of the mail server account. -FOLDER= Full path of the storage folder for attachments and messages OdiSendMail Can send an email through an SMTP server. Required Properties: -MAILHOST= IP address of the SMTP server -FROM= Address of the sender of the message. -TO= List of email addresses of the recipients, separated by commas. Ex: Get IP address of the SMTP server from : C:\Documents and Settings\nikhil.battula>nslookup Default Server: appsav.appsahyd.com Address: 192.168.1.4 >set type=mx >yahoo.com Server: appsav.appsahyd.com Address: 192.168.1.4 Non-authoritative answer: yahoo.com MX preference = 1, mail exchanger = g.mx.mail.yahoo.com yahoo.com MX preference = 1, mail exchanger = h.mx.mail.yahoo.com yahoo.com MX preference = 1, mail exchanger = i.mx.mail.yahoo.com yahoo.com MX preference = 1, mail exchanger = j.mx.mail.yahoo.com
MX preference = 1, mail exchanger = k.mx.mail.yahoo.com MX preference = 1, mail exchanger = a.mx.mail.yahoo.com MX preference = 1, mail exchanger = b.mx.mail.yahoo.com MX preference = 1, mail exchanger = c.mx.mail.yahoo.com MX preference = 1, mail exchanger = d.mx.mail.yahoo.com MX preference = 1, mail exchanger = e.mx.mail.yahoo.com MX preference = 1, mail exchanger = f.mx.mail.yahoo.com internet address = 98.137.54.238 internet address = 66.94.236.34 internet address = 74.6.140.64 internet address = 67.195.168.31 internet address = 74.6.136.65 internet address = 206.190.54.127 internet address = 209.191.88.254 internet address = 67.195.168.230 internet address = 98.137.54.237
1.Open Designer Module 2.Expand package,go to Diagram tab,choose OdiSendMail tool in the toolbox and place it in the dark area.
Give SMTP IP address for mailserver option. From: To: Subject: Attachments: In the Command tab we get the command as OdiSendMail "-MAILHOST=209.191.88.254" "-FROM=ujahnavi_apps@yahoo.com" "SUBJECT=hello janu" "-TO=ujahnavi_apps@yahoo.com" .. click execute,OK Check Status in the Operator Module
5)Files
OdiFileAppend Concatenates a set of files into a single file. OdiFileCopy Copies files or a folder into files or a folder on the machine of the execution agent. OdiFileDelete Deletes, based on the file modification time, files or a directory on the machine running the agent which executes it. OdiFileMove Moves files or a directory into files or a directory on the machine of the execution agent. This command can also be used to rename files or directories. OdiFileWait Scans the directory DIR and waits for files matching the mask given in PATTERN . When the specified files are found, the action indicated by the parameter ACTION is triggered. The execution agent is in standby mode waiting to receive FILECOUNT files unless the timeout limit, given by TIMEOUT is reached. The directory is scanned every POLLINT milliseconds. OdiMkDir Creates a directory. If the parent directory does not exist, this command will recursively create the parent directories. OdiOutFile
Generates a text file on the machine on which the execution agent is running. OdiUnZip Used to unzip an archive file to to a directory. OdiZip Creates ZIP files from a directory or several files.
6)Event Detection
OdiFileWait Scans the directory DIR and waits for files matching the mask given in PATTERN . OdiReadMail Can be used to read emails as well as their attachments from a POP or IMAP internet mail service. OdiWaitForData Waits for a number of rows in a table or a set of tables. This can also be applied to a number of objects containing data, such as views. OdiWaitForTable Waits for a table to be created and populated with a pre-defined number of rows. Ex: 1.Open Designer Module 2.Expand package,go to Diagram tab,choose OdiWaitForData tool in the toolbox and place it in the dark area. Click on the tool and enter the properties in properties panel Context GLOBAL Global Row Count 1 Logical Schema mb_lschema Polling Interval 1000 Table Name [EMP100] Timeout 0 Timeout without Errors - YES Unit Row Count -10 3. Drag and drop the interface which you want to execute on success of OdiWaitForData tool and connect them. 4. Click on execute.
OdiWaitForLogData Waits for a number of modifications to occur on a journalized table or a list of journalized tables. OdiWaitForTable Waits for a table to be created and populated with a pre-defined number of rows.
3.17 Scheduling
Scheduling is required to automate the whole loading process with out our involvement in production environment. Scheduling Requirements : Need to create both the Physical and Logical Agents. Need to Configure the ODIPARAMS.bat file Need to create Scenario. Create Scheduling. Launching Scheduler Agent. Note: Agent Creation is done from Topology Manager. 1. Configuring ODIPARAMS batch File : For this go to the installation directory and give: edit odiparams.bat We need to modify the following parameters. ODI_SECU_DRIVER=oracle.jdbc.driver.OracleDriver(Jdbc driver used to connect to master repository) ODI_SECU_URL=jdbc:oracle:thin:@SHARP:1522:DEV01(Jdbc URL used to connect to master repository) ODI_SECU_USER=apps(data base account used to connect to master repository) ODI_SECU_ENCODED_PASS=d,yHhBVmdXKhkqnulNETykhjm(encrypted password of the database account) ODI_SECU_WORK_REP=WREP_DEV01(Name of the work repository to connect) ODI_USER=SUPERVISOR(ODI user used to launch a scenario) ODI_ENCODED_PASS=hZyHzHjVTvdbzORf4uVj2y(ODI user encrypted password) Encoding the Password: Use the following command to encode the password. Agent ENCODE <password>
2. Create a scenario for this project 3. Creating Scheduling Go to the Scenario, in that go to the Scheduling node, Right click on Scheduling and select insert scheduling
Assign the Context, agent, log level. Under the Status section check the active radio button, and Under the Execution section check the radio button Hourly.
Launching Scheduler Agent: ( Go to the bin folder of oracle -oracle->oracledi->bin) Launch the scheduler agent by running the following command. agentscheduler "-port=20910" "-v=5" (or) agentscheduler -name=<agent_name> -port=20910
After running the AgentScheduler we get the status as number of items scheduled for execution.
Go to the Physical agent in the topology manager and double click on the physical agent, in this we have three buttons test, update scheduling, scheduling information. click on the test button to test the Agent whether successful or not.
If So go to the Scheduling Information button and click on it we get the information like what are the scenarios that are scheduled and at what time they are going to execute
4. Security Manager
Using the Security Manager module, you can manage security in Oracle Data Integrator. The Security Manager module allows users and profiles to be created. It is used to assign user rights for methods (edit, delete, etc) on generic objects (data server, datatypes, etc), and to fine-tune these rights on the object instances (Server 1, Server 2, etc). The Security Manager module stores this information in a master repository. This information can be used by all the other modules. The Security Manager GUI appears as follow:
Creating a New Profile Creating a New Profile To create a new profile: Select profiles in the tree. Right click and select Insert profile. Enter the name. Click OK.
Deleting a Profile To delete a profile: Select the profile to be deleted. Right click and select Delete. Click OK. The profile disappears from the tree.
Creating a New User To create a new user: Select users from the tree. Right click and select Insert User. Enter the name , the initials and the user's password (by clicking on the input a password button) . Click OK.
The user is displayed in the tree. Deleting a User To delete a user: Select the user to be deleted. Right click and select Delete. Click on OK. The user disappears from the tree.
Assigning a Profile to a User To assign a profile to a user: Expand the tree to display the user you wish to assign the profile to. Select the profile you wish to assign, then drag and drop it onto the user in the tree. Click OK. The profile is assigned to the user. Removing a Profile from a User To remove a profile from a user: Expand the tree to display the profile you wish to delete, under the user branch. Select the profile (under the user) to be deleted from the tree. Right click and select Delete.
Click OK. The profile is removed from the user. Assigning an Authorization by Profile or User To assign an authorization by profile or by user: Expand the tree to display the user or the profile you wish to assign the authorization to. Under the object select the method you wish to assign then drag and drop it onto the user or the profile . Click OK. The authorization is assigned to the user or the profile. To assign authorizations on all the methods of an object to a profile or a user: Expand the tree to display the user or the profile you wish to assign the authorization to. Select the object to which you wish to assign all the methods then drag and drop it onto the user or the profile . Click OK. The authorizations on all the methods are assigned to the user or the profile. Deleting an Authorization by Profile or User To delete an authorization by profile or user: Select the method you wish to delete under the user or profile branch. Right click and select Delete. Click OK. The authorization is removed from the user or the profile. To delete the authorizations for all the methods of an object from a profile or a user: Select the object whose rights assignment you wish to delete under the user or the profile branch Right click and select Delete. Click OK. The authorizations are removed from the user or the profile. Assigning an Authorization by Object Instance To assign an authorization by object instance to a user: Expand the Security Manager tree to display the user to whom you want to add the authorization. Open Designer and expand its tree to display the object instance to which you want to assign an authorization. Select the object instance in the Designer tree, then drag and drop it onto the user in Security Manager . Fine-tune the rights in the authorization by object instance window which appears. You may want to implement following simple privileges policies on methods that you select in the list: to grant all these methods in all repositories, press the Allow selected methods in all repositories button. to deny all these methods in all repositories, press the Deny selected methods in all repositories button. to grant all these methods in certain work repositories, press the Allow selected methods in selected repositories button, then select the repositories in the list. Click OK. Authorization by object instance is assigned to the user or the profile. Note: Any method for which the user has generic rights is not listed in the authorization by object instance window.
Deleting an Authorization by Object Instance To delete an authorization by object instance from a user: Open the authorization by object instance window for the instance whose rights you want to change. Click the Deny All button. Click OK. The authorizations are removed from the user. If, after this operation, the user no longer has any rights on an instance, the instance disappears from the tree in Security Manager. You can remove all unused object authorizations by object instance using the Security Clean-up Tool. Cleaning up Unused Authorizations Authorizations by object instance are stored in the Master Repository. However, if these objects are deleted from all work repositories, the authorization are not deleted. Indeed, you may wish to retain certain unused authorizations if they refer to objects currently stored in an exported file or in a stored solution, for instance. The Security Clean-up Tool should be used periodically to remove these unused authorizations from the master repository. Unused authorizations are removed if they refer to objects that do not exist in the Master Repository or in any Work Repository. Note : All work repositories attached to the master repository must be accessible in order to check the existence of the objects in these repositories. Note: Oracle Data Integrator objects themselves are never removed - only security privilege information. To remove unused authorizations: Select File > Clean up Security Settings... from the menu. On the Cleanable tab, select any useless security settings you wish to remove. Security settings that cannot be cleaned are shown on the Non-cleanable tab Use the Select All button to select all settings for cleaning. Press Delete Selected Security Settings . Defining the Password Policy The password policy consists of a set of rules applied on user passwords. This set of rules is checked when the password is defined by the user. To define the password policy: In Security Manager , select File > Password Policy... The Password Policy window appears. In this window, a list of rules is displayed. Click the Add Rule button. A new rule definition window appears. A rule is a set of condition that are checked on passwords. Set a Name and a Description for this new rule. Add conditions on the password value or length. You can define for example a minimum length for the passwords from this window. Select if you want at least one condition or all of them to be valid to consider that the password meets the rule. Click OK . Add as many rules as necessary, and check those of the rules that you want active in your current policy. Only password meeting all the rules are considered as valid for the current policy. You can also define a period of validity for the passwords. Passwords older than this number of days will automatically expire. The users will have to change them. Click OK to update the password policy.
Setting User Parameters Oracle Data Integrator saves user parameters such as default directories, windows positions,etc. User parameters are saved in the userpref.xml file in /bin . To set user parameters: Select User Parameters from the File menu. In the Editing User Parameters window, change the value of parameters as required. Click OK to save and close the window . A list of the possible user parameters is available in the Reference Manual. Exporting an Object Exporting and importing allows you to transfer Oracle Data Integrator Objects from one repository to another. To export an object from Oracle Data Integrator: Select the object to be exported in the appropriate Oracle Data Integrator tree view. Right-click on the object, and select Export... . If this menu item does not appear, then this type of object does not have the export feature. Set the Export parameters, and click OK . You must at least specify the Export Name . If the Export Directory is not specified, then the export file is created in the Default Export Directory . The object is exported as an XML file in the specified location. Importing an Object Importing and exporting allows you to transfer objects (Interfaces, Knowledge Modules, Models, ...) from one repository to another. To import an object in Oracle Data Integrator: Select the appropriate object for import in the tree. If such an object does not exist, then create it. Right-click on the object, and select import , then the type of the object you wish to import. Specify the Export Directory , and select one or more files in Import Name . Select the Import Mode , then click on OK . The XML-formatted files are imported into the work repository, and become visible in Oracle Data Integrator. Exporting the Topology/Security Settings Exporting then importing the topology or security allows you to transfer a domain from one master repository to another. The domains that can be exported are given below: Topology : the full topology (logical and physical architectures). Logical Topology : technologies (connection, datatype or language information), logical agents and logical schemas. Security : users, profiles, privileges and hosts. This export is possible from Security Manager . Execution Environment : technologies, datatypes, contexts, physical schemas and agents. To export the topology/security: In Topology Manager (or Security Manager), select one of the following menu options of the File > Export menu : Topology... Logical Topology... Security Settings... Execution Environment...
Fill in the export parameters: Export to directory : Directory in which the export file(s) will be created. Export to zip file : When this option is selected, a unique compressed file containing all export files will be created. Otherwise, a set of export files is created. Zip File name : Name of the compressed file if the option Make a Compressed Export is selected. Character Set : Encoding specified in the export file. Parameter encoding in the XML file header. Java Character Set: Java character set used to generate the file. Click OK . The export file(s) are created in the Export Directory specified. Importing the Topology/Security Settings Exporting then importing the topology or security allows you to transfer a domain from one master repository to another. To import a topology export: In Topology Manager, click on File > Import then one of the following options: Topology ... Logical Topology... Execution environment... Select the Import Mode , the import Repository or Zip File. Click OK . The specified file(s) are imported into the master repository. To import a Security export: In Topology Manager, click on File > Import > Security Settings... then one of the following options: Select the Import Mode , the import Repository or Zip File. Click OK . The specified file(s) are imported into the master repository.
5.Operator Module
Introduction Operator is the module responsible for monitoring and production management. It is designed for production operators and shows the execution logs with errors counts, the number of rows processed, execution statistics etc. At design time, developers use Operator for debugging purposes Session A session is an execution of a scenario, an interface, a package or a procedure. A session is made up of steps which are themselves made up of tasks. A Session, step or task has six possible status values: Done, Error, Running, Warning (Tasks only) & Queued (Sessions only). Filtering Sessions in Operator Filtering log sessions allows you to display only certain execution sessions in Operator , by filtering on parameters such as the user, status or duration of sessions. Sessions that do not meet the current filter are hidden from view, but they are not removed from the log. To filter out certain sessions: - Open Operator. - Click on the Filter button on the toolbar. - Select the appropriate values for the following parameters. The settings by default select all sessions. - Click Apply for a preview of the current filter. - Click Ok . To deactivate the filter: - Click on the filter button on the toolbar if the Filter button appears to be pushed in. Displaying Scheduling Information The Scheduling Information allows you to visualize the agents' scheduled tasks. To Display the Scheduling information from Operator: - Open Operator - Click on the Scheduling Information button in the toolbar To Display the Scheduling information from Topology Manager: - Open Topology Manager - In Physical Architecture view, select the Physical Agent you wish to display the planning. - Right click and select Scheduling Information
Purging the Log Purging the log allows you to remove the information of past sessions. To purge the Log: - Open Operator - Click on the Purge Log button in the toolbar, or select File > Purge Log in the menu bar. - Set the date and time range for the sessions to delete, and optionally specify context , agent and user filters. - Click OK . Executing a Scenario from Operator To launch a scenario from Operator: - Select the required scenario in the Scenarios view (in Operator). - Right-click, then select Execute - Select the Context and the required Agent . To execute the scenario on the local machine, choose the Local (without agent) ) option. - Click OK . Before running a scenario, you need to have the scenario generated from Designer or imported from a file. To delete a scenario: - Right click the scenario and select Delete. - Click OK to confirm the deletion. Restarting a Session Sessions in status Error or Waiting can be restarted from Operator or through a command line.. To restart a session from Operator: - Open Operator . - Select the session to restart from the tree view. - Right-click then select Restart . - Click OK. To restart a session from an OS command: - Open Windows command prompt. - Change to the /bin directory under the Oracle Data Integrator installation directory. - Type in the following command: restartsession.bat <session number> ["-V=<verbose level>"] for Windows
Importing Scenarios in Production A scenario generated from Designer can be imported in a development or execution repository through the Operator Module, to run in production. To import a scenario in production: - Open Operator. - Select File > Import Scenario... in the menu bar, or right-click in the Scenarios view and select Import Scenario... . - Specify the Import Directory, and select one or more scenarios to import. - Select the Import Mode. - Check the Import schedules option if you want to import the schedules exported with the scenario as well. - Click OK. To import a solution: - Go to the Solutions view. - Right-click the tree view and select Import Solution... - Specify the Import Directory , and select one or more solutions to import. - Select the Import Mode. - Click OK. - Double click the solution to open it. - Select one or more scenarios that you wish to restore, from either the Principal Elements or Required Elements views. - Click the Restore button. Scenario Folders In Operator, scenarios can be grouped into scenario folders to facilitate organization. To create a scenario folder: - Go to the Scenarios view - Right-click anywhere on the tree view and select Insert scenario folder or click the Add Scenario Folder button. - Drag and drop scenarios to move them into the scenario folder. To delete a scenario folder: - Right click the scenario folder and select Delete
Using Session Folders Session folders automatically group sessions that were launched with certain keywords. Then, any session launched with all the matching keywords is automatically categorized beneath it. To create a new session folder:
- Go to the Session List view. - Right click the Keywords node and select Insert Session Folder . - Specify a Folder Name . - Specify one or more keywords. After entering each keyword, click the Add button to add it to the list. To launch a scenario with keywords: - Use the SnpsStartScen tool with the -KEYWORDS parameter. Keywords should be separated by commas. Importing and Exporting the Log Importing log data allows you to import into your Work Repository log files that have been exported for archiving purposes. To import the log: - Open Operator - Select File > Import > Import the log... in the menu bar. - In the Import of the Journal window select if you want to import the journal from a folder or from a ZIP file . - Specify the folder or ZIP file . - Click OK . Exporting log data allows you to export log files for archiving purposes. To export the log: - Open Operator - Select File > Export the log... in the menu bar. - In the Import of the Log window set the export parameters described below, and click OK .