Showing posts with label ODI. Show all posts
Showing posts with label ODI. Show all posts

Thursday, 21 March 2013

ODI By Examples




THE FOLLOWING ARE THE GUIDES

ODI FAQ
ODI 11g FAQ
Frequently asked questions for Oracle Data Integrator 11g
Oracle By Example

Product Technical Briefs

Getting Started Guides
Getting Started with SAP ABAP ERP Adapter
This document provides you with an overview of the possibilities offered by SAP ERP Knowledge Modules and how to use them in Oracle Data Integrator 10g.
Getting Started with SAP ABAP BW Adapter
This document provides you with an overview of the possibilities offered by SAP BW Knowledge Modules and how to use them in Oracle Data Integrator 10g.

Sample Code
Data Integration Sample Code (Knowledge Modules, ODI SDK Samples)
ODI Webcasts
Statement of Direction

Oracle Data Integrator Architecture


The architecture of Oracle Data Integrator relies on different components that collaborate together, as described in Figure: Functional Architecture Overview.



Repositories

The central component of the architecture is the Oracle Data Integrator Repository. It stores configuration information about the IT infrastructure, metadata of all applications, projects, scenarios, and the execution logs. Many instances of the repository can coexist in the IT infrastructure. The architecture of the repository is designed to allow several separated environments that exchange metadata and scenarios (for example: Development, Test, Maintenance and Production environments). In the figure above, two repositories are represented: one for the development environment, and another one for the production environment. The repository also acts as a version control system where objects are archived and assigned a version number. The Oracle Data Integrator Repository can be installed on an OLTP relational database.

The Oracle Data Integrator Repository is composed of a master repository and several Work Repositories. Objects developed or configured through the user interfaces are stored in one of these repository types.

There is usually only one master repository that stores the following information:

Security information including users, profiles and rights for the ODI platform

Topology information including technologies, server definitions, schemas, contexts, languages etc.

Versioned and archived objects.

The Work Repository is the one that contains actual developed objects. Several work repositories may coexist in the same ODI installation (for example, to have separate environments or to match a particular versioning life cycle). A Work Repository stores information for:

Models, including schema definition, datastores structures and metadata, fields and columns definitions, data quality constraints, cross references, data lineage etc.

Projects, including business rules, packages, procedures, folders, Knowledge Modules, variables etc.

Scenario execution, including scenarios, scheduling information and logs.

When the Work Repository contains only the execution information (typically for production purposes), it is then called an Execution Repository.


User Interfaces


Administrators, Developers and Operators use the Oracle Data Integrator Studio to access the repositories. This Fusion Client Platform (FCP) based UI is used for administering the infrastructure (security and topology), reverse-engineering the metadata, developing projects, scheduling, operating and monitoring executions.

Business users (as well as developers, administrators and operators), can have read access to the repository, perform topology configuration and production operations through a web based UI called Oracle Data Integrator Console. This Web application can deployed in a Java EE application server such as Oracle WebLogic.

ODI Studio provides four Navigators for managing the different aspects and steps of an ODI integration project:
Topology Navigator
Designer Navigator
Operator Navigator
Security Navigator

Topology Navigator

Topology Navigator is used to manage the data describing the information system's physical and logical architecture. Through Topology Navigator you can manage the topology of your information system, the technologies and their datatypes, the data servers linked to these technologies and the schemas they contain, the contexts, the language and the agents, as well as the repositories. The site, machine, and data server descriptions will enable Oracle Data Integrator to execute the same interfaces in different environments.

Designer Navigator

Designer Navigator is used to design data integrity checks and to build transformations such as for example:

Automatic reverse-engineering of existing applications or databases

Graphical development and maintenance of transformation and integration interfaces

Visualization of data flows in the interfaces

Automatic documentation generation

Customization of the generated code

The main objects you handle through Designer Navigator are Models and Projects.

Operator Navigator

Operator Navigator is the production management and monitoring tool. It is designed for IT production operators. Through Operator Navigator, you can manage your interface executions in the sessions, as well as the scenarios in production.

Security Navigator

Security Navigator is the tool for managing the security information in Oracle Data Integrator. Through Security Navigator you can create users and profiles and assign user rights for methods (edit, delete, etc) on generic objects (data server, datatypes, etc), and fine-tune these rights on the object instances (Server 1, Server 2, etc).


Design-time Projects


A typical project is composed of several steps and milestones.

Some of these are:

Define the business needs

Identify and declare the sources and targets in the Topology

Design and Reverse-engineer source and target data structures in the form of data models

Implement data quality rules on these data models and perform static checks on these data models to validate the data quality rules

Develop integration interfaces using datastores from these data models as sources and target

Develop additional components for tasks that cannot be achieved using interfaces, such as Receiving and sending e-mails, handling files (copy, compress, rename and such), executing web services

Integrate interfaces and additional components for building Package workflows

Version your work and release it in the form of scenarios

Schedule and operate scenarios.

Oracle Data Integrator will help you cover most of these steps, from source data investigation to metadata lineage, and through loading and data quality audit. With its repository, Oracle Data Integrator will centralize the specification and development efforts and provide a unique architecture on which the project can rely to succeed.

Run-Time Agent

At design time, developers generate scenarios from the business rules that they have designed. The code of these scenarios is then retrieved from the repository by the Run-Time Agent. This agent then connects to the data servers and orchestrates the code execution on these servers. It retrieves the return codes and messages for the execution, as well as additional logging information – such as the number of processed, execution time etc. - in the Repository.

The Agent comes in two different flavors:

The Java EE Agent can be deployed as a web application and benefit from the features of an application server.

The Standalone Agent runs in a simple Java Machine and can be deployed where needed to perform the integration flows.

Both these agents are multi-threaded java programs that support load balancing and can be distributed across the information system. This agent holds its own execution schedule which can be defined in Oracle Data Integrator, and can also be called from an external scheduler. It can also be invoked from a Java API or a web service interface.


An integration interface is an Oracle Data Integrator object stored that enables the loading of one target datastore with data transformed from one or more source datastores, based on declarative rules implemented as mappings, joins, filters and constraints.

An integration interface also references the Knowledge Modules (code templates) that will be used to generate the integration process.

Datastores


A datastore is a data structure that can be used as a source or a target in an integration interface. It can be:

a table stored in a relational database

an ASCII or EBCDIC file (delimited, or fixed length)

a node from a XML file

a JMS topic or queue from a Message Oriented Middleware

a node from a enterprise directory

an API that returns data in the form of an array of records

Regardless of the underlying technology, all data sources appear in Oracle Data Integrator in the form of datastores that can be manipulated and integrated in the same way. The datastores are grouped into data models. These models contain all the declarative rules –metadata - attached to datastores such as constraints.


Declarative Rules


The declarative rules that make up an interface can be expressed in human language, as shown in the following example: Data is coming from two Microsoft SQL Server tables (ORDERS joined to ORDER_LINES) and is combined with data from the CORRECTIONS file. The target SALES Oracle table must match some constraints such as the uniqueness of the ID column and valid reference to the SALES_REP table.

Data must be transformed and aggregated according to some mappings expressed in human language as shown in Figure: Example of a business problem.



Data Flow


Business rules defined in the interface are automatically converted into a data flow that will carry out the joins filters, mappings, and constraints from source data to target tables.

By default, Oracle Data Integrator will use the Target RBDMS as a staging area for loading source data into temporary tables and applying all the required mappings, staging filters, joins and constraints. The staging area is a separate area in the RDBMS (a user/database) where Oracle Data Integrator creates its temporary objects and executes some of the rules (mapping, joins, final filters, aggregations etc.). When performing the operations this way, Oracle Data Integrator behaves like an E-LT as it first extracts and loads the temporary tables and then finishes the transformations in the target RDBMS.

In some particular cases, when source volumes are small (less than 500,000 records), this staging area can be located in memory in Oracle Data Integrator's in-memory relational database – In-Memory Engine. Oracle Data Integrator would then behave like a traditional ETL tool.

Figure: Oracle Data Integrator Knowledge Modules in action shows the data flow automatically generated by Oracle Data Integrator to load the final SALES table. The business rules will be transformed into code by the Knowledge Modules (KM). The code produced will generate several steps. Some of these steps will extract and load the data from the sources to the staging area (Loading Knowledge Modules - LKM). Others will transform and integrate the data from the staging area to the target table (Integration Knowledge Module - IKM). To ensure data quality, the Check Knowledge Module (CKM) will apply the user defined constraints to the staging data to isolate erroneous records in the Errors table.





Oracle Data Integrator Knowledge Modules contain the actual code that will be executed by the various servers of the infrastructure. Some of the code contained in the Knowledge Modules is generic. It makes calls to the Oracle Data Integrator Substitution API that will be bound at run-time to the business-rules and generates the final code that will be executed.

At design time, declarative rules are defined in the interfaces and Knowledge Modules are only selected and configured.
At run-time, code is generated and every Oracle Data Integrator API call in the Knowledge Modules (enclosed by <% and %>) is replaced with its corresponding object name or expression, with respect to the metadata provided in the Repository. The generated code is orchestrated by Oracle Data Integrator run-time component - the Agent – on the source and target systems to make them perform the processing, as defined in the E-LT approach.

Internal ID


This post is all about explaining how the internal Id in ODI plays and acts and what is the best practice and what actually happens when most of the error occurs.
Internal Id is an important element in ODI when its comes to Import Export of Object  and so a good understanding of it can help us save time and above all errors related to Export Import . Internal ID apart being an incrementing code also have relationship to Work and Master Repository.
If you are keen on knowing how the internal Id objects have relation to Work (WR)and Master Repository. Here is the key , when you define your 2 or 3 digit number as the Internal ID to Repository,  this number(WR Internal ID )  acts like a base in all the object.
Lets say my internal id for Work Repository is 200 .
My project will have the Internal Id – 1200
My Folder will have Internal Id of -2200
Interface1 – 2200
Interface2- 3200
Interface3- 1200
Variable1- 2200.
If you have notice all the  different objects have suffix 200( WR Internal ID) to their incrementing number.
Note : ODI can have same internal Id for different objects but never for the same object , since while storing into the SNP work repository tables it increments accordingly by 1000 or other number as defined in its code.
Similarly its go for the Master Repository.
Lets say my Master Repository internal id 50 .
Data Server 1- 2050
Data Server 2 – 3050
Physical Schema1 – 1050
Physical Schema2 – 2050
Agent1-1050
[Please note some of the Default Data server created while Initial ODI installation( Ex - XML_GEO_DIM , File Generic etc ) will have different Internal ID not matching to your Master Repository IDs. ]
We hope till now , you would have got an good understanding of ODI objects , Repository Internal ID and their relationship .
Now let us cover the importance and the simple reason why Repositories needs to have different Internal ID and which is the highly suggested and the best practise to do with few simple scenarios.
Scenario 1. Two DWR having the same internal ID.
Let  us call them DWR1 and DWR2. They both have the same Internal Id of 200 connected to different Master Repository.
Let say the development is being carried on both the repository or even the Development and Testing environment are both DWR .
Let say we  have reveresed engineered datastore
Source Data Store – ABC (121200)
Target Data Store – TRG (321200)
and created the interface
Interface – XMT_Interface (143200)
The  interface is ready and some body in the DWR2  needs the same Interface for their development build, but unfortunately some one have already reverse engineered the same in the DWR and their internal id as follows .
Source Data Store – ABC (122200)
Target Data Store – TRG (322200)
When the Developer brings in the Interface XMT_Interface thinking that already the Data store exists in the DWR2 , it will fail because of improper links , since  Interface will look for  Datastore ABC (121200) and TRG (321200) and KM and when they are different it error out.
To solve this  either the developer have to bring in Duplicate mode and also change the Source Data stores. In short he pretty much needs to rebuild the whole interface again.
The other sub scenario can be Interface with the same ID can exists with different name and code. Again the Developer need to import in duplicate and assign the required datastore , in short rework .
Scenario 2. Two DWR having different internal ID.
Let me call them DWR1 and DWR2. They both have the different internal id of 200 and 201 respectively.
Now let say the developer is moving the Interface from the DWR1 to DWR 2. Well even now the interface will fail , not becuase of  Interface id but becuase of the datastore can be different in the DWR2.
To solve this we need to import the data store too but that’s like having two different data stores for single purpose.
Well here the steps to handle in such a scenarios.
Step1. Sync Global Variable and other Global ODI objects , Model and Datastore when imported in One DWR to another DWR immediately or at the End of the Day.
Step2. Import the Codes in this order Project, KM, Local Variable, Procedure ,Interface and Packages for sync [ if required ]
Well with these above conditions in place, the Developer can easily migrate the code since
Source Data Store – ABC (121200)
Target Data Store – TRG (321200)
Will be present and also being a different internal id we can use Insert_Update and import the codes.
Above all, what ever is the case, have different internal Id for Work Repository.
Scenario -3 Importing from DWR to EWR having the same internal ID. (100)
Lets take an Example where a Developer in the EWR makes a duplicate of the present scenario ( 12100) and ODI starts to define a new Internal ID but here comes the issue  , that
scenario internal id can be already present  or in future when trying to import makes an Error ,since we have created a duplicate interface with that ID.
Scenario -4 Importing from DWR to EWR having the different internal ID. (200 and 201 )
If in case the EWR would have had a different internal id DWR (200) and EWR (201) , so when the Developer make a duplicate the new id will be with some thing like this (12201) and so its does not updates or modify or error out because of an existing object or even when importing scenarios in the future ,since the base ODI have scenarios with numbers ending in 200 .
Similarly the Internal ID can impact for different ODI objects in Topology and Security Manager, so use different Internal ID.
I hope this post have given you an understanding and the relationship of Internal Id , its generation method and why the objects gets error out.
Moral of the Post – Create Different internal Id for both Work Repository and Master Repository always .
Look for odiexperts.com for more tips, tricks and best practices.

ODI 11g Agent Logging / Tracing


Hi Reader!
How are you? I hope you are having a really great day!
Here we have a new post about Agents… enjoy it!
In ODI 10g tracing was simple as starting the agent with Trace level( -V=2) Parameters where as in ODI 11g the tracing information needs to modified in the ODI-logging-config.XML
This post describes the different parameters and settings to configure ODI-logging-config.xml.
ODI-logging-config.xml is used to configure the ODI logging/tracing system .The different mapping between Java log levels and OJDL (Oracle Java Debugging Level) log levels is as follows:
Java log levelOJDL log level
SEVERE intValue()+100INCIDENT_ERROR:1
SEVEREERROR:1
WARNINGWARNING:1
INFONOTIFICATION:1
CONFIGNOTIFICATION:16
FINETRACE:1
FINERTRACE:16
FINESTTRACE:32
and you can find the sufficient level of information about different parameters in the initial few lines of the ODI-logging-config.xml.
image
There are two ways you can edit the ODI-logging-config.xml . One at ODI Studio level and the other at Agent Level.
For ODI Studio: $ODI_HOME/oracledi/client/odi/bin/ODI-logging-config.xml
For ODI Standalone Agent  : $ODI_HOME/oracledi/agent/bin/ODI-logging-config.xml .
Lets configure this xml with Trace level 32, highest level for Standalone Agent.

Step 1 . Change Log Folder and other settings  (Optional)

By default Logs are generate at the default agent folder usually%ODI_HOME%\oracledi\agent\log\${LOG_FILE}
If you wish to assign a different folder , you can modify the highlighted line and provide the appropriate path. You can also change the other setting if required.
image

Step 2 .  Add Tracing Level

Add the following line under the loggers

<logger level="TRACE:32" name="oracle.odi.agent" useparenthandlers="false">
<handler name="ODI-file-handler" />
<handler name="ODI-console-handler" />
</logger>
image

Step 3 . Stop and Start the Agent again

Note :- The higher the tracing value the maximum the logging and so will be the file Size.

Configuring the ODI Logging at ODI Studio level (ie. Local(No Agent) )

In ODI 11g we can configure the logging configuration similar to above at ODI Studio level.
In order to do so go to ODI 11g Home and browse into Folder oracledi\client\odi\bin and edit ODI-logging-config.xml
$ODI_11G_HOME\oracledi\client\odi\bin\ODI-logging-config.xml and the logs are generated by default at
$ODI_11G_HOME\oracledi\client\log\studio.log
Similar to the above add appropriate tracing parameters and change other parameters if required

<logger name="oracle.odi.agent" level="TRACE:32" useParentHandlers="false">
<handler name="ODI-file-handler"/>
<handler name="ODI-console-handler"/>
</logger>

Close and restart your ODI Studio and logging/tracing parameters will be reflected.
Thanks for visiting us.

Email Reporting in ODI

Success/Failure Notification Of scenario/Package through Email.


I would describe how to generate scenario notification through email.
In figure below there is main package which contains two scenario's A & B and two email notifications one for success & other for failure.




Failure email notification will be generated when the execution of the main package fails & similarly the Success email notification will be generated when the package is executed successfully. 


1.Just Drag the Scenario A & B within the main package for which success/failure notification is to be sent.


2. From tool box drag the OdiSendMail & name it as SUCCESS.
3. From tool box drag the OdiSendMail & name it as FAILURE.
4. Open the properties tab of SUCCESS  add the following details.



Mail Server: 162.22.120.56
From:         ODI@gmail.com
To:                  [neeraj.singh@gmail.com]
Cc:                neeraj.singh@gmail.com
BCc:              neeraj.singh@gmail.com
Subject::  SUCESSFULLY EXECUTED:<%=odiRef.getSession( "SCEN_NAME" )%>  Load.
Attcahment: ABC.xml
Body:
Session Name: <%=odiRef.getSession( "SCEN_NAME" )%>
Session Version: <%=odiRef.getSession( "SCEN_VERSION" )%>
Session No: <%=odiRef.getSession( "SESS_NO" )%> 
Context: <%=odiRef.getSession( "CONTEXT_NAME" )%>
Session Begin: <%=odiRef.getSession("SESS_BEG")%>


4. Open the properties tab of FAILURE add the following details.



Mail Server: 162.22.120.56
From:         ODI@gmail.com
To:                  [neeraj.singh@gmail.com]
Cc:                neeraj.singh@gmail.com
BCc:              neeraj.singh@gmail.com
Subject::  FAILED :<%=odiRef.getSession( "SCEN_NAME" )%>  Load.
Body:
Session Name: <%=odiRef.getSession( "SCEN_NAME" )%>
Session Version: <%=odiRef.getSession( "SCEN_VERSION" )%>
Session No: <%=odiRef.getSession( "SESS_NO" )%> 
Context: <%=odiRef.getSession( "CONTEXT_NAME" )%>
Session Begin: <%=odiRef.getSession("SESS_BEG")%>





5. Link Scenario  "B" to the step "SUCCESS"  through the "OK" ( Next step on success) present on the upper pane 
6. Link Scenario  "B" to the step "FAILURE"  through the "KO" ( Next step on failure) present on the upper pane 




7. Now generate the scenario for the main package example "C"  with version "001".




Execute the Scenarion "C".


If it is successfully executed you will recieve the success email.






If it is failed you will recieve the Failure email.










You can add & modify the notification reports. For this you can do some tricky things
for example you do the following tasks.
can export the log through attachment.
can create the log of the previous step through "<%=odiRef.getPrevStepLog( <> )%>"
can the add the target schema & source schema deatils through


"<%=odiRef.getSchemaName("D")%>" & "<%=odiRef.getCatalogName( "D")%>"

Previous Session/Scenario/Package details

odiRef.getPrevStepLog: oracle data Integrator


In this artice i will try to explain how to extract the log/details for mutiple scenarios in ODI.
What is odiRef.getPrevStepLog ?
This ODI inbuilt function helps to identify or create reports at package/scenario level.
odiRef.getPrevStepLog : Returns information about the most recently executed step in a package. The information requested is specified through the parameter. If there is no previous step (for example, if the getPrevStepLog step is executed from outside a package), the exception "No previous step" is raised. 


I will try to explain one of the scenario where it can be used.
for an example suppose I have a main package/scenario "COPY_OF_PKG_LOAD_SUZ_CORNERSTONE" which includes/calls  Scenario "COPY_OF_PKG_SUZ_ORA_PGSQL version 001".

I want to create the log for the Scenario "COPY_OF_PKG_SUZ_ORA_PGSQL version 001".
Add these codes (in the image below )in the email body ( the mail notification step which i have already posted in previous article "Email Reporting in ODI" )  or the ouput log file & add it to the main pacakage "COPY_OF_PKG_LOAD_SUZ_CORNERSTONE"



I have added these deatils in my SUCCESS & FAILURE notification email step and added it to the main package "COPY_OF_PKG_LOAD_SUZ_CORNERSTONE"



Previous Session/Scenarion/Package details


MESSAGE: <%=odiRef.getPrevStepLog("MESSAGE")%>
step name: <%=odiRef.getPrevStepLog("STEP_NAME")%>
Session No: <%=odiRef.getPrevStepLog("SESS_NO")%>
Steps Number: <%=odiRef.getPrevStepLog("NNO")%>
Step Type: <%=odiRef.getPrevStepLog("STEP_TYPE")%>
Step Begin: <%=odiRef.getPrevStepLog("BEGIN")%>
Step End: <%=odiRef.getPrevStepLog("END")%>
Step Duration: <%=odiRef.getPrevStepLog("DURATION")%>
Step Status: <%=odiRef.getPrevStepLog("STATUS")%>



MESSAGE: Error log if there is any error / or empty if COPY_OF_PKG_SUZ_ORA_PGSQL version 001 executed successfully.  Error message returned by previous step, if any. Blank string if no error.
step name:   Name of the previous step of the package "COPY_OF_PKG_SUZ_ORA_PGSQL version 001'
Session No: Session number of previous step
Steps Number:  Number of steps
Step Type:  
  • F: Interface
  • VD: Variable declaration
  • VS: Set/Increment variable
  • VE: Evaluate variable
  • V: Refresh variable
  • T: Procedure
Step Begin:  The date and time that the step began
Step End: The date and time that the step terminated
Step Duration: Time the step took to execute in seconds
Step Status: Returns the one-letter code indicating the status with which the previous step terminated. The state R (Running) is never returned.
  • D: Done (success)
  • E: Error
  • Q: Queued
  • W: Waiting
  • M: Warning



I created the Scenario for the package "COPY_OF_PKG_LOAD_SUZ_CORNERSTONE" & executed it. 
THe OUPUT of odiRef.getPrevStepLog step which i recieved through email is :



Failure loading of Package


Session Name:           COPY_OF_PKG_LOAD_SUZ_CORNERSTONE
Session Version:  002
Session No:       21768001
Context:          Development

Failure details

step name:  Execution of the Scenario COPY_OF_PKG_SUZ_ORA_PGSQL version 001
Session No: 21768001
Steps Number:     1
Step Type:  SE
Step Begin: 2010-11-08 01:57:03.644
Step End:   2010-11-08 01:57:05.84
Step Duration:    2
Step Status:      E

Error Message:    java.lang.Exception: The scenario did not end properly.
      at com.sunopsis.dwg.dbobj.SnpScen.a(SnpScen.java)
      at com.sunopsis.dwg.dbobj.SnpScen.localExecuteSync(SnpScen.java)
      at com.sunopsis.dwg.tools.StartScen.actionExecute(StartScen.java)
      at com.sunopsis.dwg.function.SnpsFunctionBaseRepositoryConnected.execute(SnpsFunctionBaseRepositoryConnected.java)
      at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execIntegratedFunction(SnpSessTaskSql.java)
      at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTaskTrt(SnpSessTaskSql.java)
      at com.sunopsis.dwg.dbobj.SnpSessTaskSqlS.treatTaskTrt(SnpSessTaskSqlS.java)
      at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java)
      at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java)
      at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java)
      at com.sunopsis.dwg.cmd.DwgCommandScenario.treatCommand(DwgCommandScenario.java)
      at com.sunopsis.dwg.cmd.DwgCommandBase.execute(DwgCommandBase.java)
      at com.sunopsis.dwg.cmd.e.k(e.java)
      at com.sunopsis.dwg.cmd.h.A(h.java)
      at com.sunopsis.dwg.cmd.e.run(e.java)
      at java.lang.Thread.run(Thread.java:619)