PDF To Word
PDF To Word
PDF To Word
Version 6.4
The Programs (which include both the software and documentation) contain proprietary information;
they are provided under a license agreement containing restrictions on use and disclosure and are also
protected by copyright, patent, and other intellectual and industrial property laws. Reverse engineering,
disassembly, or decompilation of the Programs, except to the extent required to obtain interoperability
with other independently created software or as specified by law, is prohibited.
The information contained in this document is subject to change without notice. If you find any
problems in the documentation, please report them to us in writing. This document is not
warranted to be error-free. Except as may be expressly permitted in your license agreement for
these Programs, no part of these Programs may be reproduced or transmitted in any form or by
any means, electronic or mechanical, for any purpose.
If the Programs are delivered to the United States Government or anyone licensing or using the
Programs on behalf of the United States Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, endpoints, and related documentation and
technical data delivered to U.S. Government customers are "commercial computer software" or
"commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-
specific supplemental regulations. As such, use, duplication, disclosure, modification, and
adaptation of the Programs, including documentation and technical data, shall be subject to the
licensing restrictions set forth in the applicable Attunity license agreement, and, to the extent
applicable, the additional rights set forth in FAR 52.227-19, Commercial Computer Software—
Restricted Rights (June 1987). Attunity Ltd., 70 Blanchard Road, Burlington, MA 01803
The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other
inherently dangerous applications. It shall be the licensee's responsibility to take all appropriate
fail-safe, backup, redundancy and other measures to ensure the safe use of such applications if
the Programs are used for such purposes, and we disclaim liability for any damages caused by
such use of the Programs.
Attunity is a registered trademark of Attunity Ltd and/or its affiliates. Other names may be
trademarks of their respective owners.
The Programs may provide links to Web sites and access to content, products, and services from third
parties. Attunity is not responsible for the availability of, or any content provided on, third-party Web
sites. You bear all risks associated with the use of such content. If you choose to purchase any products
or services from a third party, the relationship is directly between you and the third party. Attunity is not
responsible for: (a) the quality of third-party products or services; or (b) fulfilling any of the terms of the
agreement with the third party, including delivery of products or services and warranty obligations
related to purchased products or services. Attunity is not responsible for any loss or damage of any sort
that you may incur from dealing with any third party.
Contents
1 | Introduction 45
Replication Explained 45
Attunity Replicate 46
Limitations and Considerations 47
System Architecture 49
Replication Tasks 50
Using Multiple Tasks 50
3 | Security Considerations 77
Securing Access to the Attunity Replicate Web UI 77
Setting Up Replicate Console HTTPS Support 78
Checking if an SSL Certificate is Installed 78
Using the Self-Signed Certificate 78
Glossary 1187
Index 1188
Note The term "endpoint" is used generically throughout this guide to refer to a data
repository that can be used as a source and/or target in an Attunity Replicate task.
Examples of such repositories include relational databases (such as Oracle) and files.
In this chapter:
▶ Replication Explained
▶ Attunity Replicate
▶ Limitations and Considerations
▶ System Architecture
▶ Replication Tasks
▶ Full Load and CDC Processes
▶ Replication Topologies
Replication Explained
Replication is a process that keeps two or more collections of computerized
information identically synchronized. It facilitates:
Load reduction: Keeping a complete or partial copy of a collection on a
different server reduces the load on the main server.
Improved service: Accessing a copy of the data can provide better service to
users than having them access the original data..
Restricted data access: If some users should only have access to a subset of data,
replicating only part of a collection makes it easy to enforce security restrictions.
Geographic distribution: Making only a subset of data relevant to a specific node
(or location) available is beneficial in widely distributed enterprises (such as a chain
of retail stores or warehouses). You can still make all data available at a central
location for less frequent use.
Integrity: The target data must reflect the complete result of all changes made to
the source data during the replication process.
Consistency: If a change affects different tables, rows, or files, the copy must
reflect these changes consistently across all affected tables, rows, or files.
Latency: The replication process must aim at keeping latency at a minimum.
Ideally, it should not exceed a few seconds.
Attunity Replicate
Attunity Replicate is a simple, powerful, easy-to-implement solution that
provides replication between various endpoints. Replicate lets you:
Full Load Replication: Creates files or tables at the target endpoint, automatically
defines the metadata that is required at the target, and populates the tables with
data from the source
Change Processing, also called Change Data Capture (CDC): Captures changes in
the source data or metadata as they occur and applies them to the target endpoint
as soon as possible in near-real time
Replication is log based, which means that it reads only the changes. This reduces
the impact on the source endpoints.
When replicating a table which has no Primary Key on the source endpoint, LOB
columns are removed from the table at the target endpoint as no Unique Index
column is created in the source table.
When the Limit LOB size to option is enabled, replication of structured data LOBs
(e.g. XML, JSON, IMAGE, etc.) may truncate (and thereby invalidate) the structured
data in the target LOB.
In Batch Optimized Apply mode, if the target table has more columns than the
source table, any values in the extra columns will be replaced with NULL.
The workaround is to create two tasks. One task for the target table(s) with extra
columns and the other task for the source table(s) which have the same number of
columns as the target tables. Then, run the task for the target table(s) with extra
columns in Transactional Apply mode and run the other task (where the target tables
When Replicate creates a new table in the target endpoint, it defines only one index
on the table. The index will either be the Primary Key or the first Unique Key
(according to alphabetical order) of the table. No other indexes will be defined in the
target. If additional indexes are required, these will need to be defined manually.
If a Unique Index/Primary Key in any of the source tables contains NULL values
in multiple rows, UPDATE and DELETE operations on one of the rows will UPDATE
/DELETE all of the target rows (in the Unique Index/Primary Key) that have a
NULL value.
LOB columns are always created as nullable on the target database. If you create
the target table(s) manually, then you must set all LOB columns to nullable.
If you stop a task after Full Load completes, perform some changes on the source
tables, and later resume the task from timestamp (by selecting the Start
processing changes from run option), some changes may not be replicated to the
target. This usually only happens if the transaction logs in the source database have
been deleted due to a log purge policy. In this case, Replicate will resume the task
from the last change in the current transaction log.
When replicating tables without a Primary Key, there is no way to verify whether
a record already exists on the target. This may result in data inconsistency when
UPDATE and DELETE operations are performed on the target database.
Replication of calculated values is not supported during Change Processing.
If a task fails with a recoverable error on the target while it is starting, it will not
read changes from the source.
Cached changes may be duplicated in a target table that does not have a
Unique Index.
A unique index consisting of several ascending and descending columns will always
be replicated to the target as ascending columns. In other words, the descending
columns will become ascending columns.
When the source table contains an identity column, Replicate does not create
the identity column on the target table. In this case, the table will need to be
created manually on the target endpoint.
Replication of tables with the same name as any of the Replicate Control tables is
not supported. For a list of the Control Table names, see Control Tables.
In this diagram, the source data and metadata are part of the source server. The
transaction log reader can be on the source server (for efficiency) or on the Attunity
Replicate server (for zero footprint on the source). Filtering and compression of the
source rows/logs can occur on the source or Attunity Replicate servers.
In the initial load process, Attunity Replicate reads a filtered stream of rows (with
relevant columns only) and passes them to the transformation process for further
filtering and subsequent writing to the target endpoint (in the expected output format).
The CDC process obtains a stream of filtered events or changes in data or metadata
from the transaction log file. It then buffers all changes for a given transaction into a
single unit before forwarding them to the target when the transaction commits. During
the initial load process, CDC also buffers all changes that occur within a transaction until
all affected tables have been loaded.
The Designer/Console server, which is part of the Replication server, is a Web-based
application that serves as the user interface for dealing with designing or modifying
the replication system and displaying and controlling its operation.
You can add new tables to an existing target without reloading the existing tables.
Similarly, you can add or drop columns in previously populated target tables
without reloading.
The CDC process captures changes in the source data or metadata as they occur and
applies them to the target endpoint as soon as possible in near real time. It captures and
applies the changes as units of single committed transactions and can update several
different target tables as the result of a single source commit. This guarantees
transactional integrity in the target endpoint. The CDC process for any file or table starts
as soon as the data load process for the file or table begins.
CDC operates by reading the recovery log file of the source endpoint management
system and grouping together the entries for each transaction. The process employs
techniques that ensure efficiency without seriously impacting the latency of the target
data. If the CDC process cannot apply the changes to the target within a reasonable
amount of time (for example when the target is not accessible), it buffers the changes on
the Replication server for as long as necessary. There is no need to re-read the source
DBMS logs, which may take a long time.
Replication Topologies
Attunity Replicate supports the following topologies for replication tasks:
One to One
Logical Independence
Hub and Spoke
One to One
In a one-one topology, there is one source and one target endpoint. When the source and
target endpoints are distinct, Attunity Replicate guarantees transactional integrity and
Caution: If the same row in a table is updated by two different replication tasks, the
result of two-way synchronization may be unpredictable. A problem can occur even if
two different rows are referentially related, that is if some application updates a row
based on reading a value in a different row. If the rows are updated concurrently on
1
the source and the target, the result may be unpredictable . Such occurrences are
rare, but they can occur.
Logical Independence
Two-way replication works best when updates of a row on a source and on a target are
entirely autonomous and do not affect each other. There is an assumption that any table
or a horizontal or vertical segment of a partitioned table can only be updated in one
source. Attunity Replicate allows updating the same row in several places, but in this
case, the columns being updated must be distinct. Another assumption is that if a data
value in one row depends on or is derived from a value in another row, the values can
be changed only on the same server but nowhere else (except by the Replicator). This is
called logical independence. With logical independence, concurrent update conflicts
cannot occur during replication.
1
CDC has no way of knowing exactly when a row was read by an application on one system relative to its
having been changed on another system. Read operations are typically not logged.
Important: To work, Attunity Replicate needs to be set up with the proper security
configuration. It is therefore strongly recommended to review Security
Considerations before using the product for the first time.
In this chapter:
▶ Installation Prerequisites
▶ Installing or Upgrading Attunity Replicate on Windows
▶ Installing Attunity Replicate on Linux
Installation Prerequisites
This section describes how to prepare your system to use Attunity Replicate. The
requirements differ according to the platform on which you want to install Attunity
Replicate and according to the desired Attunity Replicate UI Server configuration. For
more information on the available UI Server configurations, see Attunity Replicate UI
Server Configurations.
Software Requirements
Supported Endpoints
Software Requirements
This section describes what software is required to work with Attunity Replicate.
Windows Software Requirements
Linux Software Requirements
Windows Permissions
Attunity Replicate needs to be installed as an Administrator.
The following privileges are required to start the Attunity Replicate UI Server service (which
is run as a local system service), but are dropped as soon as the service is started:
SE_CREATE_GLOBAL_NAME
SE_CREATE_PAGEFILE_NAME
SE_CREATE_PERMANENT_NAME
SE_CREATE_SYMBOLIC_LINK_NAME
SE_CREATE_TOKEN_NAME
SE_DEBUG_NAME
SE_ENABLE_DELEGATION_NAME
SE_IMPERSONATE_NAME
SE_INC_BASE_PRIORITY_NAME
SE_INCREASE_QUOTA_NAME
SE_INC_WORKING_SET_NAME
SE_LOAD_DRIVER_NAME
SE_LOCK_MEMORY_NAME
SE_MACHINE_ACCOUNT_NAME
SE_MANAGE_VOLUME_NAME
SE_PROF_SINGLE_PROCESS_NAME
Network 1 Gb 10 Gb Two 10
Gb
Supported Endpoints
To replicate data using Attunity Replicate, you must be sure to have a supported version
of the endpoint you are working with available. For information about the endpoints you
can use with Attunity Replicate, see Supported Platforms and Endpoints .
Later, if you need to start or stop the Attunity Replicate Server, see the following section:
Starting and Stopping the Attunity Replicate Server on Windows
Note In the setup wizard’s Replication Server Location screen, one of the options
is Connect to a remote Linux Attunity Replicate Server. You should only select
this option if you have already installed Attunity Replicate Server on a Linux machine.
If you select this option, you will be prompted for the IP address and port number of
the Linux machine in the following screen.
For more information on installing Attunity Replicate Server on Linux, see
Installing Attunity Replicate on Linux.
All of the data that is created when you use Attunity Replicate is stored in a directory
called data. By default, this directory is located in the installation directory where you
install Attunity Replicate. If you want to create the data directory in a different location,
select this option in the installation wizard.
If you elect to create the data directory in a different location, all command line
actions must include the -d your_data_directory_path parameter where
your_data_ directory_path is the location of the "data" directory.
Upgrade Instructions
Method 1:
Back up the Replicate "data" directory to a location outside the product folder. This
method is recommended in most cases. However, if your "data" folder is very large
and there is not enough space to back it up, then use Method 2 below.
3. If you notice an issue with the upgrade, you can either revert to the previous
version as described below or do one of the following depending on which backup
method you chose above.
If you chose Method 1:
1. Back up the Replicate "data" directory to a location outside the product folder.
2. Uninstall the product and then reinstall to the same directory.
3. Once the installation is complete, stop all Replicate services.
4. Overwrite the "data" directory with the "data" directory that you backed up earlier.
5. Start all Replicate services.
From the Start menu on the Windows computer where Attunity Replicate is
installed, find Attunity Replicate; then select either Stop Attunity Replicate
Server or Start Attunity Replicate Server.
Note Before commencing the installation, make sure that the prerequisites have
been met.
1. From the directory containing the Attunity Replicate setup file, run the
following command (note that this will also install Attunity Replicate):
AttunityReplicate_<version-build>_X64.exe /r
/f1<my_response_file> where:
Example:
AttunityReplicate_<version-build>_X64.exe /r
/f1C:\Replicate_ install.iss
Example:
C:\>AttunityReplicate_<version-build>_X64.exe /s
/f1C:\temp\1\Replicate_ install.iss /f2C:\temp\1\silent_x64_install.log
If the installation was successful, the log file should contain the following rows:
[ResponseResult]
ResultCode=0
Example:
C:\>AttunityReplicate_<version-build>_X64.exe /s
/f1C:\temp\1\Replicate_ upgrade.iss /f2C:\temp\1\silent_x64_up.log
If the upgrade was successful, the log file should contain the following rows:
[ResponseResult]
ResultCode=0
Dlg0={9C614355-28A0-4C2A-98DF-DB9FD674826F}-SdWelcomeMaint-
0 Count=3 Dlg1={9C614355-28A0-4C2A-98DF-DB9FD674826F}-
MessageBox-0 Dlg2={9C614355-28A0-4C2A-98DF-DB9FD674826F}-
SdFinish-0 [{9C614355-28A0-4C2A-98DF-DB9FD674826F}-
98DF-DB9FD674826F}-SdFinish-0] Result=1
bOpt2=0
Syntax:
"C:\Program Files (x86)\InstallShield Installation Information\<directory_
containing_setup_file>\setup.exe" /s /f1RESPONSE_FILE /f2LOG_FILE
Note The directory containing the Replicate setup file always ends with the
following string: DB9FD674826F
Example:
C:\>"C:\Program Files (x86)\InstallShield Installation Information\
{9C614355-28A0-4C2A-98DF-DB9FD674826F}\setup.exe" /s
/f1C:\temp\response.iss /f2C:\temp\1\silent_uninstall.log
If the uninstall was successful, the log file should contain the following rows:
[ResponseResult]
ResultCode=0
1. Stop the Attunity Replicate UI Server and Attunity Replicate Server services.
a. Browse to:
HKEY_LOCAL_
MACHINE\SYSTEM\CurrentControlSet\services\AttunityReplicateConsole
"C:\Program Files\Attunity\Replicate\bin\RepUiCtl.exe" -d
"C:\Program Files\Attunity\Replicate\Data2" service run
c. Browse to:
HKEY_LOCAL_
MACHINE\SYSTEM\CurrentControlSet\services\AttunityReplicateServer
Note For all command line operations involving files such as arep_login.sh, only
the bash shell is supported.
Note The commands for installing, upgrading and uninstalling Attunity Replicate
must be run as root or using the sudo command.
All of the commands and examples in this section assume that Attunity Replicate
is being installed/upgraded/uninstalled as root.
Instances can be installed as Linux services, which means that the instance will be
stopped in an orderly fashion when a server is shut down and restarted when a server is
rebooted. Installing Replicate creates an initial instance, named areplicate, which is
installed as a service.
Additional instances can be created after Replicate is installed. An additional instance
will not be installed as a service if installed by a non-root user. A new instance will not
be run when it is created, allowing for configuration first.
Replicate instances get their environment from several files:
Each instance has a unique name, uses its own ports, and has its own data directory. This
means that when running commands such as repctl, you need to specify the instance-
specific data directory in the command line so that the correct instance is affected.
Notes
Then, copy the Attunity Replicate RPM file to any location on the Linux computer.
Example:
user=mike group=sales verbose=true rpm -i[vh] --prefix
/opt/mydir/ areplicate-6.4.0-102.x86_64.rpm iport=1024 rport=1025
For description of the optional parameters, see the Command Parameters table below.
Example:
data=/opt/mydata
Parameter Description
/opt/mydir/attunity/replicate
Note The Attunity Replicate/lib directory must precede the /usr/lib64 directory
in the LD_LIBRARY_PATH environment variable.
The environment variable is set in the <product_dir>/bin/arep_login.sh file.
Example:
export LD_LIBRARY_PATH=/opt/attunity/replicate/lib:/usr/lib64
Example:
/opt/attunity/replicate/bin/areplicate status
Run the following command (shown using the default installation path):
/opt/attunity/replicate/bin/instancename start
Example:
Run the following command (shown using the default installation path):
/opt/attunity/replicate/bin/instancename stop
Example:
/opt/attunity/replicate/bin/areplicate stop
Note When upgrading from a version prior to Replicate 6.1 where the data directory
is in a non-default location, you need to add the following parameter to the upgrade
command:
data=existing-data-directory
Note Upgrading Replicate on SUSE Linux requires adding the --nodeps parameter
to the command, as in the following example:
rpm -U[vh] --nodeps areplicate-6.4.0-102.x86_64.rpm
Note When upgrading from a version prior to Replicate6.1 where the data directory
is in a non-default location, you need to add the following parameter to the upgrade
command:
data=existing-data-directory
Note As in the initial installation, if the new or existing user and/or group is not
defined locally (namely, in Active Directory), you must include the
nocredentials=true parameter in the command. Otherwise, the upgrade will fail.
Note When upgrading Replicate, all instances will be reinstalled and started, even
if they were not running prior to the upgrade.
1:areplicate warning:
/opt/attunity/replicate/bin/repctl.cfg created
as /opt/attunity/replicate/bin/repctl.cfg.rpmnew
########################################### [100%]
command: rm -f <product_dir>/bin/repctl.cfg.rpmnew
Example:
./areplicate start
To ensure that Attunity Replicate was removed from the computer, run the
following command:
rpm -q areplicate
Note Uninstalling Replicate will not delete a Replicate instance's data directory,
the services_list.txt file, or any other modified files.
In this section:
Verify that the specified service name, port numbers, and data directory are
not already in use
Create start/stop links for run levels as specified in the script (chkconfig)
Create a symbolic link to /etc/init.d/instancename in the Replicate
bin directory (same name)
If a server password is specified, the script will run repctl to set the admin UI
server password
Notes
Uninstalling an instance will stop the instance's processes, but leave the
instance's data directory in place.
Example:
<product_dir>/bin/areplicate uninstall
3. Move the data directory from its current location (e.g. <product_dir>/data) to
your requested location.
4. Reinstall the service, specifying the new location for the data directory:
data=data_dir iport=number rport=number
<product_dir>/bin/arep.sh install instancename
Example:
<product_dir>/bin/areplicate start
In this chapter:
▶ Securing Access to the Attunity Replicate Web UI
▶ Setting Up Replicate Console HTTPS Support
▶ Setting Up Attunity Replicate Server HTTPS Support
▶ Changing the Server Password
▶ Protecting Replicate Passwords
▶ Encrypting the User Permissions File
▶ Securing Connections to Endpoints
▶ Application Security
▶ Using External Credentials
The Attunity Replicate Server on Windows or Linux can also serve the Web UI
directly, but supports just a single user with a fixed role of administrator (’admin’).
Connecting to this server is done using HTTPS with Basic Authentication.
See also Configuration 3: Replicate UI Console and Replicate Server Running on Linux.
IP:port : 192.168.1.13:443
IP:port : 192.168.1.11:443
IP:port : [fe80::285d:599c:4a55:1092%11]:443
IP:port : [fe80::3d0e:fb1c:f6c3:bc52%23]:443
With a valid SSL certificate installed, the Attunity Replicate web user interface will
automatically be available for secure access from a web browser using the following URL:
https://<machine-name>/attunityreplicate
Note Some corporate security policies prohibit the use of self-signed certificates. In
such cases, it is incumbent upon the IT Security department to provide and install
the appropriate SSL server certificate (as is the practice with other Windows
products such as IIS and SharePoint). If a self-signed certificate was installed and
needs to be removed, then the following command can be used:
$ netsh http delete sslcert ipport=192.168.1.13:443
Note When working with multiple instances, instead of creating a separate set of
certificates for each instance in <product-dir>/<data-directory>/ssl/data, you
can create a single set of certificates in <product-dir>/ssl/data. This way, instead
of managing multiple sets of certificates for each instance, you only need to
create/manage a single set of certificates.
You can replace the default self-signed certificates with you own, as follows:
Example:
/clear:12345
When Attunity Replicate Server starts it will scramble the private key passphrase
as shown in Examples of the Scrambled Private Key Password.
5. Start the Attunity Replicate Server service.
For information on stopping and starting Attunity Replicate Server, see Installing or
Upgrading Attunity Replicate on Windows and Installing Attunity Replicate on Linux.
Before starting, make sure that the following prerequisites have been met:
1. Stop the Attunity Replicate Server and Attunity Replicate UI Server services.
2. Open a command prompt (using the "Run as administrator" option) and change
the path to the Replicate bin directory. The default path is:
C:\Program Files\Attunity\Replicate\bin.
12. Scroll down the Details tab until you see the Thumbprint details and copy them
to the clipboard.
13. Open a command prompt and run the following commands:
Syntax:
¢ netsh http add sslcert ipport=0.0.0.0:443 certhash=[YOUR_CERTIFICATE_
THUMBPRINT_WITHOUT_SPACES] appid={4dc3e181-e14b-4a21-b022-59fc669b0914}
Example:
netsh http add sslcert ipport=0.0.0.0:443
certhash=5f6eccba751a75120cd0117389248ef3ca716e61 appid={4dc3e181-
e14b-4a21-b022-59fc669b0914}
Syntax:
¢ netsh http add sslcert ipport=[::]:443 certhash=[YOUR_CERTIFICATE_
THUMBPRINT_WITHOUT_SPACES] appid={4dc3e181-e14b-4a21-b022-59fc669b0914}
Example:
netsh http add sslcert ipport=[::]:443
certhash=5f6eccba751a75120cd0117389248ef3ca716e61 appid={4dc3e181-
e14b-4a21-b022-59fc669b0914}
14. Close the command prompt and Microsoft Management Console.
15. Start the Attunity Replicate Server and Attunity Replicate UI Server services.
Note When running multiple Replicate Linux instances, this procedure needs to
be repeated for each instance (as each instance has its own data directory).
For information on installing multiple Replicate Linux instances, see Replicate
Instances and Services on Linux .
where data-directory is the name of the Replicate data directory (by default: data).
Where:
'Z' is a fixed character
'x' a protection method indicator
'hhhhhh…' is a hexadecimal representation of the encrypted secret
Upon import, if a secret is provided in clear text (i.e. not in the format shown above), it
is automatically encrypted and stored in the protected format.
Note When running multiple Replicate Linux instances, you need to enter the
server admin password for each instance. For details, see Installing an Instance
of Replicate as a Service.
The default location for the master key file is the product data folder - by default
<product-dir>/data - or data directories when multiple instances of Replicate are
installed on Linux. If the server admin password does not exist, it is automatically
created with randomly generated passwords that are safe but unknown. The user
can change the server admin password as well as the master key to known values,
if needed (e.g to connect to the replication server remotely).
For more information, see Changing the Server Password.
When Replicate is set to run on a cluster using the same settings and storage, the
mk.dat file should also be the same (or shared). Similarly, if the Replicate settings are
exported in order to be moved to a different machine, the mk.dat file should also be
moved in order to avoid the need to reenter all secrets.
The procedure for changing the master key as well as measures that can be taken to
protect the file containing the master key are described below in Changing and
Protecting the Master Key.
Note When running multiple Replicate Linux instances, this procedure needs to
be repeated for each instance (as each instance has its own data directory).
For information on multiple Replicate Linux instances, see Replicate Instances
and Services on Linux .
4. Change the working directory to the product "bin" directory and then run the
following command (on Linux run ./repctl …):
repctl [-d data-directory] setmasterkey your_new_master_key
[master_ key_scope=scope]
where data-directory is the name of the Replicate data directory (by default: data).
Note In order not to have the key appear in the shell history, you can use
the command interface (on Linux run ./repctl):
repctl {enter}
quit {enter}
See Protecting the Master Key File from Misuse for the master_key_scope options.
Example:
For more information on importing and exporting Replicate tasks, see Exporting Tasks.
<product_dir>\Data\GlobalRepo.sqlite
To prevent unauthorized access of this file, you can encrypt it using the procedure
described below. After you perform the procedure, the repository file will be
encrypted with the AES-256 bit cipher.
Note The length of any passwords specified during the procedure must be at least
32 characters.
to: <product_dir>\bin
Example:
repuictl.exe masterukey set --password ANqaGYERP3UKmGLK6UNuMqrkAGxwH8FM
Example:
repuictl.exe repository secure --on --master-user-
password ANqaGYERP3UKmGLK6UNuMqrkAGxwH8FM
Attunity does not implement the network protocol (see important exceptions below) and
for this reason, Attunity Replicate generally relies on the vendor of the source or target
endpoint to offer encryption. When setting up endpoint connections, the user is able to
specify any connection properties required for the use of encryption; these properties
are, invariably, vendor-specific properties. In some cases, use of encryption requires
system-level settings (such as adding a certificate to the machine trust store) which are
outside of the scope of Attunity Replicate. Users are referred to the operation manual of
the source or target endpoint for details on how to set up encrypted client
communication with the server.
One exception to the previous paragraph is for endpoints based on the Attunity
Connect product (endpoints on zOS, iSeries, HP NonStop and HP OpenVMS). In this
case, the network encryption is implemented by the Attunity Connect product and is
based on the AES encryption algorithm with a 256-bit key.
Application Security
As a leading provider of enterprise-class big data management solutions, Attunity
understands that application security is of paramount importance. With the integration
of Static Code Analysis into the development lifecycle, the code is rigorously tested for
vulnerabilities before each product release.
<INSTALL_DIR>\addons\samples\MyPasswordProvider.c
<INSTALL_DIR>\addons\include\MyPasswordProvider.h
2. Replace the code below this line (starting AR_AO_LOG and ending AR_ADDONS_STATUS_
"lib_path": "MyPasswordProvider\\MyPasswordProvider.dll",
2. Edit the lib_path according to the location and name of your DLL.
3. Save the file as addons_def.json.
For a list of supported endpoint versions, see Supported Platforms and Endpoints .
In this chapter:
▶ Supported Replicate Endpoints
▶ Using ARC CDC Agents as Endpoints
▶ Replicate Data Types
▶ Supported DDL Statements
▶ Configuring Replicate to Automatically Replace the User-Entered Password
For a list of supported source and target endpoints, see Supported Platforms and
Endpoints .
You can also use CDC Agents in the Attunity Integration Suite as a source endpoint.
For a list of supported ARC CDC Agents and information on how to use them with
Replicate, see Using ARC CDC Agents as Endpoints.
Table 4.1 |
Endpoints Supported by ARC
Relational Endpoints Non-Relational Endpoints
SQL/MP HP NonStop Enscribe
RMS
VSAM
IBM IMS
For information on how to work with ARC, see Using ARC CDC Solutions in
Attunity Replicate .
Table 4.2 |
Replicate Data Types
Replicate Data Description
Types
STRING A character string
HH:MM:SS
DATETIME A timestamp value: Year, Month, Day, Hour, Minute, Second,
Fractional Seconds
YYYY:MM:DD HH:MM:SS.F(9)
INT1 A one-byte, signed integer
1. Captures ALTER TABLE DDLs from the transaction log without identifying the DDL
type (ADD/DROP/MODIFY COLUMN).
2. Reads the new table metadata from the source backend.
3. Compares the previous table metadata with the new table metadata in order to
determine the change. Note that a single change may include multiple DDL
operations performed on the backend.
4. Uses the new table metadata to parse the subsequent DML events.
When a rapid sequence of DDL-DMLs-DDL occurs, Replicate may read the table
metadata after the second DDL which may (on rare occasions) result in missing
data due to incorrect parsing of DML events from the log.
If you change the name of a table used in a task and then stop the task, Replicate
will not capture any changes made to that table after the task is resumed.
Note This feature cannot be used when the user name is "sa".
To utilize this feature, the password must be defined both in the Replicate endpoint
settings and on the actual database, in the following format:
replace:your_password
Example:
replace:k$RJdg7!S&ib
1. In the User name field, enter the user name in the following
format: ref:endpoint_name
Specify:
54lakrfgnier3!
In this chapter:
▶ Accessing the Attunity Replicate Console
▶ Tasks View
▶ Server View
▶ List Actions
Click Start and from the All Programs section point to Attunity Replicate and
select Attunity Replicate Console.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 100
Note When you connect to the Attunity Replicate Console, your browser will prompt
you for a username and password. The username and password that you need to
specify depends whether Replicate Server is installed on Windows or Linux.
Attunity Replicate Server on Windows: Your domain username and password.
Attunity Replicate Server on Linux: Either specify your PAM credentials or,
if PAM is not set up in your environment, specify admin as your username and
the Replicate Server password as your password.
For information on setting the Replicate Server password, see Security Considerations.
Note When the Attunity Replicate machine is located in a subdomain, the URL in the
ServiceConfiguration.xml file will contain localhost instead of the machine
name. In order to connect remotely, to the Attunity Replicate machine, you need to
replace localhost with the actual machine name or IP address.
To access the Attunity Replicate Console from a remote computer, type the
following address in the address bar of your Web browser:
Attunity Replicate Server on Windows:
https://<computer name>/attunityreplicate
Where <computer name> is the name or IP address of the computer where the
Attunity Replicate Server is installed and <port> is the C UI Server port (3552 by
default). For more information on the C UI Server component, see Attunity Replicate
UI Server Configurations.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 101
Note The person logged in to the computer where you are accessing the Console must be
an authorized Attunity Replicate user. For more information, see User Permissions.
Note When the Attunity Replicate .NET UI Server is running on one machine and the
Attunity Replicate Server is running on another, the Attunity Replicate Server
password must be the same on both machines. The password is used during the SSL
handshake to establish a secure connection between the participating machines.
For information on setting the password, see Changing the Server Password.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 102
Configuration 3: Replicate UI Console and Replicate Server
Running on Linux
In this configuration, the UI Console and the Web server (Attunity Replicate Server)
are hosted on two separate Linux machines, though it is also possible to install them
on a single machine.
Note that in such a configuration, the ability to assign different roles (as described in
User Permissions) is not supported. In other words, all users will have the admin role.
PAM Prerequisites
To establish a secure connection using PAM, make sure that the following
prerequisites have been met:
The Attunity user or group (or the user/group set during the installation) must
be granted permission to read the file: etc/shadow. Note that this prerequisite is
only required when Attunity Replicate is installed on two machines.
Edit the repctl.cfg file and modify the path to the fully qualified name of
the libpam.so.0 library if required.
Example:
"login_pam_libpam_full_path":"/lib64/libpam.so.0",
"login_pam_service_name": "system-auth"
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 103
Multiple Users Connecting to a Single Console
Multiple users can connect to a single Attunity Replicate Console using a Web browser,
as follows:
1. Install Attunity Replicate on the computer that will serve as the Attunity
Replicate Console.
2. If Attunity Replicate Server is installed on another computer (Linux for example),
on the console machine, edit the Attunity Replicate URL (and port if required) in
the ServiceConfiguration.xml file to point to that machine.
By default, the file is located in the following directory:
C:\Program Files\Attunity\Replicate\data
3. Open the Windows Services console and restart the Attunity Replicate
Console service.
4. Connect as described in Accessing the Attunity Replicate Console above.
Tasks View
The Tasks view is the default view that opens when you launch Attunity Replicate for
the first time, as shown in the following figure:
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 104
It lists all replication tasks you have defined. You use this view to view, edit, run,
and delete tasks, or to create new tasks.
This view includes the following elements:
Toolbar running along the top. It includes buttons that let you create a new
task, open, delete, run, or stop an existing task, configure advanced run
options, and manage endpoint connections. See also Adding Tasks.
Tasks already defined in the system, listed in the left pane.
Icons view, where each icon indicates the current state of the tasks.
Details view, which displays a table with additional information about each task
including their current state. Note that the state icons are the same as described
in the Task Icons table, but without the part of the icon.
To toggle between these views, you can select Icons or Details from the drop-
down list in the top right of the Console.
For information about creating a task, see Defining Tasks.
The Console displays each open task on its own tab along the top. For
more information, see Viewing Specific Tasks.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 105
Endpoints map in the right pane, which illustrates the endpoints for the task
selected in the left pane. Any notifications (if defined) and log messages will be
shown in the Messages pane below the map.
Messages pane below the endpoints diagram on the right. This pane includes a
Notifications tab for progress messages and a Log Messages tab for warnings
and error messages issued when Replicate encounters a problem. For more
information, see Reading Messages about a Task and Creating a New Notification.
Select Tasks from the drop-down list in the top left, below the Attunity Replicate logo.
Indicates that the task has stopped due to an error. When you select
the task, Replicate displays a list of errors on the Log Messages tab
at the bottom right of the console.
Indicates that the task has stopped due to a recoverable error. When you
select the task, Replicate displays a list of errors on the Log Messages
tab at the bottom right of the console.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 106
Viewing Specific Tasks
From the Tasks view, you can drill down to an individual task, provided you have
already created at least one task (see Defining Tasks for more information). Two
modes display different sets of information for each task:
Designer Mode: Default mode when you open a task. Here you define endpoints,
select tables, modify table settings (including filters and transformations), and create
global transformation rules.
Monitor Mode: Here you view replication task activities in real time, along with
log messages and notifications.
1. In the Tasks view, select the task you want to work with.
The right pane displays the task diagram on the right side of the page.
Designer Mode
In Designer mode, you define endpoints, select tables to be replicated, modify table
settings (including filters and transformations), and create global transformation
rules. This is the default mode when you open a task.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 107
Figure 5.1 | Viewing a Task in Designer Mode
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 108
Manage Endpoint Connections button: Lets you view the endpoints defined, edit
them, or add new endpoints. For more information, see Working with Endpoints.
Select and Define Tables: Lets you select the tables you want to include in your
replication task. In addition, you can use transformation and filter operations to create
new tables or to replicate parts of tables. For more information, Adding Tables and/or
Views to a Task, Using Filters, and Defining Transformations for a Single Table/View.
Global Transformations option: Lets you create transformations for all tables in
a task. For more information, see Defining Global Transformations.
Monitor Mode
In Monitor mode, you view the replication task activities in real time.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 109
The Monitor mode includes the following elements:
Run button: Lets you run the task at hand.
Manage Endpoint Connections button: Lets you view the endpoints defined, edit
them, or add new endpoints. For more information, see Working with Endpoints.
Monitor and Designer buttons: Switch between Monitor mode and Designer
mode. See also Monitoring and Controlling Replication Tasks, Designer Mode,
Defining Tasks.
Tools list: Provides access to history, log management, and status information.
Change Processing/Full Load tabs: Lets you select the information you want to focus
on. By default, Replicate displays the Full Load view (also shown in the figure).
Task Map: Illustrates the connection between the source and target endpoints for
the task. The round icon between the endpoints represents the task type, which can
indicate Full Load only, Full Load and Apply Changes, or Apply Changes only.
Messages pane: Displays notifications and logging messages. For more
information, see Reading Messages about a Task.
Server View
SERVER view lets you view and configure the Attunity Replicate Server settings.
From the drop-down list in the top left corner of the console (below the product
logo) select Server.
For information on configuring server settings, see Attunity Replicate Server Settings.
List Actions
The following table describes the various list actions you can perform. Note
that, depending on the list type, some of the actions may not be available.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 110
Table 5.2 |
List Actions
To Do This
Sort ascending or sort descending Right click the desired column and select one of the
sorting options as required.
Restore the default sorting order Right click any of the column headings and select
Default Sorting.
Export the list to a TSV file The following lists can be exported: tasks, messages,
selected tables, and processed tables (in Monitor
view). Either click the Export to TSV button above
the list or right-click any of the column headings and
select Export to TSV. Choose where to save the file
and then click Save.
Add or remove columns Right click any of the column headings and select
Column Settings. Then add or remove columns as
required.
Hide a column Right click the desired column and select Hide
Column.
Copyright © Attunity Ltd. Chapter 5 | Using the Attunity Replicate Console | Page 111
6 | Getting Started: An
Attunity Replicate Tutorial
This section guides you through setting up a basic replication task for data from an
Oracle source to a Microsoft SQL Server target.
In this chapter:
▶ What You Need
▶ Open the Attunity Replicate Console
▶ Add an Oracle Endpoint as a Source
▶ Add a Microsoft SQL Server database as a Target
▶ Add a Replication Task
▶ Run and Monitor the Replication Task
▶ View the Replicated Tables in Microsoft SQL Server
For the target: A Microsoft SQL Server database with the default tempdb
system database (used to store the target tables)
This can be installed on your local computer.
For the Attunity Replicate Console, one of the following Internet browsers:
Microsoft Internet Explorer version 11 and
above Mozilla Firefox
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 112
Google Chrome
For additional installation information, see the Installation Prerequisites.
Note You can access Attunity Replicate from any computer in your system.
To access the Console from a remote computer, type the following address in
the address bar of your Web browser:
http://<computer name>/attunityreplicate
Note The person logged in to the computer hosting the Console must be an
authorized Attunity Replicate user. For more information, see User Permissions.
Connection string: Enter the connect string to the Oracle database you work
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 113
with, in any Oracle format.
User Name: Enter the user name for the Oracle database you work with.
Password: Enter the password for the Oracle database you work with.
The figure below shows the data that you need to enter on the General tab of
the Oracle database.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 114
4. Click Test Connection to verify the information you entered and the availability
of the database.
5. Click Save to add the database.
You can also set advanced settings for the Oracle database, but this beyond the scope
of this tutorial. For more information, see Setting Advanced Connection Properties
Using Oracle LogMiner.
For information on adding other types of databases, see the chapter for the required
database. For a list of supported databases, see Supported Platforms and Endpoints .
Server name: Enter the name of the computer where your Microsoft SQL
Server database is installed.
For example, if you connect to a Microsoft SQL Server database on a
computer called bee, enter bee.
Select one of the following:
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 115
Database name: Enter tempdb, which is the name of the database to
where you are going to replicate data. If you created a new database for this
purpose, enter the name of that database.
4. Click Test Connection to verify the information you entered and the availability
of the database.
5. Click Save to add the database.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 116
You can also set advanced settings for the Microsoft SQL Server database, but this is
beyond the scope of this tutorial. For more information, see Setting Advanced
Connection Properties.
For information on adding other types of databases, see the chapter for the required
database. For a list of supported databases, see Supported Platforms and Endpoints .
By default the Oracle database includes the HR schema. You will make a copy of the
same tables in your Microsoft SQL Server tempdb. The EMPLOYEES and JOBS tables
created in Microsoft SQL Server will be identical to the Oracle tables.
For information on how to use Transformations and Filters when creating a replication
task, see Defining Transformations for a Single Table/View and Using Filters.
1. Make sure Tasks is selected in the upper left corner of the Attunity Replicate Console.
3. In the New Task dialog box, in the Name field, type My_Task and click OK.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 117
The Attunity Replicate Console displays the task on a new tab. By default, because
the task has not been set up yet, the tab opens in Designer view. The diagram on
the left serves as a drop-off point for the source and target databases you defined
previously. The right pane lets you select the tables you want to work with and
carry out transformations and filtering operations. For more information, see Tasks
View, Viewing Specific Tasks, and Defining Tasks.
If needed, you can also change the default task settings. For more information,
see Task Settings.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 118
Add the Source and Target Endpoints to the Task
This section guides you through adding the source and target endpoints to the
replication task, which is a simple drag-and-drop operation. In the Endpoints tab, the
following icons help you distinguish between source and target endpoints:
The Endpoints pane consists of All, Source, and Target tabs. Because each database
is named in a way that reflects whether it is a source or a target, click the All tab.
2. Drag the OracleSource database to the Drop source endpoint here area in
the endpoints diagram.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 119
3. Drag the sqlserver_targetdatabase to the Drop target endpoint here area.
Next, you can select the tables from the source database to use in the replication task.
For more information, see Defining Tasks.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 120
This task guides you through selecting specific tables (HR.EMPLOYEES and HR.JOBS)
from the Oracle source. Replicate takes all of the data from these tables "as is" and
copies it to the Microsoft SQL Server target.
Note If you need to copy only some of the data to the target database, you need to
use a filter. For information, see Using Filters.
If you need to copy the data into the target using different rows or columns than
those in the source, you need to use transforms. For more information, see Defining
Transformations for a Single Table/View.
1. In the right pane of the Attunity Replicate Console, click Table Selection. The
Select Tables dialog box opens.
2. In the Select Tables dialog box, do the following:
From the Schema list, select HR, and then click Search.
From the Table List, select EMPLOYEES, and then click the right arrow to
select that table.
Repeat these steps for the JOBS table.
Click OK.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 121
3. On the task tab, click Save. The task is now ready to run.
The Starting task message displays, and the console switches to Monitor view,
which includes gauges and graphs on two tabs:
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 122
Full Load tab: Indicates the status progress during the full load process.
Change Processing tab: Monitors changes that occur after the full
load completes.
For information on reading the data presented in these sections, see
Viewing Information in the Monitor.
2. Click the Select All link above the Tables graphs. Replicate displays a table below
the graphs with information about each of the tables being processed in the task.
3. Click the individual bar graphs, such as the Completed graph and the Loading graph,
to view additional information.
For information about the data supplied in these tables, see Monitoring Full-
Load Operations.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 123
To view the replicated tables in Microsoft SQL Server:
1. From the Windows Start menu, go to All Programs > Microsoft SQL Server
> Microsoft SQL Server Management Studio.
2. In the Object Explorer, find the Microsoft SQL Server target computer you are
working with.
3. Expand the databases folder for that computer, then expand the System databases
folder, then expand the tempdb database. The EMPLOYEES and JOBS tables should
now appear in the list.
4. Right-click the EMPLOYEES table and select Select Top 1000 Rows. Check that there
is data in the table.
5. Right-click the JOBS table and select Select Top 1000 Rows. Check that there is
data in the table.
Copyright © Attunity Ltd. Chapter 6 | Getting Started: An Attunity Replicate Tutorial | Page 124
7 | Defining Tasks
This section describes how to design a replication task. To design a replication task,
you must first be sure that you have configured at least one source endpoint and one
target endpoint to work with Attunity Replicate.
It is also possible to customize a task by creating new tables or columns for the target
endpoint or by selecting only some of the data from each column to be replicated. This
is done using transformations and filters.
Note A number of variables affect the amount of tasks that can be run on a single
Replicate Server, including the task configuration (e.g. how many tables are being
replicated), the size of the source tables and the hardware configuration of the
Replicate Server machine. Bearing this in mind, the number of tasks that can be run
on a single Replicate Server should not exceed 100 (and may need to be significantly
less depending on the aforementioned variables). Best practice is to perform load
testing in a Test environment before moving to Production.
In this chapter:
▶ Adding Tasks
▶ Editing and Viewing a Task Description
▶ Working with Endpoints
▶ Adding a Source and Target Endpoint to a Task
▶ Adding Tables and/or Views to a Task
▶ Editing a Replication Task
▶ Searching for Tasks
▶ Deleting a Replication Task
▶ Exporting and Importing Tasks
1. In Tasks view, click New Task. The New Task dialog box opens.
Log Stream Staging - Log Stream Staging enables a dedicated Replicate task
to save data changes from the transaction log of a single source database and
apply them to multiple targets, without the overhead of reading the logs for
each target separately.
For more information, see Using the Log Stream.
Full Load: Click to enable or disable Full Load options for this task.
When full load is enabled, Attunity Replicate loads the initial source data to
the target endpoint. By default a full load is carried out for this task. If you
want to change this setting after you begin working with this task, you make
the change in the Task Settings, Full Load tab.
Apply Changes: Click to enable or disable Apply Changes (Change Processing).
If this option is enabled, changes are stored in change tables or in an audit table.
By default, changes are not stored.
6. Click OK to close the New Task dialog box and save your settings.
Bidirectional Replication
Bidirectional replication enables organizations to synchronize data between two endpoints
(henceforth referred to as Endpoint A and Endpoint B), ensuring that both endpoints contain
identical records. The endpoints can either be the same type (e.g. Oracle-to-Oracle) or
different types (e.g. Microsoft SQL Server-to-Oracle). To implement bidirectional replication,
two Bidirectional Replication tasks need to be defined: one that captures changes made to
Endpoint A and replicates them to Endpoint B (Task 1) and another that captures changes
made to Endpoint B and replicates them to Endpoint A (Task 2). An explanation of how to set
up these tasks is provided in the following sections.
Limitations
The following limitations apply to Bidirectional replication tasks:
To ensure that the source and target endpoints are identical, transformations
and filters should not be used in bidirectional replication tasks.
The Use direct path full load option in the Oracle target endpoint settings is
not supported.
Supported Endpoints
Bidirectional tasks support the following endpoints:
Source Endpoints:
Oracle
Microsoft SQL Server
MySQL
PostgreSQL
All AIS sources
File Channel
SAP Sybase ASE
IBM DB2 for iSeries
Amazon RDS for MySQL
Amazon RDS for PostgreSQL
Amazon RDS for SQL Server
Target Endpoints:
Oracle
Microsoft SQL Server
MySQL
PostgreSQL
ODBC
File Channel
SAP Sybase ASE
2. Specify a source and target Loopback prevention table schema in the task
settings’ Loopback Prevention tab. For more information about loopback
prevention settings, see Bidirectional.
3. Run the task.
1. Define another Bidirectional Replication task that replicates data from Endpoint
B to Endpoint A.
2. Specify a source and target Loopback prevention table schema in the task
settings’ Loopback Prevention tab. For more information about loopback
prevention settings, see Bidirectional.
3. If Full Load was enabled when replicating data from Endpoint A to Endpoint B,
you must first wait for the Full Load replication to complete before running the
task. Otherwise, run the task immediately.
Task 1: MySQL Source --> FC Target Task 2: FC Source 1 --> Oracle Target
Task 3: Oracle Source --> FC Target 2 Task 4: FC Source 2 --> MySQL Target
3. Enter a
description. -OR-
4. Click OK.
Adding an Endpoint
Editing Endpoint Configuration Information
Viewing Endpoint Configuration Information
To add an endpoint:
3. Select the type of endpoint you are using. The information that you must
enter depends on which endpoint you select.
For more information, see the chapter that describes the endpoint you are using. For
a list of supported databases, see Supported Platforms and Endpoints .
Note You cannot change the following information in the endpoint window:
The name you provided for the endpoint.
The endpoint Type, for example Oracle or Microsoft SQL Server.
The endpoint role, either SOURCE or TARGET.
1. In the Manage Endpoint Connections window, select the endpoint you want to
edit. or
In the Endpoints list on the left of the Designer view, double-click the endpoint you
want to edit. Note that this option is only available when editing a specific task.
For more information, see the chapter for the specific Attunity Replicate endpoint
you are using. For information which endpoints are supported by Attunity Replicate,
see Supported Platforms and Endpoints .
Select an endpoint from the Endpoints list in the left pane; then click the tabs to
view the information.
1. In the Manage Endpoint Connections window, select the endpoint you want to
work with.
2. At the bottom of the endpoint’s General tab, click Test Connection.
If the connection fails, click View Log to view the server log entry with
information for the connection failure.
Duplicating Endpoints
You can duplicate an endpoint if you need to define a new endpoint with similar settings.
Except for the name, all endpoint settings are duplicated to the new endpoint.
1. In the left panel of the Manage Endpoint Connections window, click the
endpoint you want to duplicate.
2. Click Duplicate.
3. On the General tab, edit the name for the endpoint.
4. Make any other necessary changes.
5. Click Save; then click Close.
Deleting Endpoints
You can delete endpoints that you no longer require. Note that to delete an endpoint
that is defined as a source or target in a task, you first need to remove the endpoint
from the task.
To delete an endpoint:
In the left panel of the Manage Endpoint Connections window, Select the
endpoint and click Delete.
Create a new task. When you click OK in the Create New Task dialog box,
the task opens on a dedicated tab. For more information, see Adding Tasks.
In the Tasks view, select the task to which you want to add endpoints and
click Open. The task opens on a dedicated tab.
2. The Task map is displayed, with the available endpoints listed in the pane on the
left, as shown in the following figure.
3. Drag a source endpoint to the top circle in the task map (that contains the text
Drop source endpoint here). If dragging is not possible, make sure that the
endpoint you are using is defined as a source endpoint.
4. Drag a target endpoint to the bottom circle in the task map (that contains the text
Drop target endpoint here). If dragging is not possible, make sure that the
Notes
When replicating views, the corresponding tables are created without a primary
key. This presents an issue for Apply Changes tasks, which require the target
tables to have a primary key. Therefore, if you are also running Apply Changes
tasks (using one of the CDC-capable endpoints mentioned above), you need to
define one or more primary keys for each of the target tables using a
Note When working with ODBC with CDC and Teradata source endpoints, any views
and tables that you want to replicate must have the same context field(s). If you only
want to replicate views, then all of the views must have the same context field(s).
Once you have selected tables/views to replicate, you can run the replication task.
However, if you need to make any changes to the structure of the tables in the
target endpoint or only select specific columns, you will need to carry out one or
both of the following procedures:
Defining Transformations for a Single
To select tables/views:
1. Open the task you are working with if it is not already displayed in a dedicated
If the source endpoint does not support view selection, the Select Tables dialog
box opens. If the source endpoint supports view selection, the Select
Tables/Views dialog box opens.
See the following for information on how to work with the Select
Tables/Select Tables/Views dialog box:
Searching for Tables/Views to use in a Replication
Task Selecting Specific Tables/Views for Replication
Creating Table/View Selection Patterns Setting Load
Order
2. In the Select Tables dialog box, if the source endpoint supports view selection,
select one of the following:
All to search for tables and views
Tables to search for tables only
Views to search for views only
Note When selecting tables from the SAP Application or SAP Application
(DB) endpoint, "Business Groups" will appear instead of "Schema".
Note You can also include special characters in your search string. For more
information, see the Note in Creating a Record Selection Condition for One or
More Columns.
The Table List field displays any table/view that matches the specified.
1. In the Select Tables dialog box, from the Table List field, select one or more
tables that you want to include in the replication task.
2. To select a table, click the button with a single right-facing arrowhead (Add).
To select all tables in the Table List, click the button with two right-facing
arrowheads (Add All).
The selected tables are added to the Selected Tables list.
Note If you rename a table in the database, the Designer tab will still show the original
table name. The Monitor tab, on the other hand, will show the new table name.
1. From the Selected Tables list, select a table that you want to remove from
the replication task and then click the button with a single left-facing
arrowhead (Remove).
2. To remove all of the tables/views from the Selected Tables or Selected
Tables/Views list, click the button with two left-facing arrowheads (Remove All).
3. Click OK to close the Select Tables or Select Tables/Views dialog box.
4. Click Save to make sure that Attunity Replicate saves the table information for
this task.
Exclude dbo.PRODUCT_1%
You can also use the "_" wildcard character to match a single character. For example,
specifying Exclude m_d% will exclude all tables that begin with m and end with d%, such
as model or msdb.
When you explicitly select tables/views, all selected tables/views are replicated in full
unless you define transformations or filters for the table/view. If you need to make
Note To view all of the tables/views included when you use a table selection
pattern, click the Full Table List tab in Designer view. The Full Table List lists all
of the tables/views included in any table pattern you defined as well as all explicitly
selected tables/views. To view only patterns and explicitly selected tables/views,
click the Patterns and Selected Tables tab in Designer view.
Notes
Load order cannot be changed during a task. If you want to change the load
order, first stop the task, then change the load order as desired, and finally
reload the target.
Load order cannot be set for table patterns.
2. From the Load Order drop-down list, select one of the available priority levels
(Lowest Priority, Low Priority, Normal Priority, High Priority, and Highest Priority).
3. This step is only relevant if you are setting load order in the Select Tables
window. Click OK to save your settings and close the Select Tables window.
2. From any of the selected items' Load Order drop-down list, select one of
the available priority levels.
3. This step is only relevant if you are setting load order in the Select Tables
window. Click OK to save your settings and close the Select Tables window.
To edit a task:
The task opens, displaying the source and target endpoints and which tables have
been selected for replication.
2. Continue with any of the following procedures:
Note If you use a Microsoft SQL Server endpoint, a Microsoft SQL Server system
administrator must delete the Microsoft SQL Server Replication Publisher definitions
for the endpoint that was used in the task from SQL Server.
For more information, see the Limitations in the Microsoft SQL Server chapter.
To delete a task:
Note If you need to access the computer with Attunity Replicate from a
remote computer, you can use a telnet connection.
When you export a task to a different environment, you may need to edit the task
information. For example, you may need to change the connection string for an endpoint.
Exporting Tasks
Editing an Exported (json) File
Exporting Tasks
The following section explains how to export a task using either the Attunity
Replicate Console or the command line.
In TASKS tab, select the task you want to export and then either click the
Export toolbar button or right-click the task and select Export.
-OR-
In the TASK_NAME tab (opened when a task is being edited), click the
Export Task toolbar button.
Depending on your browser settings, one of the following will occur:
1. From the Attunity Replicate computer where you defined the task you want to
import, open the Attunity Replicate command line console by doing the following:
On Windows: Go to All Programs in the Start menu and point to Attunity
Replicate, then to Utilities and then select Attunity Replicate Command Line. A
command-line console is displayed with the correct prompt for Attunity Replicate.
Note: You can also open the Windows command-line console and change the
directory to the following:
<Full path to the Attunity Replicate root folder>\bin
For example, to use the path to the folder or directory where Attunity Replicate
is installed by default, type: C:\Program Files\Attunity\Replicate\bin.
On Linux: Run the following command in the Replicate bin
Example:
repctl -d D:\Data exportrepository task=mytask
3. Browse to the task JSON file you want to import and then click Import
1. From the Attunity Replicate computer where you want to export the task, open
the Attunity Replicate command line console by doing the following:
Go to All Programs in the Start menu and point to Attunity Replicate, then
to Utilities and then select Attunity ReplicateCommand Line.
A command-line console is displayed with the correct prompt for Attunity Replicate.
Note You can also open the Windows command-line console and change
the directory to the following:
<product_dir>\Attunity Replicate>\bin
For example to use the path to the directory where Attunity Replicate is installed
by default, type: C:\Program Files\Attunity\Replicate\bin.
Example:
repctl -d data-directory importrepository
json_file=C:\Temp\many_ tables.json
The export utility automatically appends the extension .JSON to exported files,
thus eliminating the need to add the extension manually.
Example:
repctl -d D:\MyData importrepository json_file=C:\mytask.json
If you are importing this task into a different environment, you should copy the file
to a location on the second Attunity Replicate computer and then import the file
from there.
In many cases, when you import the task into a different environment, you will
need to make changes to the task. For example, you may need to change the
connect strings for the endpoints in the task or change the user password. In that
case, you will need to edit the *.json file.
See Editing an Exported (json) File for more information.
Important: Make any changes to the *.json file before you carry out the
import operation.
To be able to use the new task, you will need to make changes to the endpoint
password and connection strings by editing the *.json file. See Making Changes to the
Endpoint Connection Information for more information.
"name": "Oracle_Source",
"type": "Oracle",
"authenticator": "
{01000000D08C9DDF0115D1118C7A00C04FC297EB010000003EA495B32CAAE14CB9777B96B3
CC00B30000000002000000000003660000A8000000100000002765A3287AB56447DA31508F7
1CE62700000000004800000A00000001000000088D5C1BBD615BEEEAF5FAC1B9B0E20800800
000075D89177A9C6F11B1400000047B3110B80920DD9EB0A5FABA05679979B78DDD0}",
"role": "SOURCE"
}, {
"name": "SQLSERVER_Target",
"type": "SQLServer",
"connection_string": "server=bee01-
xp;endpoint=tempdb;WindowsAuthentication=Y;CDCBCP=Y;FullloadBCP=Y;BCPPacket
Size=16384",
"role": "TARGET"
Important: Make sure that the connect string remains between the
quotation marks (").
2. Change the password string to the relevant password for the endpoint in the new
environment. Type the new password using plain text exactly as it is configured in
the endpoint. For example, 8yTkLMt.
When you save the file and then import it to the new environment, the password
is encrypted automatically.
{Pass%^&rd}
In this chapter:
▶ Using Amazon RDS for SQL Server as a Source
▶ Using Amazon RDS for MySQL as a Source
▶ Using Amazon RDS for PostregSQL as a Source
▶ Using ARC CDC Solutions in Attunity Replicate
▶ Using a File as a Source
▶ Using Hadoop as a Source
▶ Using IBM DB2 for iSeries as a Source
▶ Using IBM DB2 for LUW as a Source
▶ Using IBM DB2 for z/OS as a Source
▶ Using IBM Informix as a Source
▶ Using IBM Netezza as a Source
▶ Using a Microsoft Azure SQL Managed Instance as a Source
▶ Using Microsoft SQL Server as a Source
▶ Using a MySQL-Based Database as a Source
▶ Using ODBC to Connect to a Source
▶ Using ODBC with CDC as a Source
▶ Using Oracle as a Source
▶ Using PostgreSQL as a Source
▶ Using SAP Application as a Source
▶ Using SAP Application (DB) as a Source
▶ Using SAP HANA as a Source
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 150
▶ Using SAP Sybase ASE as a Source
▶ Using Teradata Database as a Source
In this section:
▶ Prerequisites
▶ Limitations
▶ Non-Supported Amazon RDS for SQL Server Security Features
▶ Required Permissions
▶ Supported Compression Methods
▶ Amazon RDS for SQL Server Source Data Types
▶ Non-Supported Data Types
▶ Homogeneous Replication
▶ Data Type Exceptions
▶ Column and Table Collation
▶ Preparing the Amazon RDS for SQL Server Database for Replication
▶ Setting up Amazon RDS for SQL Server for CDC (Change Data Capture)
▶ Preparing Amazon RDS for SQL Server Backup and Recovery
▶ Defining Amazon RDS for SQL Server Database Settings
▶ Working with Windows Authentication
▶ Setting Connection Properties
▶ Artifacts Created by Replicate in the Database
Prerequisites
Make sure that the following prerequisites have been met:
Client prerequisites (for source and target endpoints):
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 151
For all versions of Amazon RDS for SQL Server, Microsoft SQL Server Native
Client 11.0 must be installed on the Attunity Replicate Server machine.
Attunity Replicate for Linux:
First, install Microsoft ODBC Driver 13.1.1.0 for Linux on the Attunity Replicate
Server machine.
Then, on the Attunity Replicate Server machine, open a Unix shell and perform
the following steps:
Note The procedure below assumes that you have installed a single default
instance of Replicate on Linux (areplicate). If you have installed multiple
instances, replace areplicate with the name of the instance running the task with
a Microsoft SQL Server source. If several instances are running such as task, the
procedure needs to be repeated for each instance.
to: cd <product_dir>/bin
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 152
Note Replicate requires the following ODBC library:
libmsodbcsql-13.1.so.0.0
ln -s existing_library_name libmsodbcsql-13.1.so.0.0
An Amazon RDS for SQL Server account with the specific access privileges is required.
See Source Permissions for more information.
Amazon RDS for SQL Server as a source must be configured for a full backup to
work with Attunity Replicate. For more information, see Preparing Amazon RDS for
SQL Server Backup and Recovery.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 153
Limitations
When using an Amazon RDS for SQL Server source endpoint in a Replicate task,
the following imitations apply:
Amazon RDS for SQL Server moves the online logs to the backup location when
the maximum allocated storage size is reached. This may sometimes occur
before Replicate has read the logs, resulting in missing changes on the target.
Therefore, in a system with a high volume of changes (or if you notice that some
changes are not being captured), best practice is to increase the allocated storage
size as described in the following article:
https://aws.amazon.com/premiumsupport/knowledge-center/rds-sql-server-
storage-optimization/
Due to a known issue with Microsoft SQL Server 2008/2008 R2, Attunity Replicate
does not support server level audits when using Microsoft SQL Server 2008/2008 R2
as a source database.
For example, running the following command:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 154
USE [master]
GO
GO
Amazon RDS for SQL Server backup to multiple disks is not supported.
When inserting a value into SQL Server spatial data types (GEOGRAPHY and
GEOMETRY), one can either ignore the SRID (Spatial Reference System Identifier)
property - in which case the default SRID will be used (0 for GEOMETRY and 4326
for GEOGRAPHY) - or specify a different number. When replicating tables with
spatial data types, Attunity Replicate replaces the SRID that was inserted by user
with the default SRID.
Columnstore indexes are not supported.
Memory-optimized tables (using In-Memory OLTP) are not supported.
Temporal databases are not supported
Delayed durability is not supported
Table change tracking is not supported
Due to an ODBC limitation, no more than 16 columns can be part of a Primary Key.
UPDATEs to a Primary Key/Unique Index that affect multiple rows may cause
conflicts when applying the changes to the target. This may occur, for example, if
the UPDATEs are applied to the target as INSERT/DELETE operations rather than a
single UPDATE operation. In such a scenario (where UPDATEs to a Primary
Key/Unique Index affect multiple rows), working in Batch Optimized Apply Change
Processing mode is not supported as it may result in the table being ignored.
Working in Transactional Apply Change Processing mode however may result in
constraint violations. If this happens, you either need to reload the relevant table or
locate the problematic records in the attrep_apply_exceptions Control Table and
edit them manually in the target database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 155
Always Encrypted
Dynamic Data Masking
Row-Level Security
Required Permissions
To use an Amazon RDS for SQL Server source in an Attunity Replicate task, the
user specified in the Amazon RDS for SQL Server endpoint connection settings
must be a member of the db_owner database role.
Table 8.1 |
Supported Amazon RDS for SQL Server Compression Methods
Row/Page Compression Vardecimal Vardecimal Vardecimal
(at Partition Level) Storage Format Storage Format Storage Format
Yes No No No
For information on how to view the data type that is mapped in the target, see the
section for the target endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 156
Table 8.2 |
Amazon RDS for SQL Server Source Data Types with Mapping to
Attunity Replicate Data Types when the Target is not Amazon RDS for SQL Server
BIT BOOLEAN
DECIMAL NUMERIC
INT INT4
MONEY NUMERIC
(19,4)
SMALLINT INT2
SMALLMONEY NUMERIC
(10,4)
TINYINT UINT1
REAL REAL4
FLOAT REAL8
DOUBLE REAL8
DATETIME DATETIME
DATETIME2 (Amazon RDS for SQL Server 2008 and later) DATETIME
SMALLDATETIME DATETIME
DATE DATE
TIME STRING
(16)
DATETIMEOFFSET STRING
*CHAR STRING
*VARCHAR STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 157
Table 8.2 | Amazon RDS for SQL Server Source Data Types with Mapping to Attunity Rep-
licate Data Types when the Target is not Amazon RDS for SQL Server (Cont.)
*TEXT
To use this data type with Attunity Replicate, you must enable the use of
CLOBs for a specific task.
LOB columns for Amazon RDS for SQL Server tables are updated in the
target even for UPDATE statements that did not change the value of the
LOB column in Amazon RDS for SQL Server.
During CDC, CLOB data types are supported only in tables that include a
primary key.
*NCHAR WSTRING
*NTEXT
To use this data type with Attunity Replicate, you must enable the use of
NCLOBs for a specific task.
LOB columns for Amazon RDS for SQL Server tables are updated in the
target even for UPDATE statements that did not change the value of the
LOB column in Amazon RDS for SQL Server.
During CDC, NCLOB data types are supported only in tables that include a
primary key.
BINARY BYTES
VARBINARY BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 158
Table 8.2 | Amazon RDS for SQL Server Source Data Types with Mapping to Attunity Rep-
licate Data Types when the Target is not Amazon RDS for SQL Server (Cont.)
IMAGE
LOB columns for Amazon RDS for SQL Server tables are updated in the
target even for UPDATE statements that did not change the value of the
LOB column in Amazon RDS for SQL Server.
To use this data type with Attunity Replicate, you must enable the use of
BLOBs for a specific task.
BLOB data types are supported only in tables that include a primary key.
TIMESTAMP BYTES
UNIQUEIDENTIFIER STRING
HIERARCHYID STRING
(250)
XML CLOB
LOB columns for Amazon RDS for SQL Server tables are updated in the
target even for UPDATE statements that did not change the value of the
LOB column in Amazon RDS for SQL Server.
To use this data type with Attunity Replicate, you must enable the use of
NCLOBs for a specific task.
During CDC, NCLOB data types are supported only in tables that include a
primary key.
GEOMETRY CLOB
GEOGRAPHY CLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 159
CURSOR
SQL_VARIANT
TABLE
Note User-defined data types are supported according to their base-type. For example a
user-defined data type based on DATETIME is handled as a DATETIME data type.
Homogeneous Replication
When replicating from an Amazon RDS for SQL Server source to an Amazon RDS for
SQL Server target, most of the source and target data types will be identical. The
exceptions are listed in the table below.
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through
when replicating from Amazon RDS for SQL Server, see the Amazon RDS for SQL
Server to Attunity Replicate data types mapping table described earlier.
Note To prevent data truncation when replicating XML, Geometry and Geography
data types, it is strongly recommended to enable the Allow unlimited LOB size option
in the task settings.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 160
Table 8.3 |
Data Type Exceptions in Homogeneous Replication
Amazon RDS for SQL Microsoft SQL Server Microsoft SQL Server 2005-
Server Source 2005 Target 2016 Target
HIERARCHYID STRING(250)
TIMESTAMP VARBINARY
Note To support collation replication, the DBA must ensure that the collations
defined for the source Amazon RDS for SQL Server database are the same as those
defined for the supported Microsoft SQL Server-based target.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 161
Non-Nullable Columns and Primary/Unique Index Names
Setting up Amazon RDS for SQL Server for CDC (Change Data Capture)
The following section explains how to set up Amazon RDS for SQL Server for CDC.
Note When creating the Amazon RDS for SQL Server instance, full backup must
be enabled, otherwise Replicate will not be able to capture changes.
1. Log in as the master user in the DB instance and set up the database for MS-CDC
as follows:
exec msdb.dbo.rds_cdc_enable_db 'db_name'
2. To enable MS-CDC, run the following for each of the intended source tables:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 162
exec sys.sp_cdc_enable_table
@source_schema = N'schema_name',
@source_name = N'table_name',
@role_name = NULL,
@supports_net_changes = 1
GO
where schema_name and table_name are the names of the source schema and
the source table respectively.
3. Set the retention period for the changes to be available on the source by running
the following command:
EXEC sys.sp_cdc_change_job @job_type = 'capture' ,@pollinginterval
= 86400
Limitations:
CDC is supported with the transaction log only, with no option to work with backup.
Events may be lost if they are moved to backup or truncated.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 163
Note After setting the Recovery Model, it is strongly recommended not to change
it; doing so may result in loss of data.
The Attunity Replicate user is specified as the "Log on as" user for the
"Attunity Replicate Server" service account.
-OR-
Amazon RDS for SQL Server is configured to allow login for the Attunity
Replicate Server service account.
To add an Amazon RDS for SQL Server source endpoint to Attunity Replicate:
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 164
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Amazon RDS
for SQL Server database. This is optional.
4. Select SOURCE as the database role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. Specify the Server name. This is the host name or IP address of the computer
with the Amazon RDS for SQL Server instance containing the source database.
Note To override the default port, add the port to the server name, separated by
a comma. For example, if the server name is myserver.company.local and the
port is 3333, then the server name should be entered like this:
myserver.company.local,3333
Note When using Windows authentication, make sure that the user account
that is associated with the Attunity Replicate Server service has Network
read and write permissions. This must be configured by a Windows system
administrator.
If you select SQL Server authentication, type the Amazon RDS for SQL Server
authentication information (User name, Password) for the authorized user for this
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 165
Amazon RDS for SQL Server database. If you do not know this information, see
your Amazon RDS for SQL Server System Administrator.
To prevent illicit database activity by unauthorized third-parties, Replicate can be
configured to automatically replace the user-entered password with a strong
random password. For more information, see Configuring Replicate to
Automatically Replace the User-Entered Password.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the Amazon RDS for SQL Server user has the
correct access privileges. For information on how to provide the required
privileges, see Required Permissions.
8. Type the Database name or click Browse and select one from the list of available
databases. This is the name of the database from where you are replicating the data.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 166
Artifacts Created by Replicate in the Database
In order to capture changes, Attunity Replicate creates various artifacts in the dbo
schema of the Amazon RDS for Microsoft SQL Server database when the task starts. You
can safely remove them when the task completes.
The following artifact is created to track open transactions:
#attrep_dbcc_open_trans_table
The following functions are created to verify that the LSN (Log Sequence Number) is valid:
attrep_fn_LsnSegmentToHexa
attrep_fn_NumericLsnToHexa
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 167
Using Amazon RDS for MySQL as a Source
This section describes how to set up and use Amazon RDS for MySQL as a source
in a replication task.
In this section:
▶ Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Amazon RDS for MySQL Replication
▶ Enable Binary Logging
▶ Cluster Prerequisites
▶ Setting up Amazon RDS for MySQL for CDC (Change Data Capture)
▶ Replicating 4-byte UTF8 Emojis
▶ Limitations
▶ Security Requirements
▶ Data Types
▶ Homogeneous Replication
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Selecting a Schema
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the prerequisites described in this section have been met.
In this section:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 168
Setting up Amazon RDS for MySQL for CDC (Change Data Capture)
Replicating 4-byte UTF8 Emojis
Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in
the following example:
[MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 169
Table 8.4 |
Required my.ini/my.cnf Parameters for Binary Logging
Parameter Value
server_id Any value from 1.
Example:
server_id=1
log-bin=<path> Path to the binary log file (without an extension).
Example:
binlog_format=row
expire_logs_days To prevent disk space issues, it is strongly recommended not to use
the default value (0).
Example:
expire_logs_days=5
binlog_row_image Must be:
binlog_row_image=full
log_slave_updates When replicating from an Amazon RDS for MySQL slave database
server, this value should be set to true (1). If set to 0 (the default)
updates on a slave received from a master during replication are
not logged to the slave's binary log. The slave's binary log needs to
be enabled for this to have an effect.
Cluster Prerequisites
To be able to replicate clustered (NDB) tables (i.e. by connecting Attunity Replicate to
any of the cluster nodes), the following parameters must be configured in Amazon RDS
for MySQL’s my.ini (Windows) or my.cnf (UNIX) files.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 170
Table 8.5 |
Required my.ini/my.cnf Parameters for Cluster Replication
Parameter Value
ndb_log_bin Must be:
ndb_log_bin=on
This ensures that changes in clustered tables will be logged to the
binary log.
Setting up Amazon RDS for MySQL for CDC (Change Data Capture)
1. Follow the instructions in the AWS Help for creating a new Parameter Group (see
the Binary Logging Format section):
2. When creating the new Parameter Group, set the following values:
binlog_format=row
MySQL Target: The target schema character set must be set to utf8mb4.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 171
For information on defining transformations, see Using the Transform Tab.
Limitations
The following limitations apply:
Changes that are older than one year will not be captured.
Using the alter table <table_name> add column <column_name> statement to add
columns to the beginning or to the middle of a table is not supported. When the alter
table <table_name> add column <column_name> statement is used to add a column
to the middle of a table, the column will be added to the end of the table instead.
The AR_H_USER header column is currently not supported. For information on
using header columns, see Headers.
If an Amazon RDS for MySQL table contains LOBs and the task's Replicate Lob
columns option is disabled, the table will be replicated without the LOB columns.
Note that this only applies to MEDIUMBLOB, LONGBLOB, MEDIUMTEXT and
LONGTEXT columns. This limitation does not apply to BLOB, TINYBLOB, TEXT and
TINYTEXT columns.
If the Amazon RDS for MySQL database is stopped during Full Load, the Full Load
will end successfully, but the tables on the target may have less rows than the
source tables. If this should happen, either restart the task or reload the tables
with the missing rows.
A Unique Index/Primary Key created on only part of the column data is not
supported. If such a Unique Index/Primary Key is encountered, it will be created on
the entire target column.
The following is an example of a statement that creates a Unique Index/Primary
Key using only part of the column data:
CREATE INDEX partial_name ON customer (name(10));
Security Requirements
The Attunity Replicate user must have the ReplicationAdmin role with the
following privileges (according to task type):
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 172
REPLICATION CLIENT - Required for Change Processing tasks only. In other
words, Full Load only tasks do not require this privilege.
REPLICATION SLAVE - Required for Change Processing tasks only. In other words,
Full Load only tasks do not require this privilege.
SUPER - Only required in versions prior to Amazon RDS for MySQL 5.6.6.
The Attunity Replicate user must also have SELECT privileges for the source
tables designated for replication.
Data Types
The following table shows the Amazon RDS for MySQL database source data types that
are supported when using Attunity Replicate and the default mapping to Attunity
Replicate data types. When replicating to a MySQL target, the source and target data
types are the same, apart from the exceptions described in Homogeneous Replication.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.6 |
Amazon RDS for MySQL Database Source Data Types with
Mapping to Attunity Replicate Data Types
Amazon RDS for MySQL Source Data Types Attunity Replicate Data
Types
INT INT4
BIGINT INT8
MEDIUMINT INT4
TINYINT INT1
BIT BOOLEAN
LONGBLOB BLOB
MEDIUMBLOB BLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 173
Table 8.6 | Amazon RDS for MySQL Database Source Data Types with Mapping to Attun-
ity Replicate Data Types (Cont.)
Amazon RDS for MySQL Source Data Types Attunity Replicate Data
Types
TINYBLOB BYTES (255)
DATE DATE
DATETIME DATETIME
TIME STRING
TIMESTAMP DATETIME
YEAR INT2
DOUBLE REAL8
- 1.79E+308 to -2.23E-308, 0
and
2.23E-308 to 1.79E+308
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 174
Table 8.6 | Amazon RDS for MySQL Database Source Data Types with Mapping to Attun-
ity Replicate Data Types (Cont.)
Amazon RDS for MySQL Source Data Types Attunity Replicate Data
Types
*VARCHAR (45) WSTRING (45)
*CHAR WSTRING
*LONGTEXT NCLOB
*MEDIUMTEXT NCLOB
GEOMETRY BLOB
POINT BLOB
LINESTRING BLOB
POLYGON BLOB
MULTIPOINT BLOB
MULTILINESTRING BLOB
MULTIPOLYGON BLOB
GEOMETRYCOLLECTION BLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 175
Note If the DATETIME and TIMESTAMP data types are specified with a “zero” value
(i.e. 0000-00-00), you need to make sure that the target database in the replication
task supports "zero" values for the DATETIME and TIMESTAMP data types. If they are
not supported, you can use a transformation to specify a supported value (e.g. 1970.)
Otherwise, they will be recorded as null on the target.
Note The JSON data type introduced in Amazon RDS for MySQL 5.7 is not supported.
Consequently, JSON columns in the source tables will be ignored.
Homogeneous Replication
The following section describes how Replicate handles replication between an Amazon RDS
for MySQL source and an Amazon RDS for MySQL target (i.e. homogeneous replication).
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through
when replicating from Amazon RDS for MySQL, see the Amazon RDS for MySQL to
Attunity Replicate data types mapping table described earlier.
Data Types
When replicating to an Amazon RDS for MySQL target endpoint, the data types will
be identical with the following exceptions:
Amazon RDS for MySQL Source Data Amazon RDS for MySQL Target Data
Types Types
NUMERIC DECIMAL
Collation
When replicating from one Amazon RDS for MySQL endpoint to another, table and column
collations will be replicated to the target. Collatable data types are indicated by an asterisk
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 176
(*) in Table 11–3 above.
To support collation replication, the DBA must ensure that the collations defined for
the source Amazon RDS for MySQL database are the same as those defined for the
target Amazon RDS for MySQL database.
Note You can also use Amazon RDS for MySQL files as a source. For more
information, see Using the Attunity Replicate File Channel.
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your endpoint. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Amazon RDS
for MySQL database. This is optional.
4. Select SOURCE as the endpoint role.
5. From the Type drop-down list, select Amazon RDS for MySQL.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 177
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server field, enter the host name or IP address of the computer on which
the Amazon RDS for MySQL database is installed.
7. Optionally, change the default port (3306).
8. Type the Amazon RDS for MySQL authentication information (Username, Password)
for the authorized user for this Amazon RDS for MySQL database. If you do not know
this information, see your Amazon RDS for MySQL database Administrator (DBA).
Notes
Important: Make sure that the Amazon RDS for MySQL user entered in the
Amazon RDS for MySQL Authentication section has the correct access
privileges. For information on how to provide the required privileges, see
Security Requirements.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 178
Verify Full - Similar to Verify CA, but also performs host name identity
verification by checking the host name the client (i.e. Replicate) uses for
connecting to the server against the identity in the certificate that the
server sends to the client. The client checks whether the host name that it
uses for connecting matches the Common Name value in the server
certificate. The connection fails if there is a mismatch.
Client certificate path - The path to the client certificate requested by the server.
Private key path - The path to the client private key file in PEM format.
CA path - The path to the Certificate Authority (CA) that issued the client
certificate file in PEM format.
Selecting a Schema
You can choose which Amazon RDS for MySQL database to access. After configuring
the Amazon RDS for MySQL source database connection settings, open the Select
Tables dialog box (by clicking the Table Selection button on the right of the console)
and select which schema to use from the Schema drop down list.
See also Defining Tasks.
Check binary log for new events every: Specify how often to check the binary
log for changes when the endpoints is idle.
Additional ODBC connection properties: Specify any additional ODBC
connection parameters that may be required.
Note Attunity Replicate assumes that Amazon RDS for MySQL Client 5.2.6 to 5.3.x
for Linux or Amazon RDS for MySQL ODBC Client 5.2.6 to 5.3.x 64-bit for Windows
is installed on the Attunity Replicate Server machine. If a version later than 5.3.x is
installed, you need to specify the version number as an internal parameter where
provider is the Parameter and Amazon RDS for MySQL ODBC <version>
Unicode Driver is the Value (where <version> is the client version e.g. 5.4).
For instructions on setting internal parameters, see Internal Parameters.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 179
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 180
Using Amazon RDS for PostregSQL as a Source
This section describes how to set up and use an Amazon RDS for PostregSQL database
as a source in a replication task.
In this section:
▶ Source Prerequisites
▶ Client Side
▶ Required Permissions
▶ Setting up Amazon RDS for PostgreSQL for CDC (Change Data Capture)
▶ Using an Account without the "rds_superuser" Role
▶ Limitations
▶ Amazon RDS for PostregSQL Source Data Types
▶ Homogeneous Replication
▶ Replication of Partitioned Tables
▶ Data Type Considerations and Exceptions
▶ Column and Table Collation
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Removing Replicate Artifacts from the Amazon RDS for PostregSQL Database
Source Prerequisites
The following section lists the prerequisites for working with Attunity Replicate and
an Amazon RDS for PostregSQL database source.
Client Side
Attunity Replicate Server for Windows:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 181
Note PostgreSQL driver 09.06.04.00 is not supported.
Note Make sure that the PostregSQL ODBC installation folder "bin" (e.g.
"C:\Program Files\psqlODBC\0905\bin") is added to the system PATH.
Driver = /usr/pgsql-9.4/lib/psqlodbc.so
Setup = /usr/pgsql-9.4/lib/psqlodbcw.so
Debug = 0
CommLog = 1
UsageCount = 2
When the Apply Changes task option is enabled, the user specified in the
Amazon RDS for PostregSQL source endpoint’s General tab must be granted
super-user permissions.
Required Permissions
The user specified in the General tab when Setting General Connection Properties must
be granted the following permissions in the Amazon RDS for PostrgreSQL database:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 182
For Full Load replication: Standard SELECT on the source database
For Apply Changes replication: See Setting up Amazon RDS for PostgreSQL for
CDC (Change Data Capture).
To do this:
1. Choose a schema where you want the objects to be created. The default schema
is public. Ensure that the schema exists and is accessible by the NoPriv account.
2. Create the table attrep_ddl_audit by running the following command:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 183
c_user varchar(64), -- Informational: current_user
c_txn varchar(16), -- Informational: current transaction
c_tag varchar(24), -- Either 'CREATE TABLE' or 'ALTER TABLE' or
'DROP TABLE'
c_oid integer, -- For future use - TG_OBJECTID
c_name varchar(64), -- For future use - TG_OBJECTNAME
c_schema varchar(64), -- For future use - TG_SCHEMANAME. For now,
holds the current_schema
c_ddlqry text -- The DDL query associated with the current DDL
event)
4. If you are logged in with a NoPriv account, log out of the NoPriv account and log
in with an account that has the rds_superuser role assigned to it.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 184
For more information on the replication configuration parameters, see the
PostgreSQL Help.
5. Create the event trigger attrep_intercept_ddl by running the following command:
Limitations
The following limitations apply when using Amazon RDS for PostregSQL as a source:
LANGUAGE plpgsql
AS $$
BEGIN
$$;
Partitioned tables: When performing a Full Load replication of partitioned tables, the
parent table will be created and separate tables will be created for each partition.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 185
A DML on the parent partitioned table will be applied on the physical partition of
the corresponding target table only.
Primary keys are not supported on partitioned tables, which may impact
UPDATE operations.
Note In order to replicate partitioned tables from an Amazon RDS for PostregSQL
source to an Amazon RDS for PostregSQL target, you first need to manually
create the parent and child tables on the target. Then define a separate task to
replicate to those tables. In such a case, the task settings should be configured
to “Truncate before loading”. For more information on the “Truncate before
loading” option, see Full Load Settings.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.7 |
Supported Amazon RDS for PostregSQL Source Data Types with
Mapping to Attunity Replicate Data Types
Amazon RDS for PostregSQL Data Types Attunity Replicate Data Types
INTEGER INT4
SMALLINT INT2
BIGINT INT8
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 186
Table 8.7 | Supported Amazon RDS for PostregSQL Source Data Types with Mapping to
Attunity Replicate Data Types (Cont.)
Amazon RDS for PostregSQL Data Types Attunity Replicate Data Types
DECIMAL(p,s) If precision is => 0 and =< 38,
then:
NUMERIC
STRING
REAL REAL4
DOUBLE REAL8
SMALLSERIAL INT2
SERIAL INT4
BIGSERIAL INT8
MONEY NUMERIC(38,4)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 187
Table 8.7 | Supported Amazon RDS for PostregSQL Source Data Types with Mapping to
Attunity Replicate Data Types (Cont.)
Amazon RDS for PostregSQL Data Types Attunity Replicate Data Types
VARCHAR(n) WSTRING (n)
unboundedVarcharMaxSize
See also Internal Parameters.
TEXT NCLOB
BYTEA BLOB
TIMESTAMP DATETIME
Note Replicate only supports ISO formatted textual DATE formats (the default). If
other formats are used, an error will be generated. You can change the date format
in the Amazon RDS for PostregSQL.conf file or using the PGDATESTYLE
environment variable. You can also change the date format at database level.
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 188
Table 8.7 | Supported Amazon RDS for PostregSQL Source Data Types with Mapping to
Attunity Replicate Data Types (Cont.)
Amazon RDS for PostregSQL Data Types Attunity Replicate Data Types
CIDR STRING (50)
UUID STRING
TSVECTOR CLOB
TSQUERY CLOB
XML CLOB
JSON NCLOB
ARRAY NCLOB
COMPOSITE NCLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 189
Table 8.7 | Supported Amazon RDS for PostregSQL Source Data Types with Mapping to
Attunity Replicate Data Types (Cont.)
Amazon RDS for PostregSQL Data Types Attunity Replicate Data Types
CHARACTER VARYING If length is specified:
WSTRING (LENGTH)
If no length is specified:
WSTRING (8000)
TINTERVAL WSTRING(255)
Homogeneous Replication
When replicating from an Amazon RDS for PostrgreSQL source to a PostrgreSQL
target, most of the source and target data types will be identical. The exceptions are
listed in the table below.
Additionally, in homogeneous replication, source column and table collations will
be replicated to the target as described in Column and Table Collation.
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through
when replicating from Amazon RDS for PostregSQL, see the Amazon RDS for
PostregSQL to Attunity Replicate data types mapping table described earlier.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 190
Table 8.8 |
Data Type Exceptions in Homogeneous Replication
Amazon RDS for PostregSQL Source PostgreSQL-based Target
ENUM STRING
COMPOSITE STRING
A numeric array with precision and scale. A numeric array without precision and scale.
Note To support collation replication, the DBA must ensure that the collations
defined for the source Amazon RDS for PostregSQL database are the same as those
defined for the PostgreSQL-based target database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 191
3. In the Description field, type a description that helps to identify the Amazon RDS
for PostregSQL database. This is optional.
4. Select SOURCE as the database role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server name field, specify the host name of the database instance,
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the Amazon RDS for PostregSQL database user
entered in the Amazon RDS for PostregSQL database Authentication section has
the correct access privileges.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 192
9. Type the Database name or select one from the list of available endpoints. This is
the name of the Amazon RDS for PostregSQL database from which you are
replicating data.
10. Click OK to save your settings and close the dialog box.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 193
Setting Advanced Connection Properties
In the Advanced tab, you can set the following properties:
Capture DDLs: When this option is selected, the following actions occur:
Operational artifacts are created (by Replicate) in the database when the
task starts. In order to capture DDL events, Attunity Replicate creates
various artifacts in the Amazon RDS for PostregSQL database when the task
starts. You can later remove these artifacts as described in Removing
Replicate Artifacts from the Amazon RDS for PostregSQL Database.
Streamed DDL events are captured.
Create DDL artifacts in schema: The schema in which the operational
DDL database artifacts will be created. The default value is "Public".
WAL heartbeat - An Apply Changes task that is running but not capturing
changes (due to source table inactivity) will continue to occupy the LSN position in
its replication slot, thereby preventing truncation of the WAL. Since the WAL is a
server-wide resource used by all Amazon RDS for PostregSQL processes, it may
grow extremely large if no changes are captured for an extended period.
To prevent this from happening, enable the "WAL heartbeat" option. When this
option is enabled, the Amazon RDS for PostregSQL source endpoint mimics task
activity by periodically committing pseudo transactions (i.e. "Heartbeats") to the
heartbeat table, thereby advancing the task slot’s LSN position.
Schema for heartbeat artifacts: The schema in which the WAL heartbeat
table (attrep_wal_heartbeat) will be created. The default value is "public".
Heartbeat frequency (minutes): The frequency with which to
commit transactions to the heartbeat table.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 194
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Note that the event trigger does not belong to a specific schema.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 195
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 196
Using ARC CDC Solutions in Attunity Replicate
This section describes how to use an ARC (Attunity Replicate Connect) CDC Solution as
an Attunity Replicate endpoint.
Notes
In this section:
▶ Prerequisites for Using ARC CDC Solutions
▶ Additional Prerequisites when Using ARC Non-Relational Sources
▶ ARC CDC Solution Security Considerations
▶ Encrypting Communications Between Replicate and ARC Data Sources
▶ Limitations
▶ ARC Source Data Type Mapping
▶ Working with ARC CDC Solutions
▶ Create an ARC CDC Solution in Attunity Replicate Connect Studio
▶ Add the ARC Data Source to Attunity Replicate
▶ Add the ARC CDC Solution Endpoint to a Task
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 197
information on how to install ARC, see the Attunity Replicate Connect Installation
Guide for the computer platform that is relevant to the CDC Solution you are
working with.
Attunity Studio version 5.3.2 or above: Attunity Studio is used to set up a
CDC Solution. This will create the CDC Solution that can be used in the Attunity
Replicate replication task. Attunity Studio must be installed on a Windows
computer. For information on installing Attunity Studio, see the Attunity Replicate
Connect Studio Installation Guide.
When the ARC database is on DB400-AS4002: To apply deletes to the
target, journaling must be set to *BOTH.
ARC relational data sources that support table ownership: If the table owner
contains an underscore, you must create the ARC solution with a default table owner.
When the source endpoint is IBM IMS (ARC): The ARC IMS Bulk data source is
always created as IMS-DLI. You should specify the correct ARC IMS Bulk started
task in the endpoint settings. The ARC USERLIB library contains the following
started task examples:
NVIMSSRV for IMS DLI access
NVBMPSRV for IMS BMP access
For more information on creating ARC Solutions, please refer to the Attunity
Replicate Connect User Guide and Reference.
Note For information about installing the database you are working with, see
the installation guide for that database.
If the source tables contain Primary Keys, edit the source table metadata in Attunity
Studio and mark the Primary Key columns as shown in the figure below. This should
be done at the start of creating the CDC solution when importing the tables/files.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 198
For more information on selecting Primary Keys in Attunity Studio, refer to the
"Working with Metadata in Attunity Studio" chapter in the Attunity Replicate
Connect User Guide and Reference.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 199
database and the ARC Agent machine. However, when capturing changes from a non-
relational database, the encryption key needs to be defined in four different locations:
The Attunity Replicate ARC database, the ARC Agent machine, the ARC Router machine,
and the Router Authenticator.
d. To the right of the Encryption Keys list (in the lower half of the screen),
click the Add button.
The Encryption Key dialog opens.
e. Enter an encryption key name and value and then click OK.
Note Steps 2-4 apply to non-relational ARC data sources only (e.g. VSAM). If
you are working with a relational ARC data source, continue to Step 5.
2. On the Router machine, create an encryption key which has the same values as
the encryption key that you created on the Agent machine. The procedure is the
same as described in Step 1, but instead of expanding the machine on which your
ARC Solution’s Agent is installed, expand the machine on which your ARC Solution’s
Router is installed.
3. On the Router machine, define the Agent as an authenticator according to
the following steps:
a. In the Configuration tab, expand the machine on which the Router is
installed. Then, right-click your solution’s Router binding (e.g vsam_router)
and select Open.
b. In the Machines tab, click the Security
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 200
d. From the Resource type, drop-down list, select Adapter.
e. In the Resource name field, specify the name of your solution’s Agent as
it appears under the Adapters folder (e.g VSAM_ag).
f. At the bottom of the dialog box, select the Encryption key check box and
then specify the encryption key name and value in the designated fields. These
values must be the same as the encryption key values defined in Step 1.
4. In the Router’s Properties tab, expand the comm property and set
the defaultEncryptionMethod property to AES.
Note If the Properties tab is not displayed, open the Preferences dialog
box (by selecting Preferences from the Windows menu), navigate to
Studio and then select the Show advanced environment parameters
option in the Advanced tab.
5. In the Advanced tab of the Replicate ARC database, specify the encryption key
name and value. These values must be the same as the encryption key values
defined in Step 1.
For more information on the Advanced tab, see Using ARC CDC Agents as
Limitations
When working with ARC data sources, the following limitations apply:
IBM DB2 on iSeries (ARC): Table and field names that contain the "/" character
are not supported.
Only one Replicate task can work with the same ARC Agent concurrently.
Replication of DDL changes to the target endpoint is not supported.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 201
Table 8.9 |
Supported ARC Data Types with Mapping to Attunity Replicate Data Types
ARC Data Types Attunity Replicate Data Types
ARC SQL Server
INT REAL4
REAL REAL4
FLOAT REAL8
BIT INT1
TINYINT INT1
SMALLINT INT2
BIGINT NUMERIC
DECIMAL NUMERIC
NUMERIC NUMERIC
MONEY NUMERIC
SMALLMONEY NUMERIC
DATETIME DATETIME
SMALLDATETIME DATETIME
CHAR STRING
VARCHAR STRING
NCHAR STRING
NVARCHAR STRING
BINARY BYTES
VARBINARY BYTES
TIMESTAMP BYTES
UNIQUEIDENTIFER STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 202
as the Attunity Replicate database. You can then use this database as your source for
any task that you create. To use ARC CDC Solutions, carry out the following:
Create an ARC CDC Solution in Attunity Replicate Connect
Studio Add the ARC Data Source to Attunity Replicate Add the
ARC CDC Solution Endpoint to a Task
1. Using Attunity Replicate Connect Studio, create a CDC Solution using the CDC
Solution that you want to use as your source database in Attunity Replicate.
For information on creating a CDC Solution, refer to the Attunity Integration Suite
User Guide and Reference.
2. At the end of the process for creating a CDC solution, you must deploy the solution.
Do not activate the solution. Attunity Replicate activates the solution automatically
when you begin to work with the CDC Solution.
Note If you activate the solution, then disable the router and staging area
workspaces and keep the agent workspace enabled. For more information, see
the Attunity Replicate Connect User Guide and Reference.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 203
Adding a Relational ARC Data Source to Attunity Replicate
5. Select a relational ARC data source from the Type list. The ARC data sources are
listed as <Data Source> (ARC). For a list of supported relational data sources,
see Using ARC CDC Agents as Endpoints.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Host/IP field, type the name or IP Address of the computer where the
CDC Solution (data source) you defined in Attunity Studio is located.
7. In the Port field, type the port number for the port you used when creating the
CDC Solution in Attunity Studio. The default port number is 2551.
8. In the CDC Solution field, enter the name of the solution you defined when
you created the data source in Attunity Studio.
9. In the User name and Password fields, enter the username and password
required to access the database.
10. Click OK to add the database to Attunity Replicate. You can use this database as
the source database for any replication task that you create.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 204
Note To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
When you add a database to Attunity Replicate and you select a non-relational ARC
data source as the database type, the following dialog box opens.
5. Select an ARC non-relational data source from the Type list. The ARC data sources
are listed as <database> (ARC), for example RMS (ARC). For a list of supported
non-relational data sources, see Using ARC CDC Agents as Endpoints.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. When working with the RMS (ARC) data source, choose one of the following
Change Processing modes:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 205
Non relational (the default) - When this mode is selected, Replicate reads the
changes from a CSV file that contains the modified data records. Use this mode
if you need to retrieve changes to arrays and variant tables.
If you select this option, continue from Step 7 below.
Relational - When this mode is selected, Replicate reads the changes directly
from the ARC Agent. Relational mode improves performance but does not
support changes to complex data structures such as arrays and variant tables.
If you select this option, continue from Step 6 in Adding a Relational ARC
Data Source to Attunity Replicate.
7. In the Port field, type the port number for the port you used when creating the
CDC Router in Attunity Studio. The default port number is 2551.
8. In the CDC Solution field, enter the name of the solution you defined when
you created the data source in Attunity Studio.
9. If a username and password are required to access the CDC Solution Router, enter
them in the User name and Password fields in the Local ARC router section.
10. If a username and password are required to access the CDC Solution, enter them in
the User name and Password fields in the ARC on <source> machine section.
11. Required for IBM IMS (ARC) only: In the Bulk started task field, specify the
correct z/OS Started Task name for IMS/BMP or IMS/DLI. This member was copied
to the z/OS PROCLIB library from <ARC HLQ>.USERLIB. NVBMPSRV and NVIMSSRV
are the provided member names.
Note If you choose IMS/DLI, you will need to close the database to IMS/TM or
IMS/DBCTL. This option might be faster than using BMP. IMS/BMP does not
require exclusive access to the database.
12. Click OK to add the database to Attunity Replicate. You can use this database as
the source database for any replication task that you create.
Note To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 206
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
Fixed NAT: Select this to indicate that the connection is made with a fixed
network address translation.
Timeout: Enter the amount of time, in seconds, to wait for interactions before
disconnecting. 0 indicates that the system does not timeout. The default value is 0.
Event wait: Enter the maximum amount of time (in seconds) to wait for a change
event to take place before the system times out. The default value is 300.
CDC batch size: Enter the maximum number of change events that can
be transferred in a single batch. The default value is 200.
Bulk batch size: Enter the unloading batch size. The default value is 100.
Trace: Select this to enable tracing for the change processing.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 207
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 208
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 209
Using a File as a Source
This section describes how to set up and use delimited text files as a source in a
replication task. You can use the File target endpoint to export database tables to files,
which can then be used as a source in a Replicate task with a File source endpoint.
In this section:
▶ General Overview
▶ File Source Overview
▶ Reference Files
▶ Full Load Files
▶ Change Files
▶ Prerequisites
▶ Limitations
▶ Setting General Connection Properties
▶ Defining Tables and Full Load Data
▶ Setting Advanced Options
▶ Internal Parameters
▶ Settings Summary
General Overview
The Replicate File endpoint can be used either as a source or as a target. When used as a
source, the File endpoint requires the source files to be in delimited text file format. When
used as a target, the File endpoint generates the data files ether in delimited text file format
(CSV) or in JSON format (according to the format selected in the endpoint settings).
Delimited text files are used to store data in tabular format. Examples of delimited text
file formats include the CSV (Comma Separated Values) and TSV (Tab Separated Values)
formats. Some organizations may implement procedures that export data from a
database to a delimited text file while others may simply prefer this format as a
convenient way of storing tabular data.
In a delimited text file, each record in the table occupies a separate row. Delimiters
are used to mark the beginning of a new row or the beginning of a new column.
Virtually any character can be used as a delimiter, although a newline (\n) is often
used to separate rows, and commas are commonly used to separate columns.
In JSON files, each record is represented by a single line.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 210
So, for example, the following table:
Three types of delimited files are used in the Replicate Source File endpoint:
Full Load Files
Change Files
Reference Files
Reference Files
Change Files can either reside in a single location or in multiple locations. To access
Change Files that reside in different locations and/or that do not include the target table
names, you need to use a Reference File. If the rows in the Change File(s) contain the
names of the tables to which to apply the changes, then the Reference File only needs to
contain the paths to the Change Files. If each Change File contains changes for a single
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 211
table, then each of the Change File paths in the Reference File needs to be preceded by
the name of its corresponding target table.
For more information on Reference File and Reference File formats, see
Change Processing.
Each row in the Reference File should be formatted as follows:
[<table_name>],<full path to Change File>
Where [<table_name>] is required only if the referenced Change File contains changes
for a single table.
Note Reference File names cannot exceed 70 characters (no such limitation exists
for the path length). Reference File names that exceed 70 characters will be ignored
and appropriate warning will be written to the log.
Example 8.1 | Reference File: Each Change File contains changes for
a single table:
table1,c:\temp\cdc1.csv
table2,c:\temp\cdc2.csv
table3,c:\temp\cdc3.csv
Example 8.2 | Reference File: Each Change File contains changes for
multiple tables:
c:\temp\cdc1.csv
c:\temp\cdc2.csv
c:\temp\cdc3.csv
12,May,2011,female,3236776
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 212
9,March,2009,male,9648675
For more information on Full Load data files and creating tables, see Defining Tables
and Full Load Data.
Change Files
A Change File is a delimited text file that contains a record of DML changes - represented as
rows - to apply to the specified target tables. Replicate reads the Change File(s) and applies
the changes to the relevant target tables, which can either be specified in the Change File
itself or in a Reference File (see Reference Files below for details). Change Files are picked
up from the source directory according to their modification date, thereby ensuring that the
changes will be processed in the proper sequence.
Note The Change File modification date must be both newer than the task
start timestamp and newer than the last processed Change File.
(Optional) The change operation e.g. DELETE. If the operation field is absent,
INSERT is assumed.
The name of the target table to which to apply the change (only required if the
Change File contains changes for multiple tables)
(Optional) The timestamp of the change i.e. when the change occurred
(Optional) The user who applied the change
The data to change (one or more columns)
Change Files can either contain changes for multiple tables or for a single table, as
shown in the examples below.
Note To access Change Files that reside in different locations and/or that do not
include the target table names, you need to use a Reference File. For more
information on Reference Files, see Reference Files.
Note Change File names cannot exceed 70 characters (no such limitation exists for
the path length). Change File names that exceed 70 characters will be ignored and
appropriate warning will be written to the log.
Example 8.3 | Change File that contains changes for multiple tables
INSERT,table1,ts1,user,dog,cat,bird
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 213
INSERT,table2,ts1,user,dog,cat,bird
DELETE,table3,ts1,user,dog,cat,bird
Example 8.4 | Change File that contains changes for a single table
INSERT,,ts1,user,dog,cat,bird
INSERT,,ts1,user,dog,cat,bird
DELETE,,ts1,user,dog,cat,bird
Prerequisites
Before you begin to work with a File as a source in Attunity Replicate, make sure that
the following prerequisites have been met:
Attunity Replicate installed in your network
Change Files, Full Load files and Reference Files should be in delimited text file format
Source files (including the Reference File) should be accessible from the
Attunity Replicate machine.
Limitations
The following limitations apply to the File source:
Change Files that are currently being used in a Replicate task cannot be modified
while the task is in progress.
Stopping a Full Load task and then starting it again will start the task from
the beginning (and not from the point at which it was stopped).
Reload from timestamp is not supported
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 214
2. In the Name field, type a name for your endpoint. This can be any name that will
help to identify the endpoint being used.
3. In the Description field, type a description that helps to identify the File
endpoint. This is optional.
4. Select SOURCE as the endpoint role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. Configure the settings in the General tab as described in the table below.
Table 8.10 |
File Source Endpoint - General Tab Options
Option Description
File Format
Field Delimiter The delimiter used to separate columns in the source files. The
default is a comma.
Example:
mike,male
Record delimiter The delimiter used to separate records (rows) in the source files.
The default is a carriage return (\n).
Example (Using an asterisk as the row delimiter)
mike,male*sara,female
Null value The character used to indicate a null value in the source files.
mike,male,295678*sara,female,@
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 215
Table 8.10 | File Source Endpoint - General Tab Options (Cont.)
Option Description
Quote character The character used at the beginning and end of a column that
contains the column delimiter character. The default is the double-
quote character ("). When a column that contains column
delimiters is enclosed in double-quotes, the column delimiter
characters are interpreted as actual data, and not as column
delimiters.
"sunroof, power-steering"
Escape character The character used to escape a string when both the string and the
column containing the string are enclosed in quotation marks.
Note that the string’s quotation marks will be removed unless they
are escaped.
Code page Specify the code page of your source files if it is different from the
default (65001).
Ignore records Optionally, specify which header and footer rows in the source
files to ignore. Make sure that the header and footer rows to
ignore do not contain actual data.
Change Processing
Important: Changes cannot be captured from Change Files that are present
during the Full Load operation. Consequently, the Change Files should be placed
in their source location(s) only after Full Load completes.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 216
Table 8.10 | File Source Endpoint - General Tab Options (Cont.)
Option Description
Folder Select this option if your Change Files contain the target table
names and reside in a single folder. Then specify the folder
location in the designated field. You can optionally use wildcard
characters to only process files that match the specified pattern.
Example:
c:\temp\*changes.CSV
See also: Change Files.
Use Reference Files Select this option if you are using Reference Files to point to the
location of the Change Files. Then specify one of the following:
Change File path is Select this option if each of the Change File paths in the reference
preceded by table files is preceded by a table name.
name
Note Selecting this option will disable the Table name check
box in the Header columns ordinal position section.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 217
Table 8.10 | File Source Endpoint - General Tab Options (Cont.)
Option Description
Header column order Specify the position of each header column in your Change Files.
Apart from the data columns which must be positioned after the
header columns, the columns can be positioned in any order.
DELETE,table1,timestamp,user1,dog,cat,bird
Then, the column positions would be set as follows:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 218
Table 8.10 | File Source Endpoint - General Tab Options (Cont.)
Option Description
4. Select the column from the Header Columns tab and then
click OK.
5. Specify a name for the column. For example, if you
selected the $AR_H_TIMESTAMP header column, specify
"Timestamp".
6. Click Finish.
7. Repeat steps 1-6 to add the other header column.
Start data from Specify which column to start the actual data from. Note that data
column: columns must be positioned after header columns. See the
example in the description of the Header column order field
above.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 219
Note To determine if you are connected to the endpoint you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button is not
available unless the test connection fails.
To define a table:
To edit a field:
Double-click the column in the New/Edit Source Table dialog box and then edit
the values as described above.
To delete a field:
Select the column in the New/Edit Source Table dialog box and then click
the Delete Field button.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 220
To change the position of a field:
Select the field and then click the Up/Down and Move to Top/Move to
Bottom buttons as required.
To edit a table:
In the Tables tab, either double-click the table or select the table and then click
the Edit button. Edit the table as described above.
To delete a table:
In the Tables tab, select the table and then click the Delete button.
12,May,2011,female,3236776
9,March,2009,male,9648675
30,June,2002,female,3458795
Note Boolean values must be expressed as the digits 1 (TRUE) or 0 (FALSE) and
not TRUE or FALSE.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 221
Setting Advanced Options
In the Advanced tab, the following options are available.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 222
Table 8.11 | File Source Endpoint - Advanced Tab Options
Option Description
File If your source files (Full Load and/or Change Files) are not in delimited
preprocessing text format, you can convert them to the required format using a
command conversion program.
c:\temp\files\convertfile.exe
PRODUCT_INSTALLATION\prod\data\tasks\TASK_NAME\trans_
files\
Check for Specify how often to check the Change Files for updates.
changes
every
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 223
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 224
Using Hadoop as a Source
This section describes how to set up and use Hadoop as the source endpoint in a
replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Required Permissions
▶ Hadoop Endpoint Source Data Types
▶ Unsupported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Prerequisites for using Kerberos when Replicate is installed on Red Hat 7.4 or 7.5
Prerequisites
Before you begin to work with a Hadoop cluster as a data source in Attunity
Replicate, make sure that the following prerequisites have been met:
General:
The Hadoop WebHDFS must be accessible from the Attunity Replicate machine.
The Hadoop Data Nodes must be accessible from the Attunity Replicate machine.
Configure each NameNode and each DataNode with an SSL certificate (issued
by the same CA).
Place the CA certificate on the Replicate Server machine. The certificate
should be a base64-encoded PEM (OpenSSL) file.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 225
Permissions: The user specified in the Hadoop source settings must have
read permission for the HDFS directories that contain the data files.
Limitations
The following limitations apply:
Change data capture is not supported. The Hadoop endpoint will not be available for
selection if either Apply Changes or Store Changes is enabled in the task settings.
Replicating data compressed using a method other than gzip is currently
not supported.
Replicating data that is not in Text format is currently not supported.
Replicating from tables with skews, buckets or partitions is currently not supported.
Limited LOB support only.
Due to a Hive limitation, in Hive version 0.12 and earlier versions, only
alphanumeric and underscore characters are allowed in table and column names.
From Hive 0.13, column names can contain any Unicode character.
Required Permissions
The Hadoop NameNode must be accessible from the Attunity Replicate machine.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.12 |
Supported Hadoop Data Types Mapped to Attunity Replicate Data Types
Hadoop Data Types Attunity Replicate Data Types
BOOLEAN BOOL
BINARY BLOB
DATE DATE
TIMESTAMP DATETIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 226
Table 8.12 | Supported Hadoop Data Types Mapped to Attunity Replicate Data
Types (Cont.)
SMALLINT INT2
INT INT4
BIGINT INT8
FLOAT REAL4
DOUBLE REAL8
VARCHAR STRING
CHAR STRING
STRING CLOB
DECIMAL NUMERIC
ARRAYS
MAPS
STRUCTS
UNION
1. In the Attunity Replicate console, click the Manage Endpoint Connections toolbar
button to open the Manage Endpoints Connections dialog box. Then click the New
Endpoint Connection button. For more information on adding an endpoint to
Attunity Replicate, see Working with Endpoints.
2. In the Name field, type a name for your endpoint. This can be any name that will
help to identify the endpoint being used.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 227
3. In the Description field, type a description that helps to identify the Hadoop
endpoint. This is optional.
4. Select SOURCE as the endpoint Role.
6. In the Hadoop NameNode field, enter the host name or IP address of the
Hadoop NameNode machine.
To determine if you are connected to the endpoint you want to use or if the
connection information you entered is correct, click Test Connection.
a. To encrypt the data between the Replicate machine and HDFS, select Use SSL.
In order to use SSL, first make sure that the SSL prerequisites described in
Prerequisites has been met.
In the CA path field, either specify the directory containing the CA
certificate. -OR-
User name - Select to connect to the Hadoop cluster with only a user
name. Then, in the User name field, specify the name of a user
authorized to access the Hadoop cluster.
Kerberos - Select to authenticate against the Hadoop cluster using Kerberos.
Replicate automatically detects whether Attunity Replicate Server is running
on Linux or on Windows and displays the appropriate settings.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 228
Attunity Replicate Server on Linux:
Use global Kerberos ticket file - Select this option if you want to
use the same ticket for several Hadoop endpoints (source or target).
In this case, you must make sure to select this option for each
Hadoop endpoint instance that you define.
Use specific Kerberos ticket file - Select this option if you want
to use a different ticket file for each Hadoop endpoint. Then specify
the ticket file name in the designated field.
This option is especially useful if you need to perform a task-level audit of
Replicate activity (using a third-party tool) on the Hadoop NameNode. To
set this up, define several instances of the same Hadoop endpoint and
specify a unique Kerberos ticket file for each instance. Then, for each
task, simply select a different Hadoop endpoint instance.
Note
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 229
Realm: The name of the realm in which your Hadoop cluster resides.
Note When the Replicate KDC and the Hadoop KDC are in
different domains, a relationship of trust must exist between the
two domains.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 230
If you are unsure about any of the above, consult your
IT/security administrator.
For additional steps required to complete setup for Kerberos
authentication, see Using Kerberos Authentication.
User name and password - Select to connect to the Hadoop NameNode
or to the Knox Gateway (when enabled - see below) with a user name and
password. Then, in the User name and Password fields, specify the
required user name and password.
Important: Make sure that the specified user has the required
Hadoop access privileges. For information on how to provide the
required privileges, see Required Permissions.
Note To be able to select this option, first select Use SSL and then
select Password from the Authentication type drop-down list.
Knox Gateway host - The FQDN (Fully Qualified Domain Name) of the
Knox Gateway host.
Knox port - The port number to use to access the host. The default is "8443".
Knox Gateway path - The context path for the gateway. The default
is "gateway".
Note The port and path values are set in the gateway-site.xml file. If you
are unsure whether the default values have been changed, contact your IT
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 231
department.
Cluster name - The cluster name as configured in Knox. The default is "default".
10. In the HDFS section, select either WebHDFS or HttpFS as the HDFS access
method. If you are accessing MapR, it is recommended to use HttpFS.
Note When the Use Knox Gateway option is selected, the NameNode, HttpFS
Host, and Port fields described below are not relevant (and are therefore hidden).
Note This is the Active node when High Availability is enabled (see below).
a. In the HttpFS Host field, specify the IP address of the HttpFS host.
b. In the Port field, optionally change the default port (14000).
Note When the Use Knox Gateway option is selected, the Host and Port
fields described below are not relevant (and are therefore hidden).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 232
Setting Advanced Connection Properties
In the Advanced tab, you can specify source file delimiters and other properties. Note
that the source file delimiters only need to be specified if one of the following is true:
The Hive version is earlier than 0.13
The SerDe property names used to create the source data files are different from
the Hadoop defaults
The default Hadoop property names are as follows:
Table 8.13 |
Hadoop Source Endpoint - Advanced Tab Options
Option Description
Null The value used to indicate a null value in the source files.
value
Example (where @@ is the null value):
mike,male,295678
sara,female,@@
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 233
Table 8.13 | Hadoop Source Endpoint - Advanced Tab Options (Cont.)
Option Description
Escape This can either be the character used to escape the field delimiter character -
character if the source files were created using a SerDe that does not support quote
characters (see Example 1) or the character used to escape the quote
character - if the source files were created using a SerDe that supports quote
characters (see Example 2).
sunroof\,power-steering
"\"sunroof, power-steering\""
Record The delimiter used to separate records (rows) in the source files.
delimiter If the LazySimpleSerde SerDe was used, the record delimiter must be \n.
Quote The character used to escape the field delimiter character in the source files.
character When a field delimiter is escaped, it is interpreted as actual data, and not as a
field delimiter.
"sunroof,power-steering"
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 234
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
To work with Kerberos when Replicate is installed on Red Hat 7.4 or 7.5, you need
to rename the libraries as outlined in the procedure below.
mv libk5crypto.so.3 libk5crypto.so.3.org
mv libkrb5.so.3 libkrb5.so.3.org
mv libkrb5support.so.0 libkrb5support.so.0.org
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 235
Using IBM DB2 for iSeries as a Source
This section describes how to set up and use an IBM DB2 for iSeries database as the
source database in a replication task.
In this section:
▶ Prerequisites
▶ Client
▶ Change Processing
▶ Required Permissions
▶ Limitations
▶ IBM DB2 for iSeries Database Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Overriding CCSID to Character Set Mapping
▶ Adding the RRN Column to Target Tables
▶ Replicating System Names
▶ Skipping Journal Validation
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following topic lists the prerequisites for using an IBM DB2 for iSeries endpoint in
a Replicate task.
Client
Windows: To work with an IBM DB2 for iSeries database as a source in
Attunity Replicate, the iSeries ODBC driver (5770-XE1 IBM i Access for
Windows) must be installed on the Replicate Server machine.
The minimum supported version is 12.64.
Note When Replicate is installed on Windows Server 2016, use the 5733XJ1
IBM i Access Client Solutions instead.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 236
The driver is part of IBM i Access Client Solutions, which can be downloaded from
the IBM web site by authorized IBM customers. Note that it is not necessary to
install all of the IBM i Access Client Solutions components, only the ODBC driver
(which is installed regardless of which components you choose to install).
Linux: To work with an IBM DB2 for iSeries database as a source in Attunity
Replicate, the iSeries ODBC driver (5770-XL1 IBM i Access for Linux) must be
installed on the Replicate Server machine.
The minimum supported version is 12.64.
The driver is part of IBM i Access Client Solutions, which can be downloaded from
the IBM web site by authorized IBM customers. Note that it is not necessary to
install all of the IBM i Access Client Solutions components, only the ODBC driver
(which is installed regardless of which components you choose to install).
After installing the Linux Driver, edit the file /etc/odbcinst.ini and add the
following section:
[IBM i Access ODBC Driver 64-bit] Description
Setup = /opt/ibm/iaccess/lib64/libcwbodbcs.so
Threading = 0
DontDLClose = 1
UsageCount = 1
Change Processing
For DB2 iSeries version 7.1, the following IBM patch needs to be installed:
SF99701: 710 DB2 for IBM i - Level 3 (released August 2010) or PTF '5770SS1
V7R1M0 SI39820' , PTF '5770SS1 V7R1M0 SI39821'.
All of the source tables for a given Replicate task need to be journaled to the same
journal. The name of the journal and the library in which it is located must be
specified in the endpoint settings. During the task, Replicate polls this journal for
changes to the source tables.
When you start journaling the source tables, the Record images parameter can be
set to *BOTH (for capturing before and after images) or *AFTER.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 237
Note If you need to run several Replicate tasks (that replicate data from IBM
DB2 for iSeries), it is more efficient (though not essential) to create a separate
journal for each task. As only one journal can be specified per endpoint, you
would also need to define a separate endpoint for each task.
Required Permissions
The following permissions must be granted to the user specified in the General tab of the
IBM DB2 for iSeries endpoint settings:
You must also set the following Authorities and Locks for the IBM DB2 for
iSeries database:
*OBJEXIST is also required for the journal authority if any of the following are
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 238
true:
Limitations
The following limitations apply when using the IBM for DB2 iSeries endpoint as a source
in a Replicate task:
Multi-member (Partitioned) tables are not supported
When the Use table and schema system names option is enabled (in the
Advanced tab), only table/schema names that contain uppercase letters, digits,
and underscores (_) are supported.
For more information on this option, see Replicating System Names.
Usually, Replicate only applies source database changes to the target if the
transactions with the changes were committed before the Full Load started.
However, due to a known issue with the DB2 iSeries database, records that were
deleted from the source as part of an uncommitted transaction will not be
replicated to the target during Full Load.
DELETE operations in auto-commit mode will be ignored for tables journaled with
*AFTER images. In such cases, DELETE operations will only be captured if one of
the Add RRN column options is enabled and is set as the only Primary Key.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 239
IBM DB2 for iSeries Database Source Data Types
The following table shows the IBM DB2 for iSeries database source data types that are
supported when using Attunity Replicate and the default mapping from Attunity
Replicate data types.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.14 |
IBM DB2 for iSeries database Source Data Types with Mapping to
Attunity Replicate Data Types
IBM DB2 for iSeries Source Data Attunity Replicate Data Types
Types
INTEGER INT4
SMALLINT INT2
BIGINT INT8
FLOAT REAL8
DOUBLE REAL8
REAL REAL4
STRING
STRING
STRING
STRING
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 240
Table 8.14 | IBM DB2 for iSeries database Source Data Types with Mapping to
Attunity Replicate Data Types (Cont.)
IBM DB2 for iSeries Source Data Attunity Replicate Data Types
Types
BLOB BLOB
Maximum size: 2 GB
CLOB CLOB
Maximum size: 2 GB
DBCLOB CLOB
DATALINK STRING
Note You can also use IBM DB2 for iSeries File Channel files as a source. For
more information, see Using the Attunity Replicate File Channel.
1. In the Attunity Replicate console, click Add database to open the Add
Endpoints dialog box.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the IBM DB2
for iSeries database. This is optional.
4. Select SOURCE as the database role.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 241
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
Use ODBC DSN - If you choose this option, specify the IBM DB2 for
iSeries database ODBC DSN.
Use these connection properties - If you choose this option, enter the
IBM DB2 for iSeries Server (hostname or IP address).
7. Enter the IBM DB2 for iSeries authentication information (User Name, Password) for
the authorized user for this IBM DB2 for iSeries database. If you do not know this
information, see your IBM DB2 for iSeries database administrator (DBA).
Important: Make sure that the specified user has the required access privileges.
8. In the Journal Name field, enter the name of the journal containing the
source tables.
See also: Change Processing prerequisites.
9. In the Journal Library field, enter the name of the library where the journal
is located.
See also: Change Processing prerequisites.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 242
fact, it uses CCSID 1148 (ENCDIC International with EURO). In this case, you can
tell Replicate that the source definition CCSID 500 should be treated as CCSID
1148 (specifically, the character set named IBM-1148).
Note that when the source table definition specifies CCISD 65535 (meaning character set
is unknown), you must specify what character set should be assumed when reading data
from that table or column.
Note If there is a conflict between the character set mapping for a specific column
and the character set mapping defined in the endpoint settings, the column-level
character set mapping takes precedence.
For more information on overriding character set mapping at column level, see
Using the Transform Tab.
To do this:
1. In the Override CCSID to Character Set Mapping section, click the New
2. Enter the CCSID in the CCSID column and the code page in the Character
set column.
The Character set (code page) must be specified in the proper format (e.g.
ibm-1148_P100-1997). For a list of valid formats, see http://demo.icu-
project.org/icu-bin/convexp.
3. Repeat to map additional CCSID values.
Note If you edit an existing UCM file, you must also change the values of the
<code_set_name> and <icu:alias> properties. If the file does not contain an
<icu:alias> property, then you only need to change the value of the
<code_set_ name> property.
2. Create a CNV file for the UCM file by running the following command:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 243
<product_dir>\bin\makeconv.exe -v <file_name>.ucm
Example:
"c:\Program Files\Attunity\Replicate\bin\makeconv.exe" -v 1047_EX.ucm
This will create a CNV file with the same name as the UCM file (e.g. 1047_EX.cnv).
2. Mark the new RRN column on the target as the table's sole Primary Key
(using a transformation).
Note that adding an RRN column without marking it as a Primary Key will result
in the table being suspended if a DELETE operation is performed.
When you select one of the "Add RRN columns" options, both the Change Tables and
the target tables will have an extra column, ATTREP_RRN of type INTEGER, which
contains a unique value for each row. This column contains the RRN that corresponds
to each source table row.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 244
Warning: The IBM DB2 for iSeries RRN values represent the exact position of the row in
the file structure that holds the table data. When a table is reorganized, the table is
rebuilt resulting in new RRNs being allocated to each row. Consequently, reorganization of
tables where the RRN is being used as a target key should be avoided whenever possible.
If such reorganization is unavoidable, you should immediately reload the reorganized
table to prevent unpredictable behavior when changes are applied to the target (e.g.
failed DELETEs, duplicate INSERTs, and so on).
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 245
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 246
Using IBM DB2 for LUW as a Source
This section describes how to set up and use an IBM DB2 for LUW database as the
source database in a replication task.
In this section:
▶ Prerequisites
▶ Client Prerequisites
▶ IBM DB2 for LUW Server Prerequisites
▶ Replicating 4-byte UTF8 Emojis
▶ Limitations
▶ IBM DB2 for LUW Database Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you begin to work with an IBM DB2 for LUW database as a source in Attunity
Replicate, make sure that the prerequisites described in this section have been met.
In this section:
Client Prerequisites
IBM DB2 for LUW Server Prerequisites
Client Prerequisites
The IBM DB2 for LUW client prerequisites are determined by the platform on which
Attunity Replicate Server is installed (Windows or Linux).
The IBM Data Server Client version 10.5 must be installed on the Attunity Replicate
Server machine.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 247
Attunity Replicate Server On Linux
The following steps need to be performed on the Attunity Replicate Server machine:
1. Install v10.5fp9_linuxx64_server_t.tar.gz; choose to install “CLIENT”.
2. If the Attunity Replicate Server machine does not have a DB2 instance, create a
DB2 instance by running the following commands:
adduser <db2_instance_name>
/opt/ibm/db2/V10.5/instance/db2icrt <db2_instance_name>
3. Add the DB2 driver location to the Linux library path. To do this, add the following
line to the site_ arep_login.sh file:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:path/lib64
Example:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ibm/db2/V10.5/lib64
4. Create a new file named odbcinst.ini under /etc and add the following entry:
[IBM DB2 ODBC DRIVER]
Driver = /opt/ibm/db2/V10.5/lib64/libdb2o.so
fileusage=1
dontdlclose=1
6. Add the IBM DB2 for LUW endpoint as described in Setting General
Connection Properties and click Test Connection.
If you get the following error:
Cannot connect to DB2 LUW Server ODBC unknown error.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 248
settings>
SYSADM or DBADM
DATAACCESS
Change Data Capture: To enable data capture from IBM DB2 for LUW, the
source tables need to be created as follows:
CREATE / ALTER TABLE table-name …. DATA CAPTURE CHANGES [INCLUDE
LONGVAR COLUMNS];
Note When the Automatically alter tables to enable data capture option is
selected, Replicate performs the ALTER TABLE operation when the task starts to run.
If a new table that matches the table selection pattern is added (to the source
database) while the task is already running, Replicate will only capture changes from
it if it was created with the DATA CAPTURE CHANGE attribute described above.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 249
Limitations
When using IBM DB2 for LUW as a database in a Replicate task, the following
limitations currently apply:
Clustered database is not supported
Note Users can define a separate IBM DB2 for LUW database for each of
the endpoints in the cluster.
When truncating a table with multiple partitions, the number of DDL events
displayed in the Replicate console will be equal to the number of partitions.
This is because IBM DB2 for LUW records a separate DDL for each partition.
The following DDLs on partitioned tables are not supported:
ALTER TABLE ADD PARTITION
ALTER TABLE DETACH PARTITION
ALTER TABLE ATTACH PARTITION
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 250
IBM DB2 for LUW Database Source Data Types
The following table shows the IBM DB2 for LUW database source data types that are
supported when using Attunity Replicate and the default mapping from Attunity
Replicate data types.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.15 |
IBM DB2 for LUW database Source Data Types with Mapping to
Attunity Replicate Data Types
IBM DB2 for LUW Source Data Types Attunity Replicate Data Types
INTEGER INT4
SMALLINT INT2
BIGINT INT8
FLOAT REAL8
DOUBLE REAL8
REAL REAL4
REAL8
STRING
WSTRING
WSTRING
n<=255
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 251
Table 8.15 | IBM DB2 for LUW database Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
IBM DB2 for LUW Source Data Types Attunity Replicate Data Types
VARCHAR (n) STRING
n<=32k
n<=32k
DATE DATE
TIME TIME
TIMESTAMP DATETIME
BLOB BLOB
CLOB CLOB
Maximum size: 2 GB
DBCLOB CLOB
XML CLOB
Note You can also use IBM DB2 for LUW File Channel files as a source. For
more information, see Using the Attunity Replicate File Channel.
1. In the Attunity Replicate console, click Add database to open the Add
Endpoints dialog box.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 252
3. In the Description field, type a description that helps to identify the IBM DB2 for
LUW database. This is optional.
4. Select SOURCE as the database role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
Use database alias - If you choose this option, specify the IBM DB2 for
LUW database alias.
Use these connection properties - If you choose this option, enter the IBM
DB2 for LUW Server (hostname or IP address), Port and Database name in
the designated fields.
7. Enter the IBM DB2 for LUW authentication information (User Name, Password)
for the authorized user for this IBM DB2 for LUW database. If you do not know this
information, see your IBM DB2 for LUW database administrator (DBA).
Important: Make sure that the specified user has the required access privileges.
Maximum Buffer Size for Read (KB): Specify the maximum number of bytes
to read each time the log is accessed during Change Processing. If you encounter
performance issues, adjusting this number may help.
Change Data Capture: To enable data capture from IBM DB2 for LUW, the
source tables need to be created as follows:
CREATE / ALTER TABLE table-name …. DATA CAPTURE CHANGES [INCLUDE
LONGVAR COLUMNS];
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 253
You can either configure Replicate to perform this operation by selecting
Automatically alter tables to enable data capture or you can do this
manually by selecting Let DBA set up data capture.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
In order to specify the current LSN, you first need to retrieve it by issuing the
following command on the IBM DB2 for LUW server:
db2pd -logs -db <your_database>
For example:
database Partition 0 - database TEST - Active - Up 3 days 00:00:30
Logs:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 254
Pages Written 612
Log Chain ID 0
Current LSN 0x000000000452CABB
From this header, you can see that the current LSN is 0x000000000452CABB.
If you are unable to retrieve the current LSN, you can instruct Replicate to scan the log
by specifying CurrentLSN=scan (as described in Internal Parameters) instead of the
current LSN.
Note When CurrentLSN=scan, Replicate will search the log from the start until it
finds the current LSN. This may take some time depending on the size of the log.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 255
Using IBM DB2 for z/OS as a Source
This section describes how to set up and use an IBM DB2 for z/OS database as the
source endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Install the R4Z Product on z/OS
▶ ODBC Requirements
▶ Required Permissions
▶ Change Data Capture Requirements
▶ Limitations
▶ Handling Actions Resulting in Subtype 83
▶ Controlling the CDC Process
▶ Control Program Invocation Syntax
▶ Control Program Completion Codes
▶ Sample Jobs (in the INSTALL library)
▶ Enabling the CDC Process (auto-activation)
▶ Establishing R4Z CDC Services
▶ IBM DB2 for z/OS Database Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Overriding CCSID to Character Set Mapping
▶ Change Data Capture Properties
▶ Setting Internal Parameters
▶ R4Z Configuration Dependency on Host
▶ Settings Summary
▶ Sample XMIT Files “Receive” Job
Prerequisites
The following topic lists the prerequisites for working with the Attunity Replicate IBM
DB2 for z/OS endpoint.
In this topic:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 256
Install the R4Z Product on z/OS
ODBC Prerequisites
Required Permissions
Change Data Capture Prerequisites
ODBC Requirements
The Attunity Replicate IBM DB2 for z/OS source endpoint relies on the IBM Data Server
Driver for ODBC for access to data, Change Data and metadata. This section describes
the client side and server side ODBC prerequisites.
Install "IBM Data Server Driver for ODBC and CLI" for DB2 11.1 for z/OS on the
Attunity Replicate Server machine.
Note The "enableWLB" driver option is not supported. To force the connection to
automatically switch to another LPAR in a parallel sysplex (i.e. “failover”), set
“enableACR” to “true” and “enableWLB” to “false” in the driver configuration.
Bind the plan to be used for ODBC, as specified in the PLANNAME= value in the
ODBC initialization file. The default name is DSNACLI. The BIND job can be found in
member DSNTIJCL, which is in the SDSNSAMP library of the source DB2 installation.
Use the DB2CLI bind command to bind the ODBC-supplied packages to your intended
source z/OS DB2 subsystem. This action is described in Configuring your developer and
runtime environment on the IBM website. For information about the DB2CLI utility,
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 257
including an example for the bind utility, see db2cli - DB2 interactive CLI command on
the IBM website.
Required Permissions
To enable Replicate to extract data from the source tables (Full Load and Change Data
Capture), the user specified in the IBM DB2 for z/OS endpoint settings must be granted
the following permissions:
EXECUTE on the IFI reading the UDTF (only required for Change Data Capture)
SELECT on the source tables and on the DB2 catalog tables
MONITOR2 to be able to start IFI sessions (only required for Change Data Capture)
For additional details about these permissions, see Change Data Capture Requirements.
Limitations
The following limitations apply when using the IBM DB2 for z/OS endpoint in a
Replicate task:
During a task with Full Load and Apply Changes enabled, duplicate keys may be
replicated to the target. This is because records that were updated and cached
during the Full Load stage of the task may have the same timestamp as the original
records that were inserted during Full Load. Note that this usually only happens
when the time difference between the inserted record and the updated (cached)
record is very small (milliseconds).
The "enableWLB" driver option is not supported. To force the connection to
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 258
automatically switch to another LPAR in a parallel sysplex (i.e. “failover”), set
“enableACR” to “true” and “enableWLB” to “false” in the driver configuration.
DDL limitations:
Drop column: Columns cannot be dropped from tables with the DATA
CAPTURE CHANGES attribute.
Change column data type: To capture changes to column data types:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 259
Note It's possible that LOAD RESUME NO, without REPLACE, may also create the
subtype 83 log record. In such a case, the warning message shown above would
indicate an unknown variation number, with (UNIDENTIFIED) as the description.
All other actions would be the same as described above.
It is important to note that data loaded by this utility variation will not be captured.
Consequently, this utility variation should be avoided.
This program can also be used for managing the state of CDC processing, as well as
the level of traffic on the z/OS side.
From Replicate 6.1, the CDC services (known as R4Z environment in previous versions), do
not need to be “activated” prior to invocation of the CDC reader UDTF. The services are “auto-
activated” by the UDTF (when inactive). This means that the R4Z control program is no longer
responsible of activating the service. It is, however, required for the following purposes:
checking CDC and reporting the CDC service’s status, pausing CDC processing, resuming a
paused CDC, and terminating the CDC process, either normally or forcefully.
The command parameters available when running the R4ZCTL program are:
Note If you run the command without any parameters, it will return the current
status of the CDC service.
As the R4ZCTL programs requires APF-authorization, all libraries in the STEPLIB must
be APF- authorized.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 260
When running R4ZCTL with no parameter, its completion-code is set based on whether
the environment was initialized (CC=0) or not initialized (CC=1).
Where:
SERVICE=CDC-service-qualifier designates the logical scope of Replicate activity,
upon which the control program is to act.
For more information on CDC services, see Change Data Capture Requirements.
PAUSE_TASK(*|ALL|task-qualifier)
Suspends CDC retrieval for entire CDC service. Replication tasks suspended for
more than a certain time are stopped will attempt recovery multiple times. No new
tasks will be served.
RESUME_TASK(*|ALL|task-qualifier)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 261
RESUME_CDC
Resumes CDC retrieval for all instances matching the session-limits qualifier.
DUMP_TASK(*|ALL|task-qualifier)
Frees all the R4Z resident memory structures, terminates all active instances
and deletes associated resources. From this point on, all CDC requests will
return the inactive status until CDC service is auto-activated by an incoming
call to the CDC reader UDF.
A 4-digit identifier being assigned upon initiation of the task. This identifier is aimed
mainly for controlling and tracking CDC traffic of a specific task (defined in the
client’s endpoint definitions). It is being used to form the CORRID (Correlation-ID)
of the DB2 thread serving CDC – it occupies bytes 9-12 there.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 262
4 – Warning
8 – Error
1. A special UDTF is created to serve the CDC service. The created UDTF has its name
suffixed with a double-underscore (“__”) followed by the CDC-service-qualifier,
forming a name as such: <schema>.R4Z_UDTF__<CDC-service-qualifier>.
2. A WLM APPLENV is created to execute invocations of the above UDTF. The created
APPLENV may also have its name suffixed with the CDC service qualifier, to ensure
uniqueness of the APPLENV name – which is recommended; however, multiple
UDTFs may use a single APPLENV.
3. A JCL procedure is created to “host” the WLM APPLENV executions. This JCL procedure
differs from a usual WLM STC procedures in that a special DDNAME, R4ZCNFG, must be
specified, its DSN referring to the CONFIG library of the R4Z product. In this library,
there MUST be an existing member named “CDCS”, specifying the default CDC service
configurations. There MAY also exist a member named “CDCS<CDC-service-
qualifier>” with the configuration values you want to apply for this CDC service.
4. The content of each of the CDCS* members in the CONFIG library is a list of
assignment statements, one per card, in the format “keyword=value”. The
statement may follow spaces, and are space-terminated; no spaces are allowed
with the statement. Cards beginning with a hyphen (‘-‘) are treated as comment
cards; and the content following the terminating space is also treated as comment.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 263
The following are the permitted configuration keywords, the permitted minimum
and maximum values, and the default values:
You need to set variable DEFWLMAE (DEFine WLM Applic. Env.) in DFSYMLST
member to either PER (first option) or ONE (second option). Once set, you can
proceed with the installation:
Notes
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 264
If you select ONE for DEFWLMAE, jobs DO2* and DO3SRVTF should be run
one time ; if you select PER , these jobs should be run for each CDC service,
after the JCL SET variable CDCSRV is set to the CDC service’s qualifier.
Job DO3SRVDF creates a CONFIG library, and places a member named
“CDCS”, which serves as the default configuration file. After running job
DO3SRVTF the library will contain the configuration of all CDC services,
making it possible to edit them and change the configurations of a specific
CDC service.
Table 8.16 |
IBM DB2 for z/OS Database Source Data Types with Mapping to
Attunity Replicate Data Types
INTEGER INT4
SMALLINT INT2
BIGINT INT8
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 265
Table 8.16 | IBM DB2 for z/OS Database Source Data Types with Mapping to Attunity
Replicate Data Types (Cont.)
DOUBLE REAL8
REAL REAL4
REAL8
STRING
WSTRING
WSTRING
n<=255
n<=32k
n<=32k
DATE DATE
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 266
Table 8.16 | IBM DB2 for z/OS Database Source Data Types with Mapping to Attunity
Replicate Data Types (Cont.)
TIME TIME
BLOB BLOB
CLOB CLOB
Maximum size: 2 GB
DBCLOB CLOB
XML CLOB
BINARY BYTES
VARBINARY BYTES
ROWID IGNORED
Note You can also use IBM DB2 for z/OS File Channel files as a source. For
more information, see Using the Attunity Replicate File Channel.
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 267
3. In the Description field, type a description that helps to identify the Microsoft
SQL Server database. This is optional.
4. Select SOURCE as the database role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
Use ODBC DSN - If you choose this option, specify the IBM DB2 for z/OS
ODBC DSN.
Use these connection properties - If you choose this option, enter the
IBM DB2 for z/OS Server (hostname or IP address), Port and Location in
the designated fields.
If the Server is a parallel SysPlex and data sharing members reside on multiple
LPARs, the host address may specify a DVIPA - a Dynamic Virtual IP Address
(to utilize the system redundancy and load-balancing in Replicate processing).
In this case, the port numbers must be identical for all members.
The Location should be the DB2 location name defined during the installation.
This should be a relational database management system – under z/OS, either
a subsystem or a group connection. This is the logical name which serves
applications in order to designate resources managed by this system, either
using SQL CONNECT instruction, or placing it as a qualifier of a table (preceding
the schema name).
To see the location name, use “-DIS DDF” DB2 command (option 7 under
the DB2I panel in ISPF), or look in message DSNL004I in the job log of the
<ssid>MSTR address space.
7. Enter the User name and Password for an authorized user of the specified IBM
DB2 for z/OS database. For a list of the permissions that need to be granted to this
user, see Required Permissions.
If you do not know this information, consult with your IBM DB2 for z/OS
database administrator (DBA).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 268
Notes
Make sure that the specified user has the required access privileges
8. Provider: Leave the default unless it was changed during the driver installation.
Note that this should be the same as the name specified in the ODBC Data Source
Administrator.
Note If there is a conflict between the character set mapping for a specific column
and the character set mapping defined in the endpoint settings, the column-level
character set mapping takes precedence.
For more information on overriding character set mapping at column level, see
Using the Transform Tab.
To do this:
1. In the Override CCSID to Character Set Mapping section, click the New
2. Enter the CCSID in the CCSID column and the code page in the Character set
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 269
column.
The Character set (code page) must be specified in the proper format (e.g.
ibm-1148_P100-1997). For a list of valid formats, see http://demo.icu-
project.org/icu-bin/convexp.
3. Repeat to map additional CCSID values.
Perform the following procedure if your source endpoint tables are defined with
an incorrect CCSID and the correct definition is actually in a UCM file.
1. Create or download a mapping data file with the file extension .ucm.
Note If you edit an existing UCM file, you must also change the values of the
<code_set_name> and <icu:alias> properties. If the file does not contain an
<icu:alias> property, then you only need to change the value of the
<code_set_ name> property.
2. Create a CNV file for the UCM file by running the following command:
<product_dir>\bin\makeconv.exe -v <file_name>.ucm
Example:
"c:\Program Files\Attunity\Replicate\bin\makeconv.exe" -v 1047_EX.ucm
This will create a CNV file with the same name as the UCM file (e.g. 1047_EX.cnv).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 270
CDC reader UDTF name: The name of the Attunity-supplied User-Defined Table
Function, which is to access CDC data. Specify the two-part name resulting from
the values you have chosen for schema name, &R4ZSCNM and the function name,
&R4ZIFITF.
UDTF result set size (MB): Specify the maximum size to be accumulated by result
rows returned by the R4Z-supplied User-Defined Table Function, through CDC tasks
using this endpoint. A larger result set will cause less overhead time establishing the
IFI session, but will result in greater memory consumption by DB2. Specifically,
because the result set is a LOB, you may need to increase the LOBVALA limit in DB2
configuration (DSNZPARM), specifying the maximum LOB size allowed per user.
Assuming all CDC enabling tasks are specifying the same ODBC user-ID, to
accommodate for all tasks running concurrently, LOBVALA should be set to 2 ×
1024 × the following size:
SUM[(i=all endpoints) : resultset size in endpoint(i) × #_tasks using
endpoint(i)]
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 271
Setting Internal Parameters
Internal parameters are parameters that are not exposed in the UI. With the exception
of the parameters listed in Authorized Internal Parameters at the end of this section,
you should only use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
The parameter is added to the table below the search box with its default value.
5. To reset a parameter value to its default, click the "Restore default value" icon at
the end of the row.
The following table provides a list of internal parameters that can be used as required.
9 – full DEBUG
Example:
cursorhold=1;patch2=15
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 272
R4Z Configuration Dependency on Host
This table in this section provides a list of R4Z configuration values, which can be set during
installation or as part of the CDC service setup, or as part of the endpoint definition.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 273
Sample XMIT Files “Receive” Job
The following is a sample job for receiving the LOAD and INSTALL libraries.
//*******************************************************************
//* Sample JCL for receiving *
//* Attunity ReplicateIBM DB2 for z/OS endpoint installation kit *
//* 1. Add a jobcard *
//* 2. Replace all <xmit-HLQ> by the High Level Qualifier used *
//* for receive files *
//* 3. Replace all <r4z-vn-hlq> by the High Level Qualifier chosen *
//* for the installation files *
//*******************************************************************
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RECEIVE INDSN('<xmit-HLQ>.LOAD.XMIT')
NODISPLAY DATASET('<r4z-vn-hlq>.LOAD') /*
//RCVINSTL EXEC
PGM=IKJEFT01,DYNAMNBR=20 //SYSPRINT DD
SYSOUT=* //SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RECEIVE INDSN('<xmit-HLQ>.INSTALL.XMIT')
NODISPLAY DATASET('<r4z-vnd-hlq>.INSTALL') /*
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 274
Using IBM Informix as a Source
This section describes how to set up and use an IBM Informix database as the
source database in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Required Permissions
▶ IBM Informix Database Source Data Types
▶ Unsupported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you begin to work with an IBM Informix database as a source in Attunity
Replicate, make sure that the following prerequisites have been met:
Attunity Replicate machine:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 275
Note This requires DBA privileges (User 'IBM Informix' or another DBA
user).
Make sure that the database to be replicated was created with either the
WITH LOG or the WITH BUFFERED LOG property.
Limitations
When using IBM Informix as a database in a Replicate task, the following
limitations currently apply:
CDC does not capture DDL changes. Due to an IBM Informix limitation, IBM Informix
does not allow DDLs to be executed on tables with Full Row Logging enabled.
To learn how to capture DDL changes during CDC, see Automatically enable full
row logging.
Col1 INTEGER
Col2 SET
Col3 INTEGER
Important: Choosing this option will resume the task from the current time.
If a task with an IBM Informix source is stopped before any changes have been
made and then resumed, any changes that were made between the time that the
task was stopped and the time that it was resumed will be lost.
Due to a known issue with the IBM Informix CDC API, Replicate does not support
replication of tables whose names contain spaces or non-English letters.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 276
Due to a known issue with the IBM Informix transaction consistency mechanism, all
transactions should be manually stopped before starting a Full Load task.
Transactions started during the Full Load operation will be cached.
Required Permissions
In order to access the specified database, the user specified in the General tab must be a
member of the "IBM Informix" group (which has DBA privileges) on the database server.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.18 |
IBM Informix database Source Data Types with Mapping to
Attunity Replicate Data Types
SMALLINT INT2
INT8 INT8
SERIAL INT4
SERIAL8 INT8
FLOAT REAL8
DOUBLE REAL8
REAL REAL4
SMALLFLOAT REAL4
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 277
Table 8.18 | IBM Informix database Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
INTERVAL STRING
BLOB BLOB
BYTE BLOB
CLOB CLOB
LIST CLOB
MULTISET CLOB
SET CLOB
TEXT CLOB
BOOLEAN BOOLEAN
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 278
Note You can also use IBM Informix files as a source. For more information, see Using
the Attunity Replicate File Channel.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server field, enter the name of the IBM Informix server. On Windows, this must
correspond to one of the hosts defined using the setnet32.exe tool. On Linux, this
must correspond to a valid dbservername entry in the
$INFORMIXDIR/etc/sqlhosts file on the computer running the application.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 279
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
7. Enter the IBM Informix authentication information (User Name, Password) for
the authorized user for this IBM Informix database. If you do not know this
information, see your IBM Informix database administrator (DBA).
Important: Make sure that the IBM Informix user entered in the IBM Informix
Authentication section has the correct access privileges. For information on how
to provide the required privileges, see Required Permissions.
8. In the Database name field, enter the IBM Informix database name.
Automatically enable full row logging: Full Row Logging is required for CDC.
Select this option to automatically enable Full Row Logging for the tables to be
replicated. To automatically enable Full Row Logging, the user specified in the
General tab must have administrative privileges on the IBM Informix database.
Note DDL events are not captured during CDC. To perform DDL operations
on source tables in a Replicate CDC task:
Stop the Replicate task.
Disable Full Row Logging for the relevant tables as in the following example:
execute function syscdcv1:IBM Informix.cdc_set_fullrowlogging
('sysuser:IBM Informix.employees_table', 0)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 280
Max bytes per read: Specify the maximum number of bytes to read each time the log
is accessed. If you encounter performance issues, adjusting this number may help.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 281
Using IBM Netezza as a Source
This section describes how to set up and use an IBM Netezza database as a source
endpoint in a replication task.
In this section:
▶ Prerequisites
▶ IBM Netezza Data Types
▶ Setting General Connection Properties
▶ Setting Internal Parameters
▶ Settings Summary
▶ Setting Change Processing Parameters
▶ Prerequisites
▶ Limitations
▶ Configuring Change Processing Settings
Prerequisites
Prior to installing Attunity Replicate, make sure that the IBM Netezza ODBC 64-bit client
is installed on the Attunity Replicate machine.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.19 |
Supported IBM Netezza Data Types with Mapping to Attunity Replicate
Data Types
SMALLINT INT2
BYTEINT INT1
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 282
Table 8.19 | Supported IBM Netezza Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
DOUBLE REAL8
REAL REAL4
NUMERIC NUMERIC(25,0)
TIME TIME
TIMESTAMP DATETIME
DATE DATE
VCHAR STRING(25)
CHAR STRING(25)
BOOL BOOLEAN
You can do this step before any of the other steps if you want, however before you
can continue with the next step in this process, you must select the database role.
5. Select IBM Netezza as the database Type.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select the
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 283
relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server field, enter the name of the IBM Netezza server.
7. Optionally, change the default Port (5480).
8. Enter the IBM Netezza authentication information (User Name, Password) for
the authorized user for this IBM Netezza database. If you do not know this
information, see your IBM Netezza database Administrator (DBA).
9. In the Database name field, enter the name of the IBM Netezza database.
10. Click Test Connection to verify that the settings you entered are correct.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 284
Setting Internal Parameters
In the Advanced tab, you can set internal parameters.
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Prerequisites
Before you can define the settings in the Change Processing tab, you need to ensure
that at least one special "Context" column exists in your source database tables. Context
column(s) are basically columns in a table that enable Replicate to determine whether
the data has changed. You can add Context columns specifically for the purpose of
change processing (either using a script or manually) or you can use existing columns
that contain suitable "Context" data.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 285
Note You can create and reference any number of Context columns in a table as
long as the Context column names are the same for all source tables. Additionally,
each value in the Context column(s) must be unique.
In the example below, the Context column cf has been added to the table. The cf column
contains TIMESTAMPs that enable Replicate to determine whether a change occurred (by
comparing the current TIMESTAMP with the TIMESTAMP stored in its repository).
By default, all changes are assumed to be INSERTs. If UPDATE and DELETE operations
are also performed on the source tables, you can write an UPDATE and/or DELETE
expression (described below) that will enable Replicate to identify the operation type.
Limitations
The following limitations apply when Change Processing is enabled:
The "Start from timestamp" run option is not supported. For more information,
see Using Advanced Run Options.
If one of the Context columns is part of the Primary Key or Unique Index, then
UPDATE and DELETE operations are not supported.
Context columns cannot be LOB columns
DDLs are not supported
When inserting a record and then updating the same record, the task error handling
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 286
settings should be set as follows:
2. In the Columns field, specify the names of the Context columns. The column
names are case-sensitive and must be separated by commas.
Example:
context1,context2
5. Enter expressions that Replicate will use to identify UPDATE and DELETE operations. If
you do not enter any expressions or if no match is found for an expression, any row
whose context is higher (if the sorting order is Ascending) or lower (if the sorting order
is Descending) than the previous context value will be considered an INSERT.
Note Expressions must be written in the native syntax of the source database.
All examples in this section are written using PostgresSQL syntax.
Tip: Selecting the UPDATE the existing target record option in the Apply
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 287
Conflicts tab, eliminates the need to provide an UPDATE expression.
In this section:
▶ Prerequisites
▶ Preparing the Microsoft Azure SQL Managed Instance Database for Replication
▶ Limitations
▶ Non-Supported Microsoft Azure SQL Managed Instance Security Features
▶ Required Permissions
▶ Supported Compression Methods
▶ Microsoft Azure SQL Managed Instance Source Data Types
▶ Non-Supported Data Types
▶ Homogeneous Replication
▶ Data Type Exceptions
▶ Column and Table Collation
▶ Setting General Connection Properties
▶ Artifacts Created by Replicate in the Database
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 288
Prerequisites
Make sure that the following prerequisites have been met:
Client prerequisites (for source and target endpoints):
For all versions of Microsoft Azure SQL Managed Instance, Microsoft SQL Server
Native Client 11.0 must be installed on the Attunity Replicate Server machine.
You can either work with the Microsoft ODBC Driver or the Simba Microsoft SQL
Server ODBC Driver. Instructions for both are provided below.
First, install Microsoft ODBC Driver 13.1.1.0 for Linux on the Attunity Replicate
Server machine.
Then, on the Attunity Replicate Server machine, open a Unix shell and perform
the following steps:
Note The procedure below assumes that you have installed a single default
instance of Replicate on Linux (areplicate). If you have installed multiple
instances, replace areplicate with the name of the instance running the task with
a Microsoft Azure SQL Managed Instance source. If several instances are running
such as task, the procedure needs to be repeated for each instance.
to: cd <product_dir>/bin
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 289
cat site_arep_login.sh
ln -s existing_library_name libmsodbcsql-13.1.so.0.0
A Microsoft Azure SQL Managed Instance account with the specific access privileges
is required. See Source Permissions for more information.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 290
Microsoft Azure SQL Managed Instance as a source must be configured for a full
backup to work with Attunity Replicate. For more information, see Preparing
Microsoft Azure SQL Managed Instance Backup and Recovery.
When decompressing a compressed backup transaction log, Replicate writes the
decompressed log to the same location as the compressed log. Consequently, the
location of the compressed backup transaction log must have enough space to
contain the decompressed log as well.
The following section explains how to set up a Microsoft Azure SQL Managed Instance
for CDC.
Note When creating the Microsoft Azure SQL Managed Instance, full backup must
be enabled; otherwise, Replicate will not be able to capture changes.
1. Log in as the master user in the DB instance and set up the database for MS-CDC
as follows:
exec source_db_name.dbo.sp_cdc_enable_db
2. To enable MS-CDC, run the following for each of the intended source tables:
exec sys.sp_cdc_enable_table
@source_schema = N'schema_name',
@source_name = N'table_name',
@role_name = NULL
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 291
GO
where schema_name and table_name are the names of the source schema and
the source table respectively.
3. Set the retention period for the changes to be available on the source by running
the following command:
EXEC sys.sp_cdc_change_job @job_type = 'capture' ,@pollinginterval
= 86400
Limitations:
CDC is supported with the transaction log only, with no option to work with backup.
Events may be lost if they are moved to backup or truncated.
Attunity Replicate consumes changes captured from the database transaction log
(TLOG). The TLOG is maintained by Microsoft Azure SQL Managed Instance for
recovery purposes. All changes made to a database are written to the TLOG. The
following happens when recovery is required:
A backup copy of the database is made.
Logged events are taken and used in a rolling-forward process where the
recorded changes are replayed against that copy.
To prepare for backup and recovery you must make sure that the Microsoft Azure SQL
Managed Instance Recovery Model is set up. You select the Recovery Model in the
Microsoft Azure SQL Managed Instance Management Studio. This should be carried out
by a Microsoft Azure SQL Managed Instance system administrator.
The TLOG data is truncated as soon as it is no longer needed therefore the TLOG is
not persistent. However, Attunity Replicate guaranteed delivery requires persistency
in the changed data. To ensure persistency:
A full database backup must be carried out before beginning to replicate data.
The Recovery Model must be set to Bulk logged or Full.
In the database properties Options tab, set the Recovery Model to Bulk logged or Full.
In these modes, the transaction Log is more durable.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 292
Note After setting the Recovery Model, it is strongly recommended not to change
it; doing so may result in loss of data.
Set the following for the Microsoft Azure SQL Managed Instance database(s) that you
are using as a source:
From the Object Explorer in the Microsoft Azure SQL Managed Instance
Management Studio, right click the database and select Properties. In the
Options tab, set the Recovery model to Bulk logged or Full. In this mode, the
transaction Log is more durable and truncation occurs less frequently.
Ensure that there is a full database backup for each Microsoft Azure SQL
Managed Instance database that you are using as a source.
When creating a connection string, it is possible to use any parameter supported
by Microsoft Azure SQL Managed Instance. The Microsoft Azure SQL Managed
Instance system administrator must ensure that the Microsoft Azure SQL
Managed Instance instance is configured correctly so that the proper
authentication credentials are accepted.
You can configure the Attunity Replicate Microsoft Azure SQL Managed Instance endpoint
to log in to Microsoft Azure SQL Managed Instance (on Windows) using Windows
authentication.
If you choose this option, you also need to make sure that:
The Microsoft Azure SQL Managed Instance instance is set up to allow Windows log on.
The Attunity Replicate user is specified as the "Log on as" user for the
"Attunity Replicate Server" service account.
-OR-
Microsoft Azure SQL Managed Instance is configured to allow login for the
Attunity Replicate Server service account.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 293
Limitations
When using a Microsoft Azure SQL Managed Instance source endpoint in a Replicate
task, the following imitations apply:
A Secondary SQL Server database is not supported as a source database.
If you are using a Microsoft Azure SQL Managed Instance source database in a
replication task, the Microsoft Azure SQL Managed Instance Replication Publisher
definitions for the database that was used in the task are not removed when you
remove a task. A Microsoft Azure SQL Managed Instance system administrator
must delete these definitions from Microsoft Azure SQL Managed Instance.
Sparse tables are not supported.
Replicating data from indexed views is not supported.
Renaming tables using sp_rename is not supported (e.g.
sp_rename 'Sales.SalesRegion', 'SalesReg;)
Renaming columns using sp_rename is not supported (e.g.
sp_rename 'Sales.Sales.Region', 'RegID', 'COLUMN';)
TRUNCATE events will not be captured.
Changes to computed fields in a Microsoft Azure SQL Managed Instance source will
not be replicated.
Microsoft Azure SQL Managed Instance partition switching is not supported.
When using the WRITETEXT and UPDATETEXT utilities, Attunity Replicate does
not capture events applied on the source database.
The following DML pattern is not supported:
select <*> into <new_table> from <existing_table>
Due to a known issue with Microsoft SQL Server 2008/2008 R2, Attunity Replicate
does not support server level audits on Microsoft SQL Server 2008/2008 R2 as a
source database.
For example, running the following command:
USE [master]
GO
GO
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 294
Note This issue was resolved in Microsoft SQL Server 2012.
The following limitations apply when accessing the backup transaction logs:
Encrypted backups are not supported.
Backups stored at a URL or on Windows Azure are not supported.
Microsoft Azure SQL Managed Instance backup to multiple disks is not supported.
When inserting a value into SQL Server spatial data types (GEOGRAPHY and
GEOMETRY), one can either ignore the SRID (Spatial Reference System Identifier)
property - in which case the default SRID will be used (0 for GEOMETRY and 4326
for GEOGRAPHY) - or specify a different number. When replicating tables with
spatial data types, Attunity Replicate replaces the SRID that was inserted by user
with the default SRID.
Columnstore indexes are not supported.
Memory-optimized tables (using In-Memory OLTP) are not supported.
Temporal databases are not supported
Delayed durability is not supported
Table change tracking is not supported
Due to an ODBC limitation, no more than 16 columns can be part of a Primary Key.
UPDATEs to a Primary Key/Unique Index that affect multiple rows may cause
conflicts when applying the changes to the target. This may occur, for example, if
the UPDATEs are applied to the target as INSERT/DELETE operations rather than a
single UPDATE operation. In such a scenario (where UPDATEs to a Primary
Key/Unique Index affect multiple rows), working in Batch Optimized Apply Change
Processing mode is not supported as it may result in the table being ignored.
Working in Transactional Apply Change Processing mode however may result in
constraint violations. If this happens, you either need to reload the relevant table or
locate the problematic records in the attrep_apply_exceptions Control Table and
edit them manually in the target database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 295
Required Permissions
To use a Microsoft Azure SQL Managed Instance source in an Attunity Replicate task, the user
specified in the Microsoft Azure SQL Managed Instance endpoint connection settings must be
a member of both the db_owner database role and the sysAdmin fixed server role.
Table 8.20 |
Supported Microsoft Azure SQL Managed Instance Compression Methods
Row/Page Compression Vardecimal Vardecimal Vardecimal
(at Partition Level) Storage Format Storage Format Storage Format
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 296
Table 8.21 |
Microsoft Azure SQL Managed Instance Source Data Types with
Mapping to Attunity Replicate Data Types when the Target is not Microsoft Azure SQL
Managed Instance
BIT BOOLEAN
DECIMAL NUMERIC
INT INT4
SMALLINT INT2
TINYINT UINT1
REAL REAL4
FLOAT REAL8
DOUBLE REAL8
DATETIME DATETIME
SMALLDATETIME DATETIME
DATE DATE
DATETIMEOFFSET STRING
*CHAR STRING
*VARCHAR STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 297
Table 8.21 | Microsoft Azure SQL Managed Instance Source Data Types with Mapping to
Attunity Replicate Data Types when the Target is not Microsoft Azure SQL Managed
Instance (Cont.)
*TEXT
To use this data type with Attunity Replicate, you must enable
the use of CLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB
size option is enabled, LOB data types are supported only in
tables with a primary key.
*NCHAR WSTRING
*NTEXT
To use this data type with Attunity Replicate, you must enable
the use of NCLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB
size option is enabled, LOB data types are supported only in
tables with a primary key.
BINARY BYTES
VARBINARY BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 298
Table 8.21 | Microsoft Azure SQL Managed Instance Source Data Types with Mapping to
Attunity Replicate Data Types when the Target is not Microsoft Azure SQL Managed
Instance (Cont.)
IMAGE
To use this data type with Attunity Replicate, you must enable
the use of BLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB
size option is enabled, LOB data types are supported only in
tables with a primary key.
TIMESTAMP BYTES
UNIQUEIDENTIFIER STRING
HIERARCHYID HIERARCHYID -
When replicating to
Microsoft Azure SQL
Managed Instance.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 299
Table 8.21 | Microsoft Azure SQL Managed Instance Source Data Types with Mapping to
Attunity Replicate Data Types when the Target is not Microsoft Azure SQL Managed
Instance (Cont.)
To use this data type with Attunity Replicate, you must enable
the use of NCLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB
size option is enabled, LOB data types are supported only in
tables with a primary key.
GEOMETRY CLOB
GEOGRAPHY CLOB
Note User-defined data types are supported according to their base-type. For example a
user-defined data type based on DATETIME is handled as a DATETIME data type.
Homogeneous Replication
When replicating from a Microsoft Azure SQL Managed Instance source to a
supported Microsoft SQL Server-based target, most of the source and target data
types will be identical. The exceptions are listed in the table below.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 300
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through when
replicating from a Microsoft Azure SQL Managed Instance, see the Microsoft Azure SQL
Managed Instance to Attunity Replicate data types mapping table described earlier.
Note To prevent data truncation when replicating XML, Geometry and Geography
data types, it is strongly recommended to enable the Allow unlimited LOB size option
in the task settings.
Table 8.22 |
Data Type Exceptions in Homogeneous Replication
Microsoft Azure SQL Microsoft SQL Server Microsoft SQL Server
Managed Instance Source 2005 Target 2005-2016 Target
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 301
Table 8.22 | Data Type Exceptions in Homogeneous Replication (Cont.)
TIMESTAMP VARBINARY
Note To support collation replication, the DBA must ensure that the collations
defined for the source Microsoft Azure SQL Managed Instance database are the
same as those defined for the Microsoft SQL Server-based target.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 302
image
xml
6. Specify the Server name. This is the host name or IP address of the computer
with the Microsoft Azure SQL Managed Instance.
Notes
To override the default port, add the port to the server name, separated by
a comma. For example, if the server name is myserver.company.local and
the port is 3333, then the server name should be entered like this:
myserver.company.local,3333
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 303
Note When using Windows authentication, make sure that the user account
that is associated with the Attunity Replicate Server service has Network
read and write permissions. This must be configured by a Windows system
administrator.
If you select SQL Server authentication, type the Microsoft Azure SQL Managed
Instance authentication information (User name, Password) for the authorized user
for this Microsoft Azure SQL Managed Instance database. If you do not know this
information, see your Microsoft Azure SQL Managed Instance System Administrator.
To prevent illicit database activity by unauthorized third-parties, Replicate can be
configured to automatically replace the user-entered password with a strong
random password. For more information, see Configuring Replicate to
Automatically Replace the User-Entered Password.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the Microsoft Azure SQL Managed Instance user
has the correct access privileges. For information on how to provide the
required privileges, see Required Permissions.
8. Type the Database name or click Browse and select one from the list of available
databases. This is the name of the database from where you are replicating the data.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 304
In order to capture changes, Attunity Replicate creates various artifacts in the dbo
schema of the Microsoft Azure SQL Managed Instance when the task starts. You can
safely remove them when the task completes.
The following artifact is created to track open transactions:
#attrep_dbcc_open_trans_table
The following functions are created to verify that the LSN (Log Sequence Number) is valid:
attrep_fn_LsnSegmentToHexa
attrep_fn_NumericLsnToHexa
In this section:
▶ Supported Editions
▶ Prerequisites
▶ Limitations
▶ Non-Supported Microsoft SQL Server Security Features
▶ Working with Microsoft SQL Server AlwaysOn Availability Groups
▶ Prerequisites
▶ Configuring Replicate to Access the AlwaysOn Listener
▶ Limitations
▶ Accessing Backup Logs in AlwaysOn Availability Groups
▶ Required Permissions
▶ Setting up a Non-Sysadmin User
▶ Supported Compression Methods
▶ Microsoft SQL Server Source Data Types
▶ Non-Supported Data Types
▶ Homogeneous Replication
▶ Data Type Exceptions
▶ Column and Table Collation
▶ Preparing the Microsoft SQL Server Database for Replication
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 305
▶ Preparing Microsoft SQL Server Backup and Recovery
▶ Setting up Microsoft SQL Server for Replication
▶ Replicating Tables that do not have a Primary Key
▶ Defining Microsoft SQL Server Database Settings
▶ Working with Windows Authentication
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Artifacts Created by Replicate in the Database
Supported Editions
Attunity Replicate supports the following Microsoft SQL Server editions:
Enterprise Edition
Standard Edition
Workgroup Edition
Developer Edition
Prerequisites
Make sure that the following prerequisites have been met:
Client prerequisites (for source and target endpoints):
For all versions of Microsoft SQL Server, Microsoft SQL Server Native Client 11.0
must be installed on the Attunity Replicate Server machine.
Attunity Replicate for Linux:
First, install Microsoft ODBC Driver 13.1.1.0 for Linux on the Attunity Replicate
Server machine.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 306
Then, on the Attunity Replicate Server machine, open a Unix shell and perform
the following steps:
Note The procedure below assumes that you have installed a single default
instance of Replicate on Linux (areplicate). If you have installed multiple
instances, replace areplicate with the name of the instance running the task with
a Microsoft SQL Server source. If several instances are running such as task, the
procedure needs to be repeated for each instance.
to: cd <product_dir>/bin
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 307
To check which library version is currently installed
ln -s existing_library_name libmsodbcsql-13.1.so.0.0
A Microsoft SQL Server account with the specific access privileges is required.
See Source Permissions for more information.
Microsoft SQL Server as a source must be configured for a full backup to work with
Attunity Replicate. For more information, see Preparing Microsoft SQL Server
Backup and Recovery.
When decompressing a compressed backup transaction log, Replicate writes the
decompressed log to the same location as the compressed log. Consequently, the
location of the compressed backup transaction log must have enough space to
contain the decompressed log as well.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 308
Limitations
When using a Microsoft SQL Server source endpoint in a Replicate task, the
following imitations apply:
A Secondary SQL Server database is not supported as a source database.
If you are using a Microsoft SQL Server source database in a replication task, the
Microsoft SQL Server Replication Publisher definitions for the database that was
used in the task are not removed when you remove a task. A Microsoft SQL Server
system administrator must delete these definitions from Microsoft SQL Server.
Sparse tables are not supported.
Replicating data from indexed views is not supported.
Renaming tables using sp_rename is not supported (e.g.
sp_rename 'Sales.SalesRegion', 'SalesReg;)
Renaming columns using sp_rename is not supported (e.g.
sp_rename 'Sales.Sales.Region', 'RegID', 'COLUMN';)
TRUNCATE events will not be captured.
Changes to computed fields in a Microsoft SQL Server source will not be replicated.
Microsoft SQL Server partition switching is not supported.
When using the WRITETEXT and UPDATETEXT utilities, Attunity Replicate does
not capture events applied on the source database.
The following DML pattern is not supported:
select <*> into <new_table> from <existing_table>
Due to a known issue with Microsoft SQL Server 2008/2008 R2, Attunity Replicate
does not support server level audits on Microsoft SQL Server 2008/2008 R2 as a
source database.
For example, running the following command:
USE [master]
GO
GO
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 309
Note This issue was resolved in Microsoft SQL Server 2012.
The following limitations apply when accessing the backup transaction logs:
Encrypted backups are not supported.
Backups stored at a URL or on Windows Azure are not supported.
The following limitations apply when accessing the backup transaction logs at
file level:
The backup transaction logs must reside in a shared folder with the
appropriate permissions and access rights.
Active transaction logs are accessed through the Microsoft SQL Server API
(and not at file-level).
The Attunity Replicate and Microsoft SQL Server machines must reside in
the same domain.
Compressed backup transaction logs are not supported on Windows Server 2008.
Transparent Data Encryption (TDE) is not supported. Note that when accessing
the backup transaction logs using SQL Server’s native functionality (i.e. not
using file-level access), TDE encryption is supported.
Unix platforms are not supported.
Reading the backup logs from multiple stripes is not supported.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 310
Delayed durability is not supported
Table change tracking is not supported
Due to an ODBC limitation, no more than 16 columns can be part of a Primary Key.
UPDATEs to a Primary Key/Unique Index that affect multiple rows may cause
conflicts when applying the changes to the target. This may occur, for example, if
the UPDATEs are applied to the target as INSERT/DELETE operations rather than a
single UPDATE operation. In such a scenario (where UPDATEs to a Primary
Key/Unique Index affect multiple rows), working in Batch Optimized Apply Change
Processing mode is not supported as it may result in the table being ignored.
Working in Transactional Apply Change Processing mode however may result in
constraint violations. If this happens, you either need to reload the relevant table or
locate the problematic records in the attrep_apply_exceptions Control Table and
edit them manually in the target database.
Always Encrypted
Dynamic Data Masking
Row-Level Security
Prerequisites
Credentials used for connecting to individual replicas should be identical to those
used for connecting to the AlwaysOn Listener.
Replicas should be configured to allow connections.
= 'TCP://{replica_name}:{Port}'))
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 311
Configuring Replicate to Access the AlwaysOn Listener
When working with AlwaysOn Availability Groups, you need to specify the IP
address/host name and access credentials of the AlwaysOn Listener in the Microsoft
SQL Server source endpoint settings.
Limitations
The following limitations apply when working with AlwaysOn Availability Groups:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 312
Accessing Backup Logs in AlwaysOn Availability Groups
As opposed to active transaction logs which are synchronized across the AlwaysOn Availability
Group, backup transaction logs are different for each individual replica. Consequently, when
Replicate needs to access the backup transaction logs, it "asks" each Replica in turn if it has
the required logs, until the backup transaction logs are located. Note that if one of the
Replicas that Replicate is trying to reach is offline, Replicate will wait until that Replica comes
back online and then query it for the backup transaction logs.
Replicate needs to access the backup transaction logs on each Replica (or until it finds
the required logs) in the following scenarios:
Working in backup only mode.
For more information on this mode, see Read changes from backup only.
For more information on this option, see the Tables are already loaded option in
Using Advanced Run Options.
Due to latency i.e. if there is a high rate of events (changes) that Replicate is unable
to process using the active log only.
Required Permissions
To use a Microsoft SQL Server source in an Attunity Replicate task, the user specified in the
Microsoft SQL Server endpoint connection settings can be either of the following:
A member of both the db_owner database role and the sysAdmin fixed server role.
A member of the db_owner database role but not a member of sysAdmin fixed
server role. To set this up, you need to perform the procedure described in Setting
up a Non-Sysadmin User below.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 313
Limitations and Considerations
Note that a single source table can contain several data partitions.
-OR-
In a way that allows access to non-sysadmin users via the associated linked server.
set ansi_nulls
on go
set quoted_identifier
on go
if (object_id('[attrep].[split_partition_list]','TF')) is not
null drop function [attrep].[split_partition_list];
go
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 314
create function [attrep].[split_partition_list]
(
@plist varchar(8000), --A delimited list of partitions
@dlm nvarchar(1) --Delimiting character
)
returns @partitionsTable table --Table holding the BIGINT values of the
string fragments
(
pid bigint primary key
)
as
begin
declare @partition_id bigint;
declare @dlm_pos integer;
declare @dlm_len integer;
while (charindex(@dlm,@plist)>0)
begin
set @dlm_pos = charindex(@dlm,@plist);
set @partition_id = cast( ltrim(rtrim(substring(@plist,1,@dlm_pos-
1))) as bigint);
insert into @partitionsTable (pid) values (@partition_id)
set @plist =
substring(@plist,@dlm_pos+@dlm_len,len(@plist)); end
set @partition_id = cast (ltrim(rtrim(@plist)) as bigint);
insert into @partitionsTable (pid) values ( @partition_id );
return
end
GO
use
[MASTER] go
if (object_id('[attrep].[rtm_dump_dblog]','P')) is not
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 315
go
set ansi_nulls on
go
set quoted_identifier on
GO
if (@partition_list is null)
begin
RAISERROR ('Null partition list waspassed',16,1);
return
end
SELECT
[Current LSN],
[operation],
[Context],
[Transaction ID],
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 316
[Transaction Name],
[Begin Time],
[End Time],
[Flag Bits],
[PartitionID],
[Page ID],
[Slot ID],
[RowLog Contents 0],
[Log Record],
[RowLog Contents 1] --After Image
FROM
fn_dump_dblog (
@start_lsn, NULL, N'DISK', @seqno, @filename,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default,
default, default, default, default, default, default, default)
where [Current LSN] collate SQL_Latin1_General_CP1_CI_AS >
@start_lsn_ cmp collate SQL_Latin1_General_CP1_CI_AS --This aims for
implementing FN_DBLOG based on GT comparator.
and
(
( [operation] in ('LOP_BEGIN_XACT','LOP_COMMIT_XACT','LOP_ABORT_
XACT') )
or
( [operation] in ('LOP_INSERT_ROWS','LOP_DELETE_ROWS','LOP_MODIFY_
ROW')
and
( ( [context] in ('LCX_HEAP','LCX_CLUSTERED','LCX_MARK_AS_GHOST') )
or ([context] = 'LCX_TEXT_MIX' and (datalength([RowLog Contents 0]) in
(0,14,28))) ) --This one filters only TEXT_MIX of interest.\
and [PartitionID] in ( select * from master.attrep.split_partition_
list (@partition_list,','))
)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 317
or
([operation] = 'LOP_HOBT_DDL')
)
SET NOCOUNT OFF --Re-enable "rows affected display"
end
GO
Use
[master] Go
CREATE CERTIFICATE [attrep_rtm_dump_dblog_cert]
ENCRYPTION BY PASSWORD = N'choose_your_own_pwd'
WITH SUBJECT = N'Certificate for FN_DUMP_DBLOG Permissions';
Use
[master] Go
CREATE LOGIN attrep_rtm_dump_dblog_login FROM CERTIFICATE
[attrep_rtm_ dump_dblog_cert];
Note If the stored procedure is recreated, you need to add the signature again.
11. Create a user with the following permissions/roles (in each of the
following databases):
Master DB:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 318
select on sys.fn_dblog
view any definition
view server state (should be granted to the login).
execute on sp_repldone
execute on sp_replincrementlsn
execute on sp_addpublication
execute on sp_addarticle
execute on sp_articlefilter
select on [attrep].[split_partition_list]
execute on [attrep].[rtm_dump_dblog]
MSDB DB:
select on msdb.dbo.backupset
select on msdb.dbo.backupmediafamily
select on msdb.dbo.backupfile
db_owner ROLE
Table 8.23 |
Supported Microsoft SQL Server Compression Methods
Microsoft Row/Page Vardecimal Vardecimal Vardecimal
SQL Server Compression (at Storage Storage Storage
Version Partition Level) Format Format Format
Sparse Sparse
Columns Columns
Columnar
Structure
Compression
2005 No No No No
2008-2016 Yes No No No
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 319
Microsoft SQL Server Source Data Types
The Microsoft SQL Server source for Attunity Replicate supports most Microsoft SQL
Server data types. The following table shows the Microsoft SQL Server source data types
that are supported when using Attunity Replicate and the default mapping to Attunity
Replicate data types. Note that Microsoft SQL Server data types are only mapped to
Attunity Replicate data types when the target endpoint is not Microsoft SQL Server. For
information on data type mapping and collation support when the target endpoint is
Microsoft SQL Server, see Homogeneous Replication below.
For information on how to view the data type that is mapped in the target, see the
section for the target endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.24 |
Microsoft SQL Server Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not Microsoft SQL Server
BIGINT INT8
BIT BOOLEAN
DECIMAL NUMERIC
INT INT4
SMALLINT INT2
TINYINT UINT1
REAL REAL4
FLOAT REAL8
DOUBLE REAL8
DATETIME DATETIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 320
Table 8.24 | Microsoft SQL Server Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not Microsoft SQL Server (Cont.)
SMALLDATETIME DATETIME
DATE DATE
DATETIMEOFFSET STRING
*CHAR STRING
*VARCHAR STRING
*TEXT
To use this data type with Attunity Replicate, you must enable the
use of CLOBs for a specific task.
LOB columns for Microsoft SQL Server tables are updated in the
target even for UPDATE statements that did not change the value
of the LOB column in Microsoft SQL Server.
During CDC or during Full Load when the Allow unlimited LOB size
option is enabled, LOB data types are supported only in tables
with a primary key.
*NCHAR WSTRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 321
Table 8.24 | Microsoft SQL Server Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not Microsoft SQL Server (Cont.)
*NTEXT
To use this data type with Attunity Replicate, you must enable the
use of NCLOBs for a specific task.
LOB columns for Microsoft SQL Server tables are updated in the
target even for UPDATE statements that did not change the value
of the LOB column in Microsoft SQL Server.
During CDC or during Full Load when the Allow unlimited LOB size
option is enabled, LOB data types are supported only in tables
with a primary key.
BINARY BYTES
VARBINARY BYTES
IMAGE
LOB columns for Microsoft SQL Server tables are updated in the
target even for UPDATE statements that did not change the value
of the LOB column in Microsoft SQL Server.
To use this data type with Attunity Replicate, you must enable the
use of BLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size
option is enabled, LOB data types are supported only in tables
with a primary key.
TIMESTAMP BYTES
UNIQUEIDENTIFIER STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 322
Table 8.24 | Microsoft SQL Server Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not Microsoft SQL Server (Cont.)
STRING (250) -
When replicating to all
other endpoints.
XML CLOB
LOB columns for Microsoft SQL Server tables are updated in the
target even for UPDATE statements that did not change the value
of the LOB column in Microsoft SQL Server.
To use this data type with Attunity Replicate, you must enable the
use of NCLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size
option is enabled, LOB data types are supported only in tables
with a primary key.
GEOMETRY CLOB
GEOGRAPHY CLOB
Note User-defined data types are supported according to their base-type. For example a
user-defined data type based on DATETIME is handled as a DATETIME data type.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 323
Homogeneous Replication
When replicating from a Microsoft SQL Server source to a Microsoft SQL Server target,
most of the source and target data types will be identical. The exceptions are listed in
the table below.
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through
when replicating from Microsoft SQL Server, see the Microsoft SQL Server to
Attunity Replicate data types mapping table described earlier.
Note To prevent data truncation when replicating XML, Geometry and Geography
data types, it is strongly recommended to enable the Allow unlimited LOB size option
in the task settings.
Table 8.25 |
Data Type Exceptions in Homogeneous Replication
Microsoft SQL Server Source Microsoft SQL Server Microsoft SQL Server
2005 Target 2005-2016 Target
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 324
Table 8.25 | Data Type Exceptions in Homogeneous Replication (Cont.)
Microsoft SQL Server Source Microsoft SQL Server Microsoft SQL Server
2005 Target 2005-2016 Target
TIMESTAMP VARBINARY
Note To support collation replication, the DBA must ensure that the collations
defined for the source Microsoft SQL Server database are the same as those defined
for the target Microsoft SQL Server database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 325
varchar(max)
nvarchar(max)
varbinary(max)
image
xml
In the database properties Options tab, set the Recovery Model to Bulk logged or Full.
In these modes, the transaction Log is more durable.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 326
Note After setting the Recovery Model, it is strongly recommended not to change
it; doing so may result in loss of data.
1. In the Microsoft SQL Server Management Studio, right-click the Replication folder
and select Configure Distribution.
The Configure Distribution wizard opens.
In the Distributor step, select <Microsoft SQL Server Name> will act as its own
distributor; Microsoft SQL Server will create a distribution database and
log.
Note This functionality is supported only for Microsoft SQL Server Enterprise
edition and not on Microsoft SQL Server 2005.
By default, Attunity Replicate automatically sets up MS-REPLICATION for each of the source
tables in a replication task. However, MS-REPLICATION requires each of the source tables to
have a primary key, which may not always be the case. Therefore, if you need to replicate
tables that do not have a primary key, the following options are available:
Use MS-CDC
Do not Use MS-Replication or MS-CDC
Use MS-CDC
To set up MS-CDC, you first need to enable MS-CDC for the database by running
the following command:
use [DBname]
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 327
EXEC sys.sp_cdc_enable_db
Then you need to enable MS-CDC for each of the source tables by running the
following command:
EXECUTE sys.sp_cdc_enable_table @source_schema = N'MySchema', @source_name
= N'MyTable', @role_name = NULL;
Note Replicating tables that do not have a Primary Key or a Unique Index may
adversely affect performance (since additional database resources are required to
capture the changes). However, you can prevent performance issues related to the
absence of Primary Keys or a Unique Index by manually adding indexes to the
target tables.
For more information on setting up MS-CDC for specific tables, please refer to
the Microsoft website.
If your database is not set up for MS-REPLICATION or MS-CDC, you can still capture
tables that do not have a Primary Key, but bear in mind that in such a setup only
INSERT/DELETE DML events will be captured. UPDATE events will be ignored.
It is also important to note that a DELETE statement executed on an UPDATED
source record, will not be applied on the target.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 328
Working with Windows Authentication
You can configure the Attunity Replicate Microsoft SQL Server endpoint to log in
to Microsoft SQL Server (on Windows) using Windows authentication.
If you choose this option, you also need to make sure that:
The Microsoft SQL Server instance is set up to allow Windows log on.
The Attunity Replicate user is specified as the "Log on as" user for the
"Attunity Replicate Server" service account.
-OR-
Microsoft SQL Server is configured to allow login for the Attunity Replicate
Server service account.
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Microsoft
SQL Server database. This is optional.
4. Select SOURCE as the database role.
6. Specify the Server name. This is the host name or IP address of the computer
with the Microsoft SQL Server instance containing the source database.
Notes
When working with AlwaysOn Availability Groups, you need to specify the
IP address/host name and access credentials of the AlwaysOn Listener
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 329
To override the default port, add the port to the server name, separated by
a comma. For example, if the server name is myserver.company.local and
the port is 3333, then the server name should be entered like this:
myserver.company.local,3333
Note When using Windows authentication, make sure that the user account
that is associated with the Attunity Replicate Server service has Network
read and write permissions. This must be configured by a Windows system
administrator.
If you select SQL Server authentication, type the Microsoft SQL Server
authentication information (User name, Password) for the authorized user for
this Microsoft SQL Server database. If you do not know this information, see
your Microsoft SQL Server System Administrator.
To prevent illicit database activity by unauthorized third-parties, Replicate can be
configured to automatically replace the user-entered password with a strong
random password. For more information, see Configuring Replicate to
Automatically Replace the User-Entered Password.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 330
If the connection is successful a message in green is displayed. If the
connection fails, an error message is displayed at the bottom of the
dialog box.
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
Important: Make sure that the Microsoft SQL Server user has the correct access
privileges. For information on how to provide the required privileges, see
Required Permissions.
8. Type the Database name or click Browse and select one from the list of available
databases. This is the name of the database from where you are replicating the data.
Note This method also requires the Log Reader Agent to be running to enable
truncation of the Microsoft SQL Server active transaction log. Note that if the
Log Reader Agent is not running, the active log may become full, causing the
source database to be essentially "read-only" until the issue is resolved.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 331
Note When this option is selected, Replicate creates a table named
attrep_ truncation_safeguard in the source database. This is a very
small but important table whose purpose is to prevent truncation of the
transaction log by mimicking a transaction in the database. The table can
be safely deleted if there are no tasks configured with the Start
transactions in the database option.
Note This method requires the Log Reader Agent to be stopped in the database.
If the Log Reader Agent is running when the task starts, Attunity Replicate will
forcibly stop it. Alternatively, you can stop the Log Reader Agent manually,
before starting the Attunity Replicate task. For instructions on how to do this,
refer to the Microsoft SQL Server Management Studio help.
Note This option is not available when the Microsoft SQL Server Replication
job resides on a remote Distributor machine as Replicate does not have
access to the remote machine.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 332
Note that the backup files must be exported to the specified location in
standard Microsoft SQL Server format.
Change processing mode: Choose one of the following change processing modes:
Prioritize Online Logs - This is the default. Attunity Replicate will first look
for the changes in the online transaction logs. In the event that Replicate
cannot find the changes in the online transaction logs, it will look for them in
the backup transaction logs instead.
Prioritize Backup Logs - When this option is enabled, Attunity Replicate will first
look for the changes in the backup transaction logs. This can improve performance
when reading from the online transaction log is slow (e.g due to lock contention) or
when using file-level access to access the backup transaction logs.
In the event that Replicate cannot find the changes in the backup transaction
logs, it will look for them in the online transaction logs instead.
Backup Logs Only - When this option is selected, Attunity Replicate will try and
find the changes in the backup transaction logs only. Selecting this method results
in increased latency due to the interval between backups. The actual latency time
will remain constant, but will vary according to the backup schedule.
Online Logs Only - When this option is selected, Attunity Replicate will try
and find the changes in the online transaction logs only.
Replicate has file-level access to the backup log files: Select this option if
Attunity Replicate has been granted file-level access to the backup log files in
the Alternate backup folder.
Note When Attunity Replicate has file-level access to the backup transaction
logs, the following rules apply:
The Alternate backup folder must be a common shared network folder,
for example: \\temp\backup.
The Attunity Replicate Server service must be configured to log on using
the user name and password specified in the Backup folder user name
and Backup folder password fields.
To do this:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 333
In the Log On tab, select This account and then enter the user name
and password.
The specified user must be granted Read permission to the alternate
backup folder (i.e. the shared network folder).
For a complete list of the limitations affecting file-level access, see Limitations.
Backup folder user name: The user name required to access the backup
folder when Attunity Replicate has file-level access.
Backup folder password: The password required to access the backup
folder when Attunity Replicate has file-level access.
Backup folder preprocessing command: You can use a third-party utility to convert
the transaction logs to standard Microsoft SQL Server format (if they are in a different
format) and back them up to an alternate backup folder. This option should be used in
conjunction with the Alternate backup folder option described above.
The backup utility is responsible for setting the system return code (0 for
success, 1 for failure), assuming that this code is delegated as the
XP_CMDSHELL return value.
The backup utility invoked by XP_CMDSHELL must have the same security
rights as the Microsoft SQL Server service account.
XP_CMDSHELL is normally disabled. It can be enabled and disabled by using
the Policy-Based Management or by executing SP_CONFIGURE.
Using this extended procedure requires CONTROL SERVER permission (at least).
Command Usage:
The backup utility should provide Attunity Replicate with the following parameters:
Example command:
C:\Temp\YourBackupUtility.exe -B{BACKUP_INFILE} -A{ALTDIR_OUTFILE}"
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 334
Important: Directory names in the command path or file names in the
actual command that contain spaces must be enclosed in double-quotes:
Example:
Delete processed backup logs: Select this option to delete the backup logs
after they have been read.
Select virtual backup device types: When this option is selected, Attunity
Replicate will read changes from the specified virtual device(s). Usually, this
option only needs to be enabled when using a third-party backup utility (which
will be recorded as a virtual device).
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 335
Artifacts Created by Replicate in the Database
In order to capture changes, Attunity Replicate creates various artifacts in the dbo
schema of the Microsoft SQL Server database when the task starts. You can safely
remove them when the task completes.
The following artifact is created to track open transactions:
#attrep_dbcc_open_trans_table
The following functions are created to verify that the LSN (Log Sequence Number) is valid:
attrep_fn_LsnSegmentToHexa
attrep_fn_NumericLsnToHexa
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 336
Using a MySQL-Based Database as a Source
This section describes how to set up and use a MySQL-based database as a source
in a replication task.
You need to configure the Replicate MySQL endpoint when replicating from any of
the following databases:
MySQL
Percona
MariaDB
Amazon Aurora
Note The procedures for configuring connectivity to these endpoints are identical to
those described in this section, for MySQL. However, when using Percona as a source,
there is no need to perform the procedures described in Cluster Prerequisites.
In this section:
▶ Prerequisites
▶ General Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Enable Binary Logging
▶ Cluster Prerequisites
▶ Replicating 4-byte UTF8 Emojis
▶ Limitations
▶ Security Requirements
▶ MySQL Database Source Data Types
▶ Homogeneous Replication
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Selecting a Schema
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 337
Prerequisites
Make sure that the prerequisites described in this section have been met.
In this section:
General Prerequisites
Attunity Replicate Server for Windows
Attunity Replicate Server for Linux
Enable Binary Logging
Cluster Prerequisites
Replicating 4-byte UTF8 Emojis
General Prerequisites
The following is required:
A MySQL account with the required Security Requirements.
A MySQL database with the tables that you want to replicate should be accessible
in your network.
The following MySQL editions are supported:
Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in the
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 338
following example:
[MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Table 8.26 |
Required my.ini/my.cnf Parameters for Binary Logging
Parameter Value
server_id Any value from 1.
Example:
server_id=1
log-bin=<path> Path to the binary log file (without an extension).
Example:
log-bin=E:\MySql_Logs\BinLog
binlog_format Must be:
binlog_format=row
expire_logs_days To prevent disk space issues, it is strongly recommended not to use
the default value (0).
Example:
expire_logs_days=5
binlog_row_image Must be:
binlog_row_image=full
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 339
Table 8.26 | Required my.ini/my.cnf Parameters for Binary Logging (Cont.)
Parameter Value
log_slave_updates When replicating from a MySQL slave database server, this value
should be set to true (1). If set to 0 (the default) updates on a
slave received from a master during replication are not logged to
the slave's binary log. The slave's binary log needs to be enabled
for this to have an effect.
Cluster Prerequisites
To be able to replicate clustered (NDB) tables (i.e. by connecting Attunity Replicate to
any of the cluster nodes), the following parameters must be configured in MySQL’s
my.ini (Windows) or my.cnf (UNIX) files.
Table 8.27 |
Required my.ini/my.cnf Parameters for Cluster Replication
Parameter Value
ndb_log_bin Must be:
ndb_log_bin=on
This ensures that changes in clustered tables will be logged to the
binary log.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 340
Microsoft SQL Server Target: Transform the emojis from WSTRING(n) to
WSTRING (n*2).
Amazon Redshift Target: Transform the emojis from WSTRING(n) to WSTRING(n*2).
MySQL Target: The target schema character set must be set to utf8mb4.
Limitations
The following limitations apply:
A Unique Index/Primary Key created on only part of the column data is not
supported. If such a Unique Index/Primary Key is encountered, it will be created on
the entire target column.
The following is an example of a statement that creates a Unique Index/Primary
Key using only part of the column data:
CREATE INDEX partial_name ON customer (name(10));
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 341
Security Requirements
The Attunity Replicate user must have the ReplicationAdmin role with the
following privileges (according to task type):
REPLICATION CLIENT - Required for Change Processing tasks only. In
other words, Full Load only tasks do not require this privilege.
REPLICATION SLAVE - Required for Change Processing tasks only. In
other words, Full Load only tasks do not require this privilege.
SUPER - Only required in versions prior to MySQL 5.6.6.
The Attunity Replicate user must also have SELECT privileges for the source
tables designated for replication.
The Attunity Replicate user must have Standard authentication in order to work
with MySQL 8 as a source endpoint for CDC.
Table 8.28 |
MySQL Database Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not MySQL
BIGINT INT8
MEDIUMINT INT4
TINYINT INT1
BIT BOOLEAN
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 342
Table 8.28 | MySQL Database Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not MySQL (Cont.)
LONGBLOB BLOB
MEDIUMBLOB BLOB
DATE DATE
DATETIME DATETIME
TIME STRING
TIMESTAMP DATETIME
YEAR INT2
DOUBLE REAL8
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 343
Table 8.28 | MySQL Database Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not MySQL (Cont.)
- 1.79E+308 to -2.23E-308, 0
and
2.23E-308 to 1.79E+308
*CHAR WSTRING
*LONGTEXT NCLOB
*MEDIUMTEXT NCLOB
GEOMETRY BLOB
POINT BLOB
LINESTRING BLOB
POLYGON BLOB
MULTIPOINT BLOB
MULTILINESTRING BLOB
MULTIPOLYGON BLOB
GEOMETRYCOLLECTION BLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 344
Table 8.28 | MySQL Database Source Data Types with Mapping to Attunity
Replicate Data Types when the Target is not MySQL (Cont.)
Note If the DATETIME and TIMESTAMP data types are specified with a “zero” value
(i.e. 0000-00-00), you need to make sure that the target database in the replication
task supports "zero" values for the DATETIME and TIMESTAMP data types. If they are
not supported, you can use a transformation to specify a supported value (e.g. 1970.)
Otherwise, they will be recorded as null on the target.
Note The JSON data type introduced in MySQL 5.7 is not supported.
Consequently, JSON columns in the source tables will be ignored.
Homogeneous Replication
The following section describes how Replicate handles replication between a MySQL
source and a MySQL target (i.e. homogeneous replication).
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through when
replicating from MySQL, see the MySQL to Attunity Replicate data types mapping
table described earlier.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 345
Data Types
When replicating to a MySQL target endpoint, the data types will be identical with
the following exceptions:
NUMERIC DECIMAL
Collation
When replicating from one MySQL endpoint to another, table and column collations will
be replicated to the target. Collatable data types are indicated by an asterisk (*) in Table
11–3 above.
To support collation replication, the DBA must ensure that the collations defined for the
source MySQL database are the same as those defined for the target MySQL database.
Note You can also use MySQL files as a source. For more information, see Using
the Attunity Replicate File Channel.
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 346
2. In the Name field, type a name for your endpoint. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the MySQL
database. This is optional.
4. Select SOURCE as the endpoint role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server Name field, enter the host name or IP address of the computer
on which the MySQL database is installed.
7. Optionally, change the default port (3306).
8. Type the MySQL authentication information (User Name, Password) for the
authorized user for this MySQL database. If you do not know this information, see
your MySQL database Administrator (DBA).
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
If you want to set custom properties for this database, see Setting
Advanced Connection Properties.
Important: Make sure that the MySQL user entered in the MySQL
Authentication section has the correct access privileges. For information on
how to provide the required privileges, see Security Requirements.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 347
SSL Authentication Options
In the Security section, you can configure Replicate to connect to the MySQL
database using SSL.
SSL Mode: Select one of the following:
Preferred - Establishes an encrypted connection if the server supports
encrypted connections, falling back to an unencrypted connection if an
encrypted connection cannot be established.
None - Connect with a surname and password only.
Required - Establishes an encrypted connection if the server supports
encrypted connections. The connection attempt fails if an encrypted connection
cannot be established.
Verify CA - Similar to Required, but also verifies the server Certificate
Authority (CA) certificate against the configured CA certificates. The
connection attempt fails if no valid matching CA certificates are found.
Verify Full - Similar to Verify CA, but also performs host name identity
verification by checking the host name the client (i.e. Replicate) uses for
connecting to the server against the identity in the certificate that the
server sends to the client. The client checks whether the host name that it
uses for connecting matches the Common Name value in the server
certificate. The connection fails if there is a mismatch.
Client certificate path - The path to the client certificate requested by the server.
Private key path - The path to the client private key file in PEM format.
CA path - The path to the Certificate Authority (CA) that issued the client
certificate file in PEM format.
Selecting a Schema
You can choose which MySQL database to access. After configuring the MySQL source
database connection settings, open the Select Tables dialog box (by clicking the
Table Selection button on the right of the console) and select which schema to use
from the Schema drop down list.
See also Defining Tasks.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 348
Check binary log for new events every: Specify how often to check the binary
log for changes when the endpoints is idle.
Additional ODBC connection properties: Specify any additional ODBC
connection parameters that may be required.
Note Attunity Replicate assumes that MySQL Client 5.2.6 to 5.3.x for Linux or MySQL
ODBC Client 5.2.6 to 5.3.x 64-bit for Windows is installed on the Attunity Replicate
Server machine. If a version later than 5.3.x is installed, you need to specify the
version number as an internal parameter where provider is the Parameter and
MySQL ODBC <version> Unicode Driver is the Value (where <version> is the
client version e.g. 5.4).
For instructions on setting internal parameters, see Internal Parameters.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 349
Using ODBC to Connect to a Source
This section describes how to use ODBC connectivity to connect to a source endpoint.
In this section:
▶ Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Limitations
▶ ODBC Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the prerequisites for working with Attunity Replicate and
an ODBC endpoint.
Note You must use a 64-bit ODBC provider client to work with Attunity Replicate.
2. Use the ODBC Data Source Administrator to create a System DSN. The Data Source
is located in the Windows control panel.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 350
Attunity Replicate Server for Linux
The following section describes the steps you need to perform to work with Attunity
Replicate for Linux and ODBC as a source or target endpoint in a Attunity Replicate task.
1. On the Attunity Replicate Server machine, install the ODBC client that you want to
use (e.g. postgreSQL).
2. Makes sure that the /etc/odbcinst.ini file contains the correct entry for the
driver you installed, as in the following example:
[PostgeSQL]
= /usr/lib/libodbcpsqlS.so Driver64
= /usr/lib64/psqlodbc.so Setup64 =
/usr/lib64/libodbcpsqlS.so
FileUsage = 1
Note To access an IBM DB2 for LUW target using ODBC, make sure that
you specify the libdb2o.so driver (and not libdb2.so).
3. Define a DSN for the installed driver by editing the /etc/odbc.ini file, as in
the following example:
[Postgre_DSN]
Description = Test
Driver = /usr/lib64/psqlodbc.so
= 12.3.45.678
Port = 5432
Limitations
When using ODBC as a source, the following limitations apply:
UPDATES to primary key fields are not supported. To update the field, define it
as a unique index instead.
The ODBC Source endpoint supports full-load operations only.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 351
For providers that do not support batch operations, you must set the RowByRow=true
internal parameter according to the description provided in Internal Parameters.
For information on how to view the data type that is mapped in the target, see the
section for the target endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.29 |
Supported ODBC Source Data Types with Mapping to Attunity Replicate
Data Types
SQL_BIT BOOLEAN
SQL_TINYINT INT1
UINT1
SQL_SMALLINT INT2
UINT2
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 352
Table 8.29 | Supported ODBC Source Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
UINT4
SQL_BIGINT INT8
UINT8
SQL_DOUBLE REAL8
SQL_FLOAT REAL8
SQL_REAL REAL8
REAL8
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 353
Table 8.29 | Supported ODBC Source Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
REAL 8
Precision > 38
Scale < 0
Scale > 38
Scale > Precision
SQL_DATE DATE
SQL_TYPE_DATE
SQL_TIME TIME
SQL_TYPE_TIME
SQL_TIMESTAMP DATETIME
SQL_TYPE_TIMESTAMP
SQL_CHAR STRING
SQL_VARCHAR
SQL_WCHAR WSTRING
SQL_WVARCHAR
SQL_LONGVARCHAR CLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 354
Table 8.29 | Supported ODBC Source Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
SQL_BINARY BYTES
SQL_LONGVARBINARY BLOB
SQL_GUID STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 355
Table 8.29 | Supported ODBC Source Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
SQL_INTERVAL_YEAR STRINGw
SQL_INTERVAL_MONTH
SQL_INTERVAL_DAY
SQL_INTERVAL_MINUTE
SQL_INTERVAL_HOUR
SQL_INTERVAL_SECOND
SQL_INTERVAL_YEAR_TO_MONTH
SQL_INTERVAL_DAY_TO_HOUR
SQL_INTERVAL_DAY_TO_MINUTE
SQL_INTERVAL_DAY_TO_SECOND
SQL_INTERVAL_HOUR_TO_MINUTE
SQL_INTERVAL_HOUR_TO_SECOND
SQL_INTERVAL_MINUTE_TO_SECOND
BYTES
Note If column length is 0 or > 4000
If column length is 0 or > 4000:
then:
BLOB
To use this data type with Attunity
Replicate, you must enable the use
of BLOBs for a specific task.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 356
To add an ODBC source endpoint to Attunity Replicate:
If the DSN you want to use is not included in the list, make sure that the
endpoint client is installed on the computer with Attunity Replicate and that the
DSN is defined. Note that the ODBC provider client must be 64-bit. For more
information, see Prerequisites .
Note If you are using an ARC CDC Agent as the source in a Attunity
Replicate task, you cannot select the DSN for the Attunity ODBC driver as
the target. In this case, to use Attunity ODBC as a source, you must enter
the connection string manually by selecting Connection String and
following the directions for that option in this procedure.
Note
You can use the Advanced tab to add specific properties and create a
custom connect string. In this case, you do not need to enter
information in this tab. For more information on using the Advanced
tab, see Setting Advanced Connection Properties.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 357
To determine if you are connected to the endpoint you want to use
or if the connection information you entered is correct, click Test
Connection.
If the connection is successful a message in green is displayed. If
the connection fails, an error message is displayed at the bottom of
the dialog box.
To view the log entry if the connection fails, click View Log. The
server log is displayed with the information for the connection failure.
Note that this button is not available unless the test connection fails.
7. Type the authentication information (User Name, Password) for the authorized
user for the ODBC endpoint being used. For example, the IBM DB2 system
administrator if you are using a IBM DB2 provider. If you do not know this
information, see your ODBC Endpoint System Administrator.
Note
Important: Make sure that the ODBC endpoint user has the correct
access privileges for the ODBC provider being used.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 358
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 359
Using ODBC with CDC as a Source
This section describes how to use ODBC connectivity to connect to a source endpoint
in a Full Load and/or CDC task.
In this section:
▶ Prerequisites
▶ Replicate Server for Windows
▶ Replicate Server for Linux
▶ Limitations
▶ ODBC with CDC Source Data Types
▶ Setting General Connection Properties
▶ Setting Change Processing Parameters
▶ Prerequisites
▶ Limitations
▶ Configuring Change Processing Settings
Prerequisites
The following section describes the prerequisites for working with Attunity Replicate and
an ODBC source with CDC.
Note You must use a 64-bit ODBC provider client to work with Attunity Replicate
2. Use the ODBC Data Source Administrator to create a System DSN.The Data Source
is located in the Windows control panel.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 360
Replicate Server for Linux
The following section describes the steps for working with Attunity Replicate for Linux
and ODBC with CDC as a source endpoint in a Replicate task.
1. On the Attunity Replicate Server machine, install the ODBC client that you want to
use (e.g. postgreSQL).
2. Makes sure that the /etc/odbcinst.ini file contains the correct entry for the
driver you installed, as in the following example:[PostgeSQL]
Description = ODBC for PostgreSQL
= /usr/lib/libodbcpsqlS.so Driver64
= /usr/lib64/psqlodbc.so Setup64 =
/usr/lib64/libodbcpsqlS.so
FileUsage = 1
3. Define a DSN for the installed driver by editing the /etc/odbc.ini file, as in
the following example:
[Postgre_DSN]
Description = Test
Driver = /usr/lib64/psqlodbc.so
= 12.3.45.678
Port = 5432
Limitations
The following limitations apply:
UPDATES to primary key fields are not supported. To update the field, define it
as a unique index instead.
For providers that do not support batch operations, you must manually add the
RowByRow=true internal parameter according to the instruction provided in
Setting Change Processing Parameters.
The "Resume from timestamp" run option is not supported.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 361
ODBC with CDC Source Data Types
The following table shows the ODBC target data types that are supported when using
Attunity Replicate and the default mapping from Attunity Replicate data types.
For information on how to view the data type that is mapped in the target, see the
section for the target endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.30 |
Supported ODBC with CDC Source Data Types with Mapping to
Attunity Replicate Data Types
SQL_TINYINT INT1
UINT1
SQL_SMALLINT INT2
UINT2
SQL_INTEGER INT4
UINT4
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 362
Table 8.30 | Supported ODBC with CDC Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
UINT8
SQL_DOUBLE REAL8
SQL_FLOAT REAL8
SQL_REAL REAL8
REAL8
Precision > 38
Scale < 0
Scale > 38
Scale > Precision
REAL 8
Precision > 38
Scale < 0
Scale > 38
Scale > Precision
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 363
Table 8.30 | Supported ODBC with CDC Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
SQL_TYPE_DATE
SQL_TIME TIME
SQL_TYPE_TIME
SQL_TIMESTAMP DATETIME
SQL_TYPE_TIMESTAMP
SQL_CHAR STRING
SQL_VARCHAR
SQL_WCHAR WSTRING
SQL_WVARCHAR
SQL_LONGVARCHAR CLOB
SQL_WLONGVARCHAR NCLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 364
Table 8.30 | Supported ODBC with CDC Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
SQL_LONGVARBINARY BLOB
SQL_GUID STRING
SQL_INTERVAL_YEAR STRING
SQL_INTERVAL_MONTH
SQL_INTERVAL_DAY
SQL_INTERVAL_MINUTE
SQL_INTERVAL_HOUR
SQL_INTERVAL_SECOND
SQL_INTERVAL_YEAR_TO_MONTH
SQL_INTERVAL_DAY_TO_HOUR
SQL_INTERVAL_DAY_TO_MINUTE
SQL_INTERVAL_DAY_TO_SECOND
SQL_INTERVAL_HOUR_TO_MINUTE
SQL_INTERVAL_HOUR_TO_SECOND
SQL_INTERVAL_MINUTE_TO_SECOND
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 365
Table 8.30 | Supported ODBC with CDC Source Data Types with Mapping to Attunity Rep-
licate Data Types (Cont.)
To use this data type with Attunity If column length is 0 or > 4000:
Replicate, you must enable the use of BLOB
BLOBs for a specific task.
During CDC or during Full Load when the
Allow unlimited LOB size option is
enabled, LOB data types are supported
only in tables with a primary key.
1. In the Attunity Replicate console, click the Manage Endpoint Connections toolbar
button to open the Manage Endpoints Connections dialog box. Then click the New
Endpoint Connection button. For more information on adding an endpoint to
Attunity Replicate, see Working with Endpoints.
2. In the Name field, type a name for your ODBC endpoint. This can be any name
that will help to identify the endpoint being used.
3. In the Description field, type a description that helps to identify the ODBC
endpoint. This is optional.
4. Select SOURCE as the endpoint role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 366
DSN: Select this to connect to an ODBC-supported endpoint using a DSN.
When you select DSN you must select the DSN you are using from the list.
If the DSN you want to use is not included in the list, make sure that the
endpoint client is installed on the computer with Attunity Replicate and that the
DSN is defined. Note that the ODBC provider client must be 64-bit. For more
information, see Prerequisites .
Note If you are using a Replicate Connect CDC Agent as the source in a
Replicate task, you cannot select the DSN for the Attunity ODBC driver as
the target. In this case, to use Attunity ODBC as a source, you must enter
the connection string manually by selecting Connection String and
following the directions for that option in this procedure.
Note To determine if you are connected to the endpoint you want to use or if
the connection information you entered is correct, click Test Connection.
7. Type the authentication information (User Name, Password) for the authorized
user for the ODBC endpoint being used. For example, the IBM DB2 system
administrator if you are using a IBM DB2 provider. If you do not know this
information, see your ODBC Endpoint System Administrator.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 367
Note Consider the following:
Important: Make sure that the ODBC endpoint user has the correct
access privileges for the ODBC provider being used.
Prerequisites
Before you can define the settings in the Change Processing tab, you need to ensure
that at least one special "Context" column exists in your source database tables. Context
column(s) are basically columns in a table that enable Replicate to determine whether
the data has changed. You can add Context columns specifically for the purpose of
change processing (either using a script or manually) or you can use existing columns
that contain suitable "Context" data.
Note You can create and reference any number of Context columns in a table as
long as the Context column names are the same for all source tables. Additionally,
each value in the Context column(s) must be unique.
In the example below, the Context column cf has been added to the table. The cf column
contains TIMESTAMPs that enable Replicate to determine whether a change occurred (by
comparing the current TIMESTAMP with the TIMESTAMP stored in its repository).
By default, all changes are assumed to be INSERTs. If UPDATE and DELETE operations
are also performed on the source tables, you can write an UPDATE and/or DELETE
expression (described below) that will enable Replicate to identify the operation type.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 368
Figure 8.2 | Example of a Table with a Context Column
Limitations
The following limitations apply when Change Processing is enabled:
The "Start from timestamp" run option is not supported. For more information,
see Using Advanced Run Options.
If one of the Context columns is part of the Primary Key or Unique Index, then
UPDATE and DELETE operations are not supported.
Context columns cannot be LOB columns
DDLs are not supported
When inserting a record and then updating the same record, the task error
handling settings should be set as follows:
Open the <Task Name> Settings dialog box.
Select the Error Handling|Apply Conflicts tab.
Set a task-specific Apply Conflicts policy as described in Error Handling Settings.
From the No record found for applying an update drop-down list, select
INSERT the missing target record.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 369
To configure change processing settings:
2. In the Columns field, specify the names of the Context columns. The column
names are case-sensitive and must be separated by commas.
Example:
context1,context2
5. Enter expressions that Replicate will use to identify UPDATE and DELETE operations. If
you do not enter any expressions or if no match is found for an expression, any row
whose context is higher (if the sorting order is Ascending) or lower (if the sorting order
is Descending) than the previous context value will be considered an INSERT.
Note Expressions must be written in the native syntax of the source database.
All examples in this section are written using PostgresSQL syntax.
Tip: Selecting the UPDATE the existing target record option in the
Apply Conflicts tab, eliminates the need to provide an UPDATE expression.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 370
with parameters that are not exposed in the UI. As such parameters are normally
not required, they should only be used after consulting with Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 371
Using Oracle as a Source
This section describes how to set up and use an Oracle database as a source in a
replication task.
In this section:
▶ Supported Oracle Database Editions
▶ Prerequisites
▶ Limitations
▶ Required Permissions
▶ General Permissions
▶ Access Privileges when using Oracle LogMiner to Access the Redo Logs
▶ Access Privileges when using Attunity Log Reader to Access the Redo Logs
▶ Required ASM Privileges
▶ Supported Encryption Methods
▶ Supported Compression Methods
▶ Supported Standby Configurations
▶ Redo Log Files - Access Method Guidelines
▶ Handling Shrink Space Operations
▶ Replicating Nested Tables
▶ Prerequisites
▶ Supported Nested Table Types
▶ Limitations
▶ How Nested Tables are Replicated
▶ JOIN Statement Example
▶ Oracle Source Data Types
▶ Non-Supported Data Types
▶ Homogeneous Replication
▶ Preparing the Oracle Database for Replication
▶ Provide Oracle Account Access
▶ Ensure that ARCHIVELOG Mode is On
▶ Setting up Supplemental Logging
▶ Working with Oracle on Amazon RDS
▶ Setting General Connection Properties
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 372
▶ Setting Advanced Connection Properties
▶ Setting Advanced Connection Properties Using Oracle LogMiner
▶ Setting Advanced Connection Properties Using Attunity Log Reader
▶ Finding the Wallet Entries used for TDE Encryption
Prerequisites
Before you can work with an Oracle endpoint, make sure the prerequisites listed in
this section have been met.
On Windows systems, install Oracle Instant Client for Microsoft Windows (x64)
Version 11.2.0.3.0 and above.
Note Support for the XMLTYPE data type requires the full Oracle Client.
On Linux systems, install Oracle Instant Client for Linux (x86-64) Version
11.2.0.3.0 and above.
Note Support for the XMLTYPE data type requires the full Oracle Client.
In addition, if not already included in your system, you need to create a symbolic
link in the $Oracle_Home\lib directory. This link should be called libclntsh.so,
and should point to a specific version of this file.
For example, on an Oracle 12c client:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 373
lrwxrwxrwx 1 oracle oracle 63 Oct 2 14:16 libclntsh.so ->
/u01/app/oracle/home/lib/libclntsh.so.12.1
Limitations
The following limitations apply:
Long object names (over 30 bytes) are not supported.
Function-based indexes are not supported.
If you are managing supplemental logging and you carry out transformations on any
of the columns, you must be sure that supplemental logging is activated for all fields
and columns.
The AR_H_USER header column is supported only for Oracle database version
11.2.0.3 and higher. In Oracle database version 10, the value for this column may
not be correct. For information on using header columns, see Headers.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 374
Changes made by the Oracle DBMS_REDEFINITION package - e.g. table metadata
and the OBJECT_ID) - will not be captured by Attunity Replicate.
Empty BLOB/CLOB columns are mapped to NULL on the target.
During Change Processing, columns without supplemental logging that are not
updated will be inserted as NULL in the Change Table.
During Change Processing, batch updates to numeric columns defined as a
Primary Key are not supported.
Where tableX is the table name and ID is a numeric column defined as a Primary Key.
Data in LONG and LONG RAW column cannot exceed 64k. Any data that is larger
than 64k will be truncated.
Tables whose names contain apostrophes cannot be replicated.
Change Data Capture (CDC) is not supported from dynamic views.
Index-organized tables with an overflow segment are not supported in
Change Processing (CDC) tasks.
When using Oracle LogMiner to access the redo logs, the following limitations apply:
In Oracle 12 only, any changes to LOB columns are not supported (i.e. replicated).
UPDATEs to XMLTYPE and LOB columns are not supported (i.e. replicated).
SHRINK SPACE operations are not supported
Connecting to a PDB using Oracle LogMiner is not supported. Therefore, if you
want to connect to a PDB, make sure that the Access redo logs via Attunity
Log Reader option is selected in the Advanced tab.
When using Attunity Log Reader to access the redo logs, the following
limitations apply:
Table clusters are not supported.
Only table-level SHRINK SPACE operations are supported. These include the
full table, partitions, and sub-partitions.
Implementing online redo logs on raw devices is not supported.
Changes to Index-organized tables with key compression are not supported.
Required Permissions
In this section:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 375
General Permissions
Access Privileges when using Oracle LogMiner to Access the Redo Logs
Access Privileges when using Attunity Log Reader to Access the Redo
Logs Required ASM Privileges
General Permissions
To use an Oracle source in an Attunity Replicate task, the user specified in the Attunity
Replicate Oracle endpoint connection settings must be granted the following privileges
in the Oracle database:
Note If any of the required privileges cannot be granted to a V$xxx, then grant them
to the V_$xxx.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 376
SELECT on SYS.DBA_REGISTRY
SELECT on SYS.OBJ$
SELECT on SYS.ENC$
SELECT on DBA_TABLESPACES
SELECT on ALL_TAB_PARTITIONS
SELECT on ALL_ENCRYPTED_COLUMNS
If views are exposed: SELECT on ALL_VIEWS
Grant the following additional privilege (for each replicated table) when you are using
a specific table list:
SELECT on <any-replicated-table>;
Grant the following additional privilege when using a pattern for the table list:
SELECT ANY TABLE;
Grant the following additional privilege (for each replicated table) when Attunity
Replicate adds supplemental logging automatically (the default behavior) and you are
using a specific table list. For information on how to turn off supplemental logging, see
Setting Advanced Connection Properties Using Oracle LogMinerSetting Advanced
Connection Properties Using Oracle LogMiner.
ALTER on <any-replicated-table>;
Grant the following additional privilege when Attunity Replicate adds supplemental
logging automatically (the default behavior). For information on how to turn off
supplemental logging, see Setting Advanced Connection Properties Using Oracle
LogMinerSetting Advanced Connection Properties Using Oracle LogMiner.
ALTER ANY TABLE;
When accessing an Oracle standby database, the following privilege must be granted:
SELECT on V$STANDBY_LOG
Access Privileges when using Oracle LogMiner to Access the Redo Logs
If you are using Oracle LogMiner to access the Redo logs, grant the following privileges.
CREATE SESSION
EXECUTE on DBMS_LOGMNR
SELECT on V_$LOGMNR_LOGS
SELECT on V_$LOGMNR_CONTENTS
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 377
LOGMINING
Note This privilege is only required for Oracle 12c and above.
Access Privileges when using Attunity Log Reader to Access the Redo Logs
The following privileges should be granted when using Attunity Log Reader to access the
Redo logs:
CREATE SESSION
SELECT on v_$transportable_platform
Grant the SELECT on v_$transportable_platform privilege if the Redo logs are stored
in ASM and accessed by Replicate from ASM.
CREATE ANY DIRECTORY
BFILE read - Used when Replicate does not have file-level access to the Redo logs,
and the Redo logs are not accessed from ASM.
DBMS_FILE_TRANSFER package - Used to copy the Redo log files to a temporary
folder (in which case the EXECUTE ON DBMS_FILE_TRANSFER privilege needs to
be granted as well)
DBMS_FILE_GROUP package - Used to delete the Redo log files from a
temporary/alternate folder (in which case the EXECUTE ON
DBMS_FILE_GROUP privilege needs to be granted as well).
Oracle file features work together with Oracle directories. Each Oracle directory object
includes the name of the folder containing the files which need to be processed.
If you want Replicate to create and manage the Oracle directories, you need to grant the
CREATE ANY DIRECTORY privilege specified above. Note that the directory names will be
prefixed with attu_. If you do not grant this privilege, you need to create the
corresponding directories manually. If you create the directories manually and the Oracle
user specified in the Oracle Source endpoint is not the user that created the Oracle
Directories, grant the READ on DIRECTORY privilege as well.
If the Oracle source endpoint is configured to copy the Redo log files to a temporary
folder, and the Oracle user specified in the Oracle source endpoint is not the user that
created the Oracle directories, the following additional privileges are required:
READ on the Oracle directory object specified as the source directory
WRITE on the directory object specified as the destination directory in the
copy process
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 378
See also: Setting Advanced Connection Properties.
From Oracle 11g Release 2 (11.2.0.2), Attunity Replicate must be granted the
SYSASM privilege in order to access the ASM account. For older supported versions,
granting Attunity Replicate the SYSDBA privilege should be sufficient.
Note When connecting to ASM, Attunity Replicate will first try to log in as SYSDBA
and, if unsuccessful, will try to log in as SYSASM.
You can validate ASM account access by opening a command prompt and issuing
the following statements:
sqlplus asmuser/asmpassword@+asmserver as sysdba
-OR-
sqlplus asmuser/asmpassword@+asmserver as sysasm
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 379
Supported Encryption Methods
The table below lists which encryption methods Attunity Replicate supports when
working with an Oracle source database.
Table 8.31 |
Supported Oracle Encryption Methods
Redo Logs Access Method TDE Tablespace TDE Column
Attunity Log Reader Yes Yes
Table 8.32 |
Supported Oracle Compression Methods
Version Basic OLTP HCC (from Oracle 11g Others
R2)
Oracle 10 No n/a No
Note When the Oracle source endpoint is configured to use Attunity Log Reader, the
Query Low level of the HCC compression method is only supported in the Full Load
task mode.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 380
Supported Standby Configurations
The following table lists the supported Oracle Standby configurations.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 381
Physical Standby Yes The physical standby
database must have Active
Data Guard enabled
(available from Oracle 11g
and above).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 382
Attunity Support.
Logical Standby No
Hot Standby No
Monitoring Considerations:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 383
When Replicate captures changes resulting from a SHRINK SPACE operation, the task’s
Incoming Changes bar will indicate an unusually large number of changes. However,
these changes will not be reflected in the Applied Changes pie chart or the Applied
Changes Details table.
See also the Limitations section for the limitations related to SHRINK SPACE operations
in Attunity Log Reader and LogMiner mode.
Prerequisites
Make sure that you replicate parent tables for all the replicated nested tables. Both
the parent tables (the tables containing the nested table column) and the child (i.e.
nested) tables will be available for selection in Replicate.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 384
Data type
User defined Object
Limitations
Only one level of nesting is supported.
Replicate does not verify that both the parent and child table(s) are selected for
replication. In other words, it's possible to select a parent table without a child
and vice versa.
The parent table is created identical to the source. The nested column will be
defined as RAW(16) and contain a reference to its nested tables in the
NESTED_TABLE_ID column.
The child table is created identical to the source, but with an additional column
named NESTED_TABLE_ID with the same type as the parent nested column and
with the same meaning.
Creating the parent table with a column of type my_tab_t that was defined
above:
CREATE TABLE my_parent_table (id NUMBER PRIMARY KEY, col1 my_tab_t) NESTED
TABLE col1 STORE AS col1_tab;
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 385
Oracle Source Data Types
The Oracle database for Attunity Replicate supports most Oracle data types. The
following table shows the Oracle source data types that are supported when using
Attunity Replicate and the default mapping to Attunity Replicate data types.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.33 |
Supported Oracle Data Types with Mapping to Attunity Replicate Data Types
BINARY_DOUBLE REAL8
BINARY BYTES
DATE DATETIME
TIMESTAMP DATETIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 386
Table 8.33 | Supported Oracle Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
CHAR STRING
VARCHAR2 STRING
NCHAR WSTRING
NVARCHAR2 WSTRING
RAW BYTES
REAL REAL8
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 387
Table 8.33 | Supported Oracle Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
XMLTYPE CLOB
Notes
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 388
Non-Supported Data Types
Columns with the following data types are not supported and will not be replicated:
BFILE
ROWID
REF
UROWID
ANYDATA
SDO_GEOMETRY
User-defined data types
Note
Virtual columns are not supported.
As the ROWID data type is not supported, materialized views based on a
ROWID column are also not supported.
Homogeneous Replication
With the exception of the LONG and LONG RAW data types, when replicating from an
Oracle source to an Oracle target, all of the source and target data types will be
identical. The LONG data type will be mapped to CLOB and the LONG RAW data type will
be mapped to BLOB. It should be noted that, as of Oracle 9.0, the LONG and LONG RAW
data types are no longer supported by Oracle.
Additionally, Primary/Unique Index names are preserved during homogeneous replication.
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through when
replicating from Oracle, see the Oracle to Attunity Replicate data types mapping
table described earlier.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 389
Preparing the Oracle Database for Replication
The following topics describe the configuration requirements for using an Oracle
database with Attunity Replicate as a source. An Oracle DBA should know how to carry
out these tasks.
Provide Oracle Account Access
Ensure that ARCHIVELOG Mode is On
Setting up Supplemental Logging
Note that if your Oracle database instance is on Amazon RDS, a different command
needs to be executed. For more information, see Working with Oracle on Amazon RDS
and Working with Oracle on Amazon RDS in Working with Oracle on Amazon RDS.
Note You can automatically set up supplemental logging in the Advanced tab of the
Oracle database dialog box. If you select this option, you do not have to carry out the
following procedure. For more information, see Setting Advanced Connection
Properties Using Oracle LogMiner.
= 'compatible';
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 390
The returned result should be from GE to n.n.n where n.n.n is the Oracle
database version (e.g. 10.0.0).
Note For Replicate to work, the parameter value must match the real version
of the database.
Step 2: Make sure that the required supplemental logging is added for each
table
1. If a Primary Key exists, supplemental logging must be added for the Primary
Key either by using the format to add supplemental logging on the Primary
Key, or by adding supplemental logging on the Primary Key columns.
2. If no Primary Key exists and the table has a single Unique Index, then all of the Unique
Index’s columns must be added to the supplemental log. Using SUPPLEMENTAL LOG
DATA (UNIQUE INDEX) COLUMNS does not add the Unique Index columns to the log.
3. If no Primary Key exists and the table has multiple Unique Indexes, Attunity Replicate
will select the first Unique Index. Attunity Replicate will use the first index in an
alphabetically ordered ascending list. Supplemental logging must be added on the
selected index's columns. Using SUPPLEMENTAL LOG DATA (UNIQUE INDEX)
COLUMNS does not add the Unique Index columns to the log.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 391
Note When the target table Primary Key/Unique Index is different than the
source table Primary Key/Unique Index, the user needs to add supplemental
logging manually on the source table columns that comprise the target table
Primary Key/Unique Index.
5. If you change the target table primary key, the supplemental logging must be
added on the selected index's columns instead of the columns of the original
primary key/unique index.
Note If ALL COLUMNS supplemental logging has been added to the table, there is
no need to add any additional logging.
If the table has a Unique Index or a Primary Key, you also need to add supplemental
logging on each column that is involved in a filter or transformation (if those columns
are different than the Primary Key or Unique Index columns).
Note If a transformation uses only one column, this column may not be added to
a supplemental logging group. For example, "A+B" needs both columns to be
added, whereas substring(A, 10) does not need "A" to be added.
One method of setting up both Primary Key/Unique Index supplemental logging and
supplemental logging on specific columns is to add USER_LOG_GROUP supplemental
logging only on the Primary Key/Unique Index columns and on the columns that are
filtered or transformed.
For example, to replicate a table named EXAMPLE.TABLE with Primary Key ID and filter
by column NAME, you can run a command similar to the one below to create the log
group supplemental logging:
ALTER TABLE EXAMPLE.TABLE ADD SUPPLEMENTAL LOG GROUP example_log_group
(ID,NAME) ALWAYS;
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 392
Step 4: When the Insert the missing target record Apply Conflicts option is
selected, supplemental logging must be enabled for ALL the source table
columns.
Although not required, examples of additional commands that you can execute to
change the supplemental logging attributes include:
exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD','ALL');
exec rdsadmin.rdsadmin_util.alter_supplemental_logging('DROP','PRIMARY
KEY');
Setting Up Archiving
To retain archived redo logs of your Oracle database instance (which will allow
Attunity Replicate to retrieve the log information using Oracle LogMiner), execute the
following command (example 24 hours):
exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention
hours',24);
Make sure that your storage has sufficient space for the archived redo logs during
the specified period.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 393
Setting General Connection Properties
This section describes how to configure general connection properties. For an
explanation of how to configure advanced connection properties, see Setting
Advanced Connection Properties below.
Note
Oracle can also be used as a target database. For information on using Oracle as
a target, see Setting General Connection Properties.
You can also use Oracle files as a source or target. For more information,
see Using a File as a Source.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. Type the Oracle Connection String for the Oracle database you want to work
with. You can type the connect string in any Oracle format, for example:
//host:port/service name
Where:
host: This is the name or IP address for the computer with the Oracle
database that you are using. For example, johnboy_W7 or 255.255.255.0.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 394
port: (optional) This is the TNS Listener Port number for the computer with
the Oracle database that you are using. If you do not enter a port number the
default Oracle TNS Listener port is used.
service name: (optional) This is the service name for the computer with
the Oracle database you are using. If you do not enter a service name the
default service name is used.
You can also enter an Oracle Net keyword-value pair. For example:
"(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=dlsun242)
(PORT=5521)) (CONNECT_DATA=(SERVICE_NAME=bjava21)))"
If the Oracle endpoint is configured to use Attunity Log Reader and the node to
which Replicate is connected cannot access the logs created by the other cluster
nodes, you need to specify a separate connection string for each RAC instance.
When the redo logs are stored in ASM, the connection string syntax is as follows:
[<common ASM connection string>,] <thread id> <thread ASM connection
string>, <thread id> <thread ASM connection string>...
Note If no <common ASM connection string> is specified, all the RAC instances
should be defined in the ASM connection.
When using Attunity Log Reader to access the redo logs, the connection string
syntax is as follows:
<Oracle connection string>[, <thread id> <thread BFILE connection
string>, <thread id> <thread BFILE connection string> ...]
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 395
7. Type the Oracle authentication information (User Name, Password) for the
authorized user for this Oracle database. If you do not know this information, see
your Oracle database Administrator (DBA).
Important: Make sure that the Oracle user entered in the Oracle
Authentication section has the correct access privileges. For information on
how to provide the required privileges, see Required Permissions.
Note For guidelines on choosing which redo logs access method to use, see Redo
Log Files - Access Method Guidelines.
Note If your Oracle database version precedes 10.2.0.4 (i.e. version 10.1.x
to 10.2.0.3), you must use Oracle LogMiner to access the redo logs.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 396
Automatically add supplemental logging: Select this (the default)
to automatically set up supplemental logging for the Oracle database.
For more information on supplemental logging, see Setting up Supplemental Logging.
Under the Access redo logs via label, choose Oracle LogMiner. Changes will
be captured using the Oracle LogMiner utility.
Secret Store encryption entries: When some of the columns in the tables that
you intend to replicate are encrypted you need to specify the Oracle Wallet
encryption keys and their values.
See also: Finding the Wallet Entry used for TDE Column Encryption in a Specific
Table and Finding the Wallet Entries used for TDE Encryption.
Retry interval: Use the counter or type the number of seconds that the system
waits before resending a query.
Archived redo logs destination identifier: The destination of the archived
redo logs. The value should be the same as the DEST_ID number in the
V$archived_log table.
Note When working with multiple log destinations (DEST_ID), you should
specify an Archived redo logs location identifier that represents archived
logs that can be accessed by Replicate. If the Archived redo logs location
identifier is not specified, Replicate will use the minimal existing DEST_ID.
Note The "Expose NUMBER" definition in the Oracle database is used for the
NUMBER data type only (without the explicit precision and scale definition).
Use archived redo logs only: When this option is selected, Attunity Replicate will
only access the archived redo logs. If the archived redo logs ares stored on ASM
only, the Attunity Replicate user needs to be granted the ASM privileges described
in Required ASM Privileges.
Support nested tables: Select this option if you need to replicate Oracle tables
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 397
containing columns that are nested tables or defined types.
For more information on this feature and its prerequisites, see Replicating
Nested Tables.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Finding the Wallet Entry used for TDE Column Encryption in a Specific Table
This section describes how to find the correct encryption key used for TDE
column encryption in a specific table.
1. On the Oracle database, run the following query to return the object_id (e.g.
the. table ID) according to a given owner and table name:
Select object_id from all_objects where owner='<table owner>'
and object_name='<table name>' and object_type='TABLE';
2. Use the retrieved object_id in the following query to return the relevant master
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 398
mkstore –wrl <full_wallet_name> -viewEntry <entry_name>
Note For more information, see Step 5 in Finding the Wallet Entries used for
TDE Encryption.
4. Copy the master key entry and its value into the Names and Values
fields respectively.
Note If your Oracle database version precedes 10.2.0.4 (i.e. version 10.1.x
to 10.2.0.3), you must use Oracle LogMiner to access the redo logs.
Under the Access redo logs via label, choose Attunity Log Reader (the default).
Replicate will access the redo logs as a binary file.
Secret Store encryption entries: When the source tables are encrypted or
contain encrypted columns, you need to specify the Oracle Wallet encryption keys
and their values.
For information on locating the required keys, see Finding the Wallet Entries used
for TDE Encryption.
ASM Parameters (if redo logs are stored in ASM) - If the Oracle redo logs you
are using are stored using Automated Storage Management (ASM), enter the
required access information in the designated fields.
Note To access the redo logs in ASM, you also need to grant the
additional privileges described in Required ASM Privileges
See also: Best Practices when Working with Oracle ASM.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 399
ASM Connection String: The connection string to the ASM instance if
your Oracle database is using ASM.
ASM user name: The user name for the ASM user.
Replace path prefix: You can determine whether to read the redo logs from
a different root location while leaving the relative path unchanged. Continue
from Replacing the Path with a Prefix.
Replicate has file-level access to the redo log files: Select this to access and
read the redo logs directly from the file system of the local computer where
Attunity Replicate is installed.
Copy redo logs to a temporary folder: Select this to copy the redo logs to a
temporary folder and then specify the path of the redo logs on the Oracle machine.
Note When configuring multiple tasks that use the same temporary folder
(configured in the Oracle source endpoint), do not select the Delete
processed archived redo log files option. This is because Replicate uses
the original archived log names.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 400
Note When working in a RAC environment, it is strongly recommended to set up
a shared folder that is accessible by all the RAC instances. If this is not possible,
you need to define a temporary folder with the same name on each of the RAC
instances. In addition, you need to define separate Oracle and ASM connection
strings for each RAC instance.
For more information on defining RAC connection strings, see Setting
General Connection Properties.
Look for missing archived redo logs in folder: Type the full path to a location
from where you want Attunity Replicate to read the archived redo logs if they are
not found in the default location. The folder can be located anywhere in the network
where Attunity Replicate is located, but be sure that the location is accessible to the
Attunity Replicate user.
Replicate has file-level access to the specified folder: Select this to
access and read the archived redo logs directly from the file system of the
local computer where Attunity Replicate is installed.
Delete processed archived redo log files: Select this to delete the copied
archived redo log files after they have been read. This option requires the
following additional permissions for the Replicate user:
GRANT SELECT ON DBA_FILE_GROUPS
Example:
GRANT SELECT ON DBA_FILE_GROUPS to nonpriv_user;
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 401
Example:
GRANT EXECUTE ON SYS.DBMS_FILE_GROUP to nonpriv_user;
Note Verify that another file group is not using the configured temp
directory under a different Oracle user.
Retry interval: Use the counter or type the number of seconds that the system
waits before resending a query.
Archived redo logs destination identifier: The destination of the archived
redo logs. The value should be the same as the DEST_ID number in the
V$archived_log table.
Note When working with multiple log destinations (DEST_ID), you should
specify an Archived redo logs location identifier that represents archived
logs that can be accessed by Replicate. If the Archived redo logs location
identifier is not specified, Replicate will use the minimal existing DEST_ID.
Note The "Expose NUMBER" definition in the Oracle database is used for the
NUMBER data type only (without the explicit precision and scale definition).
Use archived redo logs only: When this option is selected, Attunity Replicate will
only access the archived redo logs. If the archived redo logs ares stored on ASM
only, the Attunity Replicate user needs to be granted the ASM privileges described
in Required ASM Privileges.
Support nested tables: Select this option if you need to replicate Oracle tables
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 402
containing columns that are nested tables or defined types.
For more information on this feature and its prerequisites, see Replicating
Nested Tables.
Replace path prefix: You can determine whether to read the redo logs from
a different root location while leaving the relative path unchanged.
Type the first part of the path to the current location of the redo logs. For
example, C:\OldFolder.
You can include one folder or directory level or multiple folders or directories in
this field.
With: Type the name of the folder or prefix to replace the existing prefix that
you added in the field above. For example, C:\NewFolder.
In this case, the new folder or directory called companyName replaces all of the
first three level folders that you included in the Replace path prefix field.
Apply prefix replacement to online and archived redo logs: Select this to
apply the prefix replacement to the online and archived redo logs.
Replicate has file-level access to the new location: Select this to
access and read the online and archived redo log files directly from the
file system of the local computer where Attunity Replicate is installed.
Apply prefix replacement to archived redo logs only: Select this to apply
the prefix replacement to the archived redo logs only (and not to the online
redo logs).
Replicate has file-level access to the original online location: Select this
to access and read the original online redo log files directly from the file
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 403
system of the local computer where Attunity Replicate is installed.
Replicate has file-level access to the new archive location: Select
this to access and read the archived redo log files directly from the file
system of the local computer where Attunity Replicate is installed.
Delete processed archived redo log files: Select this to delete the
copied archived redo log files after they have been read. This option
requires the following additional permissions for the Replicate user:
GRANT SELECT ON DBA_FILE_GROUPS
Example:
GRANT SELECT ON DBA_FILE_GROUPS to nonpriv_user;
Example:
GRANT EXECUTE ON SYS.DBMS_FILE_GROUP to nonpriv_user;
Note Verify that another file group is not using the configured
temp directory under a different Oracle user.
Retry interval: Use the counter or type the number of seconds that the system
waits before resending a query.
Archived redo logs destination identifier: The destination of the archived
redo logs. The value should be the same as the DEST_ID number in the
V$archived_log table.
Note When working with multiple log destinations (DEST_ID), you should
specify an Archived redo logs location identifier that represents archived
logs that can be accessed by Replicate. If the Archived redo logs location
identifier is not specified, Replicate will use the minimal existing DEST_ID.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 404
Attunity Replicate supports any precision-scale combination supported by Oracle.
By default, the NUMBER data type is converted to precision 38, scale 10.
Note The "Expose NUMBER" definition in the Oracle database is used for the
NUMBER data type only (without the explicit precision and scale definition).
Use archived redo logs only: When this option is selected, Attunity Replicate will
only access the archived redo logs. If the archived redo logs ares stored on ASM
only, the Attunity Replicate user needs to be granted the ASM privileges described
in Required ASM Privileges.
Support nested tables: Select this option if you need to replicate Oracle
tables containing columns that are nested tables or defined types.
For more information on this feature and its prerequisites, see Replicating
Nested Tables.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 405
Finding the Wallet Entries used for TDE Encryption
In order to specify the correct encryption key(s) used for TDE tablespace encryption or
TDE column encryption, you first need to find the relevant entry (or entries in the case of
multiple keys) in the Oracle Wallet containing the encryption key(s). After you find the
relevant entry or entries, copy the entry and its value (or entries and values if more than
one) into the Names and Values fields respectively.
Note To enter multiple values, first copy each entry into a text editor such as
Notepad making sure to separate the values with a comma. Then, copy the string
containing the values and commas from the text editor and paste it into the Values
field. There is no need to do this for entries. You can paste the entries directly into
the Entries field, remembering to separate each entry with a comma.
4. Use the “list” option in the Oracle mkstore utility to determine the
ORACLE.SECURITY.DB/TS.ENCRYPTION.<SUFFIX> entry name(s), as
follows: mkstore –wrl <full wallet name> -list
5. If you know which entry/entries is/are used to encrypt the Redo logs, select the
entry name(s) and use the “viewEntry” option in the Oracle mkstore utility to
determine the entry value, as follows:
mkstore –wrl <full wallet name> -viewEntry <entry name>
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 406
Note If you do not know which entry is used to encrypt the Redo logs, you can
select multiple DB or TS entries and determine their values as described above
(and then copy and paste the entry names and values into the Names and
Values fields as described in the Finding the Wallet Entries used for TDE
Encryption). If the specified entries are not correct, the task will fail and the
error message will contain the correct entry name.
Note If the DBA changes the entry while the task is running, the task will fail
and the error message will contain the new entry name. Add the new entry
(name and value) to the already specified entries and then resume the task.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 407
Using PostgreSQL as a Source
This section describes how to set up and use a PostgreSQL database as a source
in a replication task.
In this section:
▶ Source Prerequisites
▶ Client Side
▶ Server Side
▶ Required Permissions
▶ Using an Account without the "superuser" Role
▶ Source Limitations
▶ PostgreSQL Source Data Types
▶ Homogeneous Replication
▶ Replication of Partitioned Tables
▶ Data Type Considerations and Exceptions
▶ Column and Table Collation
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Artifacts Created by Replicate in the Database
Source Prerequisites
The following section lists the prerequisites for working with Attunity Replicate and
a PostgreSQL database source.
Client Side
Attunity Replicate Server for Windows:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 408
Note Make sure that the PostgreSQL ODBC installation folder "bin" (e.g.
"C:\Program Files\psqlODBC\0905\bin") is added to the system PATH.
Make sure that the /etc/odbcinst.ini file contains an entry for PostgreSQL,
as in the following example:
[PostgreSQL]
Driver = /usr/pgsql-9.4/lib/psqlodbc.so
Setup = /usr/pgsql-9.4/lib/psqlodbcw.so
Debug = 0
CommLog = 1
UsageCount = 2
When the Apply Changes task option is enabled, the user specified in the PostgreSQL
source database’s General tab must be granted super-user permissions.
Server Side
The IP address of the Attunity Replicate machine must be added to the pg_hba.conf
configuration file with the "replication" keyword in the database field.
Example:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 409
host replication all 176.123.1.212/32 trust
Make sure that the test_decoding output plugin (found in the postgresql94-
contrib package) is installed.
The following parameters and values must be set in the
postgresql.conf configuration file.
wal_level = logical
max_replication_slots >=1
The max_wal_senders parameter sets the number of concurrent tasks that can run.
http://www.postgresql.org/docs/9.4/static/runtime-config-replication.html
Required Permissions
The user specified in the General tab when Setting General Connection Properties must
be granted the following permissions in the PostgreSQL database:
For Full Load replication: Standard SELECT on the source database
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 410
Using an Account without the "superuser" Role
If you are not using an account with the "superuser" role, you need to create several
objects to capture Data Definition Language (DDL) events. Create these objects in the
selected account and then create a trigger in the master user account.
To do this:
1. Choose a schema where you want the objects to be created. The default schema
is public. Ensure that the schema exists and is accessible by the NoPriv account.
2. Create the table attrep_ddl_audit by running the following command:
(
c_key bigserial primary key,
c_time timestamp, -- Informational
c_user varchar(64), -- Informational: current_user
c_txn varchar(16), -- Informational: current transaction
c_tag varchar(24), -- Either 'CREATE TABLE' or 'ALTER TABLE' or
'DROP TABLE'
c_oid integer, -- For future use - TG_OBJECTID
c_name varchar(64), -- For future use - TG_OBJECTNAME
c_schema varchar(64), -- For future use - TG_SCHEMANAME. For now,
holds the current_schema
c_ddlqry text -- The DDL query associated with the current DDL
event)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 411
default,current_timestamp,current_user,cast(TXID_CURRENT()as
varchar(16)),tg_tag,0,'',current_schema,_qry
);
delete from <objects_schema>.attrep_ddl_audit;
end if;
END;
$$;
4. If you are logged in with a NoPriv account, log out of the NoPriv account and log
in with an account that has the superuser role assigned to it.
Source Limitations
The following limitations apply when using PostgreSQL as a source:
LANGUAGE plpgsql
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 412
AS $$
BEGIN
$$;
Partitioned tables: When performing a Full Load replication of partitioned tables, the
parent table will be created and separate tables will be created for each partition.
A DML on the parent partitioned table will be applied on the physical partition of
the corresponding target table only.
Primary keys are not supported on partitioned tables, which may impact
UPDATE operations.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 413
Table 8.34 |
Supported PostgreSQL Source Data Types with Mapping to
Attunity Replicate Data Types
SMALLINT INT2
BIGINT INT8
NUMERIC
STRING
REAL REAL4
DOUBLE REAL8
SMALLSERIAL INT2
SERIAL INT4
BIGSERIAL INT8
MONEY NUMERIC(38,4)
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 414
Table 8.34 | Supported PostgreSQL Source Data Types with Mapping to Attunity
Rep-licate Data Types (Cont.)
unboundedVarcharMaxSize
See also Internal Parameters.
TEXT NCLOB
BYTEA BLOB
TIMESTAMP DATETIME
Note Replicate only supports ISO formatted textual DATE formats (the default). If
other formats are used, an error will be generated. You can change the date format
in the postgresql.conf file or using the PGDATESTYLE environment variable. You
can also change the date format at database level.
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 415
Table 8.34 | Supported PostgreSQL Source Data Types with Mapping to Attunity
Rep-licate Data Types (Cont.)
UUID STRING
TSVECTOR CLOB
TSQUERY CLOB
XML CLOB
JSON NCLOB
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 416
Table 8.34 | Supported PostgreSQL Source Data Types with Mapping to Attunity
Rep-licate Data Types (Cont.)
COMPOSITE NCLOB
WSTRING (LENGTH)
If no length is specified:
WSTRING (8000)
TINTERVAL WSTRING(255)
Homogeneous Replication
When replicating from a PostrgreSQL source to a PostrgreSQL target, most of the source
and target data types will be identical. The exceptions are listed in the table below.
Note In homogeneous replication, the source data first passes through the Attunity
Replicate data type and is therefore subject to any limitations of that type.
For information on Replicate data types and their limitations (where relevant),
see Replicate Data Types.
For information on which Replicate data types the source data passes through
when replicating from PostgreSQL, see the PostgreSQL to Attunity Replicate data
types mapping table described earlier.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 417
separate task to replicate to those tables. In such a case, the task settings should
be configured to “Truncate before loading”. For more information on the “Truncate
before loading” option, see Full Load Settings.
Table 8.35 |
Data Type Exceptions in Homogeneous Replication
PostgreSQL Source PostgreSQL Target
ENUM STRING
COMPOSITE STRING
A numeric array with precision and scale. A numeric array without precision and scale.
See also the note about JSON data type replication in PostgreSQL Source Data Types.
Note To support collation replication, the DBA must ensure that the collations
defined for the source PostgreSQL database are the same as those defined for the
target PostgreSQL database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 418
To add a PostgreSQL endpoint source database to Attunity Replicate:
1. In the Attunity Replicate console, click Add database to open the Add
Endpoints dialog box. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your PostgreSQL database. This can be any
name that will help to identify the database being used.
3. In the Description field, type a description that helps to identify the
PostgreSQL database. This is optional.
4. Select SOURCE as the database role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. Type the Server name. This is the name or IP address of the computer with
the PostgreSQL database that you want to access.
7. Optionally, change the default port (5432).
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 419
Important: Make sure that the PostgreSQL database user entered in the
PostgreSQL database Authentication section has the correct access privileges.
9. Type the database name or select one from the list of available endpoints. This is
the name of the PostgreSQL database from which you are replicating data.
10. Click OK to save your settings and close the dialog box.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 420
Setting Advanced Connection Properties
In the Advanced tab, you can set the following properties:
Capture DDLs: When this option is selected, the following actions occur:
Operational artifacts are created (by Replicate) in the database when the task
starts. In order to capture DDL events, Attunity Replicate creates various artifacts
in the PostgreSQL database when the task starts. You can later remove these
artifacts as described in Artifacts Created by Replicate in the Database.
Streamed DDL events are captured.
Create DDL artifacts in schema: The schema in which the operational
DDL database artifacts will be created. The default value is "Public".
WAL heartbeat - An Apply Changes task that is running but not capturing
changes (due to source table inactivity) will continue to occupy the LSN position in
its replication slot, thereby preventing truncation of the WAL. Since the WAL is a
server-wide resource used by all PostgreSQL processes, it may grow extremely
large if no changes are captured for an extended period.
To prevent this from happening, enable the "WAL heartbeat" option. When this
option is enabled, the PostgreSQL source endpoint mimics task activity by
periodically committing pseudo transactions (i.e. "Heartbeats") to the heartbeat
table, thereby advancing the task slot’s LSN position.
Create WAL heartbeat table in schema: The schema in which the WAL
heartbeat table (attrep_wal_heartbeat) will be created. The default value
is "public".
Heartbeat frequency (minutes): The frequency with which to
commit transactions to the heartbeat table.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 421
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Note that the event trigger does not belong to a specific schema.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 422
Using SAP Application as a Source
This section describes how to define a SAP Application as a source endpoint in a
replication task.
In this section:
▶ Prerequisites
▶ Supported SAP Packages
▶ Set up a Source Endpoint for your SAP Application
▶ Install the SAP NetWeaver RFC Client
▶ Install the Attunity Replicate for SAP Client on the SAP Machine
▶ Managing Business Groups and Tables
▶ Target Collation
▶ Limitations
▶ SAP Application Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the prerequisites for working with the Attunity
Replicate SAP Application endpoint.
Supported SAP Packages
Set up a Source Endpoint for your SAP Application
Install the SAP NetWeaver RFC Client
Install the Attunity Replicate for SAP Client on the SAP Machine
Managing Business Groups and Tables
Target Collation
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 423
See also Set up a Source Endpoint for your SAP Application.
SAP HANA
Note Replicate supports NetWeaver RFC SDK 7.20 and 7.50, which can be
downloaded from the SAP Service Marketplace.
Windows: Extract the contents of the NWRFC_xxx.SAR file and then copy the
files from the nwrfcsdk/lib directory .dll to the Replicate bin directory.
Linux: Extract the contents of the NWRFC_xxx.SAR file and then copy the .so
files from the nwrfcsdk/lib directory to the Replicate lib directory.
Install the Attunity Replicate for SAP Client on the SAP Machine
This section describes how to install the transports that make up the Attunity Replicate
for SAP Client.
There are five transports in total, which are provided in the following ZIP files:
1. DeleteCode.zip - Required for uninstalling the Attunity Replicate for SAP Client.
2. DeletePackage.zip - Required for uninstalling the Attunity Replicate for SAP Client.
3. InstallCode.zip (e.g. K902086.ESD) - The main transport.
4. InstallCodeECC.zip (e.g. K901271.ESD) - A transport with additional logic for ECC SAP
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 424
systems .
5. InstallConfig.zip (e.g. K900012.R4S) - The configuration transport.
IMPORTANT The transports must be installed in the order they appear below.
Installing the transports in the incorrect order or omitting/skipping any of the
transports will result in unexpected issues with replication tasks.
Note If you are applying a patch or upgrading the Attunity Replicate for SAP Client,
you should only install the main transport and the ECC-specific transport (if upgrading
on an ECC system). Do not install the configuration transport again, or any
customizations made to the configuration will be lost.
Replicate for SAP delivers its own authorization object: ZR4SAP. In addition to this
authorization object, there are additional authorizations that need to be enabled for
the Attunity Replicate software.
A dialog user in SAP is required to access the Attunity Replicate for SAP Client GUI in
SAP. In addition, a communication user is required to support the RFC calls from the
Attunity Replicate software to the SAP system.
Identify existing users in SAP or create dedicated users for the Attunity Replicate software.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 425
Authorizations for Replicate
Both the dialog and communication users will need to be assigned to a role
with authorization object S_TCODE and value ZR4SAP.
The communication user will also require the following authorization objects: S_RFC
and S_OC_SEND.
There are two types of files required to import the ABAP objects into the SAP system:
the data-file and the co-file.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 426
Importing the Data-file
1. Log on to the system at the Operating System level as the <sid> adm.
2. Change the directory to /usr/sap/trans
3. Add the transport to the R/3 buffer with the following command:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 427
# tp ‘addtobuffer SID’
4. Import the transport to the target R/3 system with the following command:
# tp ‘import SID client=000 U16’
The expected result of the addtobuffer step is a successful return code of `0’.
If problems occur during the addtobuffer step, it is likely there is a problem with the files.
They may be missing, in the wrong location, or have incorrect ownership or permissions.
The expected result of the import step is a successful return code of either `0’ or `4’. A
return code of `8’, `12’ or `16’ indicates transport failure. Return codes higher than
`16’ indicate a major failure within the transport tool. If this occurs, check the present
working directory to ensure the correct location. Also, check the files for existence,
location, and proper ownership and access.
If problems exist during the import, retry the import step. If problems persist, check
the import and activation logs for failure reason. These files are in the
/usr/sap/trans/log location and named.R6U (the `?’ stands in as a wildcard).
Beginning in R/3 version 4.0, SAP allows importing transports through the SAP system
via transaction code STMS.
Note Security authorization in the SAP system must include proper access to
import the transport request.
5. Add the Transport to the import queue by selecting the following path from the
menu list:
Extras > Other Requests > Add
Add the transport request number to the proper import queue and execute. Re-
authentication of the user.s SAP User ID is likely in order to complete the step.
If an Information message is received that the "Transport request is invalid" check that
the transport number was typed correctly. Otherwise, it may indicate a problem
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 428
with the files. Verification of existence, location, permissions, or ownership may
be needed.
6. Import the Transport request by selecting the transport number from the queue,
and clicking the Import (Truck) icon from the toolbar. Set the target client to
either `000’ or any other valid client within the system and execute with the truck
icon. Once again, re-authentication of the SAP User ID may be necessary. The
transport will execute in asynchronous mode; a record of success or failure can be
found in the transport logs.
7. The system will return to the import queue screen, where the Transport results can
be checked. Select the Logs icon from the toolbar, or follow the menu path:
Request > Display > Logs
Locate the source system and verify all relevant logs. For this transport there
should be 5 logs:
DD Import
DD Activation
Import
Check Versions
ABAP/scrn. Generation
All logs should display Ended OK (return code 0) or Ended with warning (return
code 4). If any logs are missing, or display a return code of 8 or higher, follow the
instructions in step 6 to re-import the transport.
Note If you are applying a patch or upgrading the Attunity Replicate for SAP Client,
you should only install the main transport and the ECC-specific transport (if upgrading
on an ECC system). Do not install the configuration transport again, or any
customizations made to the configuration will be lost.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 429
To upgrade or patch the Attunity Replicate SAP Client on non-ECC systems
Note Do not uninstall the Replicate for SAP Client if you are also running Attunity
Gold Client, as this will uninstall some components that are shared by both products.
3. Enter your credentials for logging in to the selected SAP Application Source.
4. Enter /nzr4sap in the drop-down list at the top of the console and then press [Enter].
2. Enter a name for your Business Group and then press [Enter].
The new Business Group is added to the Business Groups list in the left pane.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 430
To duplicate a Business Group
2. In the New Bus Object field, enter a name for the new Business Group and
then press [Enter].
The duplicated Business Group is added to the Business Groups list in the left pane.
Select the Business Group you want to delete and then click the Delete toolbar button.
Managing Tables
3. Click the button above the table list to enter Edit mode.
5. Enter the Table Name (i.e. the virtual ABAP table) and the name of the
corresponding Source Table (i.e. the physical table).
6. To save your changes click the Save button in the main toolbar.
3. Click the button above the table list to enter Edit mode.
4. Select the table you want to delete.
5. Click the
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 431
button that appears to the right of the button.
6. To save your changes click the Save button in the main toolbar.
Target Collation
As SAP is case-sensitive, when a Replicate task is defined with a SAP Application
source, the target endpoints need to be set up with case-sensitive collation.
Limitations
When using SAP Application as a source endpoint in a Replicate task, the
following limitations apply:
A task with a SAP Application source and a File Channel target may replicate some
tables twice - the requested table and the underlying table. To prevent this from
happening, exclude the underlying table from the list of tables to be replicated.
When a task is defined with a SAP Application source, the Applied Changes
Details monitoring metrics in the Change Processing tab may be incorrect for
clustered and pooled tables.
Replication of views is not supported.
Table 8.36 |
Supported SAP Data Types with Mapping to Attunity Replicate Data Types
ABAB ABAB Type Description SAP Type Attunity Replicate
Type Data Type
h Table type BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 432
Table 8.36 | Supported SAP Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
INT2 (2-byte I2
integer)
INT1 (1-byte I1
integer)
NUMERIC
If length = 0:
BLOB
If length > 0:
BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 433
Table 8.36 | Supported SAP Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 434
6. From the Connection type drop-down list, select Type A.
7. In the Server name field, enter the IP address of the Application Server on which
the SAP Application source is located.
8. In the Instance number field, enter the instance number of the SAP
Application source you want to replicate.
9. In the Client field, enter the System ID of the SAP Application source you want
to replicate.
10. Enter your credentials (User Name, Password) for accessing the SAP
Application source.
Note: These are the credentials for the communication user created earlier in SAP.
11. In the Backend endpoint field, click the Browse button and then select the name
of the Attunity Replicate endpoint you configured earlier. See also Set up a Source
Endpoint for your SAP Application.
10. In the Message server service field, enter the name of the SAP message
server service as specified in the following file:
<system drive>:\WINDOWS\system32\drivers\etc\services
If you do not specify a value, the Data Provider for SAP uses the following
default name:
sapms<R/3 system name>
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 435
11. In the Client field, enter the System ID of the SAP Application source you want
to replicate.
12. Enter your credentials (User Name, Password) for accessing the SAP
Application source.
Note: These are the credentials for the communication user created earlier in SAP.
13. In the Backend endpoint field, click the Browse button and then select the name
of the Attunity Replicate endpoint you configured earlier. See also Set up a Source
Endpoint for your SAP Application.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 436
Using SAP Application (DB) as a Source
This section describes how to define SAP Application (DB) as a source endpoint
in a replication task.
You can use the SAP Application (DB) endpoint to replicate from a SAP Application with
one of the supported backend endpoints.
In this section:
▶ Prerequisites
▶ Set up a Source Endpoint for your SAP Application (DB)
▶ Install the Attunity Replicate for SAP Client on the SAP Machine
▶ Managing Business Groups and Tables
▶ Target Collation
▶ Limitations
▶ SAP Application (DB) Source Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the prerequisites for working with the Attunity
Replicate SAP Application (DB) endpoint.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 437
Microsoft SQL Server
Install the Attunity Replicate for SAP Client on the SAP Machine
This section describes how to install the transports that make up the Attunity Replicate
for SAP Client.
There are five transports in total, which are provided in the following ZIP files:
1. DeleteCode.zip - Required for uninstalling the Attunity Replicate for SAP Client.
2. DeletePackage.zip - Required for uninstalling the Attunity Replicate for SAP Client.
3. InstallCode.zip (e.g. K902086.ESD) - The main transport.
4. InstallCodeECC.zip (e.g. K901271.ESD) - A transport with additional logic for ECC
SAP systems .
5. InstallConfig.zip (e.g. K900012.R4S) - The configuration transport.
IMPORTANT The transports must be installed in the order they appear below.
Installing the transports in the incorrect order or omitting/skipping any of the
transports will result in unexpected issues with replication tasks.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 438
Note If you are applying a patch or upgrading the Attunity Replicate for SAP Client,
you should only install the main transport and the ECC-specific transport (if upgrading
on an ECC system). Do not install the configuration transport again, or any
customizations made to the configuration will be lost.
Replicate for SAP delivers its own authorization object: ZR4SAP. In addition to this
authorization object, there are additional authorizations that need to be enabled for
the Attunity Replicate software.
A dialog user in SAP is required to access the Attunity Replicate for SAP Client GUI in
SAP. In addition, a communication user is required to support the RFC calls from the
Attunity Replicate software to the SAP system.
Identify existing users in SAP or create dedicated users for the Attunity Replicate software.
Both the dialog and communication users will need to be assigned to a role
with authorization object S_TCODE and value ZR4SAP.
The communication user will also require the following authorization objects: S_RFC
and S_OC_SEND.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 439
Figure 8.7 | Authorization Object S_RFC:
There are two types of files required to import the ABAP objects into the SAP system:
the data-file and the co-file.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 440
Importing the Co-file
1. Log on to the system at the Operating System level as the <sid> adm.
2. Change the directory to /usr/sap/trans
3. Add the transport to the R/3 buffer with the following command:
# tp ‘addtobuffer SID’
4. Import the transport to the target R/3 system with the following command:
# tp ‘import SID client=000 U16’
The expected result of the addtobuffer step is a successful return code of `0’.
If problems occur during the addtobuffer step, it is likely there is a problem with the files.
They may be missing, in the wrong location, or have incorrect ownership or permissions.
The expected result of the import step is a successful return code of either `0’ or `4’. A
return code of `8’, `12’ or `16’ indicates transport failure. Return codes higher than
`16’ indicate a major failure within the transport tool. If this occurs, check the present
working directory to ensure the correct location. Also, check the files for existence,
location, and proper ownership and access.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 441
If problems exist during the import, retry the import step. If problems persist, check
the import and activation logs for failure reason. These files are in the
/usr/sap/trans/log location and named.R6U (the `?’ stands in as a wildcard).
Beginning in R/3 version 4.0, SAP allows importing transports through the SAP system
via transaction code STMS.
Note Security authorization in the SAP system must include proper access to
import the transport request.
5. Add the Transport to the import queue by selecting the following path from the
menu list:
Extras > Other Requests > Add
Add the transport request number to the proper import queue and execute. Re-
authentication of the user.s SAP User ID is likely in order to complete the step.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 442
Locate the source system and verify all relevant logs. For this transport there
should be 5 logs:
DD Import
DD Activation
Import
Check Versions
ABAP/scrn. Generation
All logs should display Ended OK (return code 0) or Ended with warning (return
code 4). If any logs are missing, or display a return code of 8 or higher, follow the
instructions in step 6 to re-import the transport.
Note If you are applying a patch or upgrading the Attunity Replicate for SAP Client,
you should only install the main transport and the ECC-specific transport (if upgrading
on an ECC system). Do not install the configuration transport again, or any
customizations made to the configuration will be lost.
Note Do not uninstall the Replicate for SAP Client if you are also running Attunity
Gold Client, as this will uninstall some components that are shared by both products.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 443
Managing Business Groups and Tables
This prerequisite is only necessary if you want to edit the default Business Groups
and/or tables before replicating them to the target endpoint.
Before you can manage business groups and tables, you first need to launch the SAP
Client UI.
3. Enter your credentials for logging in to the selected SAP Application (DB) Source.
4. Enter /nzr4sap in the drop-down list at the top of the console and then press [Enter].
2. Enter a name for your Business Group and then press [Enter].
The new Business Group is added to the Business Groups list in the left pane.
2. In the New Bus Object field, enter a name for the new Business Group and
then press [Enter].
The duplicated Business Group is added to the Business Groups list in the left pane.
Select the Business Group you want to delete and then click the Delete toolbar button.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 444
Managing Tables
3. Click the button above the table list to enter Edit mode.
5. Enter the Table Name (i.e. the virtual ABAP table) and the name of the
corresponding Source Table (i.e. the physical table).
6. To save your changes click the Save button in the main toolbar.
3. Click the button above the table list to enter Edit mode.
4. Select the table you want to delete.
6. To save your changes click the Save button in the main toolbar.
Target Collation
As SAP is case-sensitive, when a Replicate task is defined with SAP Application
(DB) source, the target endpoints need to be set up with case-sensitive collation.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 445
Limitations
When using SAP Application (DB) as a source endpoint in a Replicate task, the
following limitations apply:
In order for changes to the following DDLs to take effect, you need to stop the
task before making the changes and then resume the task after the changes
have been made:
Creating a Business Group
Adding a table to a Business Group
Removing a table from a Business Group
Replicating from an STXL table is not supported when Microsoft SQL Server is
the backend database.
Transparent table fields defined as STRING or BINARY STRING are not supported
when Microsoft SQL Server is the backend database.
Table 8.37 |
Supported SAP Data Types with Mapping to Attunity Replicate Data Types
ABAB ABAB Type Description SAP Type Attunity Replicate
Type Data Type
h Table type BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 446
Table 8.37 | Supported SAP Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
INT2 (2-byte I2
integer)
INT1 (1-byte I1
integer)
NUMERIC
If length = 0:
BLOB
If length > 0:
BYTES
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 447
Table 8.37 | Supported SAP Data Types with Mapping to Attunity Replicate Data
Types (Cont.)
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Client field, enter the System ID of the SAP Application (DB) source you
want to replicate.
7. In the SAP schema field, enter the name of the relevant SAP schema.
8. In the Backend endpoint field, click the Browse button and then select the name
of the Attunity Replicate endpoint you configured earlier. See also Set up a Source
Endpoint for your SAP Application (DB).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 448
Setting Advanced Properties
In the Advanced tab, you can set internal parameters and view a summary of
the endpoint settings.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 449
Using SAP HANA as a Source
This section describes how to set up and use a SAP HANA endpoint as a source
in a replication task.
Note The SAP HANA endpoint can either be accessed directly or via the SAP
Application endpoint. For an explanantion of how to set up the SAP Application
endpoint, see Using SAP Application as a Source
In this section:
▶ Prerequisites
▶ Port
▶ Required Clients
▶ Change Processing
▶ Limitations
▶ Permissions
▶ Supported Data Types
▶ Unsupported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Properties
▶ Trigger-based CDC
▶ Log-based CDC
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 450
Prerequisites
Before SAP HANA can be used as a source endpoint in a replication task, the
prerequisites described in this section must be met.
Port
Open inbound port number 3xx15 to the SAP HANA server where xx is the instance
number of the SAP HANA database that contains the source tables.
For example, the port for instance 90 would be 39015.
Required Clients
Windows: Install the SAP HANA ODBC 64-bit Driver 2.x for Windows on the
Replicate Server machine. The driver name is HDBODBC.
Linux: Install the SAP HANA ODBC 64-bit Driver 2.x for Linux on the Replicate
Server machine. The driver name is HDBODBC.
Add the following section to the odbcinst.ini file located in directory /etc:
[HDBODBC]
Description=64-bit HANA ODBC Driver
Driver=/opt/sap/hdbclient/libodbcHDB.so
fileUsage=1
Change Processing
To be able to capture changes committed to the source tables, Replicate requires
the following artifacts to be created in the source database (either manually by the
DBA or automatically by Replicate):
Three triggers for each of the designated source tables: The triggers capture
changes to the source tables (INSERTs, UPDATEs, and DELETEs) and write them to
the Replicate attrep_cdc_changes table described below. A separate trigger is
required for each DML operation.
The triggers will be created in the source table schema.
The Replicate attrep_cdc_changes table: This table will contain changes captured by
the triggers and will either be created in the schema of the user specified in the SAP
HANA endpoint settings, or in the schema specified in the endpoint settings.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 451
Note As Replicate periodically scans the attrep_cdc_changes table for
changes, the number of UPDATEs shown in the monitor's Change Processing
tab reflects the number of UPDATEs found (in the attrep_cdc_changes table) at
the time of scanning. This may not be the actual number of UPDATEs performed
on the source since UPDATEs are sometimes merged with INSERTs (depending
on when the attrep_cdc_changes table was last scanned).
When defining the SAP HANA source endpoint, make sure that the Create CDC
Artifacts check box in the Advanced tab is selected (the default).
1. Define the task as you would any other Replicate task, but when configuring the
SAP HANA source endpoint, clear the Create CDC Artifacts check box in the
Advanced tab.
2. Once the task is defined, select any of the Metadata only options in the
Advanced Run Options window, and then click OK.
The task will run and generate the scripts for creating the CDC artifacts in
the following location and then stop:
~/data/tasks/<task_name>/scripts
Limitations
The following limitations apply when using SAP HANA as a source:
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 452
Note This limitation does not apply when accessing SAP HANA via the
SAP Application endpoint.
Permissions
The following are the minimum permissions required:
Grant SELECT and TRIGGER on the SAP schema containing the selected source
tables to the user specified in the SAP HANA source endpoint.
If the attrep_cdc_changes table is created in a schema owned by the user
specified in the SAP HANA source endpoint settings (the default), no further
permissions are required.
If the attrep_cdc_changes table is created in a schema not owned by the user
specified in the endpoint settings, grant SELECT, INSERT, UPDATE and DELETE on
the attrep_cdc_changes table to the specified user.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.38 |
Supported SAP HANA Data Types with Mapping to Attunity Replicate
Data Types
TIME TIME
SECONDDATE DATETIME
TIMESTAMP DATETIME
TINYINT UINT1
SMALLINT INT2
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 453
Table 8.38 | Supported SAP HANA Data Types with Mapping to Attunity Replicate
Data Types (Cont.)
BIGINT INT8
SMALLDECIMAL NUMERIC
DECIMAL NUMERIC
REAL REAL4
DOUBLE REAL8
VARCHAR STRING
NVARCHAR WSTRING
ALPHANUM STRING
SHORTTEXT WSTRING
VARBINARY BYTES
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
TEXT NCLOB
BOOLEAN UINT1
ARRAY
ST_GEOMETRY
ST_POINT
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 454
To add a SAP HANA source endpoint to Attunity Replicate:
You can do this step before any of the other steps if you want, however before you
can continue with the next step in this process, you must select the database role.
5. Select SAP HANA as the database Type.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server field, specify the IP address or host name of the SAP HANA
database server.
To connect to a High Availability Cluster, specify all of the cluster nodes and
port numbers in the Server field. The nodes should be separated by a comma.
Example:
12.12.1.123:3033,12.12.1.124:3034
7. In the Instance field, enter the instance number of the SAP HANA database with
the source tables.
8. Enter the Username and Password required to access the SAP HANA database. If you
do not know this information, see your SAP HANA database Administrator (DBA).
9. Click Test Connection to verify that the specified settings are correct.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 455
Setting Advanced Properties
In the Advanced tab, as well as choosing which method to use to capture changes from
the SAP HANA, you can also specify any internal parameters that may be required.
Trigger-based CDC
When Trigger-Based CDC is selected, the following options are available:
Create CDC artifacts: When this option is enabled (the default), Replicate
creates the required CDC artifacts in the source database. Clear this check box if
you would rather create the CDC artifacts manually.
Create CDC table in schema: The name of the schema in the source
database where you want the attrep_cdc_changes table to be created.
Note The specified schema must already exist in the source database.
Log cleanup interval (min): Specify how often (in minutes) to check if
Change Records need to be deleted from the attrep_cdc_changes table.
Log retention period (min): Specify how long (in minutes) to keep Change
Records in the attrep_cdc_changes table before deleting them.
Log-based CDC
Log-based CDC is faster than Trigger-based CDC and has a smaller system footprint. When
Log-based CDC is selected, Replicate will read the changes directly from the SAP HANA
logs. In order for Replicate to be able to do this, you need to specify the location of logs in
the Online logs and Backup logs fields respectively (both fields are mandatory).
Note that if Replicate Server is not installed on the SAP HANA machine, the online logs
and backup logs must reside in Shared Folders on the SAP HANA machine. The Shared
Folders must be accessible by Replicate, and Replicate must be granted read permission
on the them.
Note When this method is selected, Replicate creates a $rowid$ column on the
target as a Unique Index insteadof the original Primary Key/Unique Index. Also, as
well as the shared limitations, Log-based CDC does not support the following:
Multitenant containers
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 456
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 457
Using SAP Sybase ASE as a Source
This section describes how to set up and use a SAP Sybase ASE database as the
source endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Required Permissions
▶ SAP Sybase ASE database Source Data Types
▶ Non-Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Removing the Truncation Point
Prerequisites
Make sure the following prerequisites have been met:
SAP Sybase ASE ODBC 64-bit client installed on the computer where Attunity
Replicate is located.
SAP Sybase ASE replication enabled for tables using the sp_setreptable command
or privileges to enable it automatically.
RepAgent must be disabled on the SAP Sybase ASE database.
When replicating to SAP Sybase ASE 15.7 installed on a Windows machine
configured with a non-Latin language (e.g. Chinese), Attunity Replicate requires
Sybase 15.7 SP121 to be installed on the SAP Sybase ASE machine.
Turn off automatic truncation using the following command:
sp_dboption mydb, "trunc log on chkpt", false
go
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 458
Limitations
The following limitations apply:
Data capture from partitions is only supported from Sybase ASE 15 and above.
Required Permissions
To use SAP Sybase ASE database as a source in a Replicate task, the following
permissions are required:
sa_role
replication_role
sybase_ts_role
If the Automatically enable Sybase replication option is enabled (in the Advanced
tab), Replicate also needs permission to run the stored procedure sp_setreptable.
For information on the Automatically enable SAP Sybase ASE replication option,
see Setting Advanced Connection Properties.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 459
SAP Sybase ASE database Source Data Types
The following table shows the SAP Sybase ASE database source data types that are
supported when using Attunity Replicate and the default mapping from Attunity
Replicate data types.
For information on how to view the data type that is mapped in the target, see the
section for the target database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.39 | SAP Sybase ASE database Source Data Types with Mapping to Attunity
Replicate Data Types
SAP Sybase ASE Source Data Types Attunity Replicate Data Types
BIGINT INT8
INT INT4
SMALLINT INT2
TINYINT UINT1
DECIMAL NUMERIC
NUMERIC NUMERIC
FLOAT REAL8
DOUBLE REAL8
REAL REAL4
MONEY NUMERIC
SMALLMONEY NUMERIC
DATETIME DATETIME
SMALLDATETIME DATETIME
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 460
Table 8.39 | SAP Sybase ASE database Source Data Types with Mapping to Attunity
Rep-licate Data Types (Cont.)
SAP Sybase ASE Source Data Types Attunity Replicate Data Types
BIGTIME TIME
CHAR STRING
UNICHAR WSTRING
NCHAR WSTRING
VARCHAR STRING
UNIVARCHAR WSTRING
NVARCHAR WSTRING
BINARY BYTES
VARBINARY BYTES
BIT BOOLEAN
TEXT CLOB
UNITEXT NCLOB
IMAGE BLOB
UDT
Note You can also use SAP Sybase ASE files as a source. For more information,
see Using the Attunity Replicate File Channel.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 461
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the SAP Sybase
ASE database. This is optional.
4. Select SOURCE as the endpoint role.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. In the Server Name field, enter the host name or IP address of the computer
on which the SAP Sybase ASE database is installed.
You can use the Advanced tab to add specific properties and create a
custom connect string. In this case, you do not need to enter information in
this tab. For more information on using the Advanced tab, see Setting
Advanced Connection Properties.
To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection
If the connection is successful a message in green is displayed. If the
connection fails, an error message is displayed at the bottom of the
dialog box.
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
8. Type the SAP Sybase ASE authentication information (User Name, Password) for
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 462
the authorized user for this SAP Sybase ASE database. If you do not know
this information, see your SAP Sybase ASE database Administrator (DBA).
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
If you want to set custom properties for this database, see Setting
Advanced Connection Properties.
Important: Make sure that the SAP Sybase ASE user entered in the SAP Sybase
ASE Authentication section has the correct access privileges. For information on
how to provide the required privileges, see Required Permissions.
9. In the Database name field, enter the SAP Sybase ASE database name.
Note If the user name or password specified in the General tab contains non-
Latin characters (e.g. Chinese), the following property is required:
charset=gb18030
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 463
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
After the truncation point has been removed, the Replicate task cannot be resumed. The
log will continue to be truncated automatically at the checkpoints (if automatic truncation
is set).
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 464
Using Teradata Database as a Source
This section describes how to set up and use Teradata Database as a source in a
replication task.
In this section:
▶ Prerequisites
▶ Replicate Server for Windows
▶ Replicate Server for Linux
▶ Required Permissions
▶ Teradata Source Data Types
▶ Setting General Connection Properties
▶ Setting Change Processing Parameters
▶ Prerequisites
▶ Limitations
▶ Configuring Change Processing Settings
Prerequisites
The following section describes the prerequisites for working with Attunity Replicate and
a Teradata Database Source.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 465
Note These instructions assume that the Teradata Database Client is installed
in the following location:
/opt/teradata/client/14.10
Important: A Replicate task cannot be defined with endpoints that use different
ODBC Driver Managers. Teradata Database source is accessed using the DataDirect
ODBC Driver Manager. With the exception of Oracle, Hadoop, File and Replicate
Connect sources (which are not subject to the above limitation), all other source
endpoints are accessed using the unixODBC Driver Manager.
To configure a task with a DataDirect source and a unixODBC target, you need to
use the Replicate File Channel. For more information about setting up a task using
the File Channel, see Using the Attunity Replicate File Channel.
Required Permissions
The user that is specified in the General tab when Setting General Connection
Properties must be registered as a user in the Teradata Database.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 466
Teradata Source Data Types
The following table shows the Teradata target data types that are supported when
using Attunity Replicate and the default mapping to the Attunity Replicate data types.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 8.40 |
Supported Teradata Source Data Types with Mapping to Attunity
Replicate Data Types
BYTE BYTES
BYTEINT INT1
BIGINT INT8
DATE DATE
DECIMAL REAL8
FLOAT REAL8
INTEGER INT4
CHAR STRING
CLOB CLOB
GRAPHIC STRING
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 467
Table 8.40 | Supported Teradata Source Data Types with Mapping to Attunity
Replicate Data Types (Cont.)
REAL REAL8
SMALLINT INT2
TIME TIME
TIMESTAMP DATETIME
VARBYTE BYTES
NUMERIC NUMERIC
1. In the Attunity Replicate console, click Add Database to open the Add
Endpoints dialog box. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your Teradata database. This can be any
name that will help to identify the database being used.
3. In the Description field, type a description that helps to identify the
Teradata database. This is optional.
4. Select SOURCE as the database role.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 468
5. Select Teradata Database as the database Type.
Note When this endpoint is used as duplicated source in a Log Stream Staging
setup, select the Read changes from log stream check box and then select
the relevant Log Stream Staging task from the drop-down list.
For information on setting up and managing Log Stream Staging tasks, see
Using the Log Stream.
6. Type the Server name. This is the name of the computer with the Teradata
Database instance you want to work with.
7. Type the Teradata Database authentication information (Username, Password)
for the authorized user for this Teradata Database. If you do not know this
information, see your Teradata Database system manager.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the Teradata Database user entered in the Teradata
Database Authentication section has the correct access privileges. For information
on how to provide the required privileges, see Required Permissions.
8. Type the Default database name or select one from the list of available endpoints.
This is the name of the Teradata Database where you are replicating the data to.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 469
then applies those changes to the target database. However, this method of change
processing is not possible with Data Warehouse endpoints such as Teradata Database
since these endpoints do not generate transaction logs.
The good news is that you can still use Replicate to capture changes from
Teradata Database - it just requires a little bit of preparation.
Prerequisites
Before you can define the settings in the Change Processing tab, you need to ensure
that at least one special "Context" column exists in your source database tables. Context
column(s) are basically columns in a table that enable Replicate to determine whether
the data has changed. You can add Context columns specifically for the purpose of
change processing (either using a script or manually) or you can use existing columns
that contain suitable "Context" data.
Note You can create and reference any number of Context columns in a table as
long as the Context column names are the same for all source tables. Additionally,
each value in the Context column(s) must be unique.
In the example below, the Context column cf has been added to the table. The cf column
contains TIMESTAMPs that enable Replicate to determine whether a change occurred (by
comparing the current TIMESTAMP with the TIMESTAMP stored in its repository).
By default, all changes are assumed to be INSERTs. If UPDATE and DELETE operations
are also performed on the source tables, you can write an UPDATE and/or DELETE
expression (described below) that will enable Replicate to identify the operation type.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 470
Figure 8.9 | Example of a Table with a Context Column
Limitations
The following limitations apply when Change Processing is enabled for the
Teradata Database source:
The "Start from timestamp" run option is not supported. For more information,
see Using Advanced Run Options.
If one of the Context columns is part of the Primary Key or Unique Index, then
UPDATE and DELETE operations are not supported.
Context columns cannot be LOB columns
DDLs are not supported
When inserting a record and then updating the same record, the task error
handling settings should be set as follows:
Open the <Task Name> Settings dialog box.
Select the Error Handling|Apply Conflicts tab.
Set a task-specific Apply Conflicts policy as described in Error Handling Settings.
From the No record found for applying an update drop-down list, select
INSERT the missing target record.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 471
To configure change processing settings:
2. In the Columns field, specify the names of the Context columns. The column
names are case-sensitive and must be separated by commas.
Example:
context1,context2
5. Enter expressions that Replicate will use to identify UPDATE and DELETE operations. If
you do not enter any expressions or if no match is found for an expression, any row
whose context is higher (if the sorting order is Ascending) or lower (if the sorting order
is Descending) than the previous context value will be considered an INSERT.
Note Expressions must be written in the native syntax of the Teradata Database
source. All examples in this section are written using PostgresSQL syntax.
Tip: [Selecting the UPDATE the existing target record option in the
Apply Conflicts tab, eliminates the need to provide an UPDATE expression.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 472
with parameters that are not exposed in the UI. As such parameters are normally
not required, they should only be used after consulting with Attunity Support.
Copyright © Attunity Ltd. Chapter 8 | Adding and Managing Source Endpoints | Page 473
9 | Adding and Managing
Target Endpoints
This topic describes how to configure target endpoint settings.
{Pass%^&rd}
In this chapter:
▶ Using Actian Vector as a Target
▶ Using Amazon EMR as a Target
▶ Using Amazon Kinesis Data Streams as a Target
▶ Using Amazon Redshift as a Target
▶ Using Amazon S3 as a Target
▶ Using a File as a Target
▶ Using Google Cloud SQL for MySQL as a Target
▶ Using Google Cloud SQL for PostgreSQL as a Target
▶ Using Google Cloud Storage as a Target
▶ Using Google Dataproc as a Target
▶ Using Hadoop as a Target
▶ Using Hortonworks Data Platform (HDP) as a Target
▶ Using HP Vertica as a Target
▶ Using IBM Netezza as a Target
▶ Using Kafka as a Target
▶ Using a Log Stream Target
▶ Using MapR Streams as a Target
▶ Using MemSQL as a Target
▶ Using Microsoft APS PDW as a Target
▶ Using Microsoft Azure ADLS as a Target
▶ Using Microsoft Azure Database for MySQL as a Target
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 474
▶ Using Microsoft Azure Database for PostgreSQL as a Target
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 475
Using Actian Vector as a Target
This section describes how to set up and use an Actian Vector database as a target
in a replication task.
In this section:
▶ Prerequisites
▶ Actian Vector Windows Environment Prerequisites
▶ Actian Vector Linux Environment Prerequisites
▶ Limitations
▶ Permissions
▶ Actian Vector Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
An Actian Vector database with the tables that are being used for replication must be
available to the system. Attunity Replicate supports using an Actian Vector database
on both Windows and Linux.
Actian Vector 3.0 must be installed on the same machine as Attunity Replicate Server.
From Actian Vector 3.5, with the addition of remote load support, Replicate and Actian
Vector do not need to be installed on the same machine. When Actian Vector is installed
on a remote machine (i.e. not on the Replicate machine), Client Runtime 3.5.1 or above
for Linux/Windows needs to be installed on the Replicate machine.
For more information about the requirements for working with Attunity Replicate,
see Installation Prerequisites.
The following sections describe the prerequisites necessary to prepare your
environment to work with Attunity Replicate and an Actian Vector database:
Actian Vector Windows Environment Prerequisites
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 476
Actian Vector Windows Environment Prerequisites
The following must be installed:
Actian Vector database on any computer in your network. Make sure to configure
the tables that you need for replication.
Actian Vector DBA Tools. This includes the Ingres ODBC driver 3.0 or above. You
must install the DBA tools on the same computer as the Actian Vector database.
For more information, see the Actian Web site.
Be sure to configure the Ingres ODBC driver. This driver supports unixODBC and is
included in the Actian Vector database installation. unixODBC is built in to most
Linux distributions.
Linux releases of Ingres include the ODBC CLI. The CLI is an ODBC manager/library
that is built specifically for Ingres. The CLI is installed on Linux platforms at the
same time as the ODBC driver. The CLI allows ODBC applications to be developed
and deployed without the need to manage or use unixODBC. Use the iiodbcadmin
tool to manage DSN definitions (or use a connection string that specifies the
database name). IIodbcadmin can be used with either unixODBC or CLI applications.
For information on configuring the ODBC driver, go to the following page on the
Actian Web site: http://community.actian.com/wiki/Ingres_UNIX_ODBC.
Limitations
The following DDL operations cannot be performed on the Actian Vector
database using Attunity Replicate:
Change schema name
The replication task will fail when applying DDL changes that were captured
from the source but not supported on the target.
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab section in Customizing Tasks .
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 477
Permissions
The Attunity Replicate user who is working with the Actian Vector database must be
registered as a user in the Actian Vector database. This is the user that is entered in the
dialog box when Setting General Connection Properties. You must grant Actian Vector
account access to this user before confiding the database in Attunity Replicate.
Note Actian Vector does not support applying changes to binary data types in Batch
optimized apply mode. For more information on Batch optimized apply mode,
see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.1 |
Supported Actian Vector Data Types with Mapping from Attunity
Replicate Data Types
VARCHAR (32000)
DATE ANSIDATE
TIME TIME
DATETIME TIMESTAMP
INT1 INTEGER1
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 478
Table 9.1 | Supported Actian Vector Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
INT2 SMALLINT
INT4 INTEGER
INT8 INTEGER8
REAL4 FLOAT4
REAL8 FLOAT
VARCHAR (Length)
VARCHAR (32000)
UINT1 SMALLINT
UINT2 INTEGER
UINT4 INTEGER8
UINT8 INTEGER8
NVARCHAR (Length)
NVARCHAR (16000)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 479
Table 9.1 | Supported Actian Vector Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
VARCHAR (32000)
VARCHAR (16000)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 480
3. In the Description field, type a description that helps to identify the
Greenplum database. This is optional.
4. Select TARGET as the database role.
6. In the Server field, enter the hostname or IP address of the machine on which
the Actian Vector database is installed.
Note You can use the Advanced tab to add specific properties and create a
custom connect string. In this case, you do not need to enter information in
this tab. For more information on using the Advanced tab, see Setting
Advanced Connection Properties.
7. Type the Actian Vector authentication information (User name, Password) for
the authorized user for this Actian Vector database. If you do not know this
information, see your Actian Vector system manager.
Note
If you are using the Advanced tab to create a custom string, make sure to
include the User Name property. A Password can also be included but is not
required. See Setting Advanced Connection Properties for more information.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 481
Setting Advanced Connection Properties
In the Advanced tab, you can set the following parameters:
Max file size (KB): Select or type the maximum file size. When the Actian Vector
file reaches this value, the data is loaded by the vmload utility. The default value is
300000 KB.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Note Hive can be configured to use the AWS Glue Data Catalog as its metastore or its
own metastore. The metastore that will be used depends on your Amazon EMR Cluster
configuration and requires no special configuration from a Replicate perspective.
In this section:
▶ Prerequisites
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 482
▶ Limitations
▶ Change Data Partitioning on Amazon EMR
▶ Amazon EMR Endpoint Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you begin to work with Amazon EMR as a target in Attunity Replicate, make
sure that the following prerequisites have been met:
General:
The Amazon S3 bucket you are using must be accessible from the
Attunity Replicate machine.
The user specified in the Amazon EMR target endpoint's Hive access settings
must have access to HiveServer2.
ODBC Driver when Replicate Server is running on Windows:
Install Amazon Hive ODBC driver 1.1.1.1001 on the Replicate Server machine.
1. Install Amazon Hive ODBC driver 1.1.1.1001 on the Replicate Server machine.
follows: DriverManagerEncoding=UTF-16
ODBCInstLib=libodbcinst.so
Permissions:
The Access type selected in the Amazon EMR endpoint's Storage settings
must have write access to the specified bucket folder.
Hive table permissions: Replicate requires permissions to perform the
following operations on Hive tables: CREATE, DROP, DESCRIBE, and ALTER
table. ALTER table may also include RENAME table, ADD/RENAME column,
and CREATE/DROP partitions on tables.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 483
Limitations
The following limitations apply:
UPDATE/DELETE DMLs are not supported during change processing. If an
UPDATE/DELETE DML was captured on the source, it will be ignored on the target
and a warning will be written to the log. If the Store Changes option is enabled in
the task settings, these records will be written to the Change Table.
Limited LOB support only.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 484
Amazon EMR Endpoint Target Data Types
The following table shows the Amazon EMR endpoint target data types that are supported
when using Attunity Replicate and the default mapping from Attunity Replicate data types.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.2 |
Supported Amazon EMR Data Types with Mapping from Attunity
Replicate Data Types
BYTES STRING
TIME TIMESTAMP
DATETIME TIMESTAMP
DATE DATE
INT1 TINYINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
REAL4 FLOAT
REAL8 DOUBLE
UINT1 SMALLINT
UINT2 INT
UINT4 BIGINT
BLOB STRING
NCLOB STRING
CLOB STRING
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 485
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 486
Setting General Connection Properties
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
a. In the Bucket name field, enter the name of your Amazon S3 bucket.
b. In the Bucket region drop-down list, select the Amazon S3 region where
your bucket is located.
c. From the Access type drop-down list, choose one of the following:
Key pair
Choose this method to authenticate with your Access Key and Secret Key.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
d. If you selected Key pair as your access method, in the Access key field,
enter the access key information for Amazon S3.
e. If you selected Key pair as your access method, in the Secret key field,
enter the secret key information for Amazon S3.
f. In the Target folder field, specify the target folder in your Amazon S3 bucket.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 487
Note Due to a Hadoop limitation, the Target folder name can only
contain ASCII characters.
None
Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3). This is
the default.
Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)
This option also requires you to specify your KMS Key ID.
http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
a. In the Host field, specify the public host name or IP address of the Hive server.
b. In the Port field, optionally change the default port.
c. From the Authentication type drop-down list, choose either Username
or Username and Password and then enter the required information in
the designated fields.
d. To access Hive using SSL, select Use SSL and then specify the full path to a
CA certificate file in PEM format in the CA path field.
e. In the Database field, specify the name of the Hive target database.
Table 9.3 |
Amazon EMR Target - Advanced Properties
Setting Description
File Format Expand this section to specify or view the file format
settings.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 488
Table 9.3 | Amazon EMR Target - Advanced Properties (Cont.)
Setting Description
Field delimiter The delimiter that will be used to separate fields in the target file. The
default is \001. Note that field delimiters must be in Octal format
(e.g. \247)
Null value The value that will be used to indicate a null value in the target file.
The null value is supported from Hive 0.13.
mike,male,295678
sara,female,@
Escape character The escape character is used to escape the field delimiter character.
When a field delimiter is escaped, it is interpreted as actual data, and
not as a field delimiter.
sunroof\,power-steering
"\"sunroof, power-steering\""
Add metadata When the target storage format is set to Text, you can optionally add
header a header row to the data files. The header row can contain the source
column names and/or the intermediate (i.e. Replicate) data types.
Example of a target file with a header row when both With column
names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 489
Table 9.3 | Amazon EMR Target - Advanced Properties (Cont.)
Setting Description
File Attributes Expand this section to specify or view the file attributes.
Maximum file size Specify the maximum file size of each target file. When the data
reaches the maximum size, the file will be closed and written to the
specified target folder.
File size Specify the minimum size of the data required to create a file in idle
reaches state.
Elapsed time Specify the maximum time to wait before applying the changes in idle
reaches state.
Note To facilitate rapid delivery of DDL messages, files are uploaded immediately,
regardless of the specified File size reaches or Elapsed time reaches values.
Use proxy server Select this option to access Amazon S3 via a proxy server.
User name The user name for accessing the proxy server.
Scheme Select which protocol to use to access the server (HTTP or HTTPS). In
order to use HTTPS, you must first install the CA certificate that
signed the proxy’s certificate on the Replicate Server machine, as
follows:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 490
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 491
Using Amazon Kinesis Data Streams as a Target
This section describes how to set up and use Amazon Kinesis Data Streams as a target
endpoint in a replication task. In a task with a Amazon Kinesis Data Streams target
endpoint, each source record is transformed into a message which is then written (with
an optional message key) to a shard in the specified stream.
In this section:
▶ Prerequisites
▶ Transaction Processing by the Consumer
▶ How it Works
▶ Transaction Consistency from a Consumer Perspective
▶ Limitations
▶ Supported Data Types
▶ Mapping from Attunity Replicate Data types to Avro
▶ Setting General Connection Properties
▶ Overriding the Default Settings
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ The Attunity Envelope
▶ Decoding a Self-Describing Message
▶ Decoding a Message by Referenced Schema ID
▶ Typical Consumer Logic
▶ Metadata and Data Messages
▶ Metadata Message
▶ Data Message
Prerequisites
Before you can use Amazon Kinesis Data Streams as a target endpoint in a Replicate
task, the following prerequisites must be met:
Replicate connects to AWS using SSL. This requires an appropriate CA certificate to
reside on the Replicate Server machine; otherwise, the connection will fail. The
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 492
purpose of the CA certificate is to authenticate the ownership of the AWS
server certificate.
On Windows, the required CA certificate is always present whereas on Linux it may
sometimes be missing. Therefore, if you are using Replicate for Linux, make sure
that the required CA certificate exists in the following location:
/etc/pki/tls/certs/ca-bundle.crt
If it does not exist, the simplest solution is to copy the certificates bundle
from another Linux machine.
The target streams must already exist before starting the replication task.
The AWS account specified in the General tab must have the following permissions:
Note All strings that begin with YOUR should be replaced with the actual value.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kinesis:PutRecord",
"kinesis:PutRecords",
"kinesis:DescribeStream"
],
"Resource": "arn:aws:kinesis:YOUR_AWS_REGION:YOUR_ACCOUNT_
NAME:stream/YOUR_STREAM_NAME"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "kinesis:ListStreams",
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 493
"Resource": "*"
}
]
}
Additionally, if the Kinesis Stream was configured in Amazon to encrypt the data
at rest, the following additional KMS permissions should be set:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:YOUR_AWS_REGION:YOUR_ACCOUNT_
NAME:key/YOUR_KEY_GUID"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "kms:GenerateDataKey",
"Resource": "*"
}
]
}
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 494
various ways such as applying them to the target tables or storing them in
dedicated Change Tables.
Each CDC message has both a transaction ID as well as change sequence. As the
change sequence is a monotonically growing number, sorting events by change
sequence always achieves chronological order. Grouping the sorted events by
transaction ID then results in transactions containing chronologically sorted changes.
However, as Amazon Kinesis Data Streams is a messaging infrastructure, applying
changes is not feasible while storing changes in tables is meaningless. The Replicate
Amazon Kinesis Data Streams endpoint, therefore, takes a different approach, which is
to report all transactional events as messages.
How it Works
Each change in the source system is translated to a data message containing the details
of the change including the transaction ID and change sequence in the source. The data
message also includes the changed columns before and after the change.
Once a data message is ready to be sent to Amazon Kinesis Data Streams, the stream
and shard it should go to are determined by analyzing the endpoint settings and any
transformation settings. For example, the user might decide to configure the endpoint
in such a way that every table is sent to a different stream and set the partition
strategy to "Random", meaning that each message (within the same table) will be sent
to a different shard.
Although the simple way may work, it’s not very efficient at the task level as all messages
end up in the same stream and shard, not necessarily utilizing the full parallelism of the
Amazon Kinesis Data Streams cluster. This may be a non-issue if there are multiple tasks,
each taking advantage of a different stream/shard. In such as scenario, the gathering of
messages from those tasks may very well utilize the cluster optimally.
The more generic way where data may be spread over multiple streams and shards means
that some intermediate buffer such as memory, a table in a relational database, or even
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 495
other streams would need to be used to collect information about transactions. Then, the
transactions would need to be rebuilt by periodically (every few minutes/hours) sorting
the events collected from Replicate’s Amazon Kinesis Data Streams output by the
change sequence and grouping them by transaction ID.
Limitations
When defining a task with Amazon Kinesis Data Streams as the target endpoint,
the following limitations apply:
The Amazon Kinesis Data Streams target endpoint does not support unlimited LOB
size. Therefore, when replicating from source tables with LOB columns, do not
select the Allow unlimited LOB size option.
For more information on defining LOB settings, see Target Metadata.
Batch optimized apply mode is not supported. If this mode is set, the task
will automatically switch to Transactional apply mode and issue an
appropriate warning.
For more information on these modes, see Change Processing Tuning.
The Ignore ALTER Apply Changes setting is not supported for changes to source
data types and table renaming.
Column names must begin with [A-Za-z_] (letters or an underscore) followed by
[A-Za-z0-9_] (letters, digits, or an underscore). For example, _Test_ is a valid
column name whereas &Test is not.
If a source column name does not adhere to this rule, then a transformation should
be used to rename the column.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 496
Note When using the JSON message format, binary values are represented
as hexadecimal digits.
Table 9.4 |
Supported Amazon Kinesis Data Streams Target Data Types with
Mapping from Attunity Replicate Data Types
DATE DATE
TIME TIME
DATETIME DATETIME
BLOB BLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 497
Table 9.4 | Supported Amazon Kinesis Data Streams Target Data Types with
Mapping from Attunity Replicate Data Types (Cont.)
CLOB CLOB
NCLOB NCLOB
Table 9.5 |
Supported Avro Data Types with Mapping from Attunity Replicate Data Types
STRING
TIME STRING
TIMESTAMP STRING
STRING STRING
WSTRING STRING
CLOB STRING
NCLOB STRING
NUMERIC STRING
BYTES BYTES
BLOB BYTES
REAL4 FLOAT
REAL8 DOUBLE
INT1 INT
INT2 INT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 498
Table 9.5 | Supported Avro Data Types with Mapping from Attunity Replicate Data
Types (Cont.)
UINT1 INT
UINT2 INT
UINT4 LONG
INT8 LONG
UINT8 STRING
BOOLEAN BOOLEAN
The Name, Description, Type and Role fields are displayed on the right.
Key pair
Choose this method to authenticate with your Access Key and Secret Key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 499
IAM Roles for EC2.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
Access key: If you selected Key pair as your access method, enter your access
key for Amazon Kinesis Data Streams.
Secret key: If you selected Key pair as your access method, enter your secret
key for Amazon Kinesis Data Streams.
8. In the Message Properties section, select JSON or Avro as the message Format.
Note Attunity provides an Avro Message Decoder SDK for consuming Avro
messages produced by Attunity Replicate. You can download the SDK together
with the Avro Message Decoder Developer's Guide as a ZIP file from the
Customer Zone.
An understanding of the Attunity envelope schema is a prerequisite for consuming
Avro messages produced by Attunity Replicate. If you do not wish to use the SDK,
see The Attunity Envelope for a description of the Attunity envelope schema.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 500
Note The partition key is represented as a string, regardless of the
selected data message format (JSON/Avro).
Schema and table name - For each message, the partition key will
contain a combination of schema and table name (e.g. "dbo+Employees").
Messages consisting of the same schema and table name will be written
to the same partition.
Primary key columns - For each message, the partition key will
contain the value of the primary key column.
Messages consisting of the same primary key value will be written to
the same partition.
10. In the Metadata Message Publishing section, specify whether or where to
publish the message metadata.
From the Publish drop-down list, select one of the following options:
If you select this option, either type the Specific stream name or use
the Browse button to select the desired stream.
Note Before you can define such a transformation, you first need to add a
source endpoint to the task and select the tables you want to replicate.
To define a transformation:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 501
Note The columns listed below (prefixed with a $) instruct Replicate to route the
message to the desired stream and will not be included in the actual message itself.
4. Define an expression for the new column that returns the following values:
For a $stream column, the expression should return the topic name.
For a $key column, the expression should return the message key contents.
Note that the expression must return a non-empty value.
For information on creating expressions, see Using the Expression Builder (for
Filters, Transformations, and Global Transformations).
Option Description
User name The user name for accessing the proxy server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 502
Table 9.6 | Advanced Tab Options (Cont.)
Option Description
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 503
"type":"record",
"name":"MessageEnvelope",
"fields":[
{"name":"magic","type":{"type":"fixed","name":"Magic","size":5}},
{"name":"type","type":"string"},
{"name":"headers","type":["null",{"type":"map","values":"string"}]},
{"name":"messageSchemaId","type":["null","string"]},
{"name":"messageSchema","type":["null","string"]},
{"name":"message","type":"bytes"}
The constant "atMSG" is used to identify this form of message. The "atMSG" constant
should be used to validate that this message is indeed an Attunity envelope message.
Describes the enveloped message type. This can be one of two values: MD
which stands for metadata message and DT which stands for data message.
headers (map of string key and value)
A free for use map for various properties set at the application level. Currently,
no headers are set by Attunity Replicate but this may change in future versions.
An embedded UTF-8 encoded Avro JSON schema with which the message field can
be serialized. This field is used exclusively with the messageSchemaId field.
message (bytes)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 504
Given the envelope schema, it is possible for anyone using this schema to properly
decode the envelope messages from Amazon Kinesis Data Streams.
Once the envelope message has been decoded, there are two possible scenarios:
The method for logically decoding messages in both scenarios is described below.
To avoid sending schema with each data message, each schema has a 32 bytes long ID.
When a data message based on a previously sent data message schema (via the
metadata message) is constructed, the messageSchema field is set to null and the
messageSchemaId field is set to the 32 bytes ID of the schema instead. The application
responsibility is to locate the data schema sent earlier in the metadata message and use
that schema to decode the data message contained in the message field.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 505
Consumer. Attunity Replicate offers the ability to define a specific stream as the
schema stream and different streams for the table data.
The customer's consumer code should read metadata messages from the schema stream
and then save the data schemas and any other information the consumer wishes to
access later in a customer defined zone. Another set of customer consumers should read
data messages from the various data streams, and access the data schemas zone as
required to retrieve the data schemas required for decoding the data messages.
When consuming data messages and metadata messages from several streams and partitions
in a multi-thread/process manner, a situation may arise where a given consumer may
attempt to read a data message before the corresponding metadata message has been read.
As it is not possible to read a data message before its corresponding metadata message, the
consumer's logic should wait a reasonable amount of time until the corresponding metadata
message has been read. If the metadata message is still not available after waiting for a
reasonable amount of time, the consumer should handle this as an unexpected error and
activate the planned error policy. An example of such a policy could be saving the message in
a dedicated “delayed” stream for later processing.
As a rule of thumb, the number of metadata messages will be much lower (in the
magnitude of 1:10000 or more) than the number of data messages. So, assuming
a metadata consumer is active, the gap between metadata message and data
message should be no more than a few seconds (usually, milliseconds).
Metadata Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 506
Field Type Description
{columns} Structure For each column, a record with the below prop-
erties.
length Integer The maximum size of the data (in bytes) permitted
for the column.
precision Integer For NUMERIC data type, the maximum number of
digits required to represent the value.
scale Integer For NUMERIC data type, the maximum number of
digits to the right of the decimal point permitted for
a number.
primaryKeyPosition Integer The position of the column in the table’s Primary
Key. or Unique Index. The value is zero if the
column is not part of the table’s Primary Key.
dataSchema String The Avro schema for deserializing the Data mes-
sages.
Data Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 507
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 508
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 509
Field Type Description
beforeData Structure The data of the table record, before the change
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 510
Using Amazon Redshift as a Target
This section describes how to set up and use Amazon Redshift as a target in a
replication task. Amazon Redshift is located in the cloud and is accessed through an
Amazon Web Services (AWS) account.
In this section:
▶ Introducing the Amazon Redshift Target Endpoint for Attunity Replicate
▶ Limitations
▶ Amazon Redshift Database Prerequisites
▶ Get Started with Amazon Redshift
▶ Sign up for an Amazon S3 Bucket
▶ Security
▶ Amazon Redshift Data Types
▶ Setting General Connection Parameters
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
In the first stage of the replication process, Attunity Replicate moves the data files
created by the source database into an Amazon S3 bucket. The files are then loaded into
the proper tables in the Amazon Redshift data warehouse (using the "copy" command).
The Amazon Redshift database provides full automation for:
Schema generation and data type mapping
Full load of source database tables
Incremental load of changes made to source tables
Application of schema changes (DDL) made to the source tables.
Synchronization between full load and CDC processes.
Manual control is also available if needed.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 511
Limitations
When using the Amazon Target endpoint, the following limitation(s) apply:
Download and install the Windows or Linux SQL client tools (according to your
Replicate Server platform) necessary to connect to the Amazon Redshift cluster.
Attunity Replicate requires that you download a 64-bit ODBC driver.
By default, Attunity Replicate uses the Amazon Redshift (x64) driver. If you use a
different driver, you must change this in the Amazon Redshift database settings in
the Attunity Replicate Console. For more information, see Setting Advanced
Connection Properties.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 512
Note To avoid conflicts when installing the driver on a Linux machine, Attunity
Replicate must be installed before you install the driver. Install the Amazon
Redshift ODBC driver with --force in the command, as in the following example:
rpm -ivh AmazonRedshiftODBC-64bit-1.2.6.1006-1.x86_64.rpm --force
ODBCInstLib=libodbcinst.so
Bucket access credentials: Make a note of the bucket name, region, access key
and secret access key - you will need to provide them in the Amazon Redshift
target endpoint settings.
Bucket access permissions: requires read/write/delete permissions to the Amazon
S3 bucket.
Security
Ensure that the following security prerequisites have been met:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 513
If it does not exist, the simplest solution is to copy the certificates bundle
from another Linux machine.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.7 |
Supported Amazon Redshift Data Types with Mapping from Attunity
Replicate Data Types
DATE DATE
TIME VARCHAR(20)
TIMESTAMP (s)
VARCHAR (37)
INT1 INT2
INT2 INT2
INT4 INT4
INT8 INT8
NUMERIC (p,s)
VARCHAR (Length)
REAL4 FLOAT4
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 514
Table 9.7 | Supported Amazon Redshift Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
UINT1 INT2
UINT2 INT2
UINT4 INT4
NVARCHAR (65535)
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab description in Customizing Tasks .
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 515
Table 9.7 | Supported Amazon Redshift Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
Redshift cluster: Type the name of the Amazon Redshift cluster you are using.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 516
values, refer to your AWS account or the Amazon Redshift System Administrator
for your enterprise.
8. Enter the following Amazon S3 staging information. You may need to click
the Amazon S3 staging header to see the information.
Bucket name: Type the name of the Amazon S3 bucket where you are
copying files to.
Bucket region: Select the Amazon S3 region where the S3 buckets and
folders you are using are hosted. The default value is US East (N. Virginia).
Note: The bucket region specified must be the same region where your
Amazon Redshift database is located.
Key pair
Choose this method to authenticate with your Access Key and Secret Key.
Access key: Type the access key information for Amazon S3.
Secret key: Type the secret key information for Amazon S3.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 517
against ADFS.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_
saml.html
Folder: The name of the Amazon S3 folder to where you want your files to
be copied.
The information for these properties is available from your Amazon Web
Services (AWS) account. If you do not have these values, refer to your AWS
account or the Amazon Redshift System Administrator for your enterprise
Note
This information is case sensitive.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 518
Number of threads used to upload a file: Select the number of threads used to
upload a single file. The minimum number of threads is 1. The maximum value is
64. The default value is 10.
ODBC driver: The name of the default ODBC driver you are using to connect
to Amazon Redshift. The default value is Amazon Redshift (x64).
Additional ODBC connection properties: Type any additional ODBC
connection properties if required.
Use proxy server: Select this option to access Amazon Redshift via a proxy server.
Host name: The host name of the proxy server.
Port: The port via which to access the proxy server.
User name: The user name for accessing the proxy server.
Password: The password for accessing the proxy server.
Scheme:
Select which protocol to use to access the server (HTTP or HTTPS). In order
to use HTTPS, you must first install the CA certificate that signed the proxy’s
certificate on the Replicate Server machine, as follows:
On Windows: Add the CA certificate to the Trusted Root
Certification Authorities store of Local Computer
On Linux: Add the CA certificate to /etc/pki/tls/certs/ca-bundle.crt
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 519
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 520
Using Amazon S3 as a Target
This chapter describes how to set up and use Amazon S3 as a target in a replication task.
In this section:
▶ Prerequisites
▶ Amazon S3 Target Overview
▶ DDL Handling
▶ Limitations
▶ Change Data Partitioning
▶ Amazon S3 Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Generating Reference Files
▶ Content-Type and Content-Encoding Properties
Prerequisites
Before you can use Amazon S3 as a target endpoint in a Replicate task, the
following prerequisites must be met:
The Attunity Replicate Server machine must be set to the correct time (UTC).
If it does not exist, the simplest solution is to copy the certificates bundle
from another Linux machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 521
You must have an Amazon S3 bucket that is accessible from the Replicate
Server machine.
For information on signing up for Amazon S3, see http://aws.amazon.com/s3/.
Bucket access credentials: Make a note of the bucket name, region, access key
and secret access key - you will need to provide them in the Attunity Replicate
Amazon S3 target settings.
Bucket access permissions: Attunity Replicate requires the following bucket
access permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1497347821000",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
},
{
"Sid": "Stmt1497344984000",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/target_path",
"arn:aws:s3:::YOUR_BUCKET_NAME/target_path/*"
]
}
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 522
]
}
Note If the target path is the bucket root, just specify “/target_path” with an
empty string.
Note When Parallel Load is used, the naming convention for Full Load files is
slightly different:
LOAD_$(SegmenteID)_$(IncreasingCounter)
Example:
LOAD_1_00000001 | LOAD_1_00000002 | LOAD_1_00000003 | LOAD_2_00000001 |
LOAD_2_00000002
Note When the Create metadata files in the target folder option is enabled, a
corresponding metadata file is created using the same naming format, but with a
.dfm extension.
For each source table, a folder is created in the specified Amazon S3 bucket. The data
files are created on the Replicate Server machine and are then uploaded to the specified
Amazon S3 bucket once the File Attributes (Full Load) and Change Processing upload
conditions have been met.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 523
DDL Handling
When a DDL change is captured, Replicate will close the data file and also create a DFM
file if the Create metadata files in the target folder option is enabled. When the next
batch of changes arrives, Replicate will create a new data file containing the changes.
Note that the DFM file created for the new data file will match the new table structure.
Limitations
The following limitations apply to the Amazon S3 target endpoint:
The following DDLs are supported only: Truncate table, Drop table, Create table,
Add Column, Rename Column, Drop Column, and Convert Data Type.
Full LOB Mode is not supported
UPDATE and DELETE statements are not supported in Apply Changes replication mode
{Target Directory}
{Table_1}
{Partition_1}
Data files
DFM files
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
{Table_2}
{Partition_1}
Data files
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 524
DFM files
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
Information about the partitions is written to the attrep_cdc_partitions Control Table.
For information about this table, see Change Data Partitions.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.8 |
Supported Amazon S3 Target Data Types with Mapping from
Attunity Replicate Data Types
DATE DATE
TIME TIME
DATETIME DATETIME
BLOB BLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 525
Table 9.8 | Supported Amazon S3 Target Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 526
Table 9.9 |
Amazon S3 Target Endpoint - General Tab Options
Option Description
Amazon S3 Storage
Bucket region Select the Amazon S3 region where your bucket is located.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_
providers_saml.html
Access key Enter the access key information for Amazon S3.
Secret key Enter the secret key information for Amazon S3.
ADFS URL The URL to an Active Directory Federation Services page, responsible for
returning a SAML claims document to be sent over to AWS.
AD principal The principal (user) name to use when identifying against ADFS
name
The format should be: user.name@domain
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 527
Table 9.9 | Amazon S3 Target Endpoint - General Tab Options (Cont.)
Option Description
AD principal The principal password to use when identifying against ADFS
password
IdP ARN The Amazon Resource Name (ARN) of the Active Directory issuing the
SAML claims document. This is required as it enables AWS to identify the
signer of the SAML document and verify its signature.
SAML Role The Amazon Resource Name (ARN) of the specific role the returned cre-
ARN dentials should be assigned.
Role external ID: The value of the external ID condition in the target
role’s trust policy.
File Attributes
Delimiters can be standard characters or a hexadecimal (hex) value. Note that the
"0x" prefix must be used to denote a hexadecimal delimiter (e.g. 0x01 = SOH). In the
Field delimiter, Record delimiter and Null value fields, the delimiter can consist of
concatenated hex values (e.g. 0x0102 = SOHSTX), whereas in the Quote character
and Escape character fields, it can only be a single hex value.
Note The hexadecimal number 0x00 is not supported (i.e. only 0x01-0xFF
are supported).
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 528
Table 9.9 | Amazon S3 Target Endpoint - General Tab Options (Cont.)
Option Description
Format You can choose to create the target files in CSV or JSON format.
{ "book_id": 456, "title": "Winnie the Pooh", "price": 6.49, "is_hardcover": true }
{ "book_id": 789, "title": "The Cat in the Hat", "price": 7.23, "is_hardcover": true }
Note Changing the format (i.e. from CSV to JSON or from JSON to
CSV) while the task is in a stopped state and then resuming the
task, is not supported.
Field delimiter The delimiter that will be used to separate fields (columns) in the
target files. The default is a comma.
"mike","male"
Record The delimiter that will be used to separate records (rows) in the target
delimiter files. The default is a newline (\n).
Example:
"mike","male"\n
"sara","female"\n
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 529
Table 9.9 | Amazon S3 Target Endpoint - General Tab Options (Cont.)
Option Description
Null value The string that will be used to indicate a null value in the target files.
"mike","male",295678\n
"sara","female",@\n
Quote The character that will be used at the beginning and end of a text column.
character The default is the double-quote character ("). When a column that
contains column delimiters is enclosed in double-quotes, the column
delimiter characters are interpreted as actual data, and not as column
delimiters.
@mike@,@male@
Escape The character used to escape a quote character in the actual data.
character
Example (where" is the quote character and \ is the escape
character):
Add metadata When the target storage format is set to Text, you can optionally add a
header header row to the data files. The header row can contain the source
column names and/or the intermediate (i.e. Replicate) data types.
Example of a target file with a header row when both With column
names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 530
Table 9.9 | Amazon S3 Target Endpoint - General Tab Options (Cont.)
Option Description
Maximum file The maximum size a file can reach before it is closed (and optionally
size compressed). This value applies both to data files and to Reference Files.
Compress files Choose GZIP to compress the target files or NONE (the default) to leave
using them uncompressed.
Change Processing
File size Specify the maximum size of Change Data to accumulate before
reaches uploading the file to Amazon S3.
Elapsed time Specify the maximum time to wait before applying the changes.
reaches
Metadata Files
Create When this option is selected, for each data file, a matching metadata file
metadata files with a .dfm extension will be created under the specified target folder.
in the target The metadata files (which are in standard JSON format) provide
folder additional information about the task/data such as the source endpoint
type, the source table name, the number of records in the data file, and
so on.
For a full description of the metadata file as well as possible uses,
see Metadata File Description .
Data Encryption
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 531
Table 9.9 | Amazon S3 Target Endpoint - General Tab Options (Cont.)
Option Description
This option also requires you to specify your KMS Key ID.
Note As part of connection testing process, Replicate uploads a test file to the specified
Amazon S3 Target folder and then deletes it once a connection has been established.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 532
Table 9.10 | Amazon S3 Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can process the final target files using a custom command. The command will be
run whenever a data file is created.
Note If the Generate a reference file option is selected, a row (specifying the
file's location) will be added to the Reference File only after the command
completes successfully.
Working directory - The directory where you want the command to run.
${METADATA_FILENAME} - The full path to the DFM file containing the metadata.
Note If the CSV/DFM file paths contain spaces, you must enclose
these parameters with quotation marks (e.g "${FILENAME}").
The post-processing command must return a proper exit code. You can either use the
standard exit code values described below or set a custom exit code value as described
in Setting Post Command Exit Codes with an Internal Parameter.
0 - Success
1 - Recoverable error. The task will recover from the point of failure according to
the settings in the Environmental Errors tab.
2 - Table error. If a table error occurs, Replicate will handle the error according
to the settings in the Table Errors tab.
3 (or any other value e.g. -100) - Fatal error. The task will fail and not
attempt recovery.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 533
Table 9.10 | Amazon S3 Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can use internal parameters to set exit codes with custom values. This is
especially useful if your application already uses the standard exit code values.
See Standard Post Command Exit Codes above for a description of the exit codes.
successExitCode
recoverableErrorExitCode
tableErrorExitCode
fatalErrorExitCode
Select this option to generate a Reference File (on Replicate Server) containing the
full path to the Apply Changes data files.
Note The reference file only points to the location of the Apply Changes files, and
not the Full Load files.
Reference File(s) folder - The folder on the Replicate machine in which the
Reference File will be created.
Example:
c:\temp\
User name The user name for accessing the proxy server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 534
Table 9.10 | Amazon S3 Target Endpoint - Advanced Tab Options (Cont.)
Option Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 535
Internal Parameters
Internal parameters are parameters that are not exposed in the UI and should only be
used if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your setting by clicking the View Setting Summary link. This
is useful if you need to send a summary of your setting to Attunity Support.
Example:
AmazonS300000001.csv
Note The counter suffix increases incrementally each time a new Reference File is
generated (i.e. when the file reaches the maximum size defined in the General tab).
Once a new Reference File has been generated, you can delete the old reference
file(s) if required.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 536
Whenever an Apply Changes data file is created, a new row is added to the Reference
File in the following format:
<Source_Table_Name>,<bucket_name>/<path>/<file_name>
Example:
employees,bigdata/new/files/my.company/20170611-120144192.csv
Note that if the Post-process files option in the Advanced tab is also enabled,
the Reference File will be generated after the post-processing completes.
Content-
Replicate Artifact File Format Compression Content-Type
Encoding
Data file CSV None text/csv; char- N/A
set=utf-8
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 537
Using a File as a Target
This section describes how to set up and use delimited text files as a target in a
replication task. You can use the File target endpoint to export database tables to files,
which can then be used as a source in a Replicate task with a File source endpoint.
In this section:
▶ File Target Overview
▶ DDL Handling
▶ Limitations
▶ Change Data Partitioning
▶ File Target Data Types
▶ Setting General Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Generating Reference Files
Delimited text files are used to store data in tabular format. Examples of delimited text
file formats include the CSV (Comma Separated Values) and TSV (Tab Separated Values)
formats. Some organizations may implement procedures that export data from a
database to a delimited text file while others may simply prefer this format as a
convenient way of storing tabular data.
In a delimited text file, each record in the table occupies a separate row. Delimiters
are used to mark the beginning of a new row or the beginning of a new column.
Virtually any character can be used as a delimiter, although a newline (\n) is often
used to separate rows, and commas are commonly used to separate columns.
In JSON files, each record is represented by a single line.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 538
book_id title price is_hardcover
When using a File as a target in a Replicate task, both the Full Load and the CDC data
are written to CSV or JSON files (depending on the endpoint settings). While the
explanations in this topic relate to CSV files, the same is true for JSON files.
Full Load files are named using incremental counters e.g. LOAD00001.csv, LOAD
00002.csv, etc. whereas Apply Changes files are named using timestamps e.g. 20141029-
1134010000.csv.
Note When Parallel Load is used, the naming convention for Full Load files is
slightly different:
LOAD_$(SegmenteID)_$(IncreasingCounter)
Example:
LOAD_1_00000001 | LOAD_1_00000002 | LOAD_1_00000003 | LOAD_2_00000001 |
LOAD_2_00000002
Note
The Apply Changes CSV files appear with a .tmp extension while they are in
idle state. For more information on idle state, see Change Processing.
When the Create metadata files in the target folder option is enabled, a
corresponding metadata file is created using the same naming format, but with
a .dfm extension.
For each source table, a folder is created under the specified target folder. All files -
i.e. Full Load, Apply Changes, and Metadata (if enabled) - are written to the relevant
folder, according to the settings defined in the File target’s General tab.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 539
After a task completes, you can define another task with a File source endpoint that
uses the generated CSV files.
DDL Handling
When a DDL change is captured, Replicate will close the data file and also create a DFM
file if the Create metadata files in the target folder option is enabled. When the next
batch of changes arrives, Replicate will create a new data file containing the changes.
Note that the DFM file created for the new data file will match the new table structure.
Limitations
The following limitations apply to the File target endpoint:
Only the following DDLs are supported: Truncate table, Drop table, Create table,
Add Column, Rename Column, Drop Column, and Convert Data Type.
Full LOB Mode is not supported
UPDATE and DELETE statements are not supported in Apply Changes replication mode
{Target Directory}
{Table_1}
{Partition_1}
Data files
DFM files
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 540
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
{Table_2}
{Partition_1}
Data files
DFM files
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
Information about the partitions is written to the attrep_cdc_partitions Control Table.
For information about this table, see Change Data Partitions.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.11 |
Supported File Target Data Types with Mapping from Attunity
Replicate Data Types
TIME TIME
DATETIME DATETIME
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 541
Table 9.11 | Supported File Target Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 542
3. In the Description field, type a description that helps to identify the File
endpoint. This is optional.
4. Select TARGET as the endpoint role.
5. Select File as the endpoint Type.
6. In the Target folder field, specify the full path of the folder to which you the
target files to be written.
7. Configure the remaining settings in the General tab as described in the
following table.
Table 9.12 |
File Target Endpoint - General Tab Options
Option Description
File Format
Delimiters can be standard characters or a hexadecimal (hex) value. Note that the
"0x" prefix must be used to denote a hexadecimal delimiter (e.g. 0x01 = SOH). In the
Field delimiter, Record delimiter and Null value fields, the delimiter can consist of
concatenated hex values (e.g. 0x0102 = SOHSTX), whereas in the Quote character
and Escape character fields, it can only be a single hex value.
Note The hexadecimal number 0x00 is not supported (i.e. only 0x01-0xFF
are supported).
Format You can choose to create the target files in CSV or JSON format.
{ "book_id": 456, "title": "Winnie the Pooh", "price": 6.49, "is_hardcover": true }
{ "book_id": 789, "title": "The Cat in the Hat", "price": 7.23, "is_hardcover": true }
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 543
Table 9.12 | File Target Endpoint - General Tab Options (Cont.)
Option Description
Field delimiter The delimiter that will be used to separate fields (columns) in the
target files. The default is a comma.
"mike","male"
Record delimiter The delimiter that will be used to separate records (rows) in the target
files. The default is a newline (\n).
Example:
"mike","male"\n
"sara","female"\n
Null value The string that will be used to indicate a null value in the target files.
"mike","male",295678\n
"sara","female",@\n
Quote character The character that will be used at the beginning and end of a text
column. The default is the double-quote character ("). When a column
that contains column delimiters is enclosed in double-quotes, the
column delimiter characters are interpreted as actual data, and not as
column delimiters.
@mike@,@male@
Escape character The character used to escape a quote character in the actual data.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 544
Table 9.12 | File Target Endpoint - General Tab Options (Cont.)
Option Description
Code page Specify the code page of your target files if it is different from the
default (65001).
Add metadata When the target storage format is set to Text, you can optionally add a
header header row to the data files. The header row can contain the source
column names and/or the intermediate (i.e. Replicate) data types.
Example of a target file with a header row when both With column
names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
File Attributes
Maximum file The maximum size a file can reach before it is closed (and optionally
size compressed). This value applies both to data files and to Reference
Files.
Compress files Choose GZIP to compress the target files or NONE (the default) to
using leave them uncompressed.
Change Processing
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 545
Table 9.12 | File Target Endpoint - General Tab Options (Cont.)
Option Description
Consider state Specify how long to wait before considering the state to be idle. In idle
idle when no state, you can apply changes to files using data that has already been
changes have processed if the specified size and time conditions are met (see
been processed below).
for
File size reaches Specify the maximum size of the data required in order to apply
changes to the target file in idle state.
Elapsed time Specify the maximum time to wait before applying the changes in idle
reaches state.
Allow a single By default, a single transaction will not be split across multiple files,
transaction to be regardless of the values specified in the File size reaches and
split into Elapsed time reaches fields. This is important for organizations who
multiple files require files to contain transactions in their entirety. However, this
may also result in very large file sizes. For example, if the File size
reaches value is 32 MB and Replicate starts to apply changes for a
new 2 GB transaction at 31 MB, the target file will only be closed at
2.031 GB.
You should therefore select this option if it is critical that the values in
the File size reaches and Elapsed time reaches fields are adhered
to (even if it means splitting a transaction across multiple files).
Metadata Files
Create metadata When this option is selected, for each data file, a matching metadata
files in the target file with a .dfm extension will be created under the specified target
folder folder. The metadata files (which are in standard JSON format) provide
additional information about the task/data such as the source endpoint
type, the source table name, the number of records in the data file,
and so on.
For a full description of the metadata file as well as possible uses, see
Metadata File Description .
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 546
Note To determine if you are connected to the endpoint you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button is not
available unless the test connection fails.
Table 9.13 |
File Target Endpoint - Advanced Tab Options
Option Description
Generate Select this option to generate a Reference File containing the full path to the
reference Apply Changes data files.
files
Note The reference file only points to the location of the Apply Changes
files, and not the Full Load files.
For information on using reference files with the File source endpoint, see
Reference Files.
Reference The folder on the Replicate machine in which the Reference File will be
file folder created.
Note Multiple tasks with file target endpoints that have the same target
directory are not supported (as each task will attempt to write to the same
reference file).
Example:
c:\temp\
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 547
Table 9.13 | File Target Endpoint - Advanced Tab Options (Cont.)
Option Description
Post- You can process the final target files using a custom command. The
process command will be run whenever a data file is created.
files
Note If the Generate a reference file option is selected, a row
(specifying the file's location) will be added to the Reference File
only after the command completes successfully.
${FILENAME} - The full path to the CSV file containing the full load
or CDC data.
${METADATA_FILENAME} - The full path to the DFM file containing
the metadata.
Note If the CSV/DFM file paths contain spaces, you must enclose
The post-processing command must return a proper exit code. You can
either use the standard exit code values described below or set a custom
exit code value as described in Setting Post Command Exit Codes with an
Internal Parameter.
0 - Success
1 - Recoverable error. The task will recover from the point of
failure according to the settings in the Environmental Errors tab.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 548
Table 9.13 | File Target Endpoint - Advanced Tab Options (Cont.)
Option Description
2 - Table error. If a table error occurs, Replicate will handle the error
according to the settings in the Table Errors tab.
3 (or any other value e.g. -100) - Fatal error. The task will fail and not
attempt recovery.
You can use internal parameters to set exit codes with custom values. This is
especially useful if your application already uses the standard exit code
values.
See Standard Post Command Exit Codes above for a description of the exit
codes.
successExitCode
recoverableErrorExitCode
tableErrorExitCode
fatalErrorExitCode
For instructions on setting internal parameters, see Internal Parameters.
After post You can decide what to do with the original target files after post-processing
processing completes:
completes
Do nothing - Leaves the files in their original location
Delete files - Deletes the files from the disk
Replace file extension with - Replaces the file extension with the
specified extension.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI and should only be
used if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 549
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your setting by clicking the View Setting Summary link. This
is useful if you need to send a summary of your setting to Attunity Support.
Example:
FileTarget00000001.csv
Note The counter suffix increases incrementally each time a new Reference File is
generated (i.e. when the file reaches the maximum size defined in the General tab).
Once a new Reference File has been generated, you can delete the old reference
file(s) if required.
Whenever an Apply Changes data file is created, a new row is added to the Reference
File in the following format:
<Source_Table_Name>,<full_path_to_data_file>
Example:
MyTable,c:\temp\filetarget\dbo.MyTable\20170102-091759447.csv
Note that if the Post-process files option in the Advanced tab is also enabled,
the Reference File will be generated after the post-processing completes.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 550
Note
When both the Post-process files and the Delete files (after post-processing
completes) options are enabled, the reference file will not be generated.
If the Archive files to folder (after post-processing completes) option is
selected, the reference file will be updated to reflect the archive location of
the data files.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 551
Using Google Cloud SQL for MySQL as a Target
This section describes how to set up and use a Google Cloud SQL for MySQL target
endpoint in a replication task.
In this section:
▶ Prerequisites
▶ General Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Limitations
▶ Security Requirements
▶ Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the prerequisites described in this section have been met.
In this section:
General Prerequisites
Attunity Replicate Server for Windows
Attunity Replicate Server for Linux
General Prerequisites
The following is required:
A Google Cloud SQL for MySQL account with the required Security Requirements.
A Google Cloud SQL for MySQL database with the tables that you want to
replicate should be accessible in your network.
The following Google Cloud SQL for MySQL editions are supported:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 552
Google Cloud SQL for MySQL Enterprise Edition
Google Cloud SQL for MySQL Cluster Carrier Grade Edition
In order to work with Google Cloud SQL for MYSQL 8 as a target endpoint, you need to:
Set the parameter secure_file_priv = NULL in the MySQL database.
If you are using LOB columns, set the parameter sort_buffer_size = 128M
in the Google Cloud SQL for MySQL database.
Notes
Attunity Replicate assumes that MySQL ODBC Client 5.2.6 to 8.0.x 64-bit for
Windows is installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal parameter
where driver is the Parameter and MySQL ODBC <version> Unicode Driver is the
Value (where <version> is the client version e.g. 8.1).
For instructions on setting internal parameters, see Attunity Replicate Server
for Windows below.
MySQL ODBC drivers 8.0 to 8.0.13 are not supported.
Notes
Attunity Replicate assumes that MySQL Client 5.2.6 to 8.0.x for Linux is
installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal
parameter where driver is the Parameter and MySQL ODBC <version>
Unicode Driver is the Value (where <version> is the client version e.g.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 553
8.1).
Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in
the following example:
[MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Limitations
The following limitations apply:
When only the LOB column in the source table is updated, Replicate will not update
the corresponding target LOB column. The target LOB column will only be updated if
at least one other column is also updated in the same transaction.
Due to the way MySQL operates, when loading data to a Google Cloud SQL for MySQL
target during a Full Load task, duplicate key errors will not be reported to the logs.
When updating a column's value to its existing value, a zero rows affected is
returned from MySQL (unlike Oracle and Microsoft SQL Server that perform an
update of one row). This generates an entry in the attrep_apply_exceptions
control table and the following warning:
Some changes from the source database had no impact when applied to
Security Requirements
You must provide Google Cloud SQL for MySQL account access to the Attunity
Replicate user. This user must have read/write privileges in the Google Cloud SQL
for MySQL database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 554
Replicate data types.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.14 |
Supported Google Cloud SQL for MySQL Data Types with Mapping
from Attunity Replicate Data Types
Attunity Replicate Data Types Google Cloud SQL for MySQL Data Types
BOOL BOOL
VARBINARY (Length)
BLOB
MEDIUMBLOB
LONGLOB
DATE DATE
TIME TIME
DATETIME (Scale)
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 555
Table 9.14 | Supported Google Cloud SQL for MySQL Data Types with Mapping from
Attunity Replicate Data Types (Cont.)
Attunity Replicate Data Types Google Cloud SQL for MySQL Data Types
DATETIME (p,s)
VARCHAR (45)
REAL4 FLOAT
REAL8 DOUBLE
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 556
Table 9.14 | Supported Google Cloud SQL for MySQL Data Types with Mapping from
Attunity Replicate Data Types (Cont.)
Attunity Replicate Data Types Google Cloud SQL for MySQL Data Types
BLOB
LONGBLOB
TEXT
TEXT
LONGTEXT
To add a Google Cloud SQL for MySQL target endpoint to Attunity Replicate:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 557
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Google
Cloud SQL for MySQL database. This is optional.
4. Select TARGET as the database role.
5. From the Type drop-down list, select Google Cloud SQL for MySQL.
6. In the Server field, enter the host name or IP address of the computer on which
the database is installed.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
8. Type the Google Cloud SQL for MySQL authentication information (User
Name, Password) for the authorized user for this database. If you do not
know this information, see your MySQL database Administrator (DBA).
9. Select one of the following Load source schemas into options:
The following database - When this option is selected, all source schemas
will be loaded into the selected database.
Multiple endpoints - When this option is selected, each of the source
schemas will be loaded into its corresponding database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 558
Setting Advanced Connection Properties
In the Advanced tab, you can set advanced properties.
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before it
is loaded into the Google Cloud SQL for MySQL target database. The default value is
32000 KB.
Use parallel loading: Select this option to improve performance when loading
data into theGoogle Cloud SQL for MySQL target database.
Use the following number of threads: Specify how many threads to use to
load the data into the Google Cloud SQL for MySQL target database. Note that
setting a large number of threads may have an adverse effect on database
performance since a separate connection is required for each thread.
Note Attunity Replicate assumes that MySQL Client 5.2.6 to 5.3.x for Linux or
MySQL ODBC Client 5.2.6 to 5.3.x 64-bit for Windows is installed on the
Attunity Replicate Server machine. If a version later than 5.3.x is installed,
you need to specify the version number as an internal parameter where
driver is the Parameter and MySQL ODBC <version> Unicode Driver is
the Value (where <version> is the client version e.g. 5.4).
For instructions on setting internal parameters, see Internal Parameters below.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 559
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 560
Using Google Cloud SQL for PostgreSQL as a Target
This section describes how to set up and use a Google Cloud SQL for PostgreSQL
target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Security Requirements
▶ Google Cloud SQL for PostgreSQL Database Target Data Types
▶ Data Types when Replicating from a PostgreSQL Source
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the client prerequisites when replicating to a Google
Cloud SQL for PostgreSQL target.
Attunity Replicate Server for Windows:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 561
Note If PgAdmin and the Greenplum client are installed on the same
Replicate Server, tasks configured to use a Google Cloud SQL for
PostgreSQL target will fail.
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
Debug = 0
CommLog = 1
UsageCount = 2
Security Requirements
The user specified in the General tab when Setting General Connection Properties must
be a registered user in the Google Cloud SQL for PostgreSQL database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 562
Replicate and the default mapping from Attunity Replicate data types. Unsupported
data types are listed below the table.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.15 |
Supported Google Cloud SQL for PostgreSQL database Data Types
with Mapping from Attunity Replicate Data Types
Attunity Replicate Data Google Cloud SQL for PostgreSQL database Data
Types Types
BOOL BOOL
BYTES BYTEA
DATE DATE
TIME TIME
TIMESTAMP
VARCHAR (37)
INT1 SMALLINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 FLOAT4
REAL8 FLOAT8
VARCHAR (65535)
UINT1 SMALLINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 563
Table 9.15 | Supported Google Cloud SQL for PostgreSQL database Data Types with
Map-ping from Attunity Replicate Data Types (Cont.)
Attunity Replicate Data Google Cloud SQL for PostgreSQL database Data
Types Types
UINT2 INTEGER
UINT4 BIGINT
UINT8 BIGINT
VARCHAR (65535)
BLOB BYTEA
NCLOB TEXT
CLOB TEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 564
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Google
Cloud SQL for PostgreSQL database. This is optional.
4. Select TARGET as the database role.
5. From the Type drop-down list, select Google Cloud SQL for PostgreSQL
6. Type the Server name. This is the name or IP address or host name of the computer
with the Google Cloud SQL for PostgreSQL database that you want to access.
7. Optionally, change the default port (5432).
8. Enter the Google Cloud SQL for PostgreSQL database authentication information
(User name, Password) of an authorized Google Cloud SQL for PostgreSQL user.
If you do not know this information, see your Google Cloud SQL for PostgreSQL
database system manager.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the specified Google Cloud SQL for
PostgreSQL database user has the correct access privileges.
9. Type the Database name or select one from the list of available endpoints. This
is the name of the Google Cloud SQL for PostgreSQL database to which you are
replicating data.
10. Click OK to save your settings and close the dialog box.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 565
SSL Authentication Options
In the Security section, you can configure Replicate to connect to the Google Cloud
SQL for PostgreSQL database using SSL.
SSL Mode: Select one of the following:
Prefer - Establishes an encrypted connection if the server supports encrypted
connections, falling back to an unencrypted connection if an encrypted
connection cannot be established.
Disable - Connect with a surname and password only.
Allow - Establish an encrypted connection if requested by the server.
Require - Establishes an encrypted connection if the server supports
encrypted connections. The connection attempt fails if an encrypted
connection cannot be established.
Verify CA - Similar to Required, but also verifies the server Certificate
Authority (CA) certificate against the configured CA certificates. The
connection attempt fails if no valid matching CA certificates are found.
Verify Full - Similar to Verify CA, but also performs host name identity
verification by checking the host name the client (i.e. Replicate) uses for
connecting to the server against the identity in the certificate that the
server sends to the client. The client checks whether the host name that it
uses for connecting matches the Common Name value in the server
certificate. The connection fails if there is a mismatch.
Client certificate path: - The path to the client certificate requested by the server.
Private key path: - The path to the client private key file in PEM format.
CA path - The path to the Certificate Authority (CA) that issued the client
certificate file in PEM format.
CRL path - The path to the CRL certificate. This file contains certificates revoked
by certificate authorities. If the server certificate appears in this list, the
connection will fail.
SSL compression - Select this option to compress the data before it is encrypted.
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before the
file is loaded into the Google Cloud SQL for PostgreSQL target database. The default
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 566
value is 32000 KB.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 567
Using Google Cloud Storage as a Target
This chapter describes how to set up and use Google Cloud Storage as a target
in a replication task.
In this section:
▶ Prerequisites
▶ Google Cloud Storage Target Overview
▶ DDL Handling
▶ Limitations
▶ Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Generating Reference Files
Prerequisites
Before you can use Google Cloud Storage as a target endpoint in a Replicate task,
the following prerequisites must be met:
Java:
1. If not already installed, install Java 64-bit on the Replicate Server machine.
2. Set the environment variable JAVA_HOME to point to the Java home
directory (e.g. C:\Program Files\Java\<version>).
3. Restart the Attunity Replicate Server service.
Permissions: The JSON credentials specified in the Google Cloud Storage
endpoint's Google Cloud Storage settings must be for an account that has read
and write access to the specified bucket and folder.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 568
files can be either CSV or JSON files. While the explanations in this section relate to
CSV files, the same is true for JSON files
Full Load files are named using incremental counters e.g. LOAD00001.csv, LOAD
00002.csv, etc. whereas Change Processing files are named using timestamps e.g.
20141029-1134010000.csv.
Note When the Create metadata files in the target folder option is enabled, a
corresponding metadata file is created using the same naming format, but with a
.dfm extension.
For each source table, a folder is created in the specified Google Cloud Storage target
folder. The data files are created on the Replicate Server machine and are then uploaded
to the specified Google Cloud Storage target folder once the File Attributes (Full Load)
and Change Processing upload conditions have been met.
DDL Handling
When a DDL change is captured, Replicate will close the data file and also create a DFM
file if the Create metadata files in the target folder option is enabled. When the next
batch of changes arrives, Replicate will create a new data file containing the changes.
Note that the DFM file created for the new data file will match the new table structure.
Limitations
The following limitations apply to the Google Cloud Storage target endpoint:
The following DDLs are supported only: Truncate table, Drop table, Create table,
Add Column, Rename Column, Drop Column, and Convert Data Type.
Full LOB Mode is not supported
UPDATE and DELETE statements are not supported in Apply Changes replication mode
Data Types
The following table shows the default mapping from Attunity Replicate data types to
Google Cloud Storage target data types. Note that the data type mapping is only
relevant if the Create metadata files in the target folder option is enabled.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 569
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.16 |
Supported Google Cloud Storage Target Data Types with Mapping
from Attunity Replicate Data Types
TIME TIME
DATETIME DATETIME
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 570
Setting General Connection Properties
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
Table 9.17 |
Google Cloud Storage Target Endpoint - General Tab Options
Option Description
Google Cloud Storage
JSON credentials The JSON credentials for the service account key with read and write
access to the Google Cloud Storage bucket.
Target folder Where to create the data files in the specified bucket.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 571
Table 9.17 | Google Cloud Storage Target Endpoint - General Tab Options (Cont.)
Option Description
File Attributes
Delimiters can be standard characters or a hexadecimal (hex) value. Note that the
"0x" prefix must be used to denote a hexadecimal delimiter (e.g. 0x01 = SOH). In the
Field delimiter, Record delimiter and Null value fields, the delimiter can consist of
concatenated hex values (e.g. 0x0102 = SOHSTX), whereas in the Quote character
and Escape character fields, it can only be a single hex value.
Note The hexadecimal number 0x00 is not supported (i.e. only 0x01-0xFF
are supported).
Format You can choose to create the target files in CSV or JSON format.
{ "book_id": 456, "title": "Winnie the Pooh", "price": 6.49, "is_hardcover": true }
{ "book_id": 789, "title": "The Cat in the Hat", "price": 7.23, "is_hardcover": true }
Note Changing the format (i.e. from CSV to JSON or from JSON
to CSV) while the task is in a stopped state and then resuming
the task, is not supported.
Field delimiter The delimiter that will be used to separate fields (columns) in the
target files. The default is a comma.
"mike","male"
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 572
Table 9.17 | Google Cloud Storage Target Endpoint - General Tab Options (Cont.)
Option Description
Record delimiter The delimiter that will be used to separate records (rows) in the target
files. The default is a newline (\n).
Example:
"mike","male"\n
"sara","female"\n
Null value The string that will be used to indicate a null value in the target files.
"mike","male",295678\n
"sara","female",@\n
Quote character The character that will be used at the beginning and end of a text
column. The default is the double-quote character ("). When a column
that contains column delimiters is enclosed in double-quotes, the
column delimiter characters are interpreted as actual data, and not as
column delimiters.
@mike@,@male@
Escape character The character used to escape a quote character in the actual data.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 573
Table 9.17 | Google Cloud Storage Target Endpoint - General Tab Options (Cont.)
Option Description
Add metadata When the target storage format is set to Text, you can optionally add a
header header row to the data files. The header row can contain the source
column names and/or the intermediate (i.e. Replicate) data types.
Example of a target file with a header row when both With column
names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
Maximum file The maximum size a file can reach before it is closed (and optionally
size compressed). This value applies both to data files and to Reference
Files.
Compress files Choose GZIP to compress the target files or NONE (the default) to
using leave them uncompressed.
Change Processing
File size reaches Specify the maximum size of Change Data to accumulate before
uploading the file to Google Cloud Storage .
Elapsed time Specify the maximum time to wait before applying the changes.
reaches
Metadata Files
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 574
Table 9.17 | Google Cloud Storage Target Endpoint - General Tab Options (Cont.)
Option Description
Create metadata When this option is selected, for each data file, a matching metadata
files in the target file with a .dfm extension will be created under the specified target
folder folder. The metadata files (which are in standard JSON format) provide
additional information about the task/data such as the source endpoint
type, the source table name, the number of records in the data file,
and so on.
For a full description of the metadata file as well as possible uses, see
Metadata File Description .
Table 9.18 | Google Cloud Storage Target Endpoint - Advanced Tab Options
Option Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 575
Table 9.18 | Google Cloud Storage Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can process the final target files using a custom command. The command will be
run whenever a data file is created.
Note If the Generate a reference file option is selected, a row (specifying the
file's location) will be added to the Reference File only after the command
completes successfully.
Working directory - The directory where you want the command to run.
${METADATA_FILENAME} - The full path to the DFM file containing the metadata.
Note If the CSV/DFM file paths contain spaces, you must enclose
these parameters with quotation marks (e.g "${FILENAME}").
The post-processing command must return a proper exit code. You can either use the
standard exit code values described below or set a custom exit code value as described
in Setting Post Command Exit Codes with an Internal Parameter below.
0 - Success
1 - Recoverable error. The task will recover from the point of failure according to
the settings in the Environmental Errors tab.
2 - Table error. If a table error occurs, Replicate will handle the error according
to the settings in the Table Errors tab.
3 (or any other value e.g. -100) - Fatal error. The task will fail and not
attempt recovery.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 576
Table 9.18 | Google Cloud Storage Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can use internal parameters to set exit codes with custom values. This is
especially useful if your application already uses the standard exit code values.
See Standard Post Command Exit Codes above for a description of the exit codes.
successExitCode
recoverableErrorExitCode
tableErrorExitCode
fatalErrorExitCode
Select this option to generate a Reference File (on Replicate Server) containing the
full path to the Apply Changes data files.
Note The reference file only points to the location of the Change Processing files,
and not the Full Load files.
Reference File(s) folder - The folder on the Replicate machine in which the
Reference File will be created.
Example:
c:\temp\
User name The user name for accessing the proxy server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 577
Table 9.18 | Google Cloud Storage Target Endpoint - Advanced Tab Options (Cont.)
Option Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 578
Internal Parameters
Internal parameters are parameters that are not exposed in the UI and should only be
used if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your setting by clicking the View Setting Summary link.
This is useful if you need to send a summary of your setting to Attunity Support.
Example:
MyGoogleCloudStorage00000001.csv
Note The counter suffix increases incrementally each time a new Reference File is
generated (i.e. which occurs when the file reaches the maximum size defined in
the General tab). Once a new Reference File has been generated, you can delete
the old reference file(s) if required.
Whenever an Change Processing data file is created, a new row is added to the
Reference File in the following format:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 579
<Source_Table_Name>,<Data_Lake_Store_name>/<path>/<file_name>
Example:
employees,mydatalakestore/new/files/my.company/20170611-120144192.csv
Note that if the Run command after upload option in the Advanced tab is also
enabled, the Reference File will be generated after the post-processing completes.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 580
Using Google Dataproc as a Target
This section describes how to set up and use Google Dataproc as the target endpoint
in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Change Data Partitioning on Google Dataproc
▶ Google Dataproc Endpoint Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you begin to work with Google Dataproc as a target in Attunity Replicate, make
sure that the following prerequisites have been met:
Java:
1. If not already installed, install Java 64-bit on the Replicate Server machine.
2. Set the environment variable JAVA_HOME to point to the Java home
directory (e.g. C:\Program Files\Java\<version>).
3. Restart the Attunity Replicate Server service.
General:
The storage location must be accessible from the Attunity Replicate machine.
The user specified in the Google Dataproc target endpoint's Hive access
settings must have access to HiveServer2.
ODBC Driver (Windows and Linux): Install Hortonworks Hive ODBC driver
2.1.10 or above, or any other Google Dataproc compatible driver on the Replicate
Server machine.
Bucket and folder permissions: The JSON Credentials specified in the Google
Dataproc endpoint's Google Cloud Storage settings must be for an account that
has read and write access to the specified bucket and target folder.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 581
Hive table permissions: Replicate requires permissions to perform the following
operations on Hive tables: CREATE, DROP, DESCRIBE, and ALTER table. ALTER
table may also include RENAME table, ADD/RENAME column, and CREATE/DROP
partitions on tables.
Limitations
The following limitations apply:
UPDATE/DELETE DMLs are not supported during change processing. If an
UPDATE/DELETE DML was captured on the source, it will be ignored on the target
and a warning will be written to the log. If the Store Changes option is enabled in
the task settings, these records will be written to the Change Table.
Limited LOB support only.
Dropping columns and changing column data types or the data type length (e.g.
VARCHAR(50) to VARCHAR(100)) is not supported.
Proxy:
Does not affect the ODBC (Hive) connection (i.e. it affects the storage
connection only).
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 582
under the Change Table directory. Whenever the specified partition timeframe ends,
a partition is created in Hive, pointing to Google Cloud Storage.
Information about the partitions is written to the attrep_cdc_partitions Control Table.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.19 |
Supported Google Dataproc Data Types with Mapping from
Attunity Replicate Data Types
BOOL BOOLEAN
BYTES STRING
TIME TIMESTAMP
DATETIME TIMESTAMP
DATE DATE
INT1 TINYINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
REAL4 FLOAT
REAL8 DOUBLE
UINT1 SMALLINT
UINT2 INT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 583
Table 9.19 | Supported Google Dataproc Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
UINT4 BIGINT
BLOB STRING
NCLOB STRING
CLOB STRING
5. Configure the remaining settings in the General tab as described in the table below.
OptionDescription
JSON credentials The JSON credentials for the service account key with read and
write access to the Google Cloud Storage bucket.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 584
Bucket name The Google Cloud Storage bucket.
Target folder Where to create the data files in the specified bucket.
Hive Access
Host The host name of the Hive server.
Authentication type Select either User name or User name and Password.
Table 9.20 |
Google Dataproc Target - Advanced Properties
Setting Description
File Format Expand this section to specify or view the file format
settings.
Field delimiter The delimiter that will be used to separate fields in the target file.
The default is \001. Note that field delimiters must be in Octal
format (e.g. \247)
Null value The value that will be used to indicate a null value in the target
file. When using the default SerDe (LazySimpleSerde), setting
the null value is supported from Hive 0.13.
mike,male,295678
sara,female,@
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 585
Table 9.20 | Google Dataproc Target - Advanced Properties (Cont.)
Setting Description
Escape character The escape character is used to escape the field delimiter
character. When a field delimiter is escaped, it is interpreted as
actual data, and not as a field delimiter.
sunroof\,power-steering
Add metadata header When the target storage format is set to Text, you can optionally
add a header row to the data files. The header row can contain
the source column names and/or the intermediate (i.e. Replicate)
data types.
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
File Attributes Expand this section to specify or view the file attributes.
Maximum file size Specify the maximum file size of each target file. When the data
reaches the maximum size, the file will be closed and written to
the specified target folder.
Compress files using Select the compression method to use on Google Cloud Storage.
File size reaches Specify the minimum size of the data required to create a file in
idle state.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 586
Table 9.20 | Google Dataproc Target - Advanced Properties (Cont.)
Setting Description
Elapsed time Specify the maximum time to wait before applying the changes in
reaches idle state.
Note To facilitate rapid delivery of DDL messages, files are uploaded immediately,
regardless of the specified File size reaches or Elapsed time reaches values.
Use proxy server Select this option to access Google Dataproc via a proxy server.
User name The user name for accessing the proxy server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 587
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 588
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 589
Using Hadoop as a Target
This section describes how to set up and use Hadoop as the target endpoint in a
replication task.
In this section:
▶ Prerequisites
▶ Prerequisites for using the Cloudera Distribution as a Hadoop Target
▶ Prerequisites for using a Linux ODBC Driver
▶ Limitations
▶ Change Data Partitioning on Hadoop
▶ Prerequisites
▶ Security Requirements
▶ Hadoop Endpoint Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Preventing ODBC Connection Timeouts
▶ Internal Parameters
▶ Settings Summary
▶ Using Kerberos Authentication
▶ Using Kerberos Authentication on Linux
▶ Using Kerberos Authentication on Windows
▶ Prerequisites for using Kerberos when Replicate is installed on Red Hat 7.4 or 7.5
Prerequisites
Before you begin to work with a Hadoop cluster as a target in Attunity Replicate, make
sure that the following prerequisites have been met:
General:
The Hadoop WebHDFS must be accessible from the Attunity Replicate machine.
The Hadoop Data Nodes must be accessible from the Attunity Replicate machine.
ODBC Access:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 590
When accessing Hive using ODBC, the following ODBC drivers are supported:
Note Cloudera ODBC drivers 2.5.20 and above do not support the
Snappy compression method.
SSL: Before you can use SSL, you first need to perform the following tasks:
Configure each NameNode and each DataNode with an SSL certificate (issued
by the same CA).
Place the CA certificate on the Replicate Server machine. The certificate
should be a base64-encoded PEM (OpenSSL) file.
Permissions: The user specified in the Hadoop target settings must have
write permission for the specified HDFS target directory.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 591
Limitations
The following limitations apply:
UPDATE/DELETE DMLs are not supported during change processing. If an
UPDATE/DELETE DML was captured on the source, it will be ignored on the target
and a warning will be written to the log. If the Store Changes option is enabled in
the task settings, these records will be written to the Change Table.
Limited LOB support only.
Dropping columns and changing column data types or the data type length (e.g.
VARCHAR(50) to VARCHAR(100)) is not supported.
Due to a Hive limitation, in Hive version 0.12 and earlier versions, only
alphanumeric and underscore characters are allowed in table and column names.
From Hive 0.13, column names can contain any Unicode character.
The Replicate Hadoop target endpoint does not support the creation of skews,
buckets or partitions in the target tables. Tables with such properties must already
exist on the target before the task starts. For more information on preparing such
tables in Hive, see Support for Partitions, Buckets and Skews.
When loading data into existing skewed, bucketed or partitioned target tables,
the following limitations apply:
The DROP and CREATE table and ARCHIVE and CREATE table options in
the task settings’ Full Load Settings tab should not be selected (as the new
table will be created without the special properties).
The Apply Changes replication option is not supported.
The following Control Tables are not supported as they require UPDATE/DELETE
operations (which are not supported by the Hadoop target endpoint):
Primary keys are not supported on Hive versions prior to 2.1. From Hive 2.1,
primary keys are supported when accessing Hive using ODBC only.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 592
Change Data Partitioning on Hadoop
When Change Data Partitioning is enabled, the Replicate Change Tables in Hive are
partitioned by the partition_name column. Data files are uploaded to HDFS, according
to the maximum size and time definition, and then stored in a directory under the
Change Table directory. Whenever the specified partition timeframe ends, a partition is
created in Hive, pointing to the HDFS directory.
Information about the partitions is written to the attrep_cdc_partitions Control Table.
Prerequisites
The prerequisites for using Change Data Partitioning with a Hadoop target endpoint are
as follows:
The target file format must be set to Text or
Security Requirements
The Hadoop NameNode (and data nodes when using WebHDFS) must be accessible
from the Attunity Replicate machine and the user specified in the Hadoop target
settings must have write permission for the specified HDFS target directory.
Hive table permissions: Replicate requires permissions to perform the following
operations on Hive tables: INSERT from SELECT, CREATE, DROP, DESCRIBE, and
ALTER table. ALTER table may also include RENAME table, ADD/RENAME column,
and CREATE/DROP partitions on tables.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 593
Table 9.21 |
Supported Hadoop Data Types with Mapping from Attunity Replicate
Data Types
BYTES STRING
TIME TIMESTAMP
DATETIME TIMESTAMP
DATE DATE
Note When Avro is selected as the Target storage format, the TIMESTAMP and
DATE data types (which are not supported by Avro) are mapped to VARCHAR(37).
INT1 TINYINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
DECIMAL (p,s)
DECIMAL
REAL4 FLOAT
REAL8 DOUBLE
VARCHAR (Length)
STRING
UINT1 SMALLINT
UINT2 INT
UINT4 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 594
Table 9.21 | Supported Hadoop Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
DECIMAL (20,0)
DECIMAL
VARCHAR (Length)
STRING
BLOB STRING
NCLOB STRING
CLOB STRING
1. Create the tables in Hive with these attributes (partitions, buckets or skews) prior
to running the task.
2. Add the following values to the
hive.security.authorization.sqlstd.confwhitelist.append property in
the Hive configuration file:
If the target tables are partitioned:
|hive.exec.dynamic.partition|hive.exec.dynamic.partition.mode
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 595
If the target tables have skews:
|hive.mapred.supports.subdirectories
Note In some Hadoop Distributions, you may need to specify the value without
the "hive" prefix.
For example, |enforce.bucketing instead of |hive.enforce.bucketing.
3. Set the Target Table Preparation task setting to Truncate before loading or
Do nothing. For more information on these settings, see Full Load Settings.
a. To encrypt the data between the Replicate machine and HDFS, select Use SSL.
In order to use SSL, first make sure that the SSL prerequisites described in
Prerequisites been met.
In the CA path field, either specify the directory containing the CA
certificate. -OR-
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 596
User name - Select to connect to the Hadoop cluster with only a user
name. Then, in the User name field, specify the name of a user
authorized to access the Hadoop cluster.
User name and password - Select to connect to the Hadoop NameNode or
to the Knox Gateway (when enabled - see below) with a user name and
password. Then, in the User name and Password fields, specify the
required user name and password.
Kerberos - Select to authenticate against the Hadoop cluster using Kerberos.
Replicate automatically detects whether Attunity Replicate Server is running
on Linux or on Windows and displays the appropriate settings.
Use global Kerberos ticket file - Select this option if you want to
use the same ticket for several Hadoop endpoints (source or target).
In this case, you must make sure to select this option for each
Hadoop endpoint instance that you define.
Use specific Kerberos ticket file - Select this option if you want
to use a different ticket file for each Hadoop endpoint (source or
target). Then specify the ticket file name in the designated field.
This option is especially useful if you need to perform a task-level audit of
Replicate activity (using a third-party tool) on the Hadoop NameNode. To
set this up, define several instances of the same Hadoop endpoint and
specify a unique Kerberos ticket file for each instance. Then, for each
task, simply select a different Hadoop endpoint instance.
Note
If your Replicate server is installed on Red Hat 7.4 or 7.5, you need
to rename the Kerberos libraries by performing the procedure
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 597
described in Prerequisites for using Kerberos when Replicate
is installed on Red Hat 7.4 or 7.5.
You need to define a global Kerberos ticket file even if you select
the Use specific Kerberos ticket file option. The global
Kerberos ticket file is used for authentication when selecting a
Hive endpoint, when testing the connection (using the Test
Connection button), and when selecting which tables to replicate.
When replicating from a Hadoop source endpoint to a Hadoop
target endpoint, both endpoints must be configured to use the
same ticket file.
Realm: The name of the realm in which your Hadoop cluster resides.
Note When the Replicate KDC and the Hadoop KDC are in
different domains, a relationship of trust must exist between the
two domains.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 598
Realm: The name of the realm/domain in which your Hadoop
cluster resides (where realm is the MIT term while domain is the
Active Directory term).
Principal: The user name to use for authentication. The principal
must be a member of the realm/domain entered above.
When Active Directory is selected - Password: The password for
the principal entered above.
When MIT is selected - Keytab file: The keytab file containing
the principal entered above.
Important: Make sure that the specified user has the required
Hadoop access privileges. For information on how to provide the
required privileges, see Security Requirements.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 599
Note To be able to select this option, first select Use SSL and then
select Password from the Authentication type drop-down list.
Knox Gateway host - The FQDN (Fully Qualified Domain Name) of the
Knox Gateway host.
Knox port - The port number to use to access the host. The default is "8443".
Knox Gateway path - The context path for the gateway. The default
is "gateway".
Note The port and path values are set in the gateway-site.xml file. If
you are unsure whether the default values have been changed, contact
your IT department.
Cluster name - The cluster name as configured in Knox. The default is "default".
7. In the HDFS section, select WebHDFS, HttpFS or NFS as the HDFS access
method. If you are accessing MapR, it is recommended to use HttpFS.
Note When the Use Knox Gateway option is selected, the NameNode, HttpFS
Host, and Port fields described below are not relevant (and are therefore hidden).
In the Target Folder field, specify where to create the data files on HDFS.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 600
In the HttpFS Host field, specify the IP address of the HttpFS host.
In the Port field, optionally change the default port (14000).
In the Target Folder field, specify where to create the data files on HDFS.
In the Target folder field, enter the path to the folder located under the
MapR cluster mount point. For example: /mapr/my.cluster.com/data
In order to do this, you first need to mount the MapR cluster using NFS.
For information on how to do this, refer to the MapR help.
Note Due to a Hadoop limitation, the Target folder name can only contain
ASCII characters.
a. From the Access Hive using drop-down list, select one of the following options:
Note When the Use Knox Gateway option is selected, the Host and
Port fields described below are not relevant (and are therefore hidden).
ODBC - Select this option to access Hive using an ODBC driver (the default).
Then continue with the Host field.
Note If you select his option, make sure that the latest 64-bit ODBC
driver for your Hadoop distribution is installed on the Attunity
Replicate Server machine.
Note When this option is selected, the target storage format must be
set to "Text".
No Access - When this option is selected, after the data files are created
on HDFS, Replicate will take no further action.
b. In the Host field, specify the IP address of the Hive machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 601
c. In the Port field, optionally change the default port.
d. In the Database field, specify the name of the Hive target database.
Table 9.22 |
Hadoop Target - Advanced Properties
Setting Description
File Format Expand this section to specify or view the file format
settings.
Note For both regular tables and Replicate Control Tables, creating and storing
the tables in text format (the default) allows data to be appended to them. This in
turn reduces the number of files created on Hadoop, improves query
performance, and reduces the number of Hive jobs running.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 602
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Target storage Select one of the following target storage formats: Text (the
format default), Avro, ORC, Parquet, Sequence.
Use Default SerDe Choose the SerDe interface to use when accessing the
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 603
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Field delimiter The delimiter that will be used to separate fields in the target file.
The default is \001. Note that field delimiters must be in Octal
format (e.g. \247)
Null value The value that will be used to indicate a null value in the target
file. When using the default SerDe (LazySimpleSerde), setting the
null value is supported from Hive 0.13.
mike,male,295678
sara,female,@
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 604
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 605
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Record delimiter The \n delimiter is used to separate records (rows) in the target
files. When using the default SerDe (LazySimpleSerde), the
record delimiter cannot be changed.
Quote character The quote character is used to escape the field delimiter
character. When a field delimiter is escaped, it is interpreted as
actual data, and not as a field delimiter. Note that the quote
character is not available when using the default SerDe
(LazySimpleSerde).
"mike,male"
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 606
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
SerDe properties Enter the SerDe properties if Other SerDe is selected and the
SerDe properties are not the same as the Hadoop defaults
(field.delim, serialization.null.format, escape.delim,
line.delim, quote.delim).
The properties should be written using the following format:
"KEY1=VALUE1,KEY2=VALUE2,KEY3=VALUE3"
The list of properties should begin and end with a quotation mark.
Example:
"separatorChar=\t,escapeChar={,quoteChar=’"
Add metadata header When the target storage format is set to Text, you can
optionally add a header row to the data files. The header row
can contain the source column names and/or the intermediate
(i.e. Replicate) data types.
Example of a target file with a header row when both With
column names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
File Attributes Expand this section to specify or view the file attributes.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 607
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Use Hadoop defaults Select to work with the default block size of your Hadoop target.
Use this block size Select to work with a different block size. The default value is 64.
(MB)
Maximum file size Specify the maximum file size of each target file. When the data
reaches the maximum size, the file will be closed and written to
the specified target folder.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 608
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
For Avro:
|hive.exec.compress.output|avro.output.codec
For Parquet:
|hive.exec.compress.output|parquet.compression
For Text:
|hive.exec.compress.output|parquet.compression
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 609
Table 9.22 | Hadoop Target - Advanced Properties (Cont.)
Setting Description
Consider state idle Specify how long to wait before considering the state to be idle. In
when no changes idle state, you can create files from data that has already been
have been processed processed if the specified size and time conditions are met (see
for below).
File size reaches Specify the minimum size of the data required to create a file in
idle state.
Elapsed time Specify the maximum time to wait before applying the changes in
reaches idle state.
Note To facilitate rapid delivery of DDL messages, files are uploaded immediately,
regardless of the specified File size reaches or Elapsed time reaches values.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 610
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Note The commands described below should be issued under the "Attunity" user
or under the user that was selected during the Replicate installation.
1. Obtain a valid TGT (Ticket-Granting Ticket) from the Kerberos KDC (Key
Distribution Center) but save the TGT to a non-default cache file. Usually, a
keytab file is used to perform non-interactive authentication to Kerberos.
Command Syntax:
kinit -kt [keytab_file] -c [cache_file_name] [principal_name]
2. This step is only required for the global Kerberos ticket file. Set the Kerberos
cache environment variable (for Replicate to use later on).
To set the environment variable:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 611
a. Add the following command to the file:
export KRB5CCNAME=cache_file_name
Example:
export KRB5CCNAME=/temp/kerberos/global.ticket
Now, whenever Attunity Replicate needs to use Kerberos authentication, it will perform
the following operations:
When Use global Kerberos ticket file is selected: Replicate will check whether
the KRB5CCNAME environment variable is set and, if so, will use the ticket(s) inside
the cache file specified by the environment variable.
When Use specific Kerberos ticket file is selected:
During design-time (e.g. when selecting tables, testing the connection, etc.),
Replicate will use the ticket(s) inside the cache file specified by the
KRB5CCNAME environment variable.
During runtime, Replicate will use the ticket file specified in the Hadoop
endpoint settings.
Note If the ticket in the cache file expires or becomes invalid, repeating the
kinit command shown in Step 1 above will write a new TGT to the cache file
and allow Attunity Replicate to continue working. This can be done without
restarting the Attunity Replicate Server.
Perform the following steps to ensure that the impersonated user (principal) has the
Log on as a batch job privilege on the Attunity Replicate server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 612
1. On the Attunity Replicate server, open the Local Security Settings (Control
Panel > System Security > Administrative Tools > Local Security Policy).
2. In the console tree, expand Local Policies and select User Rights Assignments.
3. In the details pane, double-click Log on as a batch job.
4. In the Log on as a batch job Properties dialog box, on the Local Security
Settings tab, verify that the respective user is listed. If it is not listed, click Add
User or Group, then add the user and click OK.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 613
Your changes should take effect immediately.
To set Kerberos authentication on Windows with MIT Kerberos set in one of the
endpoints:
If MIT Kerberos is set in one of the endpoints, you need to perform the following steps
to allow the Attunity Replicate server process to keep a specific privilege on startup.
By default, Attunity Replicate server drops all privileges on startup. These steps are
not required if you use Active Directory KDC.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 614
3. Modify the PrivilegesKeep string to include the value SeTcbPrivilege.
To work with Kerberos when Replicate is installed on Red Hat 7.4 or 7.5, you need
to rename the libraries as outlined in the procedure below.
mv libk5crypto.so.3 libk5crypto.so.3.org
mv libkrb5.so.3 libkrb5.so.3.org
mv libkrb5support.so.0 libkrb5support.so.0.org
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 615
4. Start the Replicate service as described in Starting and Stopping an Attunity
Replicate Instance.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 616
Using Hortonworks Data Platform (HDP) as a Target
This section describes how to set up and use Hortonworks Data Platform (HDP) as
the target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Prerequisites for using a Linux ODBC Driver
▶ Limitations
▶ Change Data Partitioning
▶ Prerequisites
▶ Security Requirements
▶ HDP Endpoint Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Using Kerberos Authentication
▶ Using Kerberos Authentication on Linux
▶ Using Kerberos Authentication on Windows
▶ Prerequisites for using Kerberos when Replicate is installed on Red Hat 7.4 or 7.5
Prerequisites
Before you begin to work with an HDP cluster as a target in Attunity Replicate, make
sure that the following prerequisites have been met:
General:
The Hive Server must be accessible from the Attunity Replicate machine.
Hortonworks Hive ODBC driver 2.1.2 or above must be installed on the
Attunity Replicate machine.
SSL: Before you can use SSL, you first need to perform the following tasks:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 617
Configure each NameNode and each DataNode with an SSL certificate (issued
by the same CA).
Place the CA certificate on the Replicate Server machine. The certificate
should be a base64-encoded PEM (OpenSSL) file.
Permissions: The user specified in the HDP target settings must have write
permission for the specified HDFS or Amazon S3 target directory (depending on
your preferred storage type).
Limitations
The following limitations apply:
UPDATE/DELETE DMLs are not supported during change processing. If an
UPDATE/DELETE DML was captured on the source, it will be ignored on the target
and a warning will be written to the log. If the Store Changes option is enabled in
the task settings, these records will be written to the Change Table.
Limited LOB support only.
Dropping columns and changing column data types or the data type length (e.g.
VARCHAR(50) to VARCHAR(100)) is not supported.
The following Control Tables are not supported as they require UPDATE/DELETE
operations (which are not supported by the Hortonworks target endpoint):
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 618
Primary keys are not supported on Hive versions prior to 2.1. From Hive 2.1,
primary keys are supported when accessing Hive using ODBC only.
Prerequisites
The prerequisites for using Change Data Partitioning with the Hortonworks Data
Platform (HDP) endpoint are as follows:
The target file format must be set to Text or
Security Requirements
The HDP NameNode (and data nodes when using WebHDFS) must be accessible
from the Attunity Replicate machine and the user specified in the HDP target
settings must have write permission for the specified HDFS target directory.
Hive table permissions: Replicate requires permissions to perform the following
operations on Hive tables: CREATE, DROP, DESCRIBE, and ALTER table. ALTER
table may also include RENAME table, ADD/RENAME column, and CREATE/DROP
partitions on tables.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 619
Table 9.23 |
Supported HDP Data Types with Mapping from Attunity Replicate
Data Types
BYTES STRING
TIME TIMESTAMP
DATETIME TIMESTAMP
DATE DATE
INT1 TINYINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
DECIMAL (p,s)
DECIMAL
REAL4 FLOAT
REAL8 DOUBLE
VARCHAR (Length)
STRING
UINT1 SMALLINT
UINT2 INT
UINT4 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 620
Table 9.23 | Supported HDP Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
DECIMAL (20,0)
DECIMAL
VARCHAR (Length)
STRING
BLOB STRING
NCLOB STRING
CLOB STRING
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 621
Note These settings are relevant for HDFS storage and Hive only.
a. To encrypt the data between the Replicate machine and HDFS, select Use SSL.
In order to use SSL, first make sure that the SSL prerequisites described in
Prerequisites been met.
In the CA path field, specify one of the following:
User name - Select to connect to the HDP cluster with only a user
name. Then, in the User name field, specify the name of a user
authorized to access the cluster.
User name and password - Select to connect to the HDP NameNode or
to the Knox Gateway (when enabled) with a user name and password.
Then, in the User name and Password fields, specify the required user
name and password.
Kerberos - Select to authenticate against the HDP cluster using Kerberos.
Replicate automatically detects whether Attunity Replicate Server is
running on Linux or on Windows and displays the appropriate settings.
Use global Kerberos ticket file - Select this option if you want to
use the same ticket for several HDP endpoints. In this case, you must
make sure to select this option for each HDP endpoint instance that
you define.
Use specific Kerberos ticket file - Select this option if you want to use
a different ticket file for each HDP endpoint. Then specify the ticket
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 622
file name in the designated field.
Note
Realm: The name of the realm in which your HDP cluster resides.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 623
Attunity Replicate Server on Windows:
Note When the Replicate KDC and the HDP KDC are in
different domains, a relationship of trust must exist between
the two domains.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 624
Important: Make sure that the specified user has the required
HDP access privileges. For information on how to provide the
required privileges, see Security Requirements.
6. If you need to access the HDP distribution through a Knox Gateway, select Use
Knox Gateway. Then provide values for the following fields:
Note To be able to select this option, first select Use SSL and then
select Password from the Authentication type drop-down list.
Knox Gateway host - The FQDN (Fully Qualified Domain Name) of the
Knox Gateway host.
Knox port - The port number to use to access the host. The default is "8443".
Knox Gateway path - The context path for the gateway. The default
is "gateway".
Note The port and path values are set in the gateway-site.xml file. If
you are unsure whether the default values have been changed, contact
your IT department.
Cluster name - The cluster name as configured in Knox. The default is "default".
Storage
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 625
Option Description
Amazon S3 Storage
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 626
Option Description
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_
roles.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_
providers_saml.html
Access key Enter the access key information for Amazon S3.
Secret key Enter the secret key information for Amazon S3.
ADFS URL The URL to an Active Directory Federation Services page, responsible
for returning a SAML claims document to be sent over to AWS.
AD principal The principal (user) name to use when identifying against ADFS
name
The format should be: user.name@domain
AD principal The principal password to use when identifying against ADFS
password
IdP ARN The Amazon Resource Name (ARN) of the Active Directory issuing the
SAML claims document. This is required as it enables AWS to identify
the signer of the SAML document and verify its signature.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 627
Option Description
SAML Role The Amazon Resource Name (ARN) of the specific role the returned
ARN credentials should be assigned.
a. From the Access Hive using drop-down list, select one of the following options:
Note When the Use Knox Gateway option is selected, the Host and
Port fields described below are not relevant (and are therefore hidden).
ODBC - Select this option to access Hive using an ODBC driver (the default).
Then continue with the Host field.
Note If you select his option, make sure that the latest 64-bit ODBC
driver for your Hadoop distribution is installed on the Attunity
Replicate Server machine.
Note When this option is selected, the target storage format must be
set to "Text".
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 628
No Access - When this option is selected, after the data files are created
on HDFS, Replicate will take no further action.
b. In the Host field, specify the IP address of the Hive machine.
c. In the Port field, optionally change the default port.
d. In the Database field, specify the name of the Hive target database.
Table 9.24 |
Hortonworks Data Platform (HDP) Target - Advanced Properties
Setting Description
File Format Expand this section to specify or view the file format
settings.
Target storage format Select the target storage format: Text (the default) or
Sequence.
Field delimiter The delimiter that will be used to separate fields in the target
file. The default is \001. Note that field delimiters must be in
Octal format (e.g. \247)
Null value The value that will be used to indicate a null value in the target
file.
mike,male,295678
sara,female,@
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 629
Table 9.24 | Hortonworks Data Platform (HDP) Target - Advanced Properties (Cont.)
Setting Description
sunroof\,power-steering
"\"sunroof, power-steering\""
Add metadata header When the target storage format is set to Text, you can
optionally add a header row to the data files. The header row
can contain the source column names and/or the intermediate
(i.e. Replicate) data types.
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
Maximum file size Specify the maximum file size of each target file. When the
data reaches the maximum size, the file will be closed and
written to the specified target folder.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 630
Table 9.24 | Hortonworks Data Platform (HDP) Target - Advanced Properties (Cont.)
Setting Description
Apply/store changes Specify how long to wait before applying or storing the
when: changes.
File size reaches Specify the minimum size of the data file required to apply or
store the changes.
Elapsed time reaches Specify the maximum time to wait before applying or storing
the changes.
Note To facilitate rapid delivery of DDL messages, files are uploaded immediately,
regardless of the specified File size reaches or Elapsed time reaches values.
Use proxy server Select this option to access the storage via a proxy server.
User name The user name for accessing the proxy server.
SSL When HDFS is the Storage type and HTTPS is the selected
CA Path Scheme, specify the location of the CA file on the Replicate
Server machine .
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 631
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Note The commands described below should be issued under the "Attunity" user
or under the user that was selected during the Replicate installation.
1. Obtain a valid TGT (Ticket-Granting Ticket) from the Kerberos KDC (Key
Distribution Center) but save the TGT to a non-default cache file. Usually, a
keytab file is used to perform non-interactive authentication to Kerberos.
Command Syntax:
kinit -kt [keytab_file] -c [cache_file_name] [principal_name]
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 632
2. This step is only required for the global Kerberos ticket file. Set the Kerberos
cache environment variable (for Replicate to use later on).
To set the environment variable:
export KRB5CCNAME=cache_file_name
Example:
export KRB5CCNAME=/temp/kerberos/global.ticket
Now, whenever Attunity Replicate needs to use Kerberos authentication, it will perform
the following operations:
When Use global Kerberos ticket file is selected: Replicate will check whether
the KRB5CCNAME environment variable is set and, if so, will use the ticket(s) inside
the cache file specified by the environment variable.
When Use specific Kerberos ticket file is selected:
During design-time (e.g. when selecting tables, testing the connection, etc.),
Replicate will use the ticket(s) inside the cache file specified by the
KRB5CCNAME environment variable.
During runtime, Replicate will use the ticket file specified in the HDP
endpoint settings.
Note If the ticket in the cache file expires or becomes invalid, repeating the
kinit command shown in Step 1 above will write a new TGT to the cache file
and allow Attunity Replicate to continue working. This can be done without
restarting the Attunity Replicate Server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 633
Using Kerberos Authentication on Windows
Before beginning, make sure that the impersonated user (principal) has access to the
Replicate Data directory (<product_dir>\Data by default) on the Attunity Replicate server.
For Active Directory KDC, the impersonated user is the user configured in the user interface.
For MIT KDC, this is the Windows user to which the MIT principal is mapped.
Perform the following steps to ensure that the impersonated user (principal) has the
Log on as a batch job privilege on the Attunity Replicate server.
1. On the Attunity Replicate server, open the Local Security Settings (Control
Panel > System Security > Administrative Tools > Local Security Policy).
2. In the console tree, expand Local Policies and select User Rights Assignments.
3. In the details pane, double-click Log on as a batch job.
4. In the Log on as a batch job Properties dialog box, on the Local Security
Settings tab, verify that the respective user is listed. If it is not listed, click Add
User or Group, then add the user and click OK.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 634
Your changes should take effect immediately.
To set Kerberos authentication on Windows with MIT Kerberos set in one of the
endpoints:
If MIT Kerberos is set in one of the endpoints, you need to perform the following steps
to allow the Attunity Replicate server process to keep a specific privilege on startup.
By default, Attunity Replicate server drops all privileges on startup. These steps are
not required if you use Active Directory KDC.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 635
3. Modify the PrivilegesKeep string to include the value SeTcbPrivilege.
To work with Kerberos when Replicate is installed on Red Hat 7.4 or 7.5, you need
to rename the libraries as outlined in the procedure below.
mv libk5crypto.so.3 libk5crypto.so.3.org
mv libkrb5.so.3 libkrb5.so.3.org
mv libkrb5support.so.0 libkrb5support.so.0.org
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 636
4. Start the Replicate service as described in Starting and Stopping an Attunity
Replicate Instance.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 637
Using HP Vertica as a Target
This section describes how to set up and use an HP Vertica database as a target
database in a replication task.
In this section:
▶ Prerequisites
▶ Replicate Server for Windows
▶ Replicate Server for Linux
▶ Limitations
▶ Security Requirements
▶ HP Vertica Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the prerequisites for working with Attunity Replicate
on Windows or Linux and the HP Vertica target database.
Linux: vertica-client-<version>.x86_64
Example:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 638
vertica-client-7.0.0-0.x86_64
Note HP Vertica 7.1 client is not compatible with database versions earlier
than HP Vertica 7.1.
2. Makes sure that the /etc/odbcinst.ini file contains the following entry for
HP Vertica, as in the following example:
[Vertica]
Driver = /opt/vertica/lib64/libverticaodbc.so
DriverODBCVer = 3.0
UsageCount = 1
Limitations
The following limitations apply to the Vertica target endpoint:
In "Batch optimized apply" Change Processing mode, binary data types (e.g.
VARBINARY) are limited to 32500 bytes on the target.
Security Requirements
You must provide HP Vertica account access to the Attunity Replicate user. The
Replicate user must also have the following privileges in the HP Vertica database:
CREATE TABLE Privileges:
USAGE privilege on the schema that contains the table or schema owner
USAGE privilege on the schema that contains the table or schema owner
USAGE privilege on the schema that contains the table or schema owner
INSERT Privileges:
UPDATE Privileges:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 639
UPDATE privilege on table
USAGE privilege on the schema that contains the table
SELECT privilege on the table when executing an UPDATE statement
that references table column values in a WHERE or SET clause
DELETE Privileges:
Note HP Vertica does not support applying changes to binary data types in Batch
optimized apply mode. For more information on Batch optimized apply mode,
see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.25 |
Supported HP Vertica Data Types with Mapping from Attunity Replicate
Data Types
BOOLEAN BOOLEAN
DATE DATE
DATETIME TIMESTAMP
INT1 INTEGER
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 640
Table 9.25 | Supported HP Vertica Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
INT2 INTEGER
INT4 INTEGER
INT8 INTEGER
REAL4 FLOAT
REAL8 FLOAT
UINT1 INTEGER
UINT2 INTEGER
UINT4 INTEGER
UINT8 INTEGER
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 641
3. In the Description field, type a description that helps to identify the HP
Vertica database. This is optional.
4. Select TARGET as the database role.
8. Type the HP Vertica authentication information (User Name, Password) for the
authorized user for this HP Vertica database. If you do not know this information,
see your HP Vertica database Administrator (DBA).
Note
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
Important: Make sure that the HP Vertica user entered in the HP Vertica
Authentication section has the correct access privileges. For information on how
to provide the required privileges, see Security Requirements.]
9. In the Database name field, enter the name of the HP Vertica database.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 642
To add internal Attunity Replicate parameters:
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 643
Using IBM Netezza as a Target
This section describes how to set up and use an IBM Netezza database as a target
database in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Security Requirements
▶ Database Privileges
▶ Table Privileges
▶ Schema Privileges
▶ View Privileges
▶ IBM Netezza Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Note
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 644
Limitations
Using IBM Netezza as a target database in an Attunity Replicate task is subject to
the following limitations:
The IBM Netezza target database uses the IBM Netezza NZLOAD utility, which does
not support loading tables with non-Latin names (e.g. Chinese). If any of your
source tables has a non-Latin name, you can map it to a table with a Latin name.
For more information on mapping table names, see Performing General Tasks for
a Single Table/View and Defining Global Transformations.
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab section in Customizing Tasks .
Note also that the size of a row in the IBM Netezza database cannot exceed 64KB.
This should be taken into consideration when specifying the maximum LOB size in
the Metadata tab.
Security Requirements
The Attunity Replicate user must be granted access to the IBM Netezza account as well
as the following privileges:
Database Privileges
LIST on <database> to <ATTUNITY USER>
Table Privileges
CREATE TABLE to <ATTUNITY USER>
LIST on TABLE to <ATTUNITY USER>
Schema Privileges
CREATE SCHEMA to <ATTUNITY USER>
LIST on SCHEMA to <ATTUNITY USER>
View Privileges
SELECT on _T_DATABASE to <ATTUNITY USER>
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 645
SELECT on _V_TABLE_DIST to <ATTUNITY USER>
SELECT on _V_RELATION_KEYDATA to <ATTUNITY USER>
LIST on _T_DATABASE to <ATTUNITY USER>
LIST on _V_SCHEMA to <ATTUNITY USER>
LIST on _V_USER to <ATTUNITY USER>
LIST on _V_TABLE to <ATTUNITY USER>
LIST on _V_TABLE_DIST to <ATTUNITY USER>
LIST on _V_RELATION_KEYDATA to <ATTUNITY USER>
Note IBM Netezza does not support applying changes to binary data types in Batch
optimized apply mode. For more information on Batch optimized apply mode,
see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.26 |
Supported IBM Netezza Data Types with Mapping from Attunity
Replicate Data Types
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 646
Table 9.26 | Supported IBM Netezza Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
TIMESTAMP
If scale 7-9:
CHARACTER VARYING(37)
INT1 BYTEINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 REAL
REAL8 DOUBLE
UINT1 SMALLINT
UINT2 INTEGER
UINT4 BIGINT
UINT8 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 647
To add an IBM Netezza target endpoint to Attunity Replicate:
8. Type the IBM Netezza authentication information (User Name, Password) for
the authorized user for this IBM Netezza database. If you do not know this
information, see your IBM Netezza database Administrator (DBA).
Note
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
Important: Make sure that the IBM Netezza user entered in the IBM Netezza
Authentication section has the correct access privileges. For information on how
to provide the required privileges, see Security Requirements.
9. In the Database name field, enter the name of the IBM Netezza database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 648
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 649
Using Kafka as a Target
This section describes how to set up and use Kafka as a target endpoint in a
replication task. In a task with a Kafka target endpoint, each source record is
transformed into a message which is then written (with an optional message key) to
a partition in the specified topic.
In this section:
▶ Transaction Processing by the Consumer
▶ How it Works
▶ Transaction Consistency from a Consumer Perspective
▶ Prerequisites
▶ Limitations
▶ Kafka Target Data Types
▶ Mapping from Attunity Replicate Data types to Avro
▶ Setting General Connection Properties
▶ Using Kerberos Authentication on Windows
▶ Overriding the Default Settings
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ The Attunity Envelope
▶ Decoding a Self-Describing Message
▶ Decoding a Message by Referenced Schema ID
▶ Typical Consumer Logic
▶ Metadata and Data Messages
▶ Metadata Message
▶ Data Message
▶ Prerequisites for using Kerberos when Replicate is installed on Red Hat 7.4 or 7.5
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 650
(topics/partitions).
During a task's CDC stage, committed changes that are detected by the Attunity
Replicate source endpoint are grouped by transaction, sorted internally in chronological
order, and then propagated to the target endpoint. The target endpoint can handle the
changes in various ways such as applying them to the target tables or storing them in
dedicated Change Tables.
Each CDC message has both a transaction ID as well as change sequence. As the
change sequence is a monotonically growing number, sorting events by change
sequence always achieves chronological order. Grouping the sorted events by
transaction ID then results in transactions containing chronologically sorted changes.
However, as Kafka is a messaging infrastructure, applying changes is not feasible while
storing changes in tables is meaningless. The Replicate Kafka endpoint, therefore, takes
a different approach, which is to report all transactional events as messages.
How it Works
Each change in the source system is translated to a data message containing the details
of the change including the transaction ID and change sequence in the source. The data
message also includes the changed columns before and after the change. As explained
above, the order in which the Kafka target writes the messages is the same as order of
changes within each transaction.
Once a data message is ready to be sent to Kafka, the topic and partition it should go to
are determined by analyzing the endpoint settings as well as potentially transformation
settings. For example, the user might decide to configure the endpoint in such a way
that every table is sent to a different topic and set the partition strategy to "Random",
meaning that each message (within the same table) will be sent to a different partition.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 651
accumulating a transaction in some intermediate memory buffer - and when a
new transaction ID is detected, mark the previous transaction as completed.
Although the simple way may work, it’s not very efficient at the task level as all
messages end up in the same topic and partition, not necessarily utilizing the full
parallelism of the Kafka cluster. This may be a non-issue if there are multiple tasks,
each taking advantage of a different topic/partition. In such as scenario, the gathering
of messages from those tasks may very well utilize the cluster optimally.
The more generic way where data may be spread over multiple topics and partitions
means that some intermediate buffer such as memory, a table in a relational database,
or even other Kafka topics would need to be used to collect information about
transactions. Then, the transactions would need to be rebuilt by periodically (every few
minutes/hours) sorting the events collected from Replicate’s Kafka output by the
change sequence and grouping them by transaction ID.
Prerequisites
Before you can use Kafka as a target endpoint in a Replicate task, the
following prerequisites must be met:
Open TCP ports to all the brokers from the Replicate Server machine
Set permissions that will allow Attunity Replicate to write to the target topics. One
way to do this is to use the Kafka ACLs script (kafka-acls).
Either create a topic named attrep_apply_exceptions before starting the
replication task or configure the brokers with auto.create.topics.enable=true.
Note that if this topic does not exist, the task will always fail when it encounters a
data error, regardless of the error handling policy.
For a description of the attrep_apply_exceptions table, see Apply Exceptions
Limitations
When defining a task with Kafka as the target endpoint, the following limitations apply:
The Kafka target endpoint does not support unlimited LOB size. Therefore, when
replicating from source tables with LOB columns, do not select the Allow
unlimited LOB size option.
For more information on defining LOB settings, see Target Metadata.
Batch optimized apply mode is not supported. If this mode is set, the task will
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 652
automatically switch to Transactional apply mode and issue an
appropriate warning.
For more information on these modes, see Change Processing Tuning.
Kafka topic names cannot exceed 255 characters (249 from Kafka 0.10) and can
only contain the following characters:
a-z|A-Z|0-9|. (dot)|_(underscore)|-(minus)
If the source table names exceed the maximum permitted length or contain
unsupported characters, you need to either modify the names before starting the
task or define a global transformation. For information on defining global
transformations, see Defining Global Transformations.
The Ignore ALTER Apply Changes setting is not supported for changes to source
data types and table renaming.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Note When using the JSON message format, binary values are represented
as hexadecimal digits.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 653
Table 9.27 |
Supported Kafka Target Data Types with Mapping from Attunity
Replicate Data Types
TIME TIME
DATETIME DATETIME
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 654
Mapping from Attunity Replicate Data types to Avro
When Avro is set as the message format, due to the limited number of data types
supported by Avro, the data type mappings will be as shown in the table below.
Note Replicate data types will only be mapped to supported Avro logical data types
if the Use logical data types for specific data types check box is selected.
Table 9.28 |
Supported Avro Data Types with Mapping from Attunity Replicate
Data Types
DATE DATE
STRING STRING
WSTRING STRING
CLOB STRING
NCLOB STRING
BYTES BYTES
BLOB BYTES
REAL4 FLOAT
REAL8 DOUBLE
INT1 INT
INT2 INT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 655
Table 9.28 | Supported Avro Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
UINT1 INT
UINT2 INT
UINT4 LONG
INT8 LONG
BOOLEAN BOOLEAN
The Name, Description, Type and Role fields are displayed on the right.
7. In the Broker servers field, specify one or more broker servers using the
following format (for high availability):
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 656
server1[:port1][,server2[:port2]]
Example:
192.168.1.100:9092,192.168.1.101:9093
Replicate will connect to the first available host. If a host is specified without a
port then port 9092 will be used as the default.
Note When using SSL or Kerberos authentication, you must specify the
broker FQDN (i.e. not the IP address).
Note All of the broker servers in your cluster need to be accessible to Replicate.
However, you do not need to specify all of the servers in the Broker servers field.
This is because Replicate only need to connect to one of the servers in order to
retrieve the connection details for the other servers in the cluster. It is therefore best
practice to specify the servers that are most likely to be available when the task is
run. The servers to which Replicate produces messages is determined by the topic
and partitioning topic and partitioning settings described below.
Note
The Use SSL and Certificate authentication options are only supported
from Kafka 0.9 and above.
The CA file, public key file and private key file must all be in PEM format.
The Kerberos and User name and password authentication methods
are only supported from Kafka 0.10 and above.
All of the broker servers in the cluster must be configured to accept
connection requests using the selected Authentication method.
Use SSL (supports TLS 1.0, 1.1 and 1.2): Select this option to encrypt the
communication between the Replicate machine and the broker server(s). If
the brokers are configured to require SSL, then you must select this option.
CA path: Specify either the full path (i.e. including the file name) to a
specific CA certificate in PEM format or the directory containing
certificate files with hash names.
Authentication: Select one of the following:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 657
None - No authentication.
Note The public and private key files must be in PEM format.
Public key file - The full path to the public key file on the
Replicate Server machine.
Private key file - The full path to the private key file on the
Replicate Server machine.
Private key password - The password for the private key file.
Note
Note Both Replicate Server and the Kafka brokers must be connected
to Active Directory KDC.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 658
Realm - The name of the domain in which the broker servers reside.
Note Attunity provides an Avro Message Decoder SDK for consuming Avro
messages produced by Attunity Replicate. You can download the SDK
together with the Avro Message Decoder Developer's Guide as a ZIP file
from the Customer Zone.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 659
When this option is not selected (the default), the message key will be in
text format.
10. In the Data Message Publishing section, set the following properties: a.
b. From the Partition strategy drop-down list, field, select either Random or By
message key. If you select Random, each message will be written to a randomly
selected partition. If you select By message key, messages will be written to
partitions based on the selected By message key (described below).
c. From the Message key drop-down list, field, select one of the following:
Note If the message Format is set to Avro and the Encode message key
in Avro format option is enabled, the message key will be an Avro record
with an Avro schema.
Schema and table name - For each message, the message key will
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 660
When By message key is selected as the Partition strategy,
messages consisting of the same schema and table name will be written
to the same partition.
Primary key columns - For each message, the message key will
contain the value of the primary key column.
When By message key is selected as the Partition strategy,
messages consisting of the same primary key value will be written to the
same partition.
10. In the Metadata Message Publishing section, specify whether or where to
publish the message metadata.
From the Publish drop-down list, select one of the following options:
When this option is selected, only the data messages will be published.
Additionally, the Wrap data messages with the Attunity Envelope option
(enabled by default) will be displayed. This option is useful for organizations that
wish to leverage the Attunity Envelope structure to process the data messages. If
you do not require the additional information provided by the Attunity Envelope
(e.g. due to existing message consumption processes), then disable this option.
If you select this option, either type the Topic name or use the Browse button to
select the desired topic. This option is required if the message format is set to Avro
since Avro-formatted messages can only be opened using the Avro schema.
If you select this option, you must also configure the Schema
Registry Connection Properties described below.
If you select this option, you must also configure the Schema
Registry Connection Properties described below.
Notes
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 661
If the topics do not exist, configure the brokers with
auto.create.topics.enable=true to enable Replicate to create the
topics during runtime. Otherwise, the task will fail.
The Confluent and Hortonworks Schema Registry options support
Avro message format only.
Example:
192.168.1.100:8081,192.168.1.101:8081
Example:
192.168.1.100:7788,192.168.1.101:7788
Use Kafka Security settings: Select this option if you want to use the
security settings that you defined for Kafka. Note that this option will only be
available if the security settings that you defined for Kafka are also supported
by the target Schema Registry.
Use SSL (supports TLS 1.0, 1.1 and 1.2): Select this option to encrypt the data
between the Replicate machine and the Schema Registry server(s). If the
servers re configured to require SSL, then you must select this option.
CA path: Specify one of the following:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 662
Authentication (Only available when publishing data schemas to
Confluent Schema Registry): Select one of the following:
None - No authentication.
Certificate - If you select this option, you also need to provide the
following information:
Note The public and private key files must be in PEM format.
Public key file - The full path to the public key file on the Replicate
Server machine.
Private key file - The full path to the private key file on the
Replicate Server machine.
Private key password - The password for the private key file.
Select a compatibility mode from the Subject compatibility mode drop-down list.
A description of the selected mode will appear below the drop-down list.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 663
Using Kerberos Authentication on Windows
Before beginning, make sure that the impersonated user (principal) has access to the
Replicate Data directory (<product_dir>\Data by default) on the Attunity Replicate
server. For Active Directory KDC, the impersonated user is the user configured in the
user interface.
Perform the following steps to ensure that the impersonated user (principal) has the
Log on as a batch job privilege on the Attunity Replicate server.
1. On the Attunity Replicate server, open the Local Security Settings (Control
Panel > System Security > Administrative Tools > Local Security Policy).
2. In the console tree, expand Local Policies and select User Rights Assignments.
3. In the details pane, double-click Log on as a batch job.
4. In the Log on as a batch job Properties dialog box, on the Local Security
Settings tab, verify that the respective user is listed. If it is not listed, click Add
User or Group, then add the user and click OK.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 664
Your changes should take effect immediately.
Note Before you can define such a transformation, you first need to add a
source endpoint to the task and select the tables you want to replicate.
To define a transformation:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 665
3. Define a transformation that adds one of the following columns:
Note The columns listed below (prefixed with a $) instruct Replicate to route the
message to the desired topic and/or partition, and will not be included in the
actual message itself.
4. Define an expression for the new column that returns the following values:
For a $topic column, the expression should return the topic name.
For a $partition column, the expression should return the partition
number. Note that an error will be returned during runtime if the partition
number does not exist.
For a $key column, the expression should return the message key contents.
Note that the expression must return a non-empty value.
For information on creating expressions, see Using the Expression Builder (for
Filters, Transformations, and Global Transformations).
In the Message maximum size field, specify the maximum size of messages that
the broker(s) are configured to receive (message.max.bytes). Replicate will not
send messages larger than the maximum size.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 666
To add internal Attunity Replicate parameters:
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
"type":"record",
"name":"MessageEnvelope",
"fields":[
{"name":"magic","type":{"type":"fixed","name":"Magic","size":5}},
{"name":"type","type":"string"},
{"name":"headers","type":["null",{"type":"map","values":"string"}]},
{"name":"messageSchemaId","type":["null","string"]},
{"name":"messageSchema","type":["null","string"]},
{"name":"message","type":"bytes"}
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 667
magic (5 bytes fixed field)
The constant "atMSG" is used to identify this form of message. The "atMSG" constant
should be used to validate that this message is indeed an Attunity envelope message.
Describes the enveloped message type. This can be one of two values: MD
which stands for metadata message and DT which stands for data message.
headers (map of string key and value)
A free for use map for various properties set at the application level. Currently,
no headers are set by Attunity Replicate but this may change in future versions.
An embedded UTF-8 encoded Avro JSON schema with which the message field can
be serialized. This field is used exclusively with the messageSchemaId field.
message (bytes)
Given the envelope schema, it is possible for anyone using this schema to properly
decode the envelope messages from Kafka.
Once the envelope message has been decoded, there are two possible scenarios:
The method for logically decoding messages in both scenarios is described below.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 668
The Attunity metadata messages which include both table metadata, lineage and data
schema description (to be referenced later by data messages) are enveloped in the
self-describing envelope.
To avoid sending schema with each data message, each schema has a 32 bytes long ID.
When a data message based on a previously sent data message schema (via the
metadata message) is constructed, the messageSchema field is set to null and the
messageSchemaId field is set to the 32 bytes ID of the schema instead. The application
responsibility is to locate the data schema sent earlier in the metadata message and use
that schema to decode the data message contained in the message field.
The customer's consumer code should read metadata messages from the schema topic
and then save the data schemas and any other information the consumer wishes to
access later in a customer defined zone. Another set of customer consumers should read
data messages from the various data topics, and access the data schemas zone as
required to retrieve the data schemas required for decoding the data messages.
When consuming data messages and metadata messages from several topics and partitions
in a multi-thread/process manner, a situation may arise where a given consumer may
attempt to read a data message before the corresponding metadata message has been read.
As it is not possible to read a data message before its corresponding metadata message, the
consumer's logic should wait a reasonable amount of time until the corresponding metadata
message has been read. If the metadata message is still not available after waiting for a
reasonable amount of time, the consumer should handle this as an unexpected error and
activate the planned error policy. An example of such a policy could be saving the message in
a dedicated “delayed” topic for later processing.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 669
As a rule of thumb, the number of metadata messages will be much lower (in the
magnitude of 1:10000 or more) than the number of data messages. So, assuming
a metadata consumer is active, the gap between metadata message and data
message should be no more than a few seconds (usually, milliseconds).
Metadata Message
{columns} Structure For each column, a record with the below prop-
erties.
length Integer The maximum size of the data (in bytes) permitted
for the column.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 670
Field Type Description
precision Integer For NUMERIC data type, the maximum number of
digits required to represent the value.
scale Integer For NUMERIC data type, the maximum number of
digits to the right of the decimal point permitted for
a number.
primaryKeyPosition Integer The position of the column in the table’s Primary
Key. or Unique Index. The value is zero if the
column is not part of the table’s Primary Key.
dataSchema String The Avro schema for deserializing the Data mes-
sages.
Data Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 671
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 672
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 673
Field Type Description
{columns} The column names and values in the current
record.
beforeData Structure The data of the table record, before the change
To work with Kerberos when Replicate is installed on Red Hat 7.4 or 7.5, you need
to rename the libraries as outlined in the procedure below.
mv libk5crypto.so.3 libk5crypto.so.3.org
mv libkrb5.so.3 libkrb5.so.3.org
mv libkrb5support.so.0 libkrb5support.so.0.org
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 674
Using a Log Stream Target
For information on using Log Stream as a target, see Using the Log Stream.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 675
Using MapR Streams as a Target
This section describes how to set up and use MapR Streams as a target endpoint in
a replication task. In a task with a MapR Streams target endpoint, each source
record is transformed into a message which is then written (with an optional
message key) to a partition in the specified topic.
In this section:
▶ Transaction Processing by the Consumer
▶ How it Works
▶ Transaction Consistency from a Consumer Perspective
▶ Prerequisites
▶ Limitations
▶ Supported Data Types
▶ Mapping from Attunity Replicate Data types to Avro
▶ Setting General Connection Properties
▶ Overriding the Default Settings
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ The Attunity Envelope
▶ Decoding a Self-Describing Message
▶ Decoding a Message by Referenced Schema ID
▶ Typical Consumer Logic
▶ Metadata and Data Messages
▶ Metadata Message
▶ Data Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 676
During a task's CDC stage, committed changes that are detected by the Attunity
Replicate source endpoint are grouped by transaction, sorted internally in chronological
order, and then propagated to the target endpoint. The target endpoint can handle the
changes in various ways such as applying them to the target tables or storing them in
dedicated Change Tables.
Each CDC message has both a transaction ID as well as change sequence. As the
change sequence is a monotonically growing number, sorting events by change
sequence always achieves chronological order. Grouping the sorted events by
transaction ID then results in transactions containing chronologically sorted changes.
However, as MapR Streams is a messaging infrastructure, applying changes is not
feasible while storing changes in tables is meaningless. The Replicate MapR Streams
endpoint, therefore, takes a different approach, which is to report all transactional
events as messages.
How it Works
Each change in the source system is translated to a data message containing the details
of the change including the transaction ID and change sequence in the source. The data
message also includes the changed columns before and after the change. As explained
above, the order in which the MapR Streams target writes the messages is the same as
order of changes within each transaction.
Once a data message is ready to be sent to MapR Streams, the topic and partition it
should go to are determined by analyzing the endpoint settings and any transformation
settings. For example, the user might decide to configure the endpoint in such a way
that every table is sent to a different topic and set the partition strategy to "Random",
meaning that each message (within the same table) will be sent to a different partition.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 677
accumulating a transaction in some intermediate memory buffer - and when a
new transaction ID is detected, mark the previous transaction as completed.
Although the simple way may work, it’s not very efficient at the task level as all
messages end up in the same topic and partition, not necessarily utilizing the full
parallelism of the MapR Streams cluster. This may be a non-issue if there are multiple
tasks, each taking advantage of a different topic/partition. In such as scenario, the
gathering of messages from those tasks may very well utilize the cluster optimally.
The more generic way where data may be spread over multiple topics and partitions
means that some intermediate buffer such as memory, a table in a relational database,
or even other topics would need to be used to collect information about transactions.
Then, the transactions would need to be rebuilt by periodically (every few
minutes/hours) sorting the events collected from Replicate’s MapR Streams output by
the change sequence and grouping them by transaction ID.
Prerequisites
Before you can use MapR Streams as a target endpoint in a Replicate task, the
following prerequisites must be met:
Install and configure MapR Client 5.2.1 or above on the Attunity Replicate
Server machine.
To verify the configuration, issue the following command in a Linux shell:
hadoop fs -ls maprfs://<cluster_name>/
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 678
Limitations
When defining a task with MapR Streams as the target endpoint, the following
limitations apply:
Unlimited LOB size is not supported. Therefore, when replicating from source
tables with LOB columns, do not select the Allow unlimited LOB size option.
For more information on defining LOB settings, see Target Metadata.
Batch optimized apply mode is not supported. If this mode is set, the task
will automatically switch to Transactional apply mode and issue an
appropriate warning.
For more information on these modes, see Change Processing Tuning.
MapR Streams topic names cannot exceed 255 characters and can only contain
the following characters:
a-z|A-Z|0-9|. (dot)|_(underscore)|-(minus)
If the source table names exceed the maximum permitted length or contain
unsupported characters, you need to either modify the names before starting the
task or define a global transformation. For information on defining global
transformations, see Defining Global Transformations.
The Ignore ALTER Apply Changes setting is not supported for changes to source
data types and table renaming.
Column names must begin with [A-Za-z_] (letters or an underscore) followed by
[A-Za-z0-9_] (letters, digits, or an underscore). For example, _Test_ is a valid
column name whereas &Test is not.
If a source column name does not adhere to this rule, then a transformation should
be used to rename the column.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 679
Note When using the JSON message format, binary values are represented
as hexadecimal digits.
Table 9.29 |
Supported MapR Streams Target Data Types with Mapping from
Attunity Replicate Data Types
TIME TIME
DATETIME DATETIME
BLOB BLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 680
Table 9.29 | Supported MapR Streams Target Data Types with Mapping from
Attunity Replicate Data Types (Cont.)
NCLOB NCLOB
Table 9.30 |
Supported Avro Data Types with Mapping from Attunity Replicate
Data Types
STRING
TIME STRING
TIMESTAMP STRING
STRING STRING
WSTRING STRING
CLOB STRING
NCLOB STRING
NUMERIC STRING
BYTES BYTES
BLOB BYTES
REAL4 FLOAT
REAL8 DOUBLE
INT1 INT
INT2 INT
INT4 INT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 681
Table 9.30 | Supported Avro Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
UINT2 INT
UINT4 LONG
INT8 LONG
UINT8 STRING
BOOLEAN BOOLEAN
The Name, Description, Type and Role fields are displayed on the right.
7. In the Cluster name field, specify the MapR Streams cluster name.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 682
a. In the Publish the data to field, choose one of the following:
Specific topic - to publish the data to a single topic. Type the stream
name followed by a colon and the topic name.
Example:
MapRStreamName:MapRTopicName
Note If the topics do not exist, the stream must be configured with
autocreate=true to enable Replicate to create the topics during
runtime. Otherwise, the task will fail.
b. From the Partition strategy drop-down list, field, select either Random or By
message key. If you select Random, each message will be written to a randomly
selected partition. If you select By message key, messages will be written to
partitions based on the selected By message key (described below).
c. From the Message key drop-down list, field, select one of the following:
Schema and table name - For each message, the message key will
contain a combination of schema and table name (e.g. "dbo+Employees").
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 683
10. In the Message Properties section, set the following
Note Attunity provides an Avro Message Decoder SDK for consuming Avro
messages produced by Attunity Replicate. You can download the SDK
together with the Avro Message Decoder Developer's Guide as a ZIP file
from the Customer Zone.
This option is required if the message format is set to Avro since Avro-
formatted messages can only be opened using the Avro schema.
Note If the topics do not exist, the stream must be configured with
autocreate=true to enable Replicate to create the topics during
runtime. Otherwise, the task will fail.
Note Before you can define such a transformation, you first need to add a
source endpoint to the task and select the tables you want to replicate.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 684
To define a transformation:
Note The columns listed below (prefixed with a $) instruct Replicate to route the
message to the desired topic and/or partition, and will not be included in the
actual message itself.
4. Define an expression for the new column that returns the following values:
For a $topic column, the expression should return the topic name.
For a $partition column, the expression should return the partition
number. Note that an error will be returned during runtime if the partition
number does not exist.
For a $key column, the expression should return the message key contents.
Note that the expression must return a non-empty value.
For information on creating expressions, see Using the Expression Builder (for
Filters, Transformations, and Global Transformations).
In the Message maximum size field, specify the maximum size of messages.
Replicate will not send messages larger than the maximum size.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 685
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
"type":"record",
"name":"MessageEnvelope",
"fields":[
{"name":"magic","type":{"type":"fixed","name":"Magic","size":5}},
{"name":"type","type":"string"},
{"name":"headers","type":["null",{"type":"map","values":"string"}]},
{"name":"messageSchemaId","type":["null","string"]},
{"name":"messageSchema","type":["null","string"]},
{"name":"message","type":"bytes"}
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 686
magic (5 bytes fixed field)
The constant "atMSG" is used to identify this form of message. The "atMSG" constant
should be used to validate that this message is indeed an Attunity envelope message.
Describes the enveloped message type. This can be one of two values: MD
which stands for metadata message and DT which stands for data message.
headers (map of string key and value)
A free for use map for various properties set at the application level. Currently,
no headers are set by Attunity Replicate but this may change in future versions.
An embedded UTF-8 encoded Avro JSON schema with which the message field can
be serialized. This field is used exclusively with the messageSchemaId field.
message (bytes)
Given the envelope schema, it is possible for anyone using this schema to properly
decode the envelope messages from MapR Streams.
Once the envelope message has been decoded, there are two possible scenarios:
The method for logically decoding messages in both scenarios is described below.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 687
The Attunity metadata messages which include both table metadata, lineage and data
schema description (to be referenced later by data messages) are enveloped in the
self-describing envelope.
To avoid sending schema with each data message, each schema has a 32 bytes long ID.
When a data message based on a previously sent data message schema (via the
metadata message) is constructed, the messageSchema field is set to null and the
messageSchemaId field is set to the 32 bytes ID of the schema instead. The application
responsibility is to locate the data schema sent earlier in the metadata message and use
that schema to decode the data message contained in the message field.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 688
as an unexpected error and activate the planned error policy. An example of such a
policy could be saving the message in a dedicated “delayed” topic for later processing.
As a rule of thumb, the number of metadata messages will be much lower (in the
magnitude of 1:10000 or more) than the number of data messages. So, assuming
a metadata consumer is active, the gap between metadata message and data
message should be no more than a few seconds (usually, milliseconds).
Metadata Message
{columns} Structure For each column, a record with the below prop-
erties.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 689
Field Type Description
length Integer The maximum size of the data (in bytes) permitted
for the column.
precision Integer For NUMERIC data type, the maximum number of
digits required to represent the value.
scale Integer For NUMERIC data type, the maximum number of
digits to the right of the decimal point permitted for
a number.
primaryKeyPosition Integer The position of the column in the table’s Primary
Key. or Unique Index. The value is zero if the
column is not part of the table’s Primary Key.
dataSchema String The Avro schema for deserializing the Data mes-
sages.
Data Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 690
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 691
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 692
Field Type Description
beforeData Structure The data of the table record, before the change
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 693
Using MemSQL as a Target
This section describes how to set up and use a MemSQL target endpoint in a
replication task.
In this section:
▶ Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Limitations
▶ Security Requirements
▶ Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the prerequisites described in this section have been met.
In this section:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 694
Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in
the following example:
[MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Limitations
When replicating to a MemSQL target, the following limitations apply:
When only the LOB column in the source table is updated, Replicate will not update
the corresponding target LOB column. The target LOB column will only be updated if
at least one other column is also updated in the same transaction.
Due to the way MemSQL operates, when loading data to a MemSQL target during a
Full Load task, duplicate key errors will not be reported to the logs.
Updating the Primary Key is not supported as the Primary Key columns in
MemSQL also serve as the shard key.
When updating a column's value to its existing value, a zero rows affected is
returned from MemSQL (unlike Oracle and Microsoft SQL Server, for example, that
perform an update of one row). This generates an entry in the
attrep_apply_exceptions control table and the following warning:
Some changes from the source database had no impact when applied to
Security Requirements
You must provide MemSQL account access to the Attunity Replicate user. This user
must have read/write privileges in the MemSQL database.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 695
Table 9.31 |
Supported MemSQL Data Types with Mapping from Attunity Replicate
Data Types
VARBINARY (Length)
BLOB
MEDIUMBLOB
LONGLOB
DATE DATE
TIME TIME
DATETIME (Scale)
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
DATETIME (p,s)
VARCHAR (45)
REAL4 FLOAT
REAL8 DOUBLE
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 696
Table 9.31 | Supported MemSQL Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
BLOB
LONGBLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 697
Table 9.31 | Supported MemSQL Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
TEXT
TEXT
LONGTEXT
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage EndpointsConnections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the
MemSQL database. This is optional.
4. Select TARGET as the database role.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 698
5. From the Type drop-down list, select MemSQL.
6. In the Server field, enter the host name or IP address of the computer on which
the MemSQL database is installed.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before it
is loaded into the MemSQL target database. The default value is 32000 KB.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 699
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 700
Using Microsoft APS PDW as a Target
This section describes how to set up and use Microsoft APS PDW as a target database
in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Security Requirements
▶ Microsoft APS PDW Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Note
A Microsoft APS PDW account with the required access privileges is required.
The following client components must be installed on the Attunity Replicate machine:
SQL Server Native Client 11.0
Microsoft SQL Server 2012 Parallel Data Warehouse Tools x64
Limitations
The following section describes the limitations of using Microsoft APS PDW as a
Replicate target.
Source columns with CHAR/VARCHAR data types and a non-Latin collation (e.g.
"Chinese_PRC_CI_AS") need to be mapped to NVARCHAR. This can be done by
defining a global transformation for all tables in the replication task or by defining a
single transformation for a specific table.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 701
For more information on defining transformations, see Defining Global
Transformations and Defining Transformations for a Single Table/View.
Microsoft APS PDW does not support empty (NULL) columns. Consequently,
when replicating a source column with an empty value, Replicate inserts a space
into the corresponding target column.
Security Requirements
You must provide Microsoft APS PDW account access to the Attunity Replicate user.
This user must have LOAD permission and applicable permissions (INSERT, UPDATE,
DELETE) on the destination table.
Table 9.32 |
Supported Microsoft APS PDW Data Types with Mapping from
Attunity Replicate Data Types
BOOLEAN BIT
DATE DATE
TIME TIME
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 702
Table 9.32 | Supported Microsoft APS PDW Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
UINT1 TINYINT
UINT2 SMALLINT
UINT4 INTEGER
UINT8 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 703
7. Optionally, change the default Port (5000).
8. Type the Microsoft APS PDW authentication information (User Name, Password)
for the authorized user for this Microsoft APS PDW database. If you do not know
this information, see your Microsoft APS PDW database Administrator (DBA).
Note
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password
properties. See Setting Advanced Connection Properties for more
information. This information is case sensitive.
If you want to set custom properties for this database, see Setting
Advanced Connection Properties.
Important: Make sure that the Microsoft APS PDW user entered in the Microsoft
APS PDW Authentication section has the correct access privileges. For information
on how to provide the required privileges, see Security Requirements.
9. In the Database name field, enter the name of the Microsoft APS PDW database.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 704
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 705
Using Microsoft Azure ADLS as a Target
This chapter describes how to set up and use Microsoft Azure ADLS as a target
in a replication task.
In this section:
▶ Prerequisites
▶ Microsoft Azure ADLS Target Overview
▶ DDL Handling
▶ Limitations
▶ Change Data Partitioning
▶ Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ Generating Reference Files
Prerequisites
Before you can use Microsoft Azure ADLS as a target endpoint in a Replicate task,
the following prerequisites must be met:
Permissions: The "Azure Active Directory application ID" specified in the Microsoft
Azure ADLS endpoint's Storage settings must have write access to the specified
ADLS storage target folder.
Supported Platforms: Attunity Replicate on Windows or Linux.
Note You do not need to install any drivers on the Attunity Replicate Server machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 706
Full Load files are named using incremental counters e.g. LOAD00001.csv, LOAD
00002.csv, etc. whereas Apply Changes files are named using timestamps e.g. 20141029-
1134010000.csv.
Note When the Create metadata files in the target folder option is enabled, a
corresponding metadata file is created using the same naming format, but with a
.dfm extension.
For each source table, a folder is created in the specified Microsoft Azure ADLS target
folder. The data files are created on the Replicate Server machine and are then uploaded
to the specified Microsoft Azure ADLS target folder once the File Attributes (Full Load)
and Change Processing upload conditions have been met.
DDL Handling
When a DDL change is captured, Replicate will close the data file and also create a DFM
file if the Create metadata files in the target folder option is enabled. When the next
batch of changes arrives, Replicate will create a new data file containing the changes.
Note that the DFM file created for the new data file will match the new table structure.
Limitations
The following limitations apply to the Microsoft Azure ADLS target endpoint:
The following DDLs are supported only: Truncate table, Drop table, Create table,
Add Column, Rename Column, Drop Column, and Convert Data Type.
Full LOB Mode is not supported
UPDATE and DELETE statements are not supported in Apply Changes replication mode
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 707
{Target Directory}
{Table_1}
{Partition_1}
Data files
DFM files
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
{Table_2}
{Partition_1}
Data files
DFM files
{Partition_2}
Data files
DFM files
{Partition_3}
Data files
DFM files
Information about the partitions is written to the attrep_cdc_partitions Control Table.
For information about this table, see Change Data Partitions.
Data Types
The following table shows the default mapping from Attunity Replicate data types to
Microsoft Azure ADLS target data types. Note that the data type mapping is only
relevant if the Create metadata files in the target folder option is enabled.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 708
Table 9.33 |
Supported Microsoft Azure ADLS Target Data Types with Mapping
from Attunity Replicate Data Types
TIME TIME
DATETIME DATETIME
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 709
To add an Microsoft Azure ADLS target endpoint to Attunity Replicate:
Table 9.34 |
Microsoft Azure ADLS Target Endpoint - General Tab Options
Option Description
ADLS Storage
SSL CA Path: The location of the CA file on the Replicate Server machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 710
Table 9.34 | Microsoft Azure ADLS Target Endpoint - General Tab Options (Cont.)
Option Description
File Attributes
Delimiters can be standard characters or a hexadecimal (hex) value. Note that the
"0x" prefix must be used to denote a hexadecimal delimiter (e.g. 0x01 = SOH). In the
Field delimiter, Record delimiter and Null value fields, the delimiter can consist of
concatenated hex values (e.g. 0x0102 = SOHSTX), whereas in the Quote character
and Escape character fields, it can only be a single hex value.
Note The hexadecimal number 0x00 is not supported (i.e. only 0x01-0xFF
are supported).
Format You can choose to create the target files in CSV or JSON format.
{ "book_id": 456, "title": "Winnie the Pooh", "price": 6.49, "is_hardcover": true }
{ "book_id": 789, "title": "The Cat in the Hat", "price": 7.23, "is_hardcover": true }
Note Changing the format (i.e. from CSV to JSON or from JSON
to CSV) while the task is in a stopped state and then resuming
the task, is not supported.
Field delimiter The delimiter that will be used to separate fields (columns) in the
target files. The default is a comma.
"mike","male"
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 711
Table 9.34 | Microsoft Azure ADLS Target Endpoint - General Tab Options (Cont.)
Option Description
Null value The string that will be used to indicate a null value in the target files.
"mike","male",295678\n
"sara","female",@\n
Escape character The character used to escape a quote character in the actual data.
Record delimiter The delimiter that will be used to separate records (rows) in the target
files. The default is a newline (\n).
Example:
"mike","male"\n
"sara","female"\n
Quote character The character that will be used at the beginning and end of a text
column. The default is the double-quote character ("). When a column
that contains column delimiters is enclosed in double-quotes, the
column delimiter characters are interpreted as actual data, and not as
column delimiters.
@mike@,@male@
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 712
Table 9.34 | Microsoft Azure ADLS Target Endpoint - General Tab Options (Cont.)
Option Description
Add metadata When the target storage format is set to Text, you can optionally add a
header header row to the data files. The header row can contain the source
column names and/or the intermediate (i.e. Replicate) data types.
Example of a target file with a header row when both With column
names and With data types are selected:
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
Maximum file The maximum size a file can reach before it is closed (and optionally
size compressed). This value applies both to data files and to Reference
Files.
Compress files Choose GZIP to compress the target files or NONE (the default) to
using leave them uncompressed.
Change Processing
File size reaches Specify the maximum size of Change Data to accumulate before
uploading the file to Microsoft Azure ADLS .
Elapsed time Specify the maximum time to wait before applying the changes.
reaches
Metadata Files
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 713
Table 9.34 | Microsoft Azure ADLS Target Endpoint - General Tab Options (Cont.)
Option Description
Create metadata When this option is selected, for each data file, a matching metadata
files in the target file with a .dfm extension will be created under the specified target
folder folder. The metadata files (which are in standard JSON format) provide
additional information about the task/data such as the source endpoint
type, the source table name, the number of records in the data file,
and so on.
For a full description of the metadata file as well as possible uses, see
Metadata File Description .
Note As part of connection testing process, Replicate uploads a test file to the
specified Microsoft Azure ADLS Target folder and then deletes it once a connection has
been established.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button is not
available unless the test connection fails.
Table 9.35 | Microsoft Azure ADLS Target Endpoint - Advanced Tab Options
Option Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 714
Table 9.35 | Microsoft Azure ADLS Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can process the final target files using a custom command. The command will be
run whenever a data file is created.
Note If the Generate a reference file option is selected, a row (specifying the
file's location) will be added to the Reference File only after the command
completes successfully.
Working directory - The directory where you want the command to run.
${METADATA_FILENAME} - The full path to the DFM file containing the metadata.
Note If the CSV/DFM file paths contain spaces, you must enclose
these parameters with quotation marks (e.g "${FILENAME}").
The post-processing command must return a proper exit code. You can either use the
standard exit code values described below or set a custom exit code value as described
in Setting Post Command Exit Codes with an Internal Parameter below.
0 - Success
1 - Recoverable error. The task will recover from the point of failure according to
the settings in the Environmental Errors tab.
2 - Table error. If a table error occurs, Replicate will handle the error according
to the settings in the Table Errors tab.
3 (or any other value e.g. -100) - Fatal error. The task will fail and not
attempt recovery.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 715
Table 9.35 | Microsoft Azure ADLS Target Endpoint - Advanced Tab Options (Cont.)
Option Description
You can use internal parameters to set exit codes with custom values. This is
especially useful if your application already uses the standard exit code values.
See Standard Post Command Exit Codes above for a description of the exit codes.
successExitCode
recoverableErrorExitCode
tableErrorExitCode
fatalErrorExitCode
Select this option to generate a Reference File (on Replicate Server) containing the
full path to the Apply Changes data files.
Note The reference file only points to the location of the Apply Changes files, and
not the Full Load files.
Reference File(s) folder - The folder on the Replicate machine in which the
Reference File will be created.
Example:
c:\temp\
User name The user name for accessing the proxy server.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 716
Table 9.35 | Microsoft Azure ADLS Target Endpoint - Advanced Tab Options (Cont.)
Option Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 717
Internal Parameters
Internal parameters are parameters that are not exposed in the UI and should only be
used if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your setting by clicking the View Setting Summary link. This
is useful if you need to send a summary of your setting to Attunity Support.
Example:
MyAzureADLS00000001.csv
Note The counter suffix increases incrementally each time a new Reference File is
generated (i.e. which occurs when the file reaches the maximum size defined in
the General tab). Once a new Reference File has been generated, you can delete
the old reference file(s) if required.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 718
Whenever an Apply Changes data file is created, a new row is added to the Reference
File in the following format:
<Source_Table_Name>,<Data_Lake_Store_name>/<path>/<file_name>
Example:
employees,mydatalakestore/new/files/my.company/20170611-120144192.csv
Note that if the Run command after upload option in the Advanced tab is also
enabled, the Reference File will be generated after the post-processing completes.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 719
Using Microsoft Azure Database for MySQL as a
Target
This section describes how to set up and use a Microsoft Azure Database for MySQL
target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ General Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Limitations
▶ Security Requirements
▶ Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the prerequisites described in this section have been met.
You must install Microsoft Azure Database for MySQL ODBC 64-bit client version 5.2.6
or later on the Attunity Replicate Server machine.
In this section:
General Prerequisites
Attunity Replicate Server for Windows
Attunity Replicate Server for Linux
General Prerequisites
The following is required:
A Microsoft Azure Database for MySQL account with the required
Security Requirements.
A Microsoft Azure Database for MySQL database with the tables that you want to
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 720
replicate should be accessible in your network.
The following Microsoft Azure Database for MySQL editions are supported:
Notes
Attunity Replicate assumes that MySQL ODBC Client 5.2.6 to 8.0.x 64-bit for
Windows is installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal parameter
where driver is the Parameter and MySQL ODBC <version> Unicode Driver is the
Value (where <version> is the client version e.g. 8.1).
For instructions on setting internal parameters, see Attunity Replicate Server
for Windows below.
MySQL ODBC drivers 8.0 to 8.0.13 are not supported.
Install Microsoft Azure Database for MySQL Connector/ODBC for Linux version 5.2.6
or later on the Attunity Replicate machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 721
Notes
Attunity Replicate assumes that MySQL Client 5.2.6 to 8.0.x for Linux is
installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal
parameter where driver is the Parameter and MySQL ODBC <version>
Unicode Driver is the Value (where <version> is the client version
e.g. 8.1).
For instructions on setting internal parameters, see Attunity Replicate
Server for Linux below.
MySQL ODBC drivers 8.0 to 8.0.13 are not supported.
Make sure that the /etc/odbcinst.ini file contains an entry for Microsoft
Azure Database for MySQL, as in the following example:
[Microsoft Azure Database for MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Limitations
The following limitations apply:
When replicating to the MariaDB target database, if the DATETIME value is set to
zero, you need to define a transformation that replaces the DATETIME with valid
values. For example:
replace($datetime_column,'9999-12-31 23:59:59','2000-01-01 22:00:00')
When only the LOB column in the source table is updated, Replicate will not update
the corresponding target LOB column. The target LOB column will only be updated if
at least one other column is also updated in the same transaction.
Due to the way Microsoft Azure Database for MySQL operates, when loading data
to a Microsoft Azure Database for MySQL target during a Full Load task, duplicate
key errors will not be reported to the logs.
When updating a column's value to its existing value, a zero rows affected is
returned from Microsoft Azure Database for MySQL (unlike Oracle and Microsoft
SQL Server that perform an update of one row). This generates an entry in the
attrep_apply_ exceptions control table and the following warning:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 722
Some changes from the source database had no impact when applied to the
target database. See attrep_apply_exceptions table for details.
Security Requirements
You must provide Microsoft Azure Database for MySQL account access to the Attunity
Replicate user. This user must have read/write privileges in the Microsoft Azure
Database for MySQL database.
Table 9.36 |
Supported Microsoft Azure Database for MySQL Data Types with
Mapping from Attunity Replicate Data Types
BOOL BOOL
VARBINARY (Length)
BLOB
MEDIUMBLOB
LONGLOB
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 723
Table 9.36 | Supported Microsoft Azure Database for MySQL Data Types with
Mapping from Attunity Replicate Data Types (Cont.)
DATETIME (Scale)
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
DATETIME (p,s)
VARCHAR (45)
REAL4 FLOAT
REAL8 DOUBLE
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 724
Table 9.36 | Supported Microsoft Azure Database for MySQL Data Types with
Mapping from Attunity Replicate Data Types (Cont.)
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
BLOB
LONGBLOB
TEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 725
Table 9.36 | Supported Microsoft Azure Database for MySQL Data Types with
Mapping from Attunity Replicate Data Types (Cont.)
TEXT
LONGTEXT
5. From the Type drop-down list, select Microsoft Azure Database for MySQL.
6. In the Server field, enter the host name or IP address of the computer on which
the Microsoft Azure Database for MySQL database is installed.
Notes
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 726
This information is case sensitive.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
8. Type the Microsoft Azure Database for MySQL authentication information (User
Name, Password) for the authorized user for this database. If you do not know
this information, see your database Administrator (DBA).
9. Select one of the following Load source schemas into options:
The following database - When this option is selected, all source schemas
will be loaded into the selected database.
Multiple endpoints - When this option is selected, each of the source
schemas will be loaded into its corresponding database.
Note Attunity Replicate assumes that Microsoft Azure Database for MySQL Client
5.2.6 to 5.3.x for Linux or Microsoft Azure Database for MySQL ODBC Client 5.2.6 to
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 727
5.3.x 64-bit for Windows is installed on the Attunity Replicate Server machine. If
a version later than 5.3.x is installed, you need to specify the version number as
an internal parameter where driver is the Parameter and Microsoft Azure
Database for MySQL ODBC <version> Unicode Driver is the Value
(where <version> is the client version e.g. 5.4).
For instructions on setting internal parameters, see Internal Parameters below.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 728
Using Microsoft Azure Database for PostgreSQL as a
Target
This section describes how to set up and use a Microsoft Azure Database for
PostgreSQL target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Security Requirements
▶ Microsoft Azure Database for PostgreSQL Target Data Types
▶ Data Types when Replicating from a PostgreSQL Source
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the client prerequisites when replicating to a
Microsoft Azure Database for PostgreSQL target.
Attunity Replicate Server for Windows:
The Microsoft Azure Database for PostgreSQL ODBC Driver: The Microsoft
Azure Database for PostgreSQL ODBC Driver psqlodbc_09_03_0300-x64-1 or
above must be installed on the Attunity Replicate machine.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 729
Note Make sure that the psql.exe path (e.g. "C:\Program Files\Microsoft
Azure Database for PostgreSQL\9.6\bin") is added to the system PATH.
Note If PgAdmin and the Greenplum client are installed on the same
Replicate Server, tasks configured to use a Microsoft Azure Database
for PostgreSQL target will fail.
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
Debug = 0
CommLog = 1
UsageCount = 2
Security Requirements
The user specified in the General tab when Setting General Connection Properties must
be a registered user in the Microsoft Azure Database for PostgreSQL database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 730
Microsoft Azure Database for PostgreSQL Target Data Types
The Microsoft Azure Database for PostgreSQL endpoint for Attunity Replicate supports
most Microsoft Azure Database for PostgreSQL database data types. The following table
shows the Microsoft Azure Database for PostgreSQL database target data types that are
supported when using Attunity Replicate and the default mapping from Attunity Replicate
data types. Unsupported data types are listed below the table.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.37 |
Supported Microsoft Azure Database for PostgreSQL Data Types
with Mapping from Attunity Replicate Data Types
BOOL BOOL
BYTES BYTEA
DATE DATE
TIME TIME
TIMESTAMP
VARCHAR (37)
INT1 SMALLINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 FLOAT4
REAL8 FLOAT8
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 731
Table 9.37 | Supported Microsoft Azure Database for PostgreSQL Data Types with
Map-ping from Attunity Replicate Data Types (Cont.)
VARCHAR (65535)
UINT1 SMALLINT
UINT2 INTEGER
UINT4 BIGINT
UINT8 BIGINT
VARCHAR (65535)
BLOB BYTEA
NCLOB TEXT
CLOB TEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 732
To add a Microsoft Azure Database for PostgreSQL target endpoint to Attunity
Replicate:
1. In Tasks view, click Manage Endpoint Connections to open the Manage
Endpoints Connections dialog box. Then click the New Endpoint Connection
button. For more information on adding an endpoint to Attunity Replicate, see
Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Microsoft
Azure Database for PostgreSQL database. This is optional.
4. Select TARGET as the database role.
5. From the Type drop-down list, select Microsoft Azure Database for PostgreSQL.
6. Type the Server name. This is the name or IP address or host name of the computer
with the Microsoft Azure Database for PostgreSQL database that you want to access.
7. Optionally, change the default port (5432).
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the specified Microsoft Azure Database for
PostgreSQL database user has the correct access privileges.
9. Type the Database name or select one from the list of available endpoints. This is
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 733
the name of the Microsoft Azure Database for PostgreSQL database to which you
are replicating data.
10. Click OK to save your settings and close the dialog box.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 734
Setting Advanced Connection Properties
In the Advanced tab, you can set the following properties:
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before
the file is loaded into the Microsoft Azure Database for PostgreSQL target database.
The default value is 32000 KB.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 735
Using Microsoft Azure Event Hubs as a Target
This section describes how to set up and use Microsoft Azure Event Hubs as a target
endpoint in a replication task. In a task with a Microsoft Azure Event Hubs target
endpoint, each source record is transformed into a message which is then written (with
an optional message key) to a partition in the specified hub.
In this section:
▶ Prerequisites
▶ Transaction Processing by the Consumer
▶ How it Works
▶ Transaction Consistency from a Consumer Perspective
▶ Limitations
▶ Supported Target Data Types
▶ Mapping from Attunity Replicate Data types to Avro
▶ Setting General Connection Properties
▶ Overriding the Default Settings
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
▶ The Attunity Envelope
▶ Decoding a Self-Describing Message
▶ Decoding a Message by Referenced Schema ID
▶ Typical Consumer Logic
▶ Metadata and Data Messages
▶ Metadata Message
▶ Data Message
Prerequisites
Before you can use Microsoft Azure Event Hubs as a target endpoint in a Replicate task,
the following prerequisites must be met:
The target hubs must already exist before starting the replication task. Note that if you
intend to use the Separate hub for each table option in the endpoint settings, the hub
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 736
name must adhere to the following format:
SourceSchemaName.HubName
Example:
HR.Employees
To be able to browse for hubs (in the General tab), the namespace Shared
Access Policy must have "Manage" permission.
If the namespace shared access policy does not have "Manage" permission, then
the hub Shared Access Policy must have at least "Send" permission.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 737
How it Works
Each change in the source system is translated to a data message containing the details
of the change including the transaction ID and change sequence in the source. The data
message also includes the changed columns before and after the change. As explained
above, the order in which the Microsoft Azure Event Hubs target writes the messages is
the same as order of changes within each transaction.
Once a data message is ready to be sent to Microsoft Azure Event Hubs, the hub and
partition it should go to are determined by analyzing the endpoint settings as well as
potentially transformation settings. For example, the user might decide to configure the
endpoint in such a way that every table is sent to a different hub and set the partition
strategy to "Random", meaning that each message (within the same table) will be sent
to a different partition.
The more generic way where data may be spread over multiple hubs and partitions
means that some intermediate buffer such as memory, a table in a relational database,
or even other hubs would need to be used to collect information about transactions.
Then, the transactions would need to be rebuilt by periodically (every few
minutes/hours) sorting the events collected from Replicate’s Microsoft Azure Event Hubs
output by the change sequence and grouping them by transaction ID.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 738
Limitations
When defining a task with Microsoft Azure Event Hubs as the target endpoint, the
following limitations apply:
The Microsoft Azure Event Hubs target endpoint does not support unlimited LOB
size. Therefore, when replicating from source tables with LOB columns, do not
select the Allow unlimited LOB size option.
For more information on defining LOB settings, see Target Metadata.
Batch optimized apply mode is not supported. If this mode is set, the task
will automatically switch to Transactional apply mode and issue an
appropriate warning.
For more information on these modes, see Change Processing Tuning.
The Ignore ALTER Apply Changes setting is not supported for changes to source
data types and table renaming.
Column names must begin with [A-Za-z_] (letters or an underscore) followed by
[A-Za-z0-9_] (letters, digits, or an underscore). For example, _Test_ is a valid
column name whereas &Test is not.
If a source column name does not adhere to this rule, then a transformation should
be used to rename the column.
For information on source data type mappings, see the section for the source endpoint
you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Note When using the JSON message format, binary values are represented
as hexadecimal digits.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 739
Table 9.38 |
Supported Microsoft Azure Event Hubs Target Data Types with
Mapping from Attunity Replicate Data Types
TIME TIME
DATETIME DATETIME
BLOB BLOB
CLOB CLOB
NCLOB NCLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 740
Mapping from Attunity Replicate Data types to Avro
When Avro is set as the message format, due to the limited number of data types
supported by Avro, the data type mappings will be as shown in the table below.
Table 9.39 |
Supported Avro Data Types with Mapping from Attunity Replicate
Data Types
DATE
STRING
TIME STRING
TIMESTAMP STRING
STRING STRING
WSTRING STRING
CLOB STRING
NCLOB STRING
NUMERIC STRING
BYTES BYTES
BLOB BYTES
REAL4 FLOAT
REAL8 DOUBLE
INT1 INT
INT2 INT
INT4 INT
UINT1 INT
UINT2 INT
UINT4 LONG
INT8 LONG
UINT8 STRING
BOOLEAN BOOLEAN
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 741
Setting General Connection Properties
This section describes how to configure general connection properties. For an
explanation of how to configure advanced connection properties, see Setting
Advanced Connection Properties below.
The Name, Description, Type and Role fields are displayed on the right.
Example: eventhubdemo
Shared Access Policy Level: Select either Namespace level or Event Hub
level according to the level defined for your Shared Access Policy.
Shared Policy Name: Enter the name of your shared access policy.
Example: RootManageSharedAccessKey
Shared Access Key: Enter your shared access primary or secondary key.
Example: BZLreXGxiWiRpGAog9Zf6b3K7ycRsImfBWqsR+SJp34=
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 742
Note Attunity provides an Avro Message Decoder SDK for consuming Avro
messages produced by Attunity Replicate. You can download the SDK together
with the Avro Message Decoder Developer's Guide as a ZIP file from the
Customer Zone.
An understanding of the Attunity envelope schema is a prerequisite for consuming
Avro messages produced by Attunity Replicate. If you do not wish to use the SDK,
see The Attunity Envelope for a description of the Attunity envelope schema.
Specific hub - to publish the data to a single hub. Either type a hub name
or use the browse button to select the desired hub.
Separate hub for each table - to publish the data to multiple hubs
corresponding to the source table names. If you select this option, the
hub name format must be as described in the prerequisites.
The target hub name consists of the source schema name and the
source table name, separated by a period (e.g. "dbo.Employees"). The
format of the target hub name is important as you will need to prepare
these hubs in advance.
b. From the Partition strategy drop-down list, field, select either Random or
By message key. If you select Random, each message will be written to a
randomly selected partition. If you select By message key, messages will be
written to partitions based on the selected Message key (described below).
c. From the Message key drop-down list, field, select one of the following:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 743
Note The message key is represented as a string, regardless of the
selected data message format (JSON/Avro).
Schema and table name - For each message, the message key will
contain a combination of schema and table name (e.g. "dbo+Employees").
If you select this option, either type the Hub name or use the Browse button to
select the desired hub. Note that the Browse button will only be available if the
Shared Access Policy Level described above is set to Namespace level.
Note Before you can define such a transformation, you first need to add a
source endpoint to the task and select the tables you want to replicate.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 744
To define a transformation:
Note The columns listed below (prefixed with a $) instruct Replicate to route
the message to the desired hub and/or partition, and will not be included in the
actual message itself.
4. Define an expression for the new column that returns the following values:
For a $hub column, the expression should return the topic name.
For a $partition column, the expression should return the partition
number. Note that an error will be returned during runtime if the partition
number does not exist.
For a $key column, the expression should return the message key contents.
Note that the expression must return a non-empty value.
For information on creating expressions, see Using the Expression Builder (for
Filters, Transformations, and Global Transformations).
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 745
Table 9.40 | Advanced Tab Options
Option Description
Use proxy server Select this option to access the namespace via a
proxy server.
User name The user name for accessing the proxy server.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 746
{
"type":"record",
"name":"MessageEnvelope",
"fields":[
{"name":"magic","type":{"type":"fixed","name":"Magic","size":5}},
{"name":"type","type":"string"},
{"name":"headers","type":["null",{"type":"map","values":"string"}]},
{"name":"messageSchemaId","type":["null","string"]},
{"name":"messageSchema","type":["null","string"]},
{"name":"message","type":"bytes"}
The constant "atMSG" is used to identify this form of message. The "atMSG" constant
should be used to validate that this message is indeed an Attunity envelope message.
Describes the enveloped message type. This can be one of two values: MD
which stands for metadata message and DT which stands for data message.
headers (map of string key and value)
A free for use map for various properties set at the application level. Currently,
no headers are set by Attunity Replicate but this may change in future versions.
messageSchemaId (null or string)
An embedded UTF-8 encoded Avro JSON schema with which the message field can
be serialized. This field is used exclusively with the messageSchemaId field.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 747
message (bytes)
Given the envelope schema, it is possible for anyone using this schema to properly
decode the envelope messages from Microsoft Azure Event Hubs.
Once the envelope message has been decoded, there are two possible scenarios:
The method for logically decoding messages in both scenarios is described below.
To avoid sending schema with each data message, each schema has a 32 bytes long ID.
When a data message based on a previously sent data message schema (via the
metadata message) is constructed, the messageSchema field is set to null and the
messageSchemaId field is set to the 32 bytes ID of the schema instead. The application
responsibility is to locate the data schema sent earlier in the metadata message and use
that schema to decode the data message contained in the message field.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 748
Typical Consumer Logic
A typical scenario involving Microsoft Azure Event Hubs involves Attunity Replicate as
the Producer of messages into Microsoft Azure Event Hubs and customer code as the
Consumer. Attunity Replicate offers the ability to define a specific hub as the schema
hub and different hubs for the table data.
The customer's consumer code should read metadata messages from the schema hub
and then save the data schemas and any other information the consumer wishes to
access later in a customer defined zone. Another set of customer consumers should read
data messages from the various data hubs, and access the data schemas zone as
required to retrieve the data schemas required for decoding the data messages.
When consuming data messages and metadata messages from several hubs and partitions in
a multi-thread/process manner, a situation may arise where a given consumer may attempt
to read a data message before the corresponding metadata message has been read. As it is
not possible to read a data message before its corresponding metadata message, the
consumer's logic should wait a reasonable amount of time until the corresponding metadata
message has been read. If the metadata message is still not available after waiting for a
reasonable amount of time, the consumer should handle this as an unexpected error and
activate the planned error policy. An example of such a policy could be saving the message
in a dedicated “delayed” hub for later processing.
As a rule of thumb, the number of metadata messages will be much lower (in the
magnitude of 1:10000 or more) than the number of data messages. So, assuming
a metadata consumer is active, the gap between metadata message and data
message should be no more than a few seconds (usually, milliseconds).
Metadata Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 749
Field Type Description
{columns} Structure For each column, a record with the below prop-
erties.
length Integer The maximum size of the data (in bytes) permitted
for the column.
precision Integer For NUMERIC data type, the maximum number of
digits required to represent the value.
scale Integer For NUMERIC data type, the maximum number of
digits to the right of the decimal point permitted for
a number.
primaryKeyPosition Integer The position of the column in the table’s Primary
Key. or Unique Index. The value is zero if the
column is not part of the table’s Primary Key.
dataSchema String The Avro schema for deserializing the Data mes-
sages.
Data Message
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 750
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 751
Field Type Description
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 752
Field Type Description
beforeData Structure The data of the table record, before the change
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 753
Using Microsoft Azure HDInsight as a Target
This section describes how to set up and use Microsoft Azure HDInsight as the
target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Change Data Partitioning on Microsoft Azure HDInsight
▶ Microsoft Azure HDInsight Endpoint Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you begin to work with Microsoft Azure HDInsight as a target in Attunity
Replicate, make sure that the following prerequisites have been met:
General:
The ADLS or Blob storage location (whichever you are using) must be
accessible from the Attunity Replicate machine.
The user specified in the Microsoft Azure HDInsight target endpoint's Hive
access settings must have access to HiveServer2.
ODBC Driver when Replicate Server is running on Windows: Microsoft Hive
ODBC driver 2.1.12 or above must be installed on the Attunity Replicate Server
machine.
ODBC Driver when Replicate Server is running on Linux:
follows: DriverManagerEncoding=UTF-16
ODBCInstLib=libodbcinst.so
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 754
Permissions: The "Storage account" (when using Blob storage) or "Azure
Active Directory application ID" (when using ADLS) specified in the Microsoft
Azure HDInsight endpoint's Storage settings must have write access to the
specified Blob/ADLS storage target folder.
Hive table permissions: Replicate requires permissions to perform the following
operations on Hive tables: CREATE, DROP, DESCRIBE, and ALTER table. ALTER
table may also include RENAME table, ADD/RENAME column, and CREATE/DROP
partitions on tables.
Limitations
The following limitations apply:
UPDATE/DELETE DMLs are not supported during change processing. If an
UPDATE/DELETE DML was captured on the source, it will be ignored on the target
and a warning will be written to the log. If the Store Changes option is enabled in
the task settings, these records will be written to the Change Table.
Limited LOB support only.
Dropping columns and changing column data types or the data type length (e.g.
VARCHAR(50) to VARCHAR(100)) is not supported.
The following Control Tables are not supported as they require UPDATE/DELETE
operations (which are not supported by the Microsoft Azure HDInsight
target endpoint):
Replication Status (requires UPDATE).
Proxy:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 755
Does not affect the ODBC (Hive) connection. Affects the storage connection
only. When using Blob storage, the HTTPS scheme is not supported
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 756
Microsoft Azure HDInsight Endpoint Target Data Types
The following table shows the Microsoft Azure HDInsight endpoint target data types
that are supported when using Attunity Replicate and the default mapping from
Attunity Replicate data types.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.41 |
Supported Microsoft Azure HDInsight Data Types with Mapping
from Attunity Replicate Data Types
BOOL BOOLEAN
BYTES STRING
TIME TIMESTAMP
DATETIME TIMESTAMP
DATE DATE
INT1 TINYINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
REAL4 FLOAT
REAL8 DOUBLE
UINT1 SMALLINT
UINT2 INT
UINT4 BIGINT
BLOB STRING
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 757
Table 9.41 | Supported Microsoft Azure HDInsight Data Types with Mapping from
Attun-ity Replicate Data Types (Cont.)
CLOB STRING
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 758
Setting General Connection Properties
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
Note The Blob storage option is not supported when Attunity Replicate is
running on Linux.
a. In the Data Lake Store name field, specify the full name of the ADLS storage.
b. In the Azure Active Directory ID field, specify the Azure Active Directory ID.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 759
c. In the Azure Active Directory application ID field, specify the Azure
Active Directory application ID.
d. In the Azure Active Directory application key field, specify the Azure Active
Directory application key.
e. In the Target folder field, specify where to create the data files on ADLS.
Note Due to a Hadoop limitation, the Target folder name can only contain
ASCII characters.
a. In the Host field, specify the host name of the Hive server.
b. Enter your Username and Password for accessing the Hive server in
the designated fields.
c. In the Database field, specify the name of the Hive target database.
Table 9.42 |
Microsoft Azure HDInsight Target - Advanced Properties
Setting Description
File Format Expand this section to specify or view the file format
settings.
Field delimiter The delimiter that will be used to separate fields in the target file.
The default is \001. Note that field delimiters must be in Octal
format (e.g. \247)
Null value The value that will be used to indicate a null value in the target
file. When using the default SerDe (LazySimpleSerde), setting
the null value is supported from Hive 0.13.
mike,male,295678
sara,female,@
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 760
Table 9.42 | Microsoft Azure HDInsight Target - Advanced Properties (Cont.)
Setting Description
Escape character The escape character is used to escape the field delimiter
character. When a field delimiter is escaped, it is interpreted as
actual data, and not as a field delimiter.
sunroof\,power-steering
Add metadata header When the target storage format is set to Text, you can optionally
add a header row to the data files. The header row can contain
the source column names and/or the intermediate (i.e. Replicate)
data types.
Position:DECIMAL(38,0),Color:VARCHAR(10)
1,"BLUE"
2,"BROWN"
3,"RED"
...
File Attributes Expand this section to specify or view the file attributes.
Maximum file size Specify the maximum file size of each target file. When the data
reaches the maximum size, the file will be closed and written to
the specified target folder.
Compress files using Select the compression method to use on ADLS/Blob storage.
File size reaches Specify the minimum size of the data required to create a file in
idle state.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 761
Table 9.42 | Microsoft Azure HDInsight Target - Advanced Properties (Cont.)
Setting Description
Elapsed time Specify the maximum time to wait before applying the changes in
reaches idle state.
Note To facilitate rapid delivery of DDL messages, files are uploaded immediately,
regardless of the specified File size reaches or Elapsed time reaches values.
Use proxy server Select this option to access Microsoft Azure HDInsight via a proxy
server.
User name The user name for accessing the proxy server.
SSL CA Path: The location of the CA certificate on the Replicate Server machine
when HTTPS is the selected Scheme.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 762
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 763
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 764
Using Microsoft Azure SQL Data Warehouse as a
Target
This section describes how to set up and use Microsoft Azure SQL Data Warehouse as a
target in a replication task. Microsoft Azure SQL Data Warehouse is located in the cloud
and is accessed through your Microsoft Azure account.
In this section:
▶ Overview
▶
▶ Limitations
▶ Microsoft Azure SQL Data Warehouse Endpoint Prerequisites
▶ Sign up for Microsoft Azure Blob Storage
▶ Sign up for Microsoft Azure SQL Data Warehouse
▶ Open the Required Firewall Port(s)
▶ Install the Required Client
▶ Microsoft Azure SQL Data Warehouse Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Overview
In the first stage of the replication process, Attunity Replicate moves the data files
created by the source database into Microsoft Azure Blob Storage where thy are stored
as CSV files. The files are then loaded into the proper tables in the Microsoft Azure SQL
Data Warehouse data warehouse (using PolyBase).
The Replicate Microsoft Azure SQL Data Warehouse database provides full automation for:
Schema generation and data type mapping
Full load of source database tables
Incremental load of changes made to source tables
Application of schema changes (DDL) made to the source tables.
Synchronization between full load and CDC processes.
Manual control is also available if needed.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 765
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 766
Limitations
The following section describes the limitations of using Microsoft Azure SQL
Data Warehouse as a Replicate target.
Source columns with CHAR/VARCHAR data types and a non-Latin collation (e.g.
"Chinese_PRC_CI_AS") need to be mapped to NVARCHAR. This can be done by
defining a global transformation for all tables in the replication task or by defining a
single transformation for a specific table.
For more information on defining transformations, see Defining Global
Transformations and Defining Transformations for a Single Table/View.
Microsoft Azure SQL Data Warehouse does not support empty (NULL)
columns. Consequently, when replicating a source column with an empty
value, Replicate inserts a space into the corresponding target column.
The rename column DDL is not supported.
Note For best performance, the Azure Blob Storage container should be in the
same region as your Microsoft Azure SQL Data Warehouse.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 767
Required Permissions
Required Permissions
Attunity Replicate performs the following operations on the replicated tables within
Microsoft Azure SQL Data Warehouse:
SELECT, INSERT, UPDATE and DELETE
Bulk Load
CREATE, ALTER, DROP (if required by the task's definition)
Unless the user is the DB Owner, the user specified in the Microsoft Azure SQL
Data Warehouse endpoint settings must be granted the above permissions.
When Replicate Server runs on a machine outside Azure - Open port 1433
for outbound traffic.
When Replicate Server runs on an AzureVM - Open the following ports
for outbound traffic:
1433
11000-11999
14000-14999
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 768
Install the Required Client
Install SQL Server Native Client 11 (for connecting to Microsoft Azure SQL
Data Warehouse) on the Attunity Replicate machine.
Table 9.43 |
Supported Microsoft Azure SQL Data Warehouse Data Types with
Mapping from Attunity Replicate Data Types
BOOL BIT
VARBINARY (8000)
DATE DATE
TIME TIME
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 FLOAT(24)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 769
Table 9.43 | Supported Microsoft Azure SQL Data Warehouse Data Types with Mapping
from Attunity Replicate Data Types (Cont.)
VARCHAR (8000)
UINT1 TINYINT
UINT2 SMALLINT
UINT4 INTEGER
UINT8 BIGINT
NVARCHAR (4000)
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab description in Customizing Tasks .
VARBINARY (8000)
Change Processing:
VARBINARY (4000)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 770
Table 9.43 | Supported Microsoft Azure SQL Data Warehouse Data Types with Mapping
from Attunity Replicate Data Types (Cont.)
VARCHAR (8000)
Change Processing:
VARCHAR (4000)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 771
7. Enter the following Microsoft Azure SQL Data Warehouse information:
Server name: Specify the name of the Microsoft Azure SQL Data
Warehouse server you are using.
Port: Specify the port number for Microsoft Azure SQL Data Warehouse.
User name: Specify the user name of a registered Microsoft Azure SQL
Data Warehouse user.
Password: Specify the password for the user entered in the User name field.
If you do not have these values, contact the Microsoft Azure account owner or
your company’s Microsoft Azure SQL Data Warehouse System Administrator.
Azure Blob Storage Access: During a replication task, Microsoft Azure SQL
Data Warehouse authenticates itself to Azure Blob Storage using an SQL
Server Credential. You can either configure Replicate to create the Credential
automatically during runtime (the default) or use an existing Credential.
Automatically create SQL Server Credential
8. Enter the following Microsoft Azure Blob Storage information. You may need to
click the Microsoft Azure Blob Storage header to see the information.
Account name: Specify the name of the Azure Blob Storage account to
which you want the files copied.
Container name: Specify the name of the Azure Blob Storage container
to which you want the files copied.
Account key: Specify the key for your Azure Blob Storage account.
Folder: Specify the name of the Azure Blob Storage folder to which you want
the files copied.
If you do not have these values, contact the Microsoft Azure Blob Storage
account owner or your company’s IT department.
Note
This information is case sensitive.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 772
If the connection is successful a message in green is displayed. If the
connection fails, an error message is displayed at the bottom of the
dialog box.
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 773
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 774
Using Microsoft Azure SQL Database as a Target
This section describes how to set up a Microsoft Azure SQL Database as a target
in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Permissions
▶ Microsoft Azure SQL Database Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the following prerequisites have been met:
Ports:
Client prerequisites:
For all versions of Microsoft SQL Server, Microsoft SQL Server Native Client 11.0
must be installed on the Attunity Replicate Server machine.
Attunity Replicate for Linux:
You can either work with the Microsoft ODBC Driver the Simba Microsoft SQL
Server ODBC Driver. Instructions for both are provided below.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 775
First, install Microsoft ODBC Driver 13.1 for Linux on the Attunity Replicate
Server machine.
Then, on the Attunity Replicate Server machine, open a Unix shell and perform
the following steps:
Note The procedure below assumes that you have installed a single default
instance of Replicate on Linux (areplicate). If you have installed multiple
instances, replace areplicate with the name of the instance running the task with
a Microsoft Azure SQL Database target. If several instances are running such as
task, the procedure needs to be repeated for each instance.
to: cd <product_dir>/bin
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 776
If the existing library has a different version number (e.g. libmsodbcsql-
13.1.so.0.2), you need to create a symbolic link between the existing
library and the required library.
ln -s existing_library_name libmsodbcsql-13.1.so.0.0
A Microsoft Azure SQL Database account with the specific access privileges
is required. See Target Permissions for more information.
Limitations
When using a Microsoft Azure SQL Database as a target in a Replicate task, The
following imitations apply:
When Use BCP for loading tables is selected in the Advanced tab (the default),
unlimited LOB columns are not supported in Batch optimized apply change
processing mode. You can work around this limitation by limiting LOB column size in
the task settings, clearing the Use BCP for loading tables option or switching to
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 777
Transactional apply mode.
When the Use BCP for loading tables option is enabled in the Advanced
tab, triggers are not executed.
When Attunity Replicate is installed on Linux, the Use BCP for loading tables
option in the Advanced tab is not supported.
Permissions
The following describes the security requirements for using Attunity Replicate with
a Microsoft Azure SQL Database target.
The Attunity Replicate user (i.e. the user specified in the Microsoft Azure SQL
Database endpoint settings) must have at least the db_owner user role on the
Microsoft Azure SQL Database you are connecting to.
A Microsoft Azure SQL Database system administrator must provide this permission for
all Attunity Replicate users.
Table 9.44 |
Microsoft Azure SQL Database Target Data Types with Mapping
from Attunity Replicate Data Types
BOOLEAN TINYINT
BYTES VARBINARY(length)
DATE DATE
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 778
Table 9.44 | Microsoft Azure SQL Database Target Data Types with Mapping from
Attun-ity Replicate Data Types (Cont.)
INT1 SMALLINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
REAL4 REAL
REAL8 FLOAT
STRING DATETIME2
UINT1 TINYINT
UINT2 SMALLINT
UINT4 INT
UINT8 BIGINT
IMAGE
To use this data type with Attunity Replicate, you must enable the use of
BLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 779
Table 9.44 | Microsoft Azure SQL Database Target Data Types with Mapping from
Attun-ity Replicate Data Types (Cont.)
TEXT
To use this data type with Attunity Replicate, you must enable the use of
CLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
NTEXT
To use this data type with Attunity Replicate, you must enable the use of
NCLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 780
5. Select Microsoft Azure SQL Database as the database Type.
6. In the SQL Azure server field, specify the host name or IP address of the computer
with the Microsoft Azure SQL Database instance containing the target database.
Note To override the default port, add the port to the server name, separated by
a comma. For example, if the server name is myserver.company.local and the
port is 3333, then the server name should be written like this:
myserver.company.local,3333
Important: Make sure that the Microsoft Azure SQL Database user has the
correct access privileges. For information on how to provide the required
privileges, see Permissions.
Note To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 781
Use BCP for loading tables: Select this to transfer data for full-load
operations using BCP.
Note
When the target table contains an identity column that does not exist in the
source table, you must disable the Use BCP for loading tables option.
BCP packet size: The maximum size of the packets (in bytes) used to transfer
data using BCP.
Filegroup for Attunity Replicate internal tables: Optionally, specify a filegroup for
the Attunity Replicate internal tables. When the replication task starts, all of the
internal Attunity Replicate control tables will be created in the specified filegroup.
GO
NAME = test1dat5,
FILENAME = 'C:\temp\DATA\t1dat5.ndf',
SIZE = 5MB,
MAXSIZE = 100MB,
FILEGROWTH = 5MB
TO FILEGROUP Test1FG1;
GO
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 782
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 783
Using Microsoft SQL Server as a Target
This section describes how to set up and use a Microsoft SQL Server database as a target
in a replication task.
Note When configuring a task with a Microsoft SQL Server target endpoint, note
the following:
If the Apply Changes task option is enabled and the change processing mode is set
to Batch optimized apply, it is recommend to enable the Apply batched changes
to multiple tables concurrently option in the Change Processing Tuningtab.
In this section:
▶ Supported Editions
▶ Prerequisites
▶ Limitations
▶ Permissions
▶ Microsoft SQL Server Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Supported Editions
Attunity Replicate supports the following Microsoft SQL Server editions:
Enterprise Edition
Standard Edition
Workgroup Edition
Developer Edition
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 784
Prerequisites
Make sure that the following prerequisites have been met:
Client prerequisites:
For all versions of Microsoft SQL Server, Microsoft SQL Server Native Client 11.0
must be installed on the Attunity Replicate Server machine.
Attunity Replicate for Linux:
First, install Microsoft ODBC Driver 13.1 for Linux on the Attunity Replicate
Server machine.
Then, on the Attunity Replicate Server machine, open a Unix shell and perform
the following steps:
Note The procedure below assumes that you have installed a single default
instance of Replicate on Linux (areplicate). If you have installed multiple
instances, replace areplicate with the name of the instance running the task with
a Microsoft SQL Server target. If several instances are running such as task, the
procedure needs to be repeated for each instance.
to: cd <product_dir>/bin
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 785
6. Start the Replicate instance:
./areplicate start
ln -s existing_library_name libmsodbcsql-13.1.so.0.0
A Microsoft SQL Server account with the specific access privileges is required.
See Target Permissions for more information.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 786
Limitations
When using a Microsoft SQL Server database as a target in a Replicate task, The
following imitations apply:
When Use BCP for loading tables is selected in the Advanced tab (the
default), unlimited LOB columns are not supported in Batch optimized apply
change processing mode. You can work around this limitation by limiting LOB
column size in the task settings, clearing the Use BCP for loading tables option
or switching to Transactional apply mode.
When the Use BCP for loading tables option is enabled in the Advanced
tab, triggers are not executed.
When Attunity Replicate is installed on Linux, the Use BCP for loading tables
option in the Advanced tab is not supported.
Microsoft SQL Server 2012 Target: When the target table is created manually with a
computed column, Full Load replication is not supported in BCP mode. Disabling the
"Use BCP for loading tables" option in the Advanced tab will resolve this issue. For
more information on BCP mode, see Setting Advanced Connection Properties.
Permissions
The following describes the security requirements for using Attunity Replicate with
a Microsoft SQL Server target.
The Attunity Replicate user must have at least the db_owner user role on the Microsoft
SQL Server database you are connecting to.
A Microsoft SQL Server system administrator must provide this permission for all
Attunity Replicate users.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 787
Table 9.45 | Microsoft SQL Server Target Data Types with Mapping from Attunity Replicate
Data Types
BYTES VARBINARY(length)
DATE
DATETIME2 (%d)
DATETIME2 (scale)
INT1 SMALLINT
INT2 SMALLINT
INT4 INT
INT8 BIGINT
REAL4 REAL
REAL8 FLOAT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 788
Table 9.45 | Microsoft SQL Server Target Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
DATETIME2
VARCHAR (length)
UINT1 INT2
UINT2 INT4
UINT4 INT8
IMAGE
To use this data type with Attunity Replicate, you must enable the use of
BLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 789
Table 9.45 | Microsoft SQL Server Target Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
TEXT
To use this data type with Attunity Replicate, you must enable the use of
CLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
NTEXT
To use this data type with Attunity Replicate, you must enable the use of
NCLOBs for a specific task.
During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary
key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 790
5. Select Microsoft SQL Server as the database Type.
6. In the Server name field, specify the host name or IP address of the computer
with the Microsoft SQL Server instance containing the target database.
Note To override the default port, add the port to the server name, separated by
a comma. For example, if the server name is myserver.company.local and the
port is 3333, then the server name should be written like this:
myserver.company.local,3333
Note When using Windows authentication, make sure that the user account that
is associated with the Attunity Replicate Server service has Network read and
write permissions. This must be configured by a Windows system administrator.
If you select SQL Server authentication, type the Microsoft SQL Server
authentication information (User name, Password) for the authorized user for this
Microsoft SQL Server database. If you do not know this information, see the
Microsoft SQL Server System Administrator.
Important: Make sure that the Microsoft SQL Server user has the correct
access privileges. For information on how to provide the required privileges,
see Permissions.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 791
Note To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
8. Type the Database name or click Browse and select one from the list of available
databases. This is the name of the database from where you are replicating the data.
Use BCP for loading tables: Select this to transfer data for full-load
operations using BCP.
Note
When the target table contains an identity column that does not exist in the
source table, you must disable the Use BCP for loading tables option.
BCP packet size: The maximum size of the packets (in bytes) used to transfer
data using BCP.
Filegroup for Attunity Replicate internal tables: Optionally, specify a filegroup for
the Attunity Replicate internal tables. When the replication task starts, all of the
internal Attunity Replicate control tables will be created in the specified filegroup.
GO
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 792
NAME = test1dat5,
FILENAME = 'C:\temp\DATA\t1dat5.ndf',
SIZE = 5MB,
MAXSIZE = 100MB,
FILEGROWTH = 5MB
TO FILEGROUP Test1FG1;
GO
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 793
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 794
Using a MySQL-Based Database as a Target
This section describes how to set up and use a MySQL target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ General Prerequisites
▶ Attunity Replicate Server for Windows
▶ Attunity Replicate Server for Linux
▶ Limitations
▶ Security Requirements
▶ Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure that the prerequisites described in this section have been met.
In this section:
General Prerequisites
Attunity Replicate Server for Windows
Attunity Replicate Server for Linux
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 795
General Prerequisites
The following is required:
A MySQL account with the required Security Requirements.
A MySQL database with the tables that you want to replicate should be accessible
in your network.
The following MySQL editions are supported:
Notes
Attunity Replicate assumes that MySQL ODBC Client 5.2.6 to 8.0.x 64-bit for
Windows is installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal parameter
where driver is the Parameter and MySQL ODBC <version> Unicode Driver is the
Value (where <version> is the client version e.g. 8.1).
For instructions on setting internal parameters, see Attunity Replicate Server
for Windows below.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 796
Attunity Replicate Server for Linux
To be able to work with Attunity Replicate for Linux and MySQL as a source endpoint in a
Replicate task, you need to:
Notes
Attunity Replicate assumes that MySQL Client 5.2.6 to 8.0.x for Linux is
installed on the Attunity Replicate Server machine. If a version later than
5.3.x is installed, you need to specify the version number as an internal
parameter where driver is the Parameter and MySQL ODBC <version>
Unicode Driver is the Value (where <version> is the client version
e.g. 8.1).
For instructions on setting internal parameters, see Attunity Replicate
Server for Linux below.
MySQL ODBC drivers 8.0 to 8.0.13 are not supported.
Make sure that the /etc/odbcinst.ini file contains an entry for MySQL, as in
the following example:
[MySQL ODBC 5.2.6 Unicode Driver]
Driver = /usr/lib64/libmyodbc5w.so
UsageCount = 1
Limitations
The following limitations apply:
When replicating to the MariaDB target database, if the DATETIME value is set to
zero, you need to define a transformation that replaces the DATETIME with valid
values. For example:
replace($datetime_column,'9999-12-31 23:59:59','2000-01-01 22:00:00')
When only the LOB column in the source table is updated, Replicate will not update
the corresponding target LOB column. The target LOB column will only be updated if
at least one other column is also updated in the same transaction.
Due to the way MySQL operates, when loading data to a MySQL target during a
Full Load task, duplicate key errors will not be reported to the logs.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 797
When updating a column's value to its existing value, a zero rows affected is
returned from MySQL (unlike Oracle and Microsoft SQL Server that perform an
update of one row). This generates an entry in the attrep_apply_exceptions
control table and the following warning:
Some changes from the source database had no impact when applied to
Security Requirements
You must provide MySQL account access to the Attunity Replicate user. This user
must have read/write privileges in the MySQL database.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.46 |
Supported MySQL Data Types with Mapping from Attunity Replicate
Data Types
BOOL BOOL
VARBINARY (Length)
BLOB
MEDIUMBLOB
LONGLOB
DATE DATE
TIME TIME
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 798
Table 9.46 | Supported MySQL Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
DATETIME (Scale)
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
DATETIME (p,s)
VARCHAR (45)
REAL4 FLOAT
REAL8 DOUBLE
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 799
Table 9.46 | Supported MySQL Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
VARCHAR (Length)
TEXT
MEDIUMTEXT
LONGTEXT
BLOB
LONGBLOB
TEXT
TEXT
LONGTEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 800
Setting General Connection Properties
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage EndpointsConnections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the MySQL
database. This is optional.
4. Select TARGET as the database role.
5. From the Type drop-down list, select MySQL for MySQL, MariaDB, Amazon
Aurora MySQL, Amazon RDS for MariaDB, and Amazon RDS for MySQL.
6. In the Server field, enter the host name or IP address of the computer on which
the MySQL database is installed.
Notes
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
8. Type the MySQL authentication information (User Name, Password) for the
authorized user for this MySQL database. If you do not know this information, see
your MySQL database Administrator (DBA).
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 801
9. Select one of the following Load source schemas into options:
The following database - When this option is selected, all source schemas
will be loaded into the selected database.
Multiple endpoints - When this option is selected, each of the source
schemas will be loaded into its corresponding database.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 802
Using ODBC to Connect to a Target
This topic describes how to use ODBC connectivity to connect to a target endpoint.
In this section:
▶ ODBC Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Note ODBC does not support applying changes to binary data types in Batch
optimized apply mode. For more information on Batch optimized apply mode,
see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.47 |
Supported ODBC Target Data Types with Mapping from Attunity
Replicate Data Types
BYTES SQL_VARBINARY
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 803
Table 9.47 | Supported ODBC Target Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
TIME SQL_TYPE_TIME
DATETIME SQL_TYPE_TIMESTAMP
INT1 SQL_SMALLINT
INT2 SQL_SMALLINT
SQL_INTEGER
Otherwise:
SQL_VARCHAR
INT8 SQL_BIGINT
NUMERIC SQL_NUMBER
REAL4 SQL_REAL
REAL8 SQL_DOUBLE
STRING SQL_VARCHAR
UINT1 SQL_TINYINT
UINT2 SQL_SMALLINT
UINT4 SQL_INTEGER
UINT8 SQL_BIGINT
WSTRING SQL_WVARCHAR
Note If the target endpoint does not support ODBC data types, data types are
mapped to SQL_VARCHAR.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 804
Setting General Connection Properties
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
If the DSN you want to use is not included in the list, make sure that the
endpoint client is installed on the computer with Attunity Replicate and that the
DSN is defined. Note that the ODBC provider client must be 64-bit. For more
information, see Prerequisites .
Note If you are using an ARC CDC Agent as the source in a Replicate task,
you cannot select the DSN for the Attunity ODBC driver as the target. In this
case, to use Attunity ODBC as a target, you must enter the connection string
manually by selecting Connection String and following the directions for
that option in this procedure.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 805
information on how to create a connection string, see the documentation for
the ODBC endpoint provider you are using.
Example of an SQL/MP Connection String:
Note
You can use the Advanced tab to add specific properties and create a
custom connect string. In this case, you do not need to enter
information in this tab. For more information on using the Advanced
tab, see Setting Advanced Connection Properties.
To determine if you are connected to the endpoint you want to use
or if the connection information you entered is correct, click Test
Connection.
If the connection is successful a message in green is displayed. If
the connection fails, an error message is displayed at the bottom of
the dialog box.
To view the log entry if the connection fails, click View Log. The
server log is displayed with the information for the connection failure.
Note that this button is not available unless the test connection fails.
7. Type the authentication information (User Name, Password) for the authorized
user for the ODBC endpoint being used. For example, the IBM DB2 system
administrator if you are using a IBM DB2 provider. If you do not know this
information, see your ODBC Endpoint System Administrator.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 806
Note
Important: Make sure that the ODBC endpoint user has the correct
access privileges for the ODBC provider being used.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 807
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 808
Using Oracle as a Target
This section describes how to set up and use an Oracle database as a target endpoint
in a replication task.
Note When replicating to an Oracle database with a full disk and/or partition
where Oracle is trying to write archived redo log files, insert operations may fail. In
such as case, no error will be shown and the task will not progress past the loading
stage. To confirm that this is indeed an Oracle Archiver error, stop and attempt to
restart the task. The task will not start and an appropriate error should be shown.
In this section:
▶ Prerequisites
▶ Limitations
▶ Security Requirements
▶ Oracle Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Before you can work with an Oracle endpoint, make sure the prerequisites listed in
this section have been met.
On Windows systems, install Oracle Instant Client for Microsoft Windows (x64)
Version 11.2.0.3.0 and above.
Note Support for the XMLTYPE data type requires the full Oracle Client.
On Linux systems, install Oracle Instant Client for Linux (x86-64) Version
11.2.0.3.0 and above.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 809
Note Support for the XMLTYPE data type requires the full Oracle Client.
In addition, if not already included in your system, you need to create a symbolic
link in the $Oracle_Home\lib directory. This link should be called libclntsh.so,
and should point to a specific version of this file.
For example, on an Oracle 12c client:
lrwxrwxrwx 1 oracle oracle 63 Oct 2 14:16 libclntsh.so ->
/u01/app/oracle/home/lib/libclntsh.so.12.1
Limitations
The following limitations apply:
The Use direct path full load option does not support the following:
Bidirectional replication
Triggers
In Batch Optimized Apply mode, loading into the net changes table uses Direct
Path which does not support XMLType.
Workaround:
Attunity Replicate cannot create a new schema on the Oracle database. To replicate
to a new schema, the new schema name must already exist on the Oracle target.
You can then specify the new schema name in the Task Settings’ Target Metadata
and Control Tables tabs as required.
Security Requirements
A user must have the following privileges granted in the Oracle database to use an
Oracle target in an Attunity Replicate task:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 810
Note If any of the required privileges cannot be granted to a V$xxx, then grant them
to the V_$xxx.
You can add the following permissions to use a specific table list:
SELECT on <any-replicated-table>
ALTER on <any-replicated-table>
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 811
The following permission must be granted for logon:
CREATE SESSION
The following permission must be granted if you are using a direct path:
LOCK ANY TABLE
If the "DROP and CREATE table" or "TRUNCATE before loading" option is selected in the
Full Load Settings tab and the target table schema is different from the Attunity
Replicate user, the following permission must be granted:
DROP ANY TABLE
To store changes in Change Tables or in an Audit Table when the target table schema is
different from the Attunity Replicate user, the following permission must be granted:
CREATE ANY INDEX
The Attunity Replicate user must also be granted read permissions for the following
DBA tables:
SELECT on DBA_USERS
SELECT on DBA_TAB_PRIVS
SELECT on DBA_OBJECTS
SELECT on DBA_SYNONYMS
SELECT on DBA_SEQUENCES
SELECT on DBA_TYPES
SELECT on DBA_INDEXES
SELECT on DBA_TABLES
SELECT on DBA_TRIGGERS
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 812
Table 9.48 | Supported Oracle Data Types with Mapping from Attunity Replicate Data Types
DATE DATETIME
REAL4 BINARY_FLOAT
REAL8 BINARY_DOUBLE
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 813
Table 9.48 | Supported Oracle Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
BLOB BLOB
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 814
Note When the source database is Oracle, the source data types will be replicated
"as is" to the Oracle target. For example, an XMLTYPE data type on the source will
be created as an XMLTYPE data type on the target.
Note The total number of columns per table supported in Batch optimized apply
mode can be expressed using the following formula:
2 * columns_in_original_table + columns_in_primary_key <= 999
So, for example, if the original tables has 25 columns and its Primary Key consists of
5 columns, then the total number of columns would be 55. If a table exceeds the
supported number of columns, Replicate will apply all of the changes in one-by-one
mode.
1. In the Attunity Replicate console, click Add database to open the Add
Endpoints dialog box. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the Oracle
database. This is optional.
4. Select TARGET as the database role.
6. Type the Oracle Connection String for the Oracle database you want to work
with. You can type the connect string in any Oracle format, for example:
//host:port/service name
Where:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 815
host: This is the name or IP address for the computer with the Oracle
database that you are using. For example, johnboy_W7 or 255.255.255.0.
port: (optional) This is the TNS Listener Port number for the computer with
the Oracle database that you are using. If you do not enter a port number the
default Oracle TNS Listener port is used.
service name: (optional) This is the service name for the computer with
the Oracle database you are using. If you do not enter a service name the
default service name is used.
You can also enter an Oracle Net keyword-value pair. For example:
"(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=dlsun242) (PORT=5521))
(CONNECT_DATA=(SERVICE_NAME=bjava21)))"
7. Type the Oracle authentication information (User Name, Password) for the
authorized user for this Oracle database. If you do not know this information, see
your Oracle database Administrator (DBA).
To prevent illicit database activity by unauthorized third-parties, Replicate can be
configured to automatically replace the user-entered password with a strong
random password. For more information, see Configuring Replicate to
Automatically Replace the User-Entered Password.
Important: Make sure that the Oracle user entered in the Oracle
Authentication section has the correct access privileges. For information on
how to provide the required privileges, see Security Requirements.
Note To determine if you are connected to the database you want to use or if
the connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 816
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button
is not available unless the test connection fails.
Use direct path full load: Select this to use the OCI direct path protocol for
bulk loading Oracle tables. This is the default selection.
Note Due to an issue with Oracle Direct Path, when this option is selected and
If target table already exists is set to Do nothing in the Full Load Settings,
the following occurs:
The first time the task runs, no error will be issued and rows with the
same Primary Key may be added to the target table.
The second time the task runs, the setting will take effect.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 817
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 818
Using Pivotal Greenplum as a Target
This section describes how to set up and use a Pivotal Greenplum database as a target
in a replication task.
In this section:
▶ An Overview of the Pivotal Greenplum Target
▶ Attunity Replicate Pivotal Greenplum Endpoint Architecture Overview
▶ Full Load
▶ CDC
▶ Full Load
▶ Applying Changes to the Pivotal Greenplum Target
▶ Transactional Apply Mode
▶ Batch-Optimized Apply Mode
▶ Required Pivotal Greenplum Software, Environments
▶ Windows Pivotal Greenplum Required Software
▶ Linux Pivotal Greenplum Required Software
▶ Required Pivotal Greenplum Configuration and Environment
▶ Provide Pivotal Greenplum Account Access
▶ Security Requirements
▶ Limitations
▶ Pivotal Greenplum Data Types
▶ Setting up the gpfdist Program as a Service
▶ Using Multiple gpfdist Programs
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 819
uses Pivotal Greenplum’s Scatter/Gather Streaming technology to help with
data integration. This technology handles large amounts of data well.
The Attunity Replicate Pivotal Greenplum database makes it possible to load data from
other heterogeneous data sources and maintain the most up to date information. This is
done by capturing changes and streaming the changes to the Pivotal Greenplum data
warehouse. This can be done with a very low impact on the source data.
The Attunity Replicate Pivotal Greenplum database provides full automation for:
The Attunity Replicate Pivotal Greenplum database integrates with the Pivotal
Greenplum database in two ways:
Pivotal Greenplum ODBC API. This is used for metadata management. The
Pivotal Greenplum ODBC API lets Attunity Replicate test the database connection,
get the table list and the table schema, build procedures that create external tables
to process a file, and invoke the procedures that load the destination table or apply
changes from the external table. During the schema generation, data types can be
mapped, such as Pivotal Greenplum to Postgres. Primary keys and distribution
clauses are generated based on the primary key.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 820
Attunity Replicate Pivotal Greenplum Endpoint Architecture
Overview
The following shows the Attunity Replicate Pivotal Greenplum endpoint system
architecture for:
Full Load
CDC
Full Load
Full load is used to setup or refresh a data warehouse on a target, by concurrently
loading large amounts of data from source tables. High-speed data extraction is initiated
from endpoints like Oracle or Microsoft SQL Server, then gpfdist and buffered load files
are used for high-speed data loading into Pivotal Greenplum. The following shows the
Pivotal Greenplum database architecture for full load.
CDC
For incremental load, Attunity Replicate uses log-based change data capture (CDC). During
CDC replication, Attunity Replicate creates external Web tables or external tables to load SQL
statements into the target Pivotal Greenplum database. The statements are then
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 821
applied to the target tables. The following shows the Pivotal Greenplum
database architecture for CDC.
Full Load
On the first run of a task the Pivotal Greenplum target writes the data being replicated to
CSV files into a folder that is defined for the task. The CSV files are named sequentially,
for example, loadNNNN, where NNNN is an incremental number starting from 0. The
maximum file size of the CSV file is set by the user when configuring the Pivotal
Greenplum database.
When the CSV file reaches its maximum size it is renamed and moved into a load folder.
It is then read by the gpfdist utility, which executes an SQL statement that loads the
data into the target table. Once the file loading is complete, the file is deleted.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 822
statements in each row returned from the external Web table. When the changes
are applied, the file is deleted.
Important: To prevent errors during replication tasks, make sure that the
Pivotal Greenplum ODBC driver has a valid license.
Ensure that the following Pivotal Greenplum environment scripts are executed when
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 823
logging into the account where Attunity Replicate is run:
greenplum_connectivity_path.sh
greenplum_clients_path.sh
greenplum_loaders_path.sh
Make sure that port 8080 is open.
Because gpfdist is a Web server program and because it needs to be accessible from
the Pivotal Greenplum database segment nodes, there are some networking
configuration settings that must be in place to allow for this access. This is
documented in the EMC Pivotal Greenplum database Administration Guide.
For further information, see Pivotal Greenplum Prerequisites for Attunity Replicate.
Note One of the Attunity Replicate computer network cards must be part of the
Pivotal Greenplum database segments network to allow communication with the
gpfdist program.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 824
Security Requirements
A user must have the following privileges granted in the Pivotal Greenplum database to
use a Pivotal Greenplum target in an Attunity Replicate task:
CREATE table/external table/Web external table
TRUNCATE table
ALTER table
INSERT/UPDATE/DELETE
Limitations
Attunity Replicate cannot update columns that are part of the distribution key.
The Pivotal Greenplum target database has limited LOB support. You cannot use an
unlimited LOB size for this database. For more information, see Pivotal Greenplum
Data Types.
Table 9.49 |
Supported Pivotal Greenplum Data Types with Mapping from
Attunity Replicate Data Types
BOOLEAN bool
BYTES bytea
DATE date
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 825
Table 9.49 | Supported Pivotal Greenplum Data Types with Mapping from Attunity Rep-
licate Data Types (Cont.)
INT2 int2
INT4 int4
INT8 int8
REAL4 float4
REAL8 float8
UINT1 int2
UINT2 int4
UINT4 int8
BLOB bytea
To use this data type with Attunity Replicate, you must enable the
use of BLOBs for a specific task.
During CDC, LOB data types are supported only in tables with a
primary key.
The Pivotal Greenplum target database has limited LOB support. You
cannot use an unlimited LOB size for this database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 826
Table 9.49 | Supported Pivotal Greenplum Data Types with Mapping from Attunity Rep-
licate Data Types (Cont.)
To use this data type with Attunity Replicate, you must enable
the use of CLOBs for a specific task.
During CDC, LOB data types are supported only in tables with
a primary key.
The Pivotal Greenplum target database has limited LOB support.
You cannot use an unlimited LOB size for this database.
For more information, see Task Settings/Metadata.
NCLOB text
To use this data type with Attunity Replicate, you must enable
the use of NCLOBs for a specific task.
During CDC, LOB data types are supported only in tables with
a primary key.
The Pivotal Greenplum target database has limited LOB support.
You cannot use an unlimited LOB size for this database.
For more information, see Task Settings/Metadata.
Note If you want to use multiple gpfdist programs, then you should set them up
on different ports. See Using Multiple gpfdist Programs.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 827
To create use a gpfdist service:
Note The Pivotal Greenplum database that you use in this command must be
configured in Attunity Replicate. If you have defined more than one Pivotal
Greenplum database, you can use any of the database names. All of the
defined endpoints on the same Attunity Replicate instance are included in the
defined service.
3. Start the service before you run any Pivotal Greenplum tasks by typing the
following at the command line prompt:
sc start AttunityReplicateServer_<name of database you defined
Note If you chose to create a data folder in a separate location from where you
installed Attunity Replicate, you must add the prefix -d <path to data
folder> before any command line task described in this section. For example,
to start the service you must type:
-d < path to data folder> sc start AttunityReplicateServer_<name
of database>
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 828
To delete the service:
Note A log file is available in the Attunity Replicate data folder and in the Pivotal
Greenplum debug files, which is accessed through the Pivotal Greenplum
database console.
1. Install gpfdist programs on the computers you want to use them. See
Pivotal Greenplum Prerequisites for Attunity Replicate.
2. Make sure that you assign a different port number for the gpfdist program in
each task. For more information, see Setting General Connection Properties.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 829
4. In the Description field, type a description that helps to identify the
Pivotal Greenplum database. This is optional.
5. Select TARGET as the database role.
7. Type the Database host name. This is the name of the computer with the
Pivotal Greenplum instance you want to work with.
Note Although the gpfdist program acts as a Webserver, it does not carry out
security checks on any requests made to it. Therefore, when you define the path
to the gpfdist program, it must be to a specific location so that no other data on
the computer is accessed.
You can use the Advanced tab to add specific properties and create a custom
connect string. In this case, you do not need to enter information in this tab.
For more information on using the Advanced tab, see Setting Advanced
Connection Properties.
8. Type the Pivotal Greenplum database Port, where the Pivotal Greenplum instance
you are working with is located. The default value is 5432.
9. Type the Pivotal Greenplum authentication information (User Name, Password)
for the authorized user for this Pivotal Greenplum database. If you do not know this
information, see your Pivotal Greenplum system manager.
Note
If you are using the Advanced tab to create a custom string, make sure to
include the User Name property. A Password can also be included but is not
required. See Setting Advanced Connection Properties for more information.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 830
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
Important: Make sure that the Pivotal Greenplum user entered in the Pivotal
Greenplum Authentication section has the correct access privileges. For information
on how to provide the required privileges, see Security Requirements.
10. Type the Database name or select one from the list of available databases. This is
the name of the Pivotal Greenplum database where you are replicating the data to.
11. Type the gpfdist hostname for the server where the gpfdist program is installed.
12. Type the gpfdist port number where the gpfdist program is listening. The
default value is 8080.
13. In the Security section:
a. To enable SSL, select the Use SSL check box.
b. In the CA Path, specify the folder containing the certificate required to
execute gpfdist.
c. In the Public key file field, specify the full path of the public key file
(i.e. including the file name). The file can reside in the same folder as
the CA certificate.
d. In the Private key file field, specify the full path of the private key file
(i.e. including the file name). The file can reside in the same folder as the
CA certificate.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 831
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before
the file is moved into the load folder. The default value is 32000 KB.
Write buffer size: (KB): Select or type the maximum amount of memory (in
Kilobytes) used to store the data before moving it to the load folder. The default
value is 1001.
ODBC driver: Type the name of the ODBC driver you are using to connect to the
Pivotal Greenplum database you are working with. The default value is
DataDirect 7.0 Pivotal Greenplum Wire Protocol.
Additional ODBC connection properties: Type any additional ODBC
connection properties if required
Use externally managed gpfdist: Select this check box to use an external
gpfdist with this Pivotal Greenplum database.
Storage folder: Type the name and location (enter full path) of the folder that
holds the CSV files for loading. This is available only if you are using an externally
managed gpfdist.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 832
Using a PostgreSQL-Based Database as a Target
This section describes how to set up and use a PostgreSQL-based target endpoint
in a replication task.
You need to configure a PostgreSQL-based endpoint when replicating to any of
the following databases:
PostgreSQL
Amazon Aurora (PostgreSQL)
Amazon RDS for PostgreSQL
In this section:
▶ Prerequisites
▶ Security Requirements
▶ PostgreSQL Database Target Data Types
▶ Data Types when Replicating from a PostgreSQL Source
▶ Setting General Connection Properties
▶ SSL Authentication Options
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the client prerequisites when replicating to a
PostgreSQL target.
Attunity Replicate Server for Windows:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 833
Note Make sure that the psql.exe path (e.g. "C:\Program
Files\PostgreSQL\9.6\bin") is added to the system PATH.
Note If PgAdmin and the Greenplum client are installed on the same
Replicate Server, tasks configured to use a PostgreSQL target will fail.
Makes sure that the /etc/odbcinst.ini file contains an entry for PostgreSQL,
as in the following example:
[PostgreSQL]
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
Debug = 0
CommLog = 1
UsageCount = 2
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 834
Security Requirements
The user specified in the General tab when Setting General Connection Properties must
be a registered user in the PostgreSQL database.
Table 9.50 |
Supported PostgreSQL database Data Types with Mapping from
Attunity Replicate Data Types
BOOL BOOL
BYTES BYTEA
DATE DATE
TIME TIME
TIMESTAMP
VARCHAR (37)
INT1 SMALLINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 FLOAT4
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 835
Table 9.50 | Supported PostgreSQL database Data Types with Mapping from
Attunity Replicate Data Types (Cont.)
VARCHAR (65535)
UINT1 SMALLINT
UINT2 INTEGER
UINT4 BIGINT
UINT8 BIGINT
VARCHAR (65535)
BLOB BYTEA
NCLOB TEXT
CLOB TEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 836
Properties below.
5. From the Type drop-down list, select PostgreSQL for PostgreSQL, Amazon
Aurora PostgreSQL, and Amazon RDS for PostgreSQL.
6. Type the Server name. This is the name or IP address or host name of the
computer with the PostgreSQL database that you want to access.
7. Optionally, change the default port (5432).
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Important: Make sure that the specified PostgreSQL database user has the
correct access privileges.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 837
9. Type the Database name or select one from the list of available endpoints. This
is the name of the PostgreSQL database to which you are replicating data.
10. Click OK to save your settings and close the dialog box.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 838
Setting Advanced Connection Properties
In the Advanced tab, you can set the following properties:
Max file size (KB): Select or type the maximum size (in KB) of a CSV file before the
file is loaded into the PostgreSQL target database. The default value is 32000 KB.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 839
Using SAP HANA as a Target
This section describes how to set up and use a SAP HANA database as a target
database in a replication task.
In this section:
▶ Prerequisites
▶ Permissions
▶ Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Windows: Install the SAP HANA ODBC 64-bit Driver 2.x for Windows on the
Replicate Server machine. The driver name is HDBODBC.
Linux: Install the SAP HANA ODBC 64-bit Driver 2.x for Linux on the Replicate
Server machine. The driver name is HDBODBC.
Add the following section to the odbcinst.ini file located in directory /etc:
[HDBODBC]
Description=64-bit HANA ODBC Driver
Driver=/opt/sap/hdbclient/libodbcHDB.so
fileUsage=1
Permissions
The user specified in the SAP HANA endpoint settings must be granted the
following permissions:
CREATE TABLES
ALTER
SELECT
INSERT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 840
DELETE
DROP
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.51 |
Supported SAP HANA Data Types with Mapping from Attunity Replicate
Data Types
DATE DATE
TIME TIME
TIMESTAMP TIMESTAMP
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 REAL
REAL8 DOUBLE
UINT1 TINYINT
UINT2 SMALLINT
UINT4 INTEGER
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 841
Table 9.51 | Supported SAP HANA Data Types with Mapping from Attunity Replicate
Data Types (Cont.)
BLOB BLOB
NCLOB NCLOB
CLOB CLOB
You can do this step before any of the other steps if you want, however before you
can continue with the next step in this process, you must select the database role.
5. Select SAP HANA as the database Type.
6. In the Server field, specify the IP address or host name of the SAP HANA
database server.
To connect to a High Availability Cluster, specify all of the cluster nodes and
port numbers in the Server field. The nodes should be separated by a comma.
Example:
12.12.1.123:3033,12.12.1.124:3034
7. In the Instance field, enter the instance number of the target SAP HANA database.
8. Enter the Username and Password required to access the SAP HANA database. If
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 842
you do not know this information, see your SAP HANA database administrator (DBA).
9. Click Test Connection to verify that the specified settings are correct.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 843
Using SAP Sybase ASE as a Target
This section describes how to set up and use a SAP Sybase ASE database as the
target endpoint in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Security Requirements
▶ SAP Sybase ASE Database Target Data Types
▶ Non-Supported Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure the following prerequisites have been met:
SAP Sybase ASE ODBC 64-bit client installed on the computer where Attunity
Replicate is located.
SAP Sybase ASE replication enabled for tables using the sp_setreptable command
or privileges to enable it automatically.
RepAgent must be disabled on the SAP Sybase ASE database.
When replicating to SAP Sybase ASE 15.7 installed on a Windows machine
configured with a non-Latin language (e.g. Chinese), Attunity Replicate requires
Sybase 15.7 SP121 to be installed on the SAP Sybase ASE machine.
Turn off automatic truncation using the following command:
sp_dboption mydb, "trunc log on chkpt", false
go
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 844
Limitations
The following limitations apply:
Only one Attunity Replicate task can be run per SAP Sybase ASE database.
Attunity Replicate tasks cannot run concurrently with SAP Sybase ASE
Replication Server against the same SAP Sybase ASE database.
Zero values located at the end of binary data type strings are truncated
when replicated to the target database. For example,
0x0000000000000000000000000100000100000000 in the source table will become
0x00000000000000000000000001000001 in the target table.
Attunity Replicate creates the target table with columns that do not allow NULL
values, if the database default is not to allow NULL values. Consequently, if a Full
Load or CDC replication task contains empty values, errors will occur.
To prevent this from happening:
Right-click the database name and select Properties from the context menu.
In the Options tab, select Allow nulls by default and then click OK.
Security Requirements
You must provide SAP Sybase ASE account access to the Attunity Replicate user. This
user must have read/write privileges in the SAP Sybase ASE database.
Note SAP Sybase ASE does not support applying changes to binary data types in
Batch optimized apply mode. For more information on Batch optimized apply
mode, see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source endpoint you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 845
Table 9.52 | Supported SAP Sybase ASE Data Types with Mapping from Attunity Replicate
Data Types
DATE DATE
TIME TIME
BIGDATETIME
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 REAL
UINT1 TINYINT
BLOB IMAGE
CLOB UNITEXT
NCLOB TEXT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 846
Non-Supported Data Types
Target SAP Sybase ASE tables with columns of the following SAP Sybase ASE data types
cannot be replicated. Replicated columns with these data types will show as null.
UDT
1. In the Attunity Replicate Console, click Manage Endpoint Connections to open the
Manage Endpoints Connections dialog box. Then click the New Endpoint
Connection button. For more information on adding an endpoint to Attunity
Replicate, see Working with Endpoints.
2. In the Name field, type a name for your database. This can be any name that will
help to identify the database being used.
3. In the Description field, type a description that helps to identify the SAP Sybase
ASE database. This is optional.
4. Select TARGET as the database role.
6. In the Server Name field, enter the host name or IP address of the computer
on which the SAP Sybase ASE database is installed.
You can use the Advanced tab to add specific properties and create a
custom connect string. In this case, you do not need to enter information in
this tab. For more information on using the Advanced tab, see Setting
Advanced Connection Properties.
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 847
If the connection is successful a message in green is displayed. If the
connection fails, an error message is displayed at the bottom of the
dialog box.
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
8. Type the SAP Sybase ASE authentication information (User Name, Password)
for the authorized user for this SAP Sybase ASE database. If you do not know
this information, see your SAP Sybase ASE database Administrator (DBA).
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
If you want to set custom properties for this database, see Setting
Advanced Connection Properties.
Important: Make sure that the SAP Sybase ASE user entered in the SAP Sybase
ASE Authentication section has the correct access privileges. For information on
how to provide the required privileges, see Security Requirements.
9. In the Database name field, enter the SAP Sybase ASE database name.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 848
Note If the user name or password specified in the General tab contains non-
Latin characters (e.g. Chinese), the following property is required:
charset=gb18030
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 849
Using SAP Sybase IQ as a Target
This section describes how to set up and use a SAP Sybase IQ database as a
target database in a replication task.
In this section:
▶ Prerequisites
▶ Limitations
▶ Security Requirements
▶ SAP Sybase IQ Target Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
Make sure the following prerequisites have been met:
Limitations
The following limitations apply:
Full LOB mode is not supported.
Replication of LOBs during Change Processing is not supported in Bulk Apply
mode (LOB values are replicated to NULL).
Security Requirements
The user specified in the SAP Sybase IQ endpoint settings must have DBA permissions
in the SAP Sybase IQ database.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 850
SAP Sybase IQ Target Data Types
The following table shows the Sybase target data types that are supported when using
Attunity Replicate and the default mapping from Attunity Replicate data types.
Note SAP Sybase IQ does not support applying changes to binary data types in
Batch optimized apply mode. For more information on Batch optimized apply
mode, see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using.
For additional information about Attunity Replicate data types, see Replicate Data Types.
Table 9.53 |
Supported SAP Sybase IQ Data Types with Mapping from Attunity
Replicate Data Types
BOOLEAN BIT
DATE DATE
TIME TIME
TIMESTAMP
VARCHAR (37)
INT1 TINYINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
REAL4 REAL
REAL8 DOUBLE
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 851
Table 9.53 | Supported SAP Sybase IQ Data Types with Mapping from Attunity
Replicate Data Types (Cont.)
UINT2 SMALLINT
UINT4 INTEGER
UINT8 BIGINT
Note The SAP Sybase IQ database requires a special license to support LOBs.
BLOB BLOB
CLOB CLOB
NCLOB CLOB
Note
Sybase can also be used as a source database. For information on using Sybase
as a source, see Using SAP Sybase ASE as a Target.
You can also use Sybase files as a source or target. For more information,
see Using the Attunity Replicate File Channel.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 852
3. In the Description field, type a description that helps to identify the
Sybase database. This is optional.
4. Select TARGET as the database role.
9. Type the Sybase authentication information (User Name, Password) for the
authorized user for this Sybase database. If you do not know this information,
see your Sybase database Administrator (DBA).
Note
This information is required. If you are using the Advanced tab to create a
custom string, make sure to include the User Name and Password properties.
See Setting Advanced Connection Properties for more information.
Important: Make sure that the Sybase user entered in the Sybase
Authentication section has the correct access privileges. For information on how
to provide the required privileges, see Security Requirements.
10. In the database field, enter the name of the SAP Sybase IQ database.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 853
To add internal Attunity Replicate parameters:
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 854
Using Snowflake on AWS as a Target
This section describes how to set up and use Snowflake on AWS as a target in a
replication task. Snowflake on AWS is located in the cloud and is accessed through an
Amazon Web Services (AWS) account.
Attunity Replicate moves the data from the source endpoint into an Amazon S3 bucket,
and then loads the data into the relevant tables in the Snowflake data warehouse (using
the "copy" command).
Note When configuring a task with a Snowflake on AWS target endpoint, note
the following:
If the Apply Changes task option is enabled and the change processing mode is set
to Batch optimized apply, it is recommend to enable the Apply batched changes
to multiple tables concurrently option in the Change Processing Tuningtab.
In this section:
▶ Prerequisites
▶ Amazon Web Services Account Prerequisites
▶ Amazon S3 Staging Prerequisites
▶ Client Prerequisites
▶ General Prerequisites
▶ Firewall Prerequisites
▶ Snowflake on AWS Data Types
▶ Setting General Connection Parameters
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 855
Prerequisites
The following describes the general prerequisites for using Snowflake on AWS as
an Attunity Replicate target endpoint.
https://docs.snowflake.net/manuals/index.html
Bucket access credentials: Make a note of the bucket name, region, access
key and secret access key - you will need to provide them in the Attunity
Replicate Snowflake on AWS target settings.
Bucket access permissions: Attunity Replicate requires
read/write/delete permissions to the Amazon S3 bucket.
Client Prerequisites
Attunity Replicate for Windows: Download and install the latest Windows 64-bit
ODBC driver for Snowflake.
Attunity Replicate for Linux: Download and install the latest Linux 64-bit ODBC
driver for Snowflake.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 856
General Prerequisites
The Attunity Replicate Server machine must be set to the correct time (UTC).
Firewall Prerequisites
Firewall port 443 needs to be opened for outbound communication.
Note Snowflake on AWS does not support applying changes to binary data types
in Batch optimized apply mode. For more information on Batch optimized
apply mode, see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.54 |
Supported Snowflake on AWS Data Types with Mapping from
Attunity Replicate Data Types
BOOL BOOLEAN
BINARY (8388608)
DATE DATE
TIME TIME
INT1 NUMBER
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 857
Table 9.54 | Supported Snowflake on AWS Data Types with Mapping from Attunity Rep-
licate Data Types (Cont.)
INT4 NUMBER
INT8 NUMBER
NUMBER (p,s)
NUMBER (Length)
REAL4 FLOAT4
REAL8 FLOAT8
VARCHAR (16777216)
UINT1 BYTEINT
UINT2 INTEGER
UINT4 INTEGER
UINT8 BIGINT
VARCHAR (65535)
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab description in Customizing Tasks .
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 858
Table 9.54 | Supported Snowflake on AWS Data Types with Mapping from Attunity Rep-
licate Data Types (Cont.)
BINARY (8388608)
7. Configure the remaining settings in the General tab as described in the table below.
OptionDescription
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 859
User name Your Snowflake user name.
Password The password for the user entered in the User name field.
Note The information for these properties is available from the account page
for Amazon Web Services (AWS) with the Snowflake on AWS cluster. If you do
not have these values, refer to your AWS account or the Snowflake on AWS
System Administrator for your enterprise.
Bucket name The name of the Amazon S3 bucket to where the files
will be copied.
Bucket region
The Amazon S3 region where the S3 buckets and folders you are
using are hosted. The default value is US East (N. Virginia).
Command 1:
create or replace file format MY_FILE_FORMAT
TYPE='CSV' field_delimiter=',' compression='GZIP'
record_delimiter='\n' null_if=(' attrep_null')
skip_header=0 FIELD_OPTIONALLY_ENCLOSED_BY='\"';
Command 2:
create or replace stage “PUBLIC”.MY_S3_STAGE
file_format=MY_FILE_FORMAT url='s3://MY_STORAGE_
URL' credentials=(aws_role='MY_IAM_ROLE');
Where:
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 861
MY_FILE_FORMAT - Can be any value.
Note
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button is not
available unless the test connection fails.
Option Description
General
Max file size Select or type the maximum size of the CSV file used to transfer data to
(MB) Snowflake. The default value is 100 MB.
ODBC driver The name of the default ODBC driver you are using to connect to
Snowflake. The default value is SnowflakeDSIIDriver.
Proxy Server
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 862
Use proxy Select this option to access Amazon S3 via a proxy server.
server
User name The user name for accessing the proxy server.
Scheme Select which protocol to use to access the server (HTTP or HTTPS). In
order to use HTTPS, you must first install the CA certificate that signed
the proxy’s certificate on the Replicate Server machine, as follows:
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 863
Using Snowflake on Azure as a Target
This section describes how to set up and use Snowflake on Azure as a target in a
replication task.
Note When configuring a task with a Snowflake on Azure target endpoint, note
the following:
If the Apply Changes task option is enabled and the change processing mode is set
to Batch optimized apply, it is recommend to enable the Apply batched changes
to multiple tables concurrently option in the Change Processing Tuningtab.
In this section:
▶ Prerequisites
▶ Microsoft Azure Blob Storage Account
▶ Client Prerequisites
▶ General Prerequisites
▶ Permissions
▶ Firewall Prerequisites
▶ Snowflake on Azure Data Types
▶ Setting General Connection Parameters
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Prerequisites
The following section describes the prerequisites for using Snowflake on Azure as
an Attunity Replicate target endpoint.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 864
Microsoft Azure Blob Storage Account
Sign up for a Microsoft Azure Blob Storage account and make a note of the account
name, access key, container name, SAS token (Shared Access Signature), and target
folder. Note that the duration of the SAS token must be the same as the Replicate task
duration. For an explanation of how to configure the SAS token, visit:
https://docs.Snowflake on Azure.net/manuals/user-guide/data-load-azure-config.html
For information on signing up for a Microsoft Azure Blob Storage account, visit:
https://www.Snowflake on Azure.com/blog/how-to-get-started-with-Snowflake on
Azure-on-azure/
Client Prerequisites
Download and install the latest Windows 64-bit ODBC driver for Snowflake on Azure.
General Prerequisites
The Attunity Replicate Server machine must be set to the correct time (UTC).
Permissions
Attunity Replicate performs the following operations on the Azure Blob
Storage container/folder:
Firewall Prerequisites
When Replicate Server runs on a machine outside Azure - Open port 1433
for outbound traffic.
When Replicate Server runs on an AzureVM - Open the following ports
for outbound traffic:
1433
11000-11999
14000-14999
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 865
Note The Snowflake on Azure endpoint does not support applying changes to
binary data types in Batch optimized apply mode. For more information on
Batch optimized apply mode, see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.56 |
Supported Snowflake on Azure Data Types with Mapping from
Attunity Replicate Data Types
BINARY (8388608)
DATE DATE
TIME TIME
INT1 NUMBER
INT2 NUMBER
INT4 NUMBER
INT8 NUMBER
NUMBER (p,s)
NUMBER (Length)
REAL4 FLOAT4
REAL8 FLOAT8
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 866
Table 9.56 | Supported Snowflake on Azure Data Types with Mapping from Attunity Rep-
licate Data Types (Cont.)
VARCHAR (16777216)
UINT1 BYTEINT
UINT2 INTEGER
UINT4 INTEGER
UINT8 BIGINT
VARCHAR (65535)
Full LOB data types are not supported. For information on including Limited-size LOB
data types in the replication, see the Metadata tab description in Customizing Tasks .
BINARY (8388608)
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 867
Setting General Connection Parameters
This section describes how to configure general connection properties. For an explanation
of how to configure advanced connection properties, see Setting Advanced Connection
Properties below.
7. Configure the remaining settings in the General tab as described in the table below.
OptionDescription
Password The password for the user entered in the User name field.
Staging
Storage Type Azure Blob storage (cannot be changed)
Storage account The name of an account with write permissions to the container.
SAS token Your SAS (Shared Access Signature) for accessing the
container.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 868
Container name The container name.
Note
To determine if you are connected to the database you want to use or if the
connection information you entered is correct, click Test Connection.
If the connection is successful a message in green is displayed. If the connection
fails, an error message is displayed at the bottom of the dialog box.
To view the log entry if the connection fails, click View Log. The server log is
displayed with the information for the connection failure. Note that this button is not
available unless the test connection fails.
Max file size Select or type the maximum size of the CSV file used to transfer data to
(MB) Snowflake on Azure. The default value is 100 MB.
ODBC driver The name of the default ODBC driver you are using to connect to
Snowflake on Azure. The default value is Snowflake on
AzureDSIIDriver.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 869
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 870
Using Teradata Database as a Target
This section describes how to set up and use Teradata Database as a target in a
replication task.
In this section:
▶ An Overview of the Teradata Database Target
▶ Teradata Database Target Load Options
▶ TPT Stream Mode
▶ TPT Load Mode
▶ Database Availability
▶ Required Teradata Database Software, Environments
▶ Replicate Server for Windows
▶ Replicate Server for Linux
▶ Providing Access to the Teradata Database
▶ Editing the Hosts File
▶ Security Requirements
▶ Teradata Database Data Types
▶ Setting General Connection Properties
▶ Setting Advanced Connection Properties
▶ Internal Parameters
▶ Settings Summary
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 871
Teradata Database Target Load Options
You can apply changes in one of two modes:
TPT Stream Mode
TPT Load Mode
Database Availability
Teradata Database with the tables that are being used for replication must be available
to the system. This can be installed on any computer in your network.
For more information about the requirements for working with Attunity Replicate,
see Installation Prerequisites.
Note Teradata Database must be installed in your network and be reachable from
the computer where Attunity Replicate is installed.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 872
Replicate Server for Windows
You must install the following on the same computer where the Attunity Replicate Server
is installed:
Teradata Database ODBC Driver for Windows version 15.00.
Teradata Database Parallel Processor API (TPT API) with the load and Stream TPT
operators. Install either version 14.00 with the latest patch or version 15.10.
Important: A Replicate task cannot be defined with endpoints that use different
ODBC Driver Managers. Teradata Database target is accessed using the DataDirect
ODBC Driver Manager. With the exception of Oracle, Hadoop, File and Replicate
Connect sources (which are not subject to the above limitation), all other source
endpoints use the unixODBC Driver Manager.
To configure a task with a unixODBC source and a DataDirect target (e.g. Microsoft
SQL Server to Teradata Database Target), you need to use the Replicate File
Channel. For more information about setting up a task using the File Channel, see
Using the Attunity Replicate File Channel.
Then verify that it contains a definition for the Teradata ODBC client:
[ODBC DRIVERS]
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 873
Teradata=Installed
[Teradata]
The output should look like this (e.g. for version 14.10):
lrwxrwxrwx 1 root root 47 Oct 28 14:58 /usr/lib64/libodbcinst.so ->
/opt/teradata/client/ODBC_64/lib/libodbcinst.so
/opt/teradata/client/ODBC_64/lib/libodbc.so
6. Add the Teradata Database name to the hosts file as described in Editing the
Hosts File.
7. Run the following commands:
export LD_LIBRARY_PATH=/usr/lib64:$TD_CLIENT_
DIR/tbuild/lib64:/opt/attunity/replicate/lib
export AREP_ODBC_DRIVER_MANAGER=/opt/teradata/client/14.10/odbc_
64/lib/libodbc.so
export ODBCINI=/opt/teradata/client/14.10/odbc_64/odbc.ini
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 874
To add the Teradata Database mappings to the hosts file:
machine. On Windows, the default path for the hosts file is:
~:\Windows\System32\drivers\etc\hosts
2. Add the following line (note the “cop1” after the database name):
<Teradata Database IP address/hostname> <Teradata Database name>cop1
Example:
123.123.123.1 teradatadbonecop1
Note Make sure that the database name added to the hosts files is the same as
the database specified in the Default database field in the Teradata Database
target database settings.
Security Requirements
A user must have the following privileges granted in the Teradata Database to use a
Teradata Database target in an Attunity Replicate task:
GRANT SELECT ON <database>
GRANT INSERT ON <database>
GRANT DELETE ON <database>
GRANT UPDATE ON <database>
GRANT EXECUTE ON <database>
GRANT EXECUTE FUNCTION ON <database>
GRANT EXECUTE PROCEDURE ON <database>
GRANT CREATE TABLE ON <database>
GRANT DROP TABLE ON <database>
GRANT CREATE VIEW ON <database>
GRANT DROP VIEW ON <database>
GRANT NONTEMPORAL on <database>
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 875
When the Stream TPT Operator is selected (in the Advanced tab), the
following privilege is also required:
GRANT CREATE MACRO ON <database>
Note Teradata Database does not support applying changes to binary data types
in Batch optimized apply mode. For more information on Batch optimized
apply mode, see Change Processing Tuning.
For information on how to view the data type that is mapped from the source, see
the section for the source database you are using. For additional information about
Attunity Replicate data types, see Replicate Data Types.
Table 9.58 |
Supported Teradata Database Data Types with Mapping from
Attunity Replicate Data Types
Note
Maximum size
is 640000.
DATE DATE
Precision is not
included unless it is
less than 0.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 876
Table 9.58 | Supported Teradata Database Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
Precision is not
included unless it is
less than 0.
INT1 BYTEINT
INT2 SMALLINT
INT4 INTEGER
INT8 BIGINT
Scale is not
included unless it is
less than 0.
REAL4 FLOAT
FLOAT is
equivalent to REAL
and DOUBLE
PRECISION.
REAL8 FLOAT
See also the note in Teradata Database Data Types below. Note: Maximum
size is 64000.
UINT1 BYTEINT
UINT2 SMALLINT
UINT4 INTEGER
UINT8 BIGINT
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 877
Table 9.58 | Supported Teradata Database Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
Full LOB data types are not supported in the Teradata Database. For information on
including Limited-size LOB data types in the replication, see Metadata. Note also that
the size of a row in the Teradata Database cannot exceed 64KB. This should be taken
into consideration when specifying the maximum LOB size in the Metadata tab.
BLOB VARBYTE
(${MAX_LOB_
SIZE})
MAX_LOB_SIZE is
the maximum LOB
size specified in
Limited-Size LOB
Mode.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 878
Table 9.58 | Supported Teradata Database Data Types with Mapping from Attunity
Rep-licate Data Types (Cont.)
Case-insensitive
character set.
MAX_LOB_SIZE is
the maximum
LOB size specified
in Limited-Size
LOB Mode.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 879
To add a Teradata Database Target to Attunity Replicate:
If you are using the Advanced tab to create a custom string, make sure to
include the USERNAME property. A Password can also be included but is not
required. See Setting Advanced Connection Properties for more information.
Important: Make sure that the Teradata Database user entered in the Teradata
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 880
Database Authentication section has the correct access privileges. For information
on how to provide the required privileges, see Security Requirements.
8. Type the Default database name or select one from the list of available
endpoints. This is the name of the Teradata Database where you are replicating
the data to. For more information, see Teradata Database Target Load Options.
TPT Operator: Select the TPT Operator used to access the Teradata Database.
The possible options are:
Load: Select this to use the TPT Load Mode.
Stream: Select this to use the TPT Stream Mode.
TPT Attributes: You can define one or more of the following attributes:
Account String: The account (database server login) that the DBA assigned to
the username for the Attunity Replicate user.
Buffer Size: The output buffer size (in KB) for sending Load parcels to the
Teradata Database.
You can enter a value from 1 to 64.
Explicit sessions range: Select this if you want set a minimum and/or
maximum number of sessions that can log on to the Teradata Database.
Maximum: The maximum number of sessions that can log on to the
Teradata Database. The default value is 1. The value cannot be higher than
the number of Access Module Processors (AMPs) available.
Minimum: The minimum number of sessions that can log on to the
Teradata Database.
Dynamic statement packing: Select this check box if you want the stream driver to
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 881
dynamically determine the maximum possible pack for the current STREAM job.
Statement packing: Use the counter or type the number of statements that
can be packed into a multiple-statement request (STREAM).
You can enter a value from 2 to 600.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 9 | Adding and Managing Target Endpoints | Page 882
10 | Using the Attunity Replicate
File Channel
This section describes how to use the Attunity Replicate File Channel as a source or
target in a replication task.
Note If you use File Channel as both source and target, the version (build) of
the Attunity Replicate servers on both sides should be identical.
In this chapter:
▶ Setting Up Attunity Replicate File Channel Tasks
▶ Working with the File Channel Data Files
▶ Attunity Replicate Installation Requirements for the File Channel
▶ Security
▶ Limitations
▶ Using the File Channel as a Source
▶ Using the File Channel as a Target
Note When using file channel, Change Tables can be enabled for the remote task
but not for the local task (enabling Change Tables for the local task will result in
remote task failure).
Local Task
You set up the local task using the File-Channel endpoint as a target. The binary
file created in this task is used as the source for one or more remote tasks using
the File-Channel source endpoint.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 883
The local task replicates data from an Attunity Replicate supported endpoint to the file
channel. If you changed the default folder for storing data files (during the installation),
then you must specify the location of the binary file created by the file channel. This
location can be anywhere in your system. For more information on setting up a local
task, see Using the File Channel as a Target.
Remote Task
Remote tasks use the File Channel as a source endpoint. You use the file created by the
local task for this source. You can replicate the data to any endpoint that is supported by
Attunity Replicate. You define the location of the File-Channel file as the remote location
where the file was created. The data is pushed over the network to the defined location
anywhere in your system. You can also define more than one location for the replicated
data. In this case, define a separate remote task for each location.
Note The attrep_cdc_partitions Control Table is not supported when using File
Channel as a source endpoint.
If you want to push the data to an endpoint that is not in your LAN, use the File
Transfer Service to send the files created in the local task to the remote location.
When you run the remote task, data is sent to the target in the following instances:
The first time you run the task as a full load.
Each time changes are made to the file. In this case, change processing takes place.
When the remote task runs, it will continuously look for the source file until the task is
stopped. When the file is found, the data is replicated to the target endpoint. If no
source file is found, an error is displayed; however, the task will continue to check for
the correct file. Therefore, it is recommended that you run the local task first to ensure
that the file exists.
Note To replicate tables that were added to the local file channel task after the
initial full load, you need to reload both the local and the remote file channel tasks.
For more information on setting up a remote task, see Using the File Channel as a Source.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 884
types (e.g. Microsoft SQL Server to Oracle and SAP Sybase ASE).
To do this:
1. For each of the target endpoints, define a separate (remote) task that replicates from
the File Channel source to the target endpoint. In the Advanced tab of the File Channel
source settings, make sure to clear the Delete processed files check box. This
ensures that the File Channel files will be available for distribution as required.
2. Define a local task that replicates from the source endpoint to a File Channel target.
3. Run the local task (this will create the File Channel files required by the remote task).
4. For each of the remote tasks, select which tables to replicate (from the File
Channel source) and optionally apply Filters and Transformations to them.
5. Run the remote tasks.
For more information on defining tasks, see Defining Tasks.
Note By default, all the metadata for the selected source tables is replicated from the
Local Task to the Remote Task. This allows you to remove, add and transform tables in
the remote task as needed. However, if you want the tables in the source and target
endpoints to be identical, you can prevent replication of the metadata (and thereby
shorten the processing time) by specifying provideremotemetadata=N in the Override
connection string parameters field of the File Channel target’s Advanced tab.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 885
Note Adding tables to the remote task is not supported in Apply Changes (CDC-only)
replication tasks. For more information on the available replication options, see
Adding Tasks.
s_msgs: This folder contains messages sent from the source side to the
replication server on the remote target side.
Messages are removed from this folder at the source side when an
acknowledgment message is received stating that the file was transferred
successfully or possibly with a timeout.
Messages are removed from this folder at the target side after they are read.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 886
files are deleted when the file-channel source endpoint finishes reading the file.
You can configure the file-channel source to keep the files, if necessary. See
Setting Advanced Connection Properties for more information.
t_status: This folder contains status updates from the target side to the source
side. Status updates appear as an infinite set of data files that are created according
to a specific schedule. These files are sent from the target by the source. The folder
contains also a fixed name file that is updated with the last created status file name.
It contains the following file:
t_status/cccccccc.fcs: This is a file channel status file (.fcs) where the
file name is a hexadecimal counter of length 8. These files will be
transferred in order with the lower numbers transferred first. If you need to
view them, you should order them by timestamp because alphabetical
ordering will not be consistent with the hexidecimal name.
File channel status files are deleted by the source after being read and by the
target when source status file indicates that this file was already processed.
You can configure the maximum amount of time that the files are kept before
a new file is created as well as the maximum file size for each file. The
minimum file size is 50 MB.
For more information, see Setting Advanced Connection Properties.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 887
You can configure the maximum amount of time that the files are kept before
being creating a new file and the maximum file size for each file. The minimum
file size is 10 MB and the minimum time that a file is kept is 5 seconds.
Security
When using the File Transfer Service, file-channel files are always transferred over
an encrypted session.
The session is encrypted as follows:
The client and server create an AES-256 session key using the Diffie-Hellman key
exchange protocol (using the OpenSSL library). After the key is created, all file
transfers between the client and the server will take place over a secure and
encrypted communication channel.
However, even though the session is encrypted, communication between the client and
the server may still be susceptible to man-in-the-middle attacks. A man-in-the-middle in
possession of the session key would be able to intercept any data transferred between
the client and the server.
To eliminate man-in-the-middle attacks, a "shared password" needs to be provided when
configuring the local and remote file channel endpoints. Once the session is established,
both the client and the server use the shared password to re-key the session key during
the next packet exchange, thereby preventing the original session key from being used
for man-in-the-middle attacks.
To sum up:
1. Strong encryption is used regardless of whether a password was provided.
2. Providing a password eliminates the risk of a man-in-the-middle attack. For
more information about the File Transfer Service, see File Transfer Service.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 888
Limitations
The following limitations apply:
The File Channel endpoint does not support Full LOB mode.
You cannot use the Full Load resume function if you are using the File Channel
endpoint. To resume a Full Load operation, you must delete the original data and
then run the task again.
You must delete the File Channel folder before restarting an Apply Changes task.
After modifying an existing transformation in a remote File Channel task, both the
local and the remote File Channel tasks need to be restarted (by selecting the
Reload Target run option in both tasks).
Control tables defined for the local File Channel task but not for the remote
File Channel task will not be created on the remote task’s target endpoint.
For information on defining Control Tables, see Control Tables.
The remote File Channel task does not support the Metadata only run options.
The remote File Channel task does not support the Stopping the Task after Full
Load task options.
If the local task fails, or is stopped, it is possible that the Full Load operation in
the remote task will not complete successfully.
The Store Changes replication option should not be enabled for local File
Channel tasks.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 889
To add the File Channel source to Attunity Replicate:
6. Type the full path to the Storage Folder where the File Channel files will be
created. The default path when not using the File Transfer Service is:
C:\Program Files\Attunity\Replicate\data\tasks\<task_name>
If you are using the File Transfer Service, the default path is:
C:\Program Files\Attunity\Replicate\data\endpoints\<file-
channel_db_ name>\fc
Note The Replicate File Transfer Service always transfers the local file channel
task’s files to the default directory on the remote system (C:\Program
Files\Attunity\Replicate\data\endpoints\<remote_file-channel_db_
name>\fc). Consequently, if you are using the File Transfer Service, ensure
that the default directory always has enough space for the incoming files.
For more information on using the File Transfer Service, see File Transfer
Service and Using Advanced Properties for a File-Channel Source.
Note The actual size of the File Channel files is usually three-to-four times
larger than the size of the source data. You should therefore make sure that the
location of the specified Storage Folder has sufficient disk space.
This folder should be in a location that is accessible from anywhere in the WAN you
are working with.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 890
Note
You can use the Advanced tab to define specific properties and create a
custom connect string. In this case, you do not need to enter information
in this tab. For more information on using the Advanced tab, see Using
Advanced Properties for a File-Channel Source.
To determine if you are connected to the endpoint you want to use or if the
connection information you entered is correct, click Test Connection.
Input files are received via file transfer service: Select this check box to
receive the source input files using the Replicate File Transfer Service.
Password: The password that will be used to establish a secure connection
with the File Channel Target.
Important: When using the File Transfer Service, an agreed upon password
is required in order to establish a secure connection between the File
Channel Source and the File Channel Target. Accordingly, the password
specified in the File Channel Source settings and the password specified in
the File Channel Target settings must be identical.
For more information about the File Transfer Service, see File Transfer Service.
Delete processed files: Select this check box to delete the File Channel files
after the data has been replicated to the target endpoint.
You should clear this check box if other tasks need to use the files.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 891
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 892
Note
The Type is different depending on the type of file you are creating, however
the information you enter is the same for all file types.
All files are used as targets, however you can use an Attunity Replicate file as
a source only after you created the file by loading data into it as a target.
6. If you changed the default data folder during installation, type the full path to the
Storage Folder (e.g. D:\data\tasks\) where the file is being created. Otherwise,
you can leave this field empty. Note that this field will be ignored when the
Transfer files to remote file channel option is enabled in the Advanced tab.
Note
You can use the Advanced tab to define specific properties and create a
custom connect string. In this case, you do not need to enter information
in this tab. For more information on using the Advanced tab, see Setting
Advanced Connection Properties.
To determine if you are connected to the endpoint you want to use or if the
connection information you entered is correct, click Test Connection.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 893
To view the log entry if the connection fails, click View Log. The server log
is displayed with the information for the connection failure. Note that this
button is not available unless the test connection fails.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 894
Max transfer streams: The maximum number of streams to use when
transferring the files. Adjust the number of streams as required to
optimize transfer speeds.
Password: The password that will be used to establish a secure connection
with the File Channel Source.
Important: When using the File Transfer Service, an agreed upon password is
required in order to establish a secure connection between the File Channel
Source and the File Channel Target. Accordingly, the password specified in the
File Channel Target settings and the password specified in the File Channel
Source(s’) settings must beidentical.
Internal Parameters
Internal parameters are parameters that are not exposed in the UI. You should only
use them if instructed by Attunity Support.
2. In the edit box, type the name of the parameter you need to add and then click it.
3. The parameter is added to the table below the search box with its default value.
4. Change the default value as required.
5. To reset the parameter value to its default, click the "Restore default value" icon at
the end of the row.
Settings Summary
You can view a summary of your settings by clicking the Setting Summary link. This
is useful if you need to send a summary of your settings to Attunity Support.
Copyright © Attunity Ltd. Chapter 10 | Using the Attunity Replicate File Channel | Page 895
11 | Using the Log Stream
This chapter explains how to use the Log Stream feature.
In this chapter:
▶ Terminology
▶ Overview
▶ Step 1: Create a Log Stream Target Endpoint
▶ Step 2: Create and Run a Log Stream Staging Task
▶ Step 3: Create a Replication Task to Transfer the Staged Data to the Target
Terminology
The following terms are used throughout this section.
Log Stream - The name of the functionality that allows users to stream and
store transaction log data as files on a disk.
Log Stream target endpoint - The name of the component that writes the
streamed log data to the Log Stream Staging folder.
Log Stream Staging folder – The location of the files created from the streamed
log data.
Log Stream Staging task - The name of the task type that writes the data to the
Log Stream Staging folder.
Replication task - The name of the task type that reads the data from the
Log Stream Staging folder and writes it to the final target endpoint.
Overview
Log Stream enables a dedicated Replicate task to save data changes from the
transaction log of a single source database and apply them to multiple targets, without
the overhead of reading the logs for each target separately. Data changes from the log
are written to a file in the Log Stream Staging folder enabling one or more targets to
access them via separate Replication tasks.
Log Stream provides the following enhancements:
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 896
When multiple CDC tasks read changes from large databases that share a single
transaction log (e.g. Oracle and IBM DB2 for z/OS), the log is polled for only one
target (the Log Stream target) instead of being polled separately for each target.
This improves performance while greatly reducing the load on the source database
and network.
Changes are continuously written to the log stream regardless of the availability of
the target endpoint. This is especially beneficial in situations where the original
source transaction logs are no longer online or cannot be accessed efficiently.
Additionally, this approach significantly reduces latency when the target endpoint
becomes available again as the logs are read from the staging folder instead of from
the source database.
As the changes are constantly streamed to the Replicate server machine, the
Replicate SLA that requires an extended log retention policy to be defined on the
source in case the target cannot be accessed is no longer applicable. This allows you
to reallocate valuable resources on the source while ensuring that log files will be
up-to-date (on the Replicate Server machine) even if the target become unavailable.
When replicating a single source database to multiple targets, each target can have
its own subset of the log stream data (tables, rows, etc.) and be started, stopped,
and scheduled independently of the others.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 897
The Flow
As mentioned above, a Log Stream setup provides benefits both when replicating to a
single target and when replicating to multiple targets. The following diagram illustrates
the end-to-end flow of a Log Stream setup with multiple target endpoints:
1. A Log Stream Staging task streams the source logs from the specified source
endpoint to a designated staging folder on Replicate Server (specified in the Log
Stream target endpoint settings).
Then, at least one Replication task:
2. Performs a Full Load of the source data to the designated target endpoint(s).
3. Transfers the changes from the Log Stream Staging folder to the final target endpoint
(s).
Note The Log Stream Staging task and replication task must both run on the
same Replicate server.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 898
Name for the endpoint. For example, Log Stream Staging Target.
Optionally, provide a description.
Select the Target role option.
From the Type drop-down list, select Log Stream.
In the Storage path field, specify the full path in which to store the streamed
data (i.e., the binary file with the record changes). You cannot specify a relative
path. Note that the Staging file folder must be on the Replicate server.
Batch: Specify when to close the batch of operations that will be applied to
the target endpoint:
Apply batch after (seconds): Length of time that the batch should run.
Default is 5 seconds. Maximum permitted time is 60 seconds.
This parameter effects the start time of the Replication task; until at least
one batch is written by the Log Stream Staging task, the Replication task
will not start.
Apply batch when number of changes reaches: Maximum number
of change operations to include in the batch. Default is 10,000.
Maximum permitted number of operations is 1,000,000.
Retention: Specify when the staging file should be deleted (note that active
files will not be deleted):
Delete staging files after (hours): Select the check box and specify
the maximum time before a file is deleted. Default is 48 hours. Maximum
permitted time is 10,000 hours.
Delete oldest files when the total size of all staging files exceeds
(MB): The maximum size that you want to allocate for the staging folder. If
the specified size is reached, Replicate will start deleting files from the
oldest to the newest until the total size falls below the upper limit.
Default is 100,000 MB. The minimum size should be at lease twice the
defined rollover size. Maximum permitted size is 10,000,000.
Rollover: Specify when to start writing to a new staging file:
Roll over file after (minutes): Number of minutes after which a new
staging file should be started. Default is 120 minutes. Maximum
permitted time is 10,080 minutes (one week).
Roll over files larger than (MB): Size of the file after which a new
staging file should be started. Default is 500 MB. Maximum permitted size is
100,000 MB.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 899
3. Click Test Connection to verify the information you entered and the availability
of the target database.
4. Click Save.
The Log Stream Staging task creates a “Staging” file in the Log Stream Staging folder.
This file contains the changes from the source database transaction log as a compressed
binary file.
Before running the Log Stream Staging task, you must define and add the source endpoint
where your data is currently stored and the target endpoint where you want to replicate the
data – in this case, the Log Stream target endpoint. After you add the endpoints to your task,
you can select the tables from the source endpoint that you want to replicate. For information
on selecting tables, see Adding Tables and/or Views to a Task.
Note
Log Stream Staging tasks can be imported or exported. For details, see
Exporting and Importing Tasks.
Log Stream Staging tasks can process files created by previous versions of
Attunity Replicate. The files will include an indication of the version number.
1. In the Tasks view, click New Task. The New Task dialog box opens.
2. Specify a name for the task.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 900
4. Click OK.
5. The next step in the Log Stream Staging task is to select the source endpoint
where your data is currently stored, and the Log Stream target endpoint.
Note
Only target endpoints of type "Log Stream" are available for selection.
Log Stream target endpoints that are already associated with other Log
Stream Staging tasks are not available for selection.
The following figure provides an example of selected source and target endpoints for a
Log Stream Staging task:
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 901
6. In the right pane in the Console, click Table Selection to select the source tables
that you want to replicate.
The task will create the metadata in the Log Stream staging folder and then
capture the changes from the specified source endpoint.
The task is now ready to run.
7. To execute the Log Stream Staging Task, click the Run button.
Note The Log Stream Staging task must be run before the Replication task
described in Step 3: Create a Replication Task to Transfer the Staged Data to
the Target.
You can view and monitor the results of the replication in real-time.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 902
Tables are already loaded. Start processing changes from: Date and Time,
Source change position (e.g. SCN or LSN), Now
Recovery: Recover using locally stored checkpoint
In the Task Settings window, under the Target Metadata tab:
Target table schema is unavailable
LOB Size settings:
The Allow unlimited LOB size option is disabled.
Limit LOB size to (KB): Default is 8 KB; maximum permitted size is 102,400 KB.
In the Task Settings window, the following tabs are not available:
Metadata > Control tables
Full Load
Change processing: Apply Changes Settings, Store Changes Settings.
Change Processing Tuning tab: Batch Tuning, Miscellaneous Tuning
Change processing mode: only Transactional Apply is available
Error handling: Data Errors, Table Errors, Apply Conflicts
Character Substitution
In the Table Settings window:
The following tabs are not available: Transform, Filter, Parallel Load, LOB
Column Handling
In the General tab, the Map to target table section is unavailable.
Global Transformations are not available.
The source_lookup Data Enrichment function is not supported. For more
information on this function, see Functions
Source endpoints that do not support CDC (such as, ODBC, Hadoop) cannot be
used as sources for Log Stream Staging tasks.
File Channel cannot be used as a source for a Log Stream Staging task.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 903
To do this, you must first duplicate of the source endpoint that has the data you want
to replicate. Then, you can create a Replication task in which the source endpoint is
the duplicated source endpoint of the Log Stream Staging task and the target is any
required target endpoint.
Note
When duplicating a source endpoint for use in a Replication task, you can define
credentials that are different to those used in the Log Stream Staging task.
Instead of duplicating the source endpoint, you can create a new endpoint and
give it the same properties as the existing one.
To duplicate a specific endpoint and connect it to the Log Stream Staging task:
1. In the left panel of the Manage Endpoint Connections window, select the
original source endpoint defined for the Log Stream Staging task.
2. Click the Duplicate button.
3. In the General tab, do the following:
a. Replace the name of the source endpoint prefixed by “Copy of …” with a
meaningful name for the duplicated endpoint (for example, Oracle1 with
Log Stream).
b. Select the check box Read changes from log stream staging folder.
Note This check box only appears if the duplicated source is connected to
the Log Stream Staging endpoint.
c. From the Log stream staging task drop-down list, select the name of
the relevant Log Stream Staging task.
Note All Log Stream Staging tasks that are related to the duplicated
source are provided in the drop-down list. If a task was never run, "Not
Run" is displayed next to its name.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 904
4. Click Test Connection to determine if you are connected to the Log Stream
staging task and if the information you entered is correct.
5. Click Save and then Close.
1. In the Tasks view, click New Task. The New Task dialog box opens.
2. Specify a meaningful name for the replication task.
3. Choose the Unidirectional replication profile option.
4. If required, select the Full Load or Store Changes task options.
5. Click Apply Changes, and then OK.
6. Select the required source with Log Stream endpoint from the list of
endpoint connections, and drag it to the top circle in the endpoints map.
Example:
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 905
Note
The Task Name links directly to the Log Stream Staging task.
7. Select the required target endpoint from the list of endpoint connections, and drag it
to the bottom circle in the task map.
8. From the right pane in the Console, click Table Selection and select the tables
that you want to replicate. You can select any of the tables that were included in
the Log Stream Staging task.
9. Click the Run button to execute the Replication task.
The Replication task will load the tables to the final target endpoint, replicating the
full load from the original source endpoint and the changes from the Log Stream
Staging folder.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 906
Replication Task Limitations and Considerations
The Bidirectional replication profile is not supported.
In the Target Metadata tab of the Task Settings dialog box, if you want to enable
the Replicate LOB columns option, the Limit LOB size to value must be the
same as the value specified in the Log Stream Staging task. Note however that if
the Replicate LOB columns option is disabled for the Log Stream Staging task, it
cannot be enabled in the Replication task.
In the Table Settings window, most tabs are available and functional with the
exception of the LOB Column Handling tab which is not available (at table level).
The source_lookup Data Enrichment function is not supported. For more
information on this function, see Functions
File Channel cannot be used as a target for a Replication task.
Copyright © Attunity Ltd. Chapter 11 | Using the Log Stream | Page 907
12 | Customizing Tasks
This section describes how to customize a replication task. For example, you can
create new tables or columns for the target endpoint or select only some of the data
from each column to be replicated. This is done using transformations and filters.
Note Although the descriptions in this section only refer to tables, the procedures
described herein are applicable to views as well. When a transformation is defined for
a view, the word "View(s)" appears in the UI instead of the word "Table(s)".
In this chapter:
▶ Table Settings
▶ Defining Global Transformations
▶ Using the Expression Builder (for Filters, Transformations, and Global
Trans-formations)
▶ Task Settings
Table Settings
In the <Table_Name> - Table Settings window, you can define how the data for
each individual table/view is replicated to the target.
Note Some of the table settings are not available in a Log Stream Staging setup.
For information on the availability of table settings in a Log Stream Staging setup,
see Using the Log Stream.
2. In Designer view, select the desired table from one of the following tabs on the right of
The Patterns and Selected Tables tab - if the desired table was
explicitly selected.
The Full Table List tab - if the desired table was selected using a table
inclusion pattern.
For information on how to define table selection patterns, see
Creating Table/View Selection Patterns.
3. Click the Table Settings button above the table list. The
Click Restore Table Defaults at the bottom left of the Table Settings window.
This option is available in all tabs.
Any changes you made will be discarded and the table's default settings will
be restored.
Note The names of modified tables will be followed by the word (changed),
enabling you to easily identify which tables have been modified.
Note Although the descriptions in this section only refer to tables, the procedures
describe herein are applicable to views as well. When a task is being performed for
a view, the word "View(s)" will appear in the UI instead of the word "Table(s)"
2. Click the General tab on the left side of the window, as shown below.
In the Map to target table section, the following options are available:
Table Schema: Specify the schema in which you want the table to be created
on the target.
Table Name: Specify a new name for the table on the target.
Table tablespace: This option is only available when the task is defined with
an Oracle target endpoint.
Specify the name of the tablespace in which you want the table to be created on
the target. By default (i.e. when this field is empty), the table will either be
created in the source table tablespace on the target (when replicating from an
Note Although the descriptions in this section only refer to tables, the procedures
describe herein are applicable to views as well. When a transformation is defined for
a view, the word "View(s)" will appear in the UI instead of the word "Table(s)".
Limitations
Transformations are subject to the following limitations:
They are not supported for calculating columns of Right-to-Left languages.
Transformations cannot be performed on columns that contain special characters (e.g.
#, \, /, -) in their name.
They cannot be performed on columns that have a pound character (#) in their name.
The only supported transformation for LOB/CLOB data types is to drop the column
on the target.
1. Select the table you want to transform and open the Table Settings window.
2. Click Transform on the left side of the window.
The following figure shows the information in the Transform tab of the
Table Settings window.
The Transform Tab in the Table Settings window has the following components:
Input: This lists the columns on which you can perform transformations.
Note When creating a transformation for the SAP Application source endpoint,
you can hover your mouse cursor over an Input column to see a tooltip with the
table’s actual name:
Output: This table shows the defined output for the columns in the table where
you are performing the transformations. It contains the following columns:
Key: This indicates whether the column is a segment of the Unique Index. A
key icon is displayed next to columns that are segments of the Unique Index.
Click the column to add and remove keys.
Name: The name of the column. To change the name of the column, select
the field with the column name you want to change and type a new name in
this column if you want to change the name of the column or if the column is
calculated (added to the table). See the table Transformation Actions for
more information.
Type: The data type for the column. To change the data type for the column,
select the field with the data type you want to change and select a new data
type. See the following table for more information.
Note Dropping a column, saving your changes, and then adding a column with the
same name and defining an expression corresponding to the dropped column's data,
is not supported. If you mistakenly drop a column, simply add the column back
again without an expression.
Table 12.1 |
Transformation Actions
To Do This
Rename a column Select the Name column for the table
column you want to change. Type in a new
name.
To Do This
Change the data type Select the Type column for the table
for a column column you want to change and select a
new data type from the drop-down list.
Make sure that the data type you select is
compatible with the data in that column.
To Do This
Delete a column From the Output list, select the row with
the column you want to delete and click the
left-facing arrow button.
To Do This
Change the data type This is required if a source column is
for a specific input defined as character type but the data
column stored in that column is binary or vice
versa.
Note Supported
with the IBM DB2 Note When the source column type is
for iSeries and IBM STRING, WSTRING, CLOB, or NCLOB,
DB2 for z/OS you must also select a Character Set,
source endpoints otherwise an error will be shown and the
only. OK button will be disabled.
To Do This
For a description of the various list actions that you can perform, see List Actions.
The following procedure is supported with the IBM DB2 for iSeries and IBM DB2 for
z/OS source endpoints only.
Note If you edit an existing UCM file, you must also change the values of the
<code_set_name> and <icu:alias> properties. If the file does not contain an
<icu:alias> property, then you only need to change the value of the
<code_set_ name> property.
2. Create a CNV file for the UCM file by running the following command:
<product_dir>\bin\makeconv.exe -v <file_name>.ucm
Example:
"c:\Program Files\Attunity\Replicate\bin\makeconv.exe" -v 1047_EX.ucm
This will create a CNV file with the same name as the UCM file (e.g. 1047_EX.cnv).
5. Select the custom character set from the Character Set drop-down list; it will appear as
the CNV file name followed by the word "Custom" e.g. 1047_EX.cnv (Custom).
You can use a column's before-image data in a transformation. This is useful if you need
to store the before-image data on the target.
To do this, simply specify the source column name in Output table's Expression
column, in the following format:
$BI__MyColumn
Where $BI__ is a mandatory prefix (that instructs Replicate to capture the before-
image data) and MyColumn is the source column name.
Although you can store the before-image data in an existing target column, it is
recommended to create a new target column (using the Add Column button) in which
to store the before-image data.
1. In the Transform tab, select the row with the column for which you want to create
an expression.
or
Table 12.2 |
SQLite Operators used by Attunity Replicate
Operator Description
|| Concatenate strings.
FIRST_NAME||LAST_NAME
Operator Description
For more information about SQLite syntax, see the SQLite documentation.
Using Filters
Filters let you include or exclude records from a replication task based on the value(s)
of the source table columns, thereby allowing you to replicate only the specific data
that you need.
In this section:
Filter Limitations
Opening the Filter Tab
Creating a Filter Condition for a Specified Column
Creating a Record Selection Condition for One or More Columns
Adding or Removing Filter Ranges
Using SQLite Syntax with Filtering
Filter Limitations
When creating a filter, the following limitations apply:
Filters are not supported for calculating columns of Right-to-Left languages.
Filters can only be applied to immutable columns.
Data Columns list: This list contains a list of the columns for the table where
you filtering data. You can use these to select the columns to use in the filtering
operations.
This list has the following tabs:
Source: This tab lists the original source columns in the table.
Header: This tab lists the available header columns. You can create filters
using these columns and include them in expressions. For information on these
header columns, see Headers.
Calculated: This tab lists the columns added to the table. You add columns
through transformations. For more information, see Defining Transformations
for a Single Table/View.
Filter Conditions table: This table has the following columns:
Name: The name of the column where you are filtering the data.
Type: The data type for the column.
Include/Exclude: Indicate whether to include or exclude the filtered data
for this column.
Ranges: Click the button on the right of the Ranges field to open the Range
Builder. For information on creating a value or ranges with the Range Builder,
see Adding or Removing Filter Ranges.
1. Select the table you want to filter and then open the Table Settings window.
2. Click the Filter tab on the left side of the window.
1. Select a column from the data columns list and then click the right-facing arrow
next to the Filter Conditions table.
To remove the column, click on it in the Filter Conditions table and then click the
left-facing arrow. Any data entered for this column in the Include/Exclude or
Values columns is also deleted.
2. Click in the Include/Exclude column to select whether to include or exclude the
data that meets this condition.
3. Click the Edit Ranges button in the Ranges column.
4. The <Name> <Include|Exclude> Ranges window opens. Continue from Adding
or Removing Filter Ranges.
When entering a string, you can use the following special characters:
%: Matches any string of zero or more characters. For example, Mc% searches for
every name that begins with Mc or %bob% includes every name that contains bob.
_:Matches a single character (as a wildcard). For example: ’Sm_th’ includes
names that begin with Sm and end with th, such as Smith or Smyth. To search
for an underscore character, use [_]".
[..]: Includes a range or set of characters. For example, [CK]ars[eo] includes
names Carsen, Karsen, Carson, and Karson or [M-Z]inger includes all words that
end in inger with the first letter between M and Z, such as Ringer, Singer, or Zinger.
For more information, see documentation on how to use Transact-SQL.
For information on what SQLite operators can be used to create Record Selection
Condition filters, see Using SQLite Syntax with Filtering.
1. From the Data Columns list, select a source column, header column or calculated
column and then click the arrow to the left of the Record Selection Condition pane.
2. Use SQLite operators, such as < or = to create the condition. Use any amount of
strings or columns as you need to create a condition.
For example $EMPLOYEE_ID < 100 AND $SALARY > 100,000
In this case only rows that satisfy both of these conditions are replicated in
the replication task.
Click Open Expression Builder. This button is located directly under the record
selection condition box. Follow the directions for creating an expression in the
section Using the Expression Builder (for Filters, Transformations, and Global
Transformations).
You can define an expression that instructs Replicate only to apply UPDATEs when a user-
defined condition has been met, for example, only when specific columns have changed.
This is useful in situations when there are many updates in the source that the user has
deemed not relevant for the target, as "irrelevant" updates will be ignored.
Limitations
Does not support columns that do not have Before-Image data (e.g. LOB columns)
Does not support the following sources (i.e. sources that do not support Before-
Image records):
ODBC with CDC
Teradata
IBM Netezza
Subject to the existing expression builder and filter limitations
Content-based filtering may result in loss of data or data corruption. For example, if the
Primary Key value changes (an UPDATE operation), the expression may ignore the
UPDATE if the columns that were specified in the expression did not change. The result
in this case would be that a "phantom" row with the old row contents will remain, even if
a change was later applied to the columns specified in the expression.
How To
Assume that you have a source table named table1 with columns c1-c10 but you are
only interested in updating columns c7, c8 and c9 on the target.
1. Open the Table Settings for table1 and select the Filter tab.
Note Although selecting the Headers tab is optional, selecting it will enable
you to add $AR_H_OPERATION to your expression (as required in Step 4 below)
simply by double-clicking the column on the left of the tab.
4. Enter the following expression in the Build Expression pane and then click OK:
($AR_H_OPERATION != 'UPDATE') OR
The above expression means that changes will be applied to c7, c8 and c9 only if
one of the following is true:
The operation is not an UPDATE.
The value of c7, c8 or c9 has changed as the result of an UPDATE operation.
Note Filter ranges that you enter manually are also displayed in the Filter Builder.
You can use the Filter Builder to delete them.
1. In the Filter tab of the Table Settings window, select a column to filter. For
more information, see Using Filters.
2. Click the button to the right of the Ranges
3. Click Add Range. Select any of the following from the drop-down list displayed.
Equal to: Select Equal to to enter a single value. The following is displayed
in the range list.
Equal to = [N]
Click the [N] and type a value in the field that is displayed.
When the value in the selected column equals the value you enter, the result
is included or excluded in the replication task depending on the option
selected in the Include/Exclude column.
Between: Click Between to enter a range of values. The following is
displayed in the range list.
Between [N] - [N]
Click each [N] and type a value in the fields that are displayed.
When the column contains the values between the two values entered, the
result is included or excluded in the replication task depending on the option
selected in the Include/Exclude column.
Less than or equal to: Select Less than or equal to and enter a
maximum value. The following is displayed in the range list.
Less than or Equal to =< [N]
Click the [N] and type a value in the field that is displayed.
When the value in the selected column is equal to or less than the value you
enter, the result is included or excluded in the replication task depending on
the option selected in the Include/Exclude column.
Greater than or equal to: Select Greater than or equal to and enter
a minimum value. The following is displayed in the range list.
Greater than or Equal to => [N]
Click the [N] and type a value in the field that is displayed.
1. In the Filter tab of the Table Settings window, select the column with the
filter condition you want to delete.
2. Click the button to the right of the Ranges column. The Ranges Builder opens.
3. Click the X next to the range you want to delete. The deleted range is removed
from the list.
Note You must put the ($) in front of each input as shown below.
Table 12.3 |
SQLite Operators used by Attunity Replicate for Filtering
Operator Description
< Is less than.
$SALARY<100000
= Is equal to
$SALARY=100000
Operator Description
IS Is the same as
$HIRE_DATE IS 2014-09-29
IS functions the same as = unless one or both of the operands are NULL.
In this case, if both operands are NULL, then the IS operator evaluates to
1 (true). If one operand is NULL and the other is not, then the IS
operator evaluates to 0 (false).
IS NOT Is not the same as
$HIRE_DATE IS NOT 2014-09-29
IS NOT functions the same as != unless one or both of the operands are
NULL. In this case, if both operands are NULL, the IS NOT operator
evaluates to 0 (false). If one operand is NULL and the other is not, then the
IS NOT operator evaluates to 1 (true).
AND Both operands are true.
$MANAGER_ID AND EMPLOYEE ID >100
For more information on how to use the SQLite syntax, see the SQLite documentation.
Parallel Load
In Full Load replication mode, you can accelerate the replication of large tables by
splitting the table into segments and loading the segments in parallel. Tables can be
segmented by data ranges, by partitions, or by sub-partitions.
Supported Endpoints
The task must be defined with a combination of the following source and target endpoints:
1. In the Parallel Load tab's Select Parallel Load Method section, select Use
Data Ranges.
2. In the Select Details section, click Select Segment
3. For all endpoints, the Unique Index column is automatically selected. Select which
additional columns whose data you wish to use to delineate the ranges and then
click OK.
b. Enter the upper data range for the segment in the selected columns.
Note When Use Data Ranges is selected, all of the table data will be replicated,
even if data ranges are not defined for all of the columns.
Usage Example
Let's assume that the following segments are defined in the Define
Segment Boundaries table:
10 30 105
20 20 120
100 12 99
Segment 1: ((COL1 < 10) OR ((COL1 = 10) AND (COL2 < 30)) OR ((COL1
= 10) AND (COL2 = 30) AND (COL3 < 105)))
Segment 2: NOT ((COL1 < 10) OR ((COL1 = 10) AND (COL2 < 30)) OR ((COL1
= 10) AND (COL2 = 30) AND (COL3 < 105))) AND ((COL1 < 20) OR ((COL1 =
20) AND (COL2 < 20)) OR ((COL1 = 20) AND (COL2 = 20) AND (COL3 < 120)))
Segment 3: NOT ((COL1 < 20) OR ((COL1 = 20) AND (COL2 < 20)) OR ((COL1
= 30) AND (COL2 = 20) AND (COL3 < 120))) AND ((COL1 < 100) OR ((COL1 =
100) AND (COL2 < 12)) OR ((COL1 = 100) AND (COL2 = 12) AND (COL3 <
99)))
Segment 4: NOT ((COL1 < 100) OR ((COL1 = 100) AND (COL2 < 12))
OR ((COL1 = 100) AND (COL2 = 12) AND (COL3 < 99)))
Note Only select this method if you are sure that the table is already partitioned.
1. In the Parallel Load tab's Select Parallel Load Method section, select
Use Partitions.
2. In the Select Partitions section, select Use all table partitions. This will
segment the table according to partitions that already exist in the source database.
3. Select one the following:
Note This option will be disabled if the source database does not
support sub-partitions.
4. Click OK.
Note Only select this method if you are sure that the table is already partitioned.
1. In the Parallel Load tab's Select Parallel Load Method section, select
Use Partitions.
Note When Specify partitions is selected, only the specified partitions will
be replicated.
5. If you specified the name of a sub-partition, select the check box in the Sub-
Partition column.
Note The check box will be disabled if the source database does not support
sub-partitions.
Note
For example: If you select a table with 6 partitions and load the table using the Use
Partitions method, 5 partitions will be loaded in parallel, corresponding with the
default number of concurrent tasks (5). When one of the sub-tasks completes its
run, it will be assigned to loading the sixth partition.
The currently set value is displayed at the bottom of the Parallel Load tab. You can
modify this value in the Maximum number of tables to load in parallel field in
the Full Load Tuning tab.
Note This option is only available for tasks defined with any combination of the
following source and target endpoints: Oracle source, Oracle target, PostgreSQL
source, PostgreSQL target, Microsoft SQL Server source, Microsoft SQL Server target,
MySQL source, and MySQL target.
Note During CDC or during Full Load when the Allow unlimited LOB size option
is enabled, LOB data types are supported only in tables with a primary key.
Option Description
Replicate LOB columns When this option is selected (the default), LOB columns
will be replicated.
Allow unlimited LOB size Select this option - also known as Full LOB mode - to
ensure that all LOBs are replicated without being
truncated. This option should be selected when all (or
nearly all) of the LOBs you wish to replicate are large (i.e.
exceed 1 GB).
Chunk size (KB) Optionally, change the size of the LOB chunks to use when
replicating the data to the target. The default chunk size
should suffice in most cases, but if you encounter
performance issues, adjusting the size may improve
performance.
Notes
Important In some scenarios, tasks configured to replicate tables with multiple LOB
columns may consume a large amount of memory. This is because Replicate allocates
memory by multiplying the Limit LOB size to value by the Commit rate during full
load value, the sum of which, it multiplies by the number of LOB columns being
replicated. So, for example, if LOB size is limited to 5 MB and the default commit rate
is used (10000 events), a task replicating 6 LOB columns will consume 30 GB of
memory. Note that other factors such as the database type and version may also
affect memory consumption.
Should you encounter memory consumption issues and suspect that a combination
of the above factors may be the cause, stop the task and lower the value in the
Commit rate during full load field. Then resume the task. Repeat this process until
acceptable performance/memory levels are reached.
These instructions apply to Change Processing and Full Load tasks.
Note Changes to a column’s LOB size while a task is running will not be reflected in the
Message Format
This tab is only available for tasks defined with a supported streaming endpoint.
When a task is defined with such an endpoint, you can specify a custom message
format that will override the default Replicate message format. This may be useful if
the consumer application needs to process the message in a particular format.
The custom message format can be defined at task level and/or at table level. When it
is defined at both task and table level, the message format defined for the table will
take precedence over the message format defined for the task.
1. Select a table.
2. Open the Table Settings window as described in Table Settings.
3. Select the Message Format tab and click the Change to Table Policy button.
4. Configure the message format as described in Message Format.
5. To use the message format defined for the task, click the Change to Task
Policy button.
For information on defining a custom message at task level, see Message Format.
Note Global transformations are not available in a Log Stream Staging setup.
For information on Log Stream Staging, see Using the Log Stream.
1. Open the task for which you want to create a global transformation.
You can click Open above the Tasks list or double-click the task.
2. If you are not in the Designer mode, click Designer at the top right of the screen. For
more information on the Task View and the Designer mode, see Designer Mode.
4. From the top of the Global Transformation Rules window, click New
Global Transformation.
The New Transformation Rules wizard opens.
5. Enter the information to define a global transformation rule. The first step is
Selecting the Transformation Type.
Note You can only create one rule for each transformation type on the same object
(e.g. a column). If you create multiple rules for a single transformation type on the
same object, only the last rule you create will be valid. For example, if you create
the following rules (in order) to rename a schema:
Rename Schema: Add Prefix
-OR-
Table or Schema:
Rename schema: Select this to change the schema name for multiple tables.
For example, if you want all HR tables to be renamed PERS.
Rename table: Select this to change the name of multiple tables. For
example, if you want all tables named SALARY to be called WAGES.
Tablespace:
Change table tablespace: Select this to change the table tablespace on the
target. You can change the table tablespace regardless of what objects it contains
or you can specify a condition for it to be renamed. For example, change all table
tablespaces that contain the table Employees in the schema Company.
By default (i.e. when this option is not selected), the tables will either be
created in the source table tablespace on the target (when replicating from an
Oracle source) or in the default database tablespace (when replicating from
any other source).
Note This option is only available for tasks with an Oracle target endpoint.
Change index tablespace: Select this to change the index tablespace on the
target. You can change the index tablespace regardless of what objects it contains
or you can specify a condition for it to be renamed. For example, change all table
tablespaces that contain the table Employees in the schema Company.
By default (i.e. when this option is not selected), the indexes will either be
created in the source table tablespace on the target (when replicating from an
Oracle source) or in the default database tablespace (when replicating from
any other source).
Note This option is only available for tasks with an Oracle target endpoint.
Column:
Drop column: Select this to drop a column with a similar name from multiple
tables.
Convert data type: Select this if you want to change a specific data type to
a different one across multiple tables. For example, if you want to change all
Integer data types to a string.
Change Table:
Change Table transformations are only available when the Store Changes
replication option is enabled.
For more information on Change Tables, see Using Change Tables.
Rename Change Table: Select this to rename the Replicate Change Table
for all tables or for any table that matches the specified schema name and/or
table name.
Rename Change Table schema: Select this to change the schema under
which the Replicate Change Table will be created, for all tables or for any table
that matches the specified schema name and/or table name.
3. Click Next to proceed to the Under what Conditions to Transform step.
Note The options displayed in this screen depend on the Transformation Type selected.
Table Always Leave the % sign to include all table names in your
name is global transformation rule.
like %
Click the % sign to add a filter. In this case you can enter
any name combination to include only tables with that
specific name in your global transformation rule.
Column Rename Column Leave the % sign to include all column names in your
name is global transformation rule.
Drop
Column like % Click the % sign to add a filter. In this case you can enter
Convert Data
any name combination to include only columns with that
Type
specific name in your global transformation rule.
You can use the % sign as a wildcard. For example, N%
includes all columns with a name that begins with the
letter N, such as NAME, NAME_FIRST, or NAME_LAST.
The % wildcard can be used in any position. For example, if you
use it at the beginning, %IES, then all column names that end
in with the string "IES" are included in the transformation rule.
The % can also be used in a middle position.
After you complete defining the transformation rule definitions, click Next to go to
the Defining the Transformation Rule step.
Note If the global transformation type you are defining is Drop Column, you do
not need to create a Transformation Rule. In this case, click Finish to add the rule
to the Global Transformation Rules list.
Note The options displayed in this screen depend on the Transformation Type selected.
Rename Schema
If your transformation type is Rename Schema, you can do the following:
Rename schema to (string)
Add a Prefix or Suffix
Remove a Prefix or Suffix
Replace a Prefix or Suffix with Different Characters
Convert Schema Name to Uppercase
Convert Schema Name to Lowercase
Rename Schema (Expression)
Use the Rename schema to: [string] option to change the name of all table schemas that
you defined in the Under what Conditions to Transform step to a different name. For
example, if you have a schema called Human_Resources and want to change all instances of
this name to HR then enter the string HR. You can enter any string in this field.
Use the Add a prefix or suffix option to add additional characters to the beginning or
end of the schema name for all schemas that fit the definition you created in the Under
what Conditions to Transform step. For example, if the schema name is HR, you can add
a suffix, such as TAR or _TAR to the schema name for all tables with that schema name.
In this case, the resulting schema name will be HRTAR or HR_TAR.
Note If you are using Oracle as your target endpoint, Attunity Replicate does not
create a new schema. Therefore, the schema name that is the result of replacing a
prefix or suffix with a different string of characters must exist in the Oracle target
endpoint. If the resulting schema name does not exist, you must create the schema
in the Oracle endpoint before carrying out this task.
For more information, see Limitations for using Oracle as a Source or Target.
Use the Remove a prefix or suffix option to remove a string of characters from the
beginning or end of a schema name for all schema that fit the definition you created in
the Under what Conditions to Transform step.
For example, you can use this option to remove the letters _REV from the schema name for
all tables in the schema HR_REV. In this case the schema name in the target will be HR.
Use the Replace a prefix or suffix option to replace a string of characters with a
different string of characters. You determine whether to replace the characters at the
beginning or end of a schema name for all schema that fit the definition you created in
the Under what Conditions to Transform step.
For example, you can use this option to replace the letters _ORIG with _REPL in the
schema name for all tables in the schema HR_ORIG. In this case the schema name in the
target will be HR_REPL.
Note If you are using Oracle as your target endpoint, Attunity Replicate does not
create a new schema. Therefore, the schema name that is the result of replacing a
prefix or suffix with a different string of characters must exist in the Oracle target
endpoint. If the resulting schema name does not exist, you must create the schema
in the Oracle endpoint before carrying out this task.
For more information, see Limitations for using Oracle as a Source or Target.
Use the convert to uppercase option to convert all of the letters in a schema name to
upper case. For example:
Schema_cat, becomes SCHEMA_CAT
Use the convert to lowercase option to convert all of the letters in a schema name to
lower case. For example:
Schema_cat, becomes schema_cat
Use the Rename schema to [expression] option to change the name of all table
schemas that you defined in the Under what Conditions to Transform step to a different
name. For example, if you have a schema called Human_Resources and want to change
all instances of this name to HR.
Note If you are using Oracle as your target endpoint, Attunity Replicate does not
create a new schema. Therefore, the schema name that is the result of replacing a
prefix or suffix with a different string of characters must exist in the Oracle target
endpoint. If the resulting schema name does not exist, you must create the schema
in the Oracle endpoint before carrying out this task.
For more information, see Limitations for using Oracle as a Source or Target.
2. Click the button to the right of the Rename schema option to open the Expression
Editor. For information on how to use the Expression Editor, see Using the
Expression Builder (for Filters, Transformations, and Global Transformations). Then
go to step 4. or
Click [expression] to activate the field and continue with step 3.
3. Type an SQLite expression or a string (in quotes) to rename the schema. For example:
"New_Schema"
’PREF_’||$SCHEMA_NAME_VAR||’_SUFF’
$SCHEMA_NAME_VAR
$TABLE_NAME_VAR
$COLUMN_NAME_VAR
$COLUMN_DATATYPE_VAR
4. Click Finish to add the rule to the Global Transformation Rules list.
Rename Table
If your transformation type is Rename Table, you can do the following:
Rename table to (string)
Add a Prefix or Suffix
Remove a Prefix or Suffix
Replace a Prefix or Suffix with Different Characters
Convert table name to uppercase
Convert table name to lowercase
Rename table (expression)
Use the Rename table to: [string] option to change the name of all tables that you
defined in the Under what Conditions to Transform step to a different name. For
example, if you have a table called EMPLOYEE and want to change all instances of this
name to EMP then enter the string EMP. You can enter any string in this field.
Use the Add a prefix or suffix option to add additional characters to the beginning or
end of the table name for all tables that fit the definition you created in the Under what
Conditions to Transform step. For example, if the table name is EMPLOYEES, you can add
a suffix, such as TAR or _TAR to the table name for all tables with that table name. In this
case, the resulting table name will be EMPLOYEESTAR or EMPLOYEES_TAR.
Use the Remove a prefix or suffix option to remove a string of characters from
the beginning or end of a table name for all tables that fit the definition you created
in the Under what Conditions to Transform step.
For example, you can use this option to remove the letters _REV from the table name for
all tables with the name EMPLOYEES. In this case the table name in the target will be
EMPLOYEES.
Use the Replace a prefix or suffix option to replace a string of characters with a
different string of characters. You determine whether to replace the characters at
the beginning or end of a table name for all tables that fit the definition you created
in the Under what Conditions to Transform step.
For example, you can use this option to replace the letters _ORIG with _REPL in the table
names for all tables called EMPLOYEE_ORIG. In this case the table name in the target will be
EMPLOYEE_REPL.
Use the convert to uppercase option to convert a table name to all upper case.
For example:
Table_cat, becomes TABLE_CAT
Use the convert to lowercase option to convert a table name to all lower case.
For example:
Table_cat, becomes table_cat
Use the Rename table to [expression] option to change the name of all tables that
fit the definition you created in the Under what Conditions to Transform step. For
example, if you have a table called EMPLOYEE and want to change all instances of this
name as defined in the previous step it to EMP.
2. Click the button to the right of the Rename table option to open the Expression
Editor. For information on how to use the Expression Editor, see Using the
Expression Builder (for Filters, Transformations, and Global Transformations). Then
go to step 4. or
Click [expression] to activate the field and continue with step 3.
3. Type an SQLite expression or a string (in quotes) to rename the table. For example:
"New_Table"
’PREF_’||$TABLE_NAME_VAR||’_SUFF’
3. You can use the following variables in the SQLite expression:
$SCHEMA_NAME_VAR
$TABLE_NAME_VAR
$COLUMN_NAME_VAR
$COLUMN_DATATYPE_VAR
Use the Rename column to: [string] option to change the name of all columns that
you defined in the Under what Conditions to Transform step to a different name. For
example, if you have a table called SALARY and want to change all instances of this
name to EMP then enter the string SAL. You can enter any string in this field.
Use the Add a prefix or suffix option to add additional characters to the beginning or
end of the column name for all columns that fit the definition you created in the Under
what Conditions to Transform step. For example, if the column name is SALARY, you can
add a suffix, such as TAR or _TAR to the table name for all tables with that table name. In
this case, the resulting table name will be SALARYTAR or SALARY_TAR.
Use the Remove a prefix or suffix option to remove a string of characters from the
beginning or end of a column name for all columns that fit the definition you created in
the Under what Conditions to Transform step.
For example, you can use this option to remove the letters _REV from the column name
for all columns with the name SALARY. In this case the column name in the target will be
SALARY.
Use the Replace a prefix or suffix option to replace a string of characters with a
different string of characters. You determine whether to replace the characters at the
beginning or end of a column name for all columns that fit the definition you created in
the Under what Conditions to Transform step.
For example, you can use this option to replace the letters _ORIG with _REPL in the
column names for all columns called SALARY_ORIG. In this case the column name in the
target will be SALARY_REPL.
Use the convert to uppercase option to convert a column name to all upper case.
For example:
Column_cat, becomes COLUMN_CAT
Use the convert to lowercase option to convert a column name to all lower case.
For example:
Column_cat, becomes column_cat
Use the Rename column to [expression] option to change the name of all tables that
fit the definition you created in the Under what Conditions to Transform step. For
example, if you have a column called SALARY and want to change it to SAL.
2. Click the button to the right of the Rename column option to open the Expression
Editor. For information on how to use the Expression Editor, see Using the
Expression Builder (for Filters, Transformations, and Global Transformations). Then
go to step 4. or
Click [expression] to activate the field and continue with step 3.
3. Type an SQLite expression or a string (in quotes) to rename the column. For example:
"New_Column"
’PREF_’||$COLUMN_NAME_VAR||’_SUFF’
Add Column
When you add a column to multiple tables, you must provide a name, define the data
type for the column and define the data that the column contains. The column that you
define here is added to all tables that fit the definition you created in step Under what
Conditions to Transform.
The following describes the information you must enter in the transformation rule page
for adding a column.
Column name: Click the [string] to activate the field. Type the name for the
column in the field. A column with this name is added to all tables that fit the
definition you created in step Under what Conditions to Transform.
Column data type: Click the drop-down for a list of data types and select a new
data type from the drop-down list. Make sure that the data type you select is
compatible with the data in that column.
For a description of available data types, see Replicate Data Types. For information
about data type mapping from the native endpoint to Attunity Replicate data types,
see the chapter for the endpoint you use. For a list of supported databases, see
Supported Platforms and Endpoints .
Computation expression: Click the button to the right of this field to open the
Drop Column
This option does not require a transformation rule. For this option you complete the
Global transformation Rule after the Under what Conditions to Transform step.
For information about data type mapping from the native endpoint to Attunity
Replicate data types, see the chapter for the endpoint you are using. For a list of
supported databases, see Supported Platforms and Endpoints .
Globally renaming a Change Table will override the Change Table suffix defined
in the task settings.
The Change Table name must be different from the source table names.
Otherwise, a table error will occur.
Use the Rename Change Table to: [string] option to change the name of all
Change Tables that you defined in the Under what Conditions to Transform step to a
different name. For example, if you have a Change Table called EMPLOYEE and want
to change all instances of this name to EMP then enter the string EMP. You can enter
any string in this field.
Use the Add a prefix or suffix option to add additional characters to the beginning or
end of the Change Table name for all Change Tables that fit the definition you created in
the Under what Conditions to Transform step. For example, if the Change Table name is
EMPLOYEES, you can add a suffix, such as TAR or _TAR to the Change Table name for all
Change Tables with that name. In this case, the resulting Change Table name will be
EMPLOYEESTAR or EMPLOYEES_TAR.
Use the Remove a prefix or suffix option to remove a string of characters from the
beginning or end of a Change Table name for all Change Tables that fit the definition
you created in the Under what Conditions to Transform step.
Use the Replace a prefix or suffix option to replace a string of characters with a
different string of characters. You determine whether to replace the characters at the
beginning or end of a Change Table name for all Change Tables that fit the definition
you created in the Under what Conditions to Transform step.
For example, you can use this option to replace the letters _ORIG with _REPL in the
Change Table names for all Change Tables called EMPLOYEE_ORIG. In this case the
Change Table name in the target will be EMPLOYEE_REPL.
Use the convert to uppercase option to convert a Change Table name to all upper case.
For example:
Table_cat, becomes TABLE_CAT
Use the convert to lowercase option to convert a Change Table name to all lower case.
For example:
Table_cat, becomes Change Table_cat
Use the Rename Change Table to [expression] option to change the name of all
Change Tables that fit the definition you created in the Under what Conditions to
Transform step. For example, if you have a Change Table called EMPLOYEE and want to
change all instances of this name as defined in the previous step it to EMP.
2. Click the button to the right of the Rename Change Table option to open the
Expression Editor. For information on how to use the Expression Editor, see Using the
Expression Builder (for Filters, Transformations, and Global Transformations). Then
or
3. Type an SQLite expression or a string (in quotes) to rename the table. For example:
"New_Change_Table_Name"
’PREF_’||$AR_M_SOURCE_TABLE_NAME||’_SUFF’
$AR_M_SOURCE_COLUMN_DATATYPE
$AR_M_SOURCE_COLUMN_NAME
$AR_M_SOURCE_SCHEMA
$AR_M_SOURCE_TABLE_NAME
Use the Rename Change Table schema to: [string] option to change the name of
all Change Table schemas that you defined in the Under what Conditions to Transform
step to a different name. For example, if you have a Change Table schema called
EMPLOYEE and want to change all instances of this name to EMP then enter the string
EMP. You can enter any string in this field.
Use the Add a prefix or suffix option to add additional characters to the beginning or end of
the Change Table schema name for all tables that fit the definition you created in the
Use the Remove a prefix or suffix option to remove a string of characters from the
beginning or end of a Change Table schema name for all tables that fit the definition
you created in the Under what Conditions to Transform step.
For example, you can use this option to remove the letters _REV from the Change
Table schema name for all Change Table schemas with the name EMPLOYEES. In this
case the Change Table schema name in the target will be EMPLOYEES.
Use the Replace a prefix or suffix option to replace a string of characters with a
different string of characters. You determine whether to replace the characters at the
beginning or end of a Change Table schema name for all tables that fit the definition
you created in the Under what Conditions to Transform step.
For example, you can use this option to replace the letters _ORIG with _REPL in the
Change Table schema names for all Change Table schemas called EMPLOYEE_ORIG. In
this case the Change Table schema name in the target will be EMPLOYEE_REPL.
Use the convert to uppercase option to convert a Change Table schema name to all
upper case. For example:
Table_cat, becomes TABLE_CAT
Use the convert to lowercase option to convert a Change Table schema name to all
lower case. For example:
Table_cat, becomes table_cat
Use the Rename Change Table schema to [expression] option to change the name of all
tables that fit the definition you created in the Under what Conditions to Transform step. For
example, if you have a Change Table schema called EMPLOYEE and want to change all
instances of the Change Table schema name as defined in the previous step to EMP.
2. Click the button to the right of the Rename table option to open the Expression
Editor. For information on how to use the Expression Editor, see Using the
Expression Builder (for Filters, Transformations, and Global Transformations). Then
go to step 4. or
Click [expression] to activate the field and continue with step 3.
3. Type an SQLite expression or a string (in quotes) to rename the table. For example:
"New_Change_Table_Schema"
’PREF_’||$AR_M_SOURCE_SCHEMA||’_SUFF’
$AR_M_SOURCE_COLUMN_DATATYPE
$AR_M_SOURCE_COLUMN_NAME
$AR_M_SOURCE_SCHEMA
$AR_M_SOURCE_TABLE_NAME
1. In the Global Transformation Rules dialog box, select the transformation rule
you want to edit.
2. Click Open (at the top of the list).
3. Make any changes you need in the wizard. For information on how to work with
each of the pages in the New transformation Rule wizard, see Defining Global
Transformations.
1. In the Global Transformation Rules dialog box, select the transformation rule
you want to edit.
2. Click Delete (above the list).
The transformation rule is removed from the list and deleted from the system.
The following sections describe the tasks you can perform in each part of the
Expression Builder:
Elements Pane (on the left): This pane contains elements that you can add to
an expression. Select elements and move them into the Expression Builder box to
create the expression. For more information, see Build an Expression.
Parse Expression Panel: This panel displays the parameters for the expression.
After you build the expression, click Parse Expression to list the expression
parameters. You can then enter a value or argument for each of the parameters.
For more information, see Parse an Expression.
The top part of the Expression panel contains the Operator toolbar. This toolbar
contains the most common operators. Click the operator you want to use to add it to
the expression. You can also add operators from the Element Pane, Operators tab.
Test Expression Panel: This panel displays the results of a test that you can
run after you provide values to each of the parameters in your expression. For
more information, see Test an Expression.
Build an Expression
The first step in using the expression builder is to build an expression. The expression
that you build is displayed in the top section of the right pane. You can open the
Expression when:
You define Defining Transformations for a Single Table/View for a single table.
You define Filters for a single table.
You use the Global transformations dialog box to Rename Schema, Rename
Table, Rename Column, or Add Column.
Note: To add operators to your expression, you can use the Operator tab in the
Element pane or the Operator buttons at the top of the Build Expression panel or any
combination of these. See Operators and Operator toolbar.
For example, to create an expression that will combine the first name and last name,
do the following:
1. In the Input Columns tab add the FIRST_NAME column to the Build
Expression box.
2. Click the concatenate (||) operator from the Operator bar at the top of the
Build Expression box.
3. In the Input Columns tab add the LAST_NAME column into the Build Expression box.
1. In the Elements Pane, select any element you want to include in your expression.
For information on the elements you can use in an expression, see Functions.
2. Add an element to the Build Expression panel by selecting it and then clicking
the arrow to the right of the element.
3. Continue to add elements as needed.
Operator toolbar
The Operator toolbar is above the Build Expression box. It contains the most
common operators so you can easily add them to an expression.
The following operators are available in the Operator toolbar:
1. Click the place in the Build Expression box where you want to add the operator.
2. Click the operator you want to add. It is added to the expression.
Parse an Expression
You can parse an expression to determine its parameters and to determine whether
the expression is valid.
To parse an expression:
If the expression is not valid, an error message is written in red at the bottom of
the Expression Builder window.
If the expression is valid, the expression parameters are displayed in the Parameter
column in the Parse Expression section. See the figure under Test an Expression.
3. Type a valid value for each of the parameters in the Value column to Test an
For example, type John for the FIRST_NAME and Smith for the LAST_NAME in
the Value column. Once you type in values, you can Test an Expression.
Test an Expression
You can use the Attunity Replicate Test procedure to display the results of a test
expression. The following figure is an example of a built expression that is evaluated
and contains a test result.
For example, type John for FIRST_NAME and Smith for LAST_NAME. The result
displayed is JohnSmith. If you want a space between the words add it to the end of
the FIRST_NAME value or the beginning of the LAST_NAME value.
Note: Testing calls to the source_lookup and target_lookup functions is
not supported.
Headers
By default, headers for source tables are not replicated to the target. You can
determine which, if any, headers to replicate when you define a transformation by
creating an expression that includes the header.
You can create a filter using header values. Header filters are applied during
change processing. See Using Filters for additional information.
Note The Headers tab in the Expression builder is available for Filters and
transformations. It is available for Global transformations only when you select
Add Columns. See Selecting the Transformation Type.
AR_V_ The time the source tables were reloaded. Current DATETIME
RELOAD_ timestamp (6)
TIME
Variables
Your expression can contain any of the variables (which will be replaced during
runtime) described in the table below.
AR_V_HOST_ The host name of the machine on which Attunity Replicate STRING
NAME Server is installed. (50)
AR_V_SOURCE_ The logical name of the source endpoint defined in the STRING
NAME endpoint settings. (50)
AR_V_TARGET_ The logical name of the target endpoint defined in the STRING
NAME endpoint settings. (50)
Operators
The sections below describe the SQLite operators you can use to build an expression with
the Expression builder. The Expression builder divides the operators into the following
Note With the exception of table-level transformations, all operator symbols must
be preceded by a space and followed by a space. For example, the expression for
concatenating a first and last name should be specified like this:
FIRST_NAME || LAST_NAME
Strings
Examples:
FIRST_NAME || LAST_NAME
Logical
The following table describes the logical SQLite operators used by the Attunity
Replicate Expression Builder.
Table 12.7 |
Logical SQLite Operators used by Attunity
Replicate Expression Builder
Operator Description
Operator Description
IS Is the same as
$HIRE_DATE IS 2014-09-29
IS functions the same as = unless one or both of the operands
are NULL. In this case, if both operands are NULL, then the IS
operator evaluates to 1 (true). If one operand is NULL and the
other is not, then the IS operator evaluates to 0 (false).
Operator Description
For example, the expression 'a' LIKE 'A' is TRUE but 'æ' LIKE
'Æ' is FALSE.)
Example 1:
Example 2:
case length($month)
GLOB The GLOB operator acts in the same way as the LIKE operator
but uses the UNIX file globbing syntax for its wildcards. GLOB is
case sensitive.
Operator Description
REGEXP The REGEXP operator is a special syntax for the regexp() user
function. No regexp() user function is defined by default and so
use of the REGEXP operator will normally result in an error
message.
left. x << n
right. x >> n
| Unary or
$SALARY>100000
Operator Description
$SALARY>=100000
= or == Is equal to
$SALARY=100000
Mathematical
The following table describes the mathematical SQLite operators used by the
Attunity Replicate Expression Builder.
Table 12.8 |
SQLite Mathematical Operators used by the Attunity
Replicate Expression Builder
Operator Description
Strings
The following table describes the string functions used by the Expression Builder in
Attunity Replicate .
Table 12.9 |
SQLite String Functions used by the Expression Builder
Function Description
lower(x) The lower(x) function returns a copy of string x with all characters converted
to lower case. The default built-in lower() function works for ASCII characters
only.
ltrim The ltrim(x,y) function returns a string formed by removing all characters that
(x,y) appear in y from the left side of x. If there is no value for y, ltrim(x) removes
spaces from the left side of x.
replace The replace(x,y,z) function returns a string formed by substituting string z for
(x,y,z) every occurrence of string y in string x.
rtrim The rtrim(x,y) function returns a string formed by removing all characters
(x,y) that appear in y from the right side of x. If there is no value for y, rtrim(x)
removes spaces from the right side of x.
Function Description
substr The substr(x,y,z) function returns a substring of input string x that begins with
(x,y,z) the y-th character and which is z characters long. If z is omitted then substr
(x,y) returns all characters through the end of the string x beginning with the
y-th. The left-most character of x is number 1. If y is negative then the first
character of the substring is found by counting from the right rather than the
left. If z is negative then the abs(z) characters preceding the y-th character
are returned. If x is a string then characters indices refer to actual UTF-8
characters. If x is a BLOB then the indices refer to bytes.
trim(x,y) The trim(x,y) function returns a string formed by removing all characters that
appear in y from both sides of x. If there is no value for y, trim(x) removes
spaces from both sides of x.
upper(x) The upper(x) function returns a copy of string x with all characters converted
to upper case.
LOBs
The following table describes the LOB functions used by the Expression Builder in
Attunity Replicate .
Table 12.10 |
SQLite Lob Functions used by the Expression Builder
Function Description
hex(x) The hex() function receives an argument as a BLOB and returns an upper-
case hexadecimal string version of the BLOB content.
randomblob The randomblob(N) function returns an N-byte BLOB that contains pseudo-
(N) random bytes. If N is less than 1 then a 1-byte random BLOB is returned.
zeroblob(N) The zeroblob(N) function returns a BLOB that consists of N bytes of 0x00.
Numeric
The following table describes the numeric functions used by the Expression Builder
in Attunity Replicate .
max The multi-argument max() function returns the argument with the maximum
(x,y...) value, or returns NULL if any argument is NULL. The multi-argument max()
function searches its arguments from left to right for an argument that defines
a collating function and uses that collating function for all string comparisons.
If none of the arguments to max() define a collating function, then the BINARY
collating function is used. Note that max() is a simple function when it has two
or more arguments but operates as an aggregate function if it has a single
argument.
min The multi-argument min() function returns the argument with the minimum
(x,y...) value. The multi-argument min() function searches its arguments from left to
right for an argument that defines a collating function and uses that collating
function for all string comparisons. If none of the arguments to min() define a
collating function, then the BINARY collating function is used. Note that min()
is a simple function when it has two or more arguments but operates as an
aggregate function if it has a single argument
NULL check
The following table describes the NULL check functions used by the Expression Builder
in Attunity Replicate .
Function Description
coalesce The coalesce() function returns a copy of its first non-NULL argument, it
(x,y...) returns NULL if all arguments are NULL. Coalesce() have at least two
arguments.
ifnull The ifnull() function returns a copy of its first non-NULL argument, it returns
(x,y) NULL if both arguments are NULL. Ifnull() must have exactly two arguments.
The ifnull() function is the same as coalesce() with two arguments.
nullif The nullif(x,y) function returns a copy of its first argument if the arguments
(x,y) are different and returns NULL if the arguments are the same. The nullif(x,y)
function searches its arguments from left to right for an argument that defines
a collating function and uses that collating function for all string comparisons.
If neither argument to nullif() defines a collating function then the BINARY is
used.
The following table describes the Date and Time functions used by the Expression Builder
in Attunity Replicate .
Table 12.13 |
SQLite Date and Time Functions used by the Attunity Replicate
Expression Builder
Function Description
datetime(timestring, Returns the date and time in the format YYYY-MM-DD HH:MM:SS.
modifier, modifier...)
julianday The julianday() function returns the number of days since noon in
(timestring, modifier, Greenwich on November 24, 4714 B.C.
modifier...)
Function Description
strftime(format, The strftime() routine returns the date formatted according to the
timestring, modifier, format string specified as the first argument. It supports the
modifier...) following variables:
%%: %
Data Enrichment
Data Enrichment functions allow the selected source tables to be augmented with data
from other records located in either the source or target endpoints. Practical
applications of data enrichment functions include code lookup or master record lookup
(e.g. social security number lookup to find a person’s name).
You can enrich the target tables with supplemental data retrieved from the source or target
endpoint by defining a transformation on the table. For more information about defining
transformations on a single table, see Defining Transformations for a Single Table/View.
Limitations
Amazon Redshift is not supported.
The table below describes the source and target lookup functions, which can be used
both for table transformations and for global transformations. For a description of the
parameters available for these functions, see Input Parameters.
Table 12.14 |
SQLite Data Enrichment Functions used by Expression Builder
Function Description
Input Parameters
The possible input parameters for the lookup functions are described in the table below.
For a usage example, see Functions.
Parameter Description
Syntax:
Full example:
source_lookup(
10000 ,
'HR' ,
'DEPARTMENTS' ,
'DEPARTMENT_NAME’ ,
Note To improve efficiency, the source/target lookup tables should be indexed for
the specified lookup fields.
In the following example, Mike needs to add the DEPARTMENT_NAME column to the
HR.JOB_ HISTORY table. The DEPARTMENT_NAME column is located in the
HR.DEPARTMENTS table in the source endpoint.
This is how the HR.JOB_HISTORY table appears before the column is added:
This is how the HR.JOB_HISTORY table appears after the Full Load completes:
1. Create a new task and select the HR.JOB_HISTORY table for replication.
2. Apply a “New Column” transformation to the HR.JOB_HISTORY table. For more
information on defining transformations, see Defining Transformations for a
Single Table/View.
3. Open the Expression Builder and choose Data Enrichment from the Functions
tab. For more information on the Expression Builder, see Using the Expression
Builder (for Filters, Transformations, and Global Transformations).
4. Select the source_lookup function and configure it as follows (using the native
syntax of the source endpoint):
If the lookup table is located in Oracle:
'DEPARTMENT_ID=:1',$DEPARTMENT_ID)
(10000,'HR','DEPARTMENTS','[DEPARTMENT_NAME]',
'[DEPARTMENT]=?',$DEPARTMENT_ID)
Where:
Operation
The following table describes the Operation functions used by the Expression Builder
in Attunity Replicate .
Function Description
Other Functions
The following table describes additional functions used by the Expression Builder in
Attunity Replicate .
like The like() function is used to implement the "Y LIKE X [ESCAPE Z]"
(x,y,z) expression. The ESCAPE (z) clause is optional. If there is a z clause, then the
like() function is invoked with three arguments. Otherwise, it is invoked with
two arguments.
typeof(x) The typeof(x) function returns a string that indicates the datatype of the
expression x: null, integer, real, text, or BLOB.
The Hash function generates a hash value for an inputted column (using the SHA-
256 algorithm) and then returns the hex value of the generated hash value.
To use the function in an expression, add the hash_sha256(x) function to the Build
Expression pane and then replace the "x" with the desired source column name (from
the Input Columns tab).
The function is especially useful for masking sensitive information. In the
expression below, for example, the Hash function has been used to obfuscate
employees' email addresses.
User-Defined Transformations
Customers that requires functionality not provided by Replicate's built-in
transformations can write their own transformations, and then access them from the
Replicate Expression Builder.
<installation_dir>\addons\samples\MyTransformation
All of the types and prototypes are defined in the ar_addon.h and
<installation_dir>\addons\samples\addon_name
Note When working in a High Availability setup, the created binaries should
be installed on all of the cluster instances.
<installation_dir>\addons\addon_name\addon_name.[dll/so]
5. Register the new function in the addon initialization function (mentioned in Step 1)
as in the following example:
USER_DEFINED_TRANSFORMATION_DEF *transdef =
GET_AR_AO_TRANSFORMATION_ DEF();
Task Settings
Task-specific replication settings can be configured in the Task Settings dialog box.
Note Some of the task settings are not available in a Log Stream Staging setup.
For information on the availability of task settings in a Log Stream Staging setup,
see Using the Log Stream.
3. In the Task Settings dialog box, select one of the following tabs according to
the setting(s) you want to configure:
Metadata
Target Metadata
Control Tables
Bidirectional
Full Load
Change Processing
Error Handling
Environmental Errors
Data Errors
Table Errors
Apply Conflicts
Logging
Character Substitution
File Uploads
Message Format
Metadata
When you click Metadata in the Task Settings dialog box, you can configure the
Target Metadata Settings for a replication task.
Target Metadata
Target table schema: (if empty, use the schema from the source table): This will
automatically add the owner prefix for the target endpoint to all tables if no source
schema is defined.
Note When replicating to a Hadoop target endpoint, the value specified in this field
will be interpreted as a database name (as opposed to a schema name).
For information on how to override these settings for individual tables, see Handling
LOB Columns.
Option Description
Replicate LOB columns When this option is selected (the default), LOB columns
will be replicated.
Allow unlimited LOB size Select this option - also known as Full LOB mode - to
ensure that all LOBs are replicated without being
truncated. This option should be selected when all (or
nearly all) of the LOBs you wish to replicate are large (i.e.
exceed 1 GB).
Chunk size (KB) Optionally, change the size of the LOB chunks to use when
replicating the data to the target. The default chunk size
should suffice in most cases, but if you encounter
performance issues, adjusting the size may improve
performance.
Notes
Important In some scenarios, tasks configured to replicate tables with multiple LOB
columns may consume a large amount of memory. This is because Replicate allocates
memory by multiplying the Limit LOB size to value by the Commit rate during full
load value, the sum of which, it multiplies by the number of LOB columns being
replicated. So, for example, if LOB size is limited to 5 MB and the default commit rate
is used (10000 events), a task replicating 6 LOB columns will consume 30 GB of
memory. Note that other factors such as the database type and version may also
affect memory consumption.
Should you encounter memory consumption issues and suspect that a combination
of the above factors may be the cause, stop the task and lower the value in the
Commit rate during full load field. Then resume the task. Repeat this process until
acceptable performance/memory levels are reached.
These instructions apply to Change Processing and Full Load tasks.
Note Changes to a column’s LOB size while a task is running will not be reflected in the
Control Tables
Control Tables provide information about the replication task as well as useful statistics
that can be used to plan and manage both the current replication task and future
replication tasks. Aside from the Apply Exceptions table which is always created, you
can choose which Control Tables to create on the target.
Create target control tables in schema: Enter the endpoint schema for the
target Control Tables. If you do not enter any information in this field, then the
tables will be created in the default location in the endpoint.
Note When this field is left empty, the target endpoint is MySQL, and the Multiple
Endpoints option is enabled, a default database named attrep_control will be created
on the MySQL server. The selected control tables will be created in this database.
For more information on the Multiple Endpoints option, see Setting General
Connection Properties.
Note When replicating to a Hadoop target endpoint, the value specified in this field
will be interpreted as a database name (as opposed to a schema name).
Create target control tables in tablespace: When the target endpoint is Oracle,
specify the tablespace where you want the target control tables to be created. If you do
not enter any information in this field, the tables will be created in the default tablespace
in the target database.
Create target control table indexes in tablespace: When the target endpoint is
Oracle, specify the tablespace where you want the control table indexes to be created.
If you do not enter any information in this field, the indexes will be created in the
same tablespace as the control tables.
Replication history time slot (minutes): The length of each time slot in the
Replication History table. The default is 5 minutes.
Note Currently, the DDL History table is only supported with the Hadoop
target endpoint.
Bidirectional
This tab is only applicable to bidirectional replication tasks. When you click Bidirectional in
the Task Settings dialog box, the Loopback Prevention tab is displayed. In bidirectional
replication, loopback prevention is a mechanism that prevents the same data from being
replicated back and forth in an endless loop. To enable loopback prevention, you need to
specify a source and target Loopback prevention table schema.
Bidirectional replication consists of two separate tasks: Task 1 captures changes made to
Endpoint A and replicates them to Endpoint B. Task 2 captures changes made to Endpoint B
and replicates them to Endpoint A. When configuring Task 1 of a bidirectional replication
Full Load
When you click Full Load in the Task Settings dialog box, you can configure the following:
Full Load Settings
Full Load Tuning
Full is ON/OFF.
Click this button to toggle full load on or off. The initial setting is determined when
Adding Tasks.
When full load is ON, Attunity Replicate loads the initial source data to the target endpoint.
Note Full load can be turned on or off at any stage even if change processing is
on. Once the task begins to process changes, the full load on/off switch is used
only as additional protection against accidental or unauthorized reload.
If target table already exists: Select one of the following from the list to
determine how you want to handle loading the target at full-load start up:
Note The option to drop or truncate the target tables is relevant only if such
operations are supported by the source endpoint.
Note Currently this option is only available for the Hadoop target endpoint.
Do nothing: Existing data and metadata of the target table will not be affected.
New data will be added to the table.
Note Replicate expects the source column data types to be compatible with the
corresponding target column data types. If you choose either TRUNCATE before
loading or Do nothing and one or more target data types are different than the
data types for the corresponding source columns, use a transformation to convert
the data types as required.
For information on creating data type transformations, see Defining
Transformations for a Single Table/View.
Create primary key or unique index after full load completes: Select this option if
you want to delay primary key or unique index creation on the target until after full load
completes.
After Full Load completes, stop the task: You can set the task to stop automatically
after Full Load completes. This is useful if you need to perform DBA operations on the
target tables before the task’s Apply Changes (i.e. CDC) phase begins.
During Full Load, any DML operations executed on the source tables are cached. When
Full Load completes, the cached changes are automatically applied to the target tables
(as long as the Before/After cached changes have been applied option(s) described
below are disabled).
Note
When configuring Replicate to stop the task after Full Load completes, note
the following:
The task will stop after Full Load completes even if there are no cached changes
to apply.
Choosing to stop the task before cached changes have been applied may
adversely affect performance, since the cached changes will only be applied to
tables (even those that have already completed Full Load) after the last table
completes Full Load.
When the Before/After cached changes have been applied option is selected
and a DDL is executed on one of the source tables during the Full Load process
(in a Full Load and Apply Changes task), Replicate will reload the table. This
effectively means that any DML operations executed on the source tables will be
replicated to the target before the task stops.
When working with the File Channel endpoint, these options should be set in
the remote File Channel task and not in the local File Channel task.
For more information on the File Channel endpoint, see Using the
Attunity Replicate File Channel.
Tuning settings
Maximum number of tables to load in parallel: Enter the maximum number
of tables to load into the target at one time. The default value is 5.
Transaction consistency timeout (seconds): Enter the number of seconds that
Attunity Replicate waits for transactions to close, if they are open when the task
starts, before beginning the Full Load operation. The default value is 600 (10
Commit rate during full load: The maximum number of events that can
be transferred together. The default value is 10000.
Change Processing
When you click Change Processing in the Task Settings dialog box, you can
configure the following:
Apply Changes Settings
Store Changes Settings
Change Processing Tuning
Click this button to toggle Apply Changes (Change Processing) on or off. The initial
setting is determined when Adding Tasks.
When Apply Changes is ON, Attunity Replicate processes the changes. You can view the
change processing in the Monitor. For more information, see Monitoring Change
Processing Operations.
Note When you turn on apply changes you must reload the task or position back to
the point of the previous reload.
DDL handling policy: Determine how to handle the target table for the change capture:
Notes
Executing a DDL on a source table during the Full Load process in a Full Load
and Apply Changes task will cause Replicate to reload the table.
The option to drop or truncate the target tables is relevant only if such
operations are supported by the source endpoint.
Click this button to toggle Store Changes on or off. The initial setting is determined
when Adding Tasks. If this option is ON, changes are stored in either Change Tables or
an audit table.
For more information about storing and applying changes, see Using an Audit Table
and Using the Change Table Model.
Note Store Changes can be turned on or off at any time without affecting anything in
the task. Changes that are processed and not stored as a result of change storage
being turned off can be recovered only by setting the task to an earlier point in time.
If Store Changes is ON, use the following options to determine how to store
changes. Changes can be stored in Change Tables or in a single Audit table. From
the Store changes in drop-down list, choose either Change tables or Audit table
according to your needs.
The following section describes the options that are available when storing changes
in Change Tables.
Suffix: Type a string to use as the suffix for all Change Tables. The default value is
__ ct.
Header column prefix: Type a string to use as the prefix for all of the Change
Table header columns. The default value is header__.
For example, the header column stream_position when using the default value
is called header__stream_position.
For more information, see Change Tables.
DDL options: Select one of the following options to determine how to handle
DDL operations on the source tables:
Apply to Change Table: Apply the DDL to the Change Table as well. For
example, when this option is enabled and a column is added to one of the
source endpoint tables, the column will also be added to the corresponding
Change Table.
Ignore: The change event from any DDL is ignored.
If Change Table exists when full load starts: Select one of the following from the
list to determine how you want to handle loading the Change Tables at full-load startup:
DROP and CREATE Change Table: The table is dropped and a new table is
created in its place.
ARCHIVE and CREATE Change Table:A copy of the existing table will be saved
to the same schema before the new table is created. The archived table name will
be appended with a timestamp, indicating when the archiving operation occurred
(e.g. Customers___ct_20170605175601).
Note Currently this option is only available for the Hadoop target endpoint.
Delete old changes and store new changes in existing Change Table: Data is
truncated and added without affecting the table metadata.
Note This feature is supported with the following target endpoints only:
Hadoop
File
Amazon S3
In a standard replication task, changes are replicated to the target in no particular order.
Change Data Partitioning enables processing of Change Data from many tables in a
consistent fashion. You can define the duration of partitions as well as the partitioning
base time, thereby ensuring overall consistency of the partitioned data (i.e. no partial
transactions, no order headers without order lines, and so on.)
The partitioned data is stored in the Replicate Change Tables. When the Change Data
Partitions table is selected (in the Control Tables tab), information about the partitions
will be recorded in the attrep_cdc_partitions Control Table on the target database. This
information can be used to identify partitioned data that needs to be further processed.
The partitioning options are as follows:
Partition every - Specify the length (in hours and minutes) of each partition.
Notes
It is recommended to specify a partition length in excess of one hour. Although
specifying a partition length less than one hour may improve latency, creating
many partitions on the target may also impact (target) performance.
If you resume a task from BEFORE the time that the last partition was
created, Replicate will write to a partition that has already been closed.
Partition base time - Partitions are created during a 24 hour time period, which is
calculated according to the specified “Partitioning base time” on the source database
(in UTC time). For example, a partition interval of 8 hours with a “Partitioning base
Note If there are existing Change Tables that were created before Change Data
Partitioning was enabled, you need to drop/rename them so that they can be
recreated with the additional "partition_name" column.
The Change Table header columns provide information about the Change Processing
operation such as the type of operation (e.g. INSERT), the commit time, and so on. If
you do not need this information, you can configure Replicate to create the Change
Tables without some or all of the header columns, thereby reducing their footprint in
the target database. To do this, clear the check boxes next to the header columns that
you wish to exclude.
Note that you cannot remove additional columns or restore columns while a task is
running. To change your initial selection, you first need to stop the task, then modify
your selection, and finally reload the target tables.
Note When Change Data Partitioning is enabled, an extra header column named
"partition_name" is added to the Change Tables and automatically selected in the UI.
As this column is required, it cannot be excluded.
The following section describes the options that are available for storing changes in
an Audit table.
Note LOB columns with unlimited size are not supported in the CHANGE_RECORD
and BU_CHANGE_RECORD fields. The other fields will be recorded but the LOB will
have a NULL value.
For a description of the audit table structure, see Using an Audit Table.
Audit table tablespace: This option is only available when the task's target
endpoint is Oracle. Enter the tablespace name on the target where you want the
Audit table to be created. If you do not enter any information in this field, then the
tables will created in the default permanent tablespace.
Audit table name: Specify a name for the Audit table.
If audit table exists when the target is reloaded: Select one of the following
to determine how you want to handle the Audit table when the target is reloaded:
DROP and CREATE audit table: The Audit table is dropped and a new table
is created in its place.
ARCHIVE and CREATE audit table: A copy of the existing table will be saved to
the same schema before the new table is created. The archived table name will be
appended with a timestamp, indicating when the archiving operation occurred (e.g.
attrep_audit_table_20170605175601).
Note Currently this option is only available for the Hadoop target endpoint.
Delete old changes and store new changes in existing audit table: Data is
truncated and added without affecting the Audit table metadata.
Keep old changes and store new changes in existing audit table: Data
and metadata of the existing Audit table are not affected.
For a description of the audit table structure, see Using an Audit Table.
Note Changes to tables without a Unique Index or Primary Key will always be
applied in Transactional apply mode.
Transactional apply: Select this to apply each transaction individually, in the order it
is committed. In this case, strict referential integrity is ensured for all tables.
Note Applying cached events in transactional mode to endpoints that do not enforce
constraints (such as HP Vertica and IBM Netezza), may result in duplicate records on
the target. This is because such endpoints do not return duplicate errors.
Batch optimized apply: Select this to commit the changes in batches. In this case,
a pre-processing action occurs to group the transactions into batches in the most
efficient way. This may affect transactional integrity. Therefore, you must select one
of the following to determine how the system will handle referential integrity issues:
Preserve transactional integrity
Note These options are not displayed in bidirectional tasks since such tasks always
use the "Preserve transactional integrity" option.
Note The following target endpoints do not support applying binary data types in
Batch Optimized Apply mode:
ODBC, SAP Sybase IQ, SAP Sybase ASE, HP Vertica, IBM Netezza, Teradata,
and Amazon Redshift.
Batch Tuning
The following options are available when Batch optimized apply is selected as the
Change Processing Mode:
Apply batched changes in intervals:
Longer than: The minimum amount of time to wait between each application
of batch changes. The default value is 1.
Increasing the Longer than value decreases the frequency with which changes are
applied to the target while increasing the size of the batches. This can improve
performance when applying changes to target endpoints that are optimized for
processing large batches, such as Teradata, HP Vertica, and Pivotal Greenplum.
But less than: The maximum amount of time to wait between each application
of batch changes (before declaring a timeout). In other words, the maximum
acceptable latency. The default value is 30. This value determines the maximum
amount of time to wait before applying the changes, after the Longer than
value has been reached.
Force apply a batch when processing memory exceeds (MB): The maximum
amount of memory to use for pre-processing in Batch optimized apply mode.
The default value is 500.
For maximum batch size, set this value to the highest amount of memory you can
allocate to Attunity Replicate. This can improve performance when applying
changes to target endpoints that are optimized for processing large batches, such
as Teradata, HP Vertica, and Pivotal Greenplum.
Limitations:
Note that this is not relevant for a Snowflake target (as Snowflake
does not support Primary Keys).
Updates: Ignore record
Escalation Action:
The Escalation action for both Data errors and Apply Conflicts
is not supported.
Control Table limitations:
The attrep_apply_exception Control Table is not supported.
Limit the number of changes applied per change processing statement to: To
limit the number of changes applied in a single change processing statement, select this
check box and then optionally change the default value. The default value is
Note This option is not available for tasks configured with a Snowflake
target endpoint.
The following options are available when Transactional apply is selected as the
Change Processing Mode:
Minimum number of changes per transaction: The minimum number of
changes to include in each transaction. The default value is 1000.
Note Replicate applies the changes to the target either when the number of
changes is equal to or greater than the Minimum number of changes per
transaction value OR when the batch timeout value is reached (see below) -
whichever occurs first. Because the frequency of changes applied to the target
is controlled by these two parameters, changes to the source records may not
immediately be reflected in the target records.
The following tuning options are available, regardless of which Change processing
mode is selected:
Offload transaction in progress to disk if:
Error Handling
Attunity Replicate handles different types of errors during its operation. The way
the system should respond to these errors depends on several aspects, including
the component where the error occurred, the type of error, and the scope of the
error. Because different sites may have different requirements for error behavior,
Attunity Replicate lets you configure the error handling.
You can also add an environment variable that instructs Replicate to create dump files
in the event of a crash. The dump files can then be used by Attunity Support to
troubleshoot the cause of the crash. For more information, see Creating Dump Files.
When you click Error Handling in the Task Settings dialog box, you can configure
the following:
Error Handling Settings:You can determine whether or not to override the global
error handling settings.
Note The option to set a Global Error Handling policy is not available when the
Apply batched changes to multiple tables concurrently option is selected
in the Task Settings' Change Processing Tuning tab.
Click the Change to Task Policy button in any of the Error Handling sub-tabs.
1. Click the Change to Global Policy button in any of the Error Handling sub-tabs.
2. Click OK when prompted to confirm your action.
Environmental Errors
Click the Environmental Errors sub-tab and then click Change to Task Policy to
configure the following:
Maximum retry count: Select this option and then specify the maximum number
of attempts to retry a task when a recoverable environmental error occurs.
To never retry a task, specify "0".
To retry the task an infinite number of times, clear the check box or specify "-
1" (the global error handling default).
When the system attempts to retry the task the designated number of times, the
task is stopped and manual intervention is required.
Data Errors
Click the Data Error sub-tab and then click Change to Task Policy to configure
the following:
For a data truncation error: Click the triangle to open the list and select what
happens when an truncation occurs in one or more specific records. You can select
one of the following from the list:
Ignore record: The task continues and the error is ignored.
Log record to the exceptions table (default): The task continues and
the error is written to the exceptions table.
Suspend table: The task continues but data from the table with the error
record is moved into an error state and its data is not replicated
Stop task: The task is stopped and manual intervention is required.
Notes Data truncation error handling is only supported in the following cases:
For other data errors: Click the triangle to open the list and select what
happens when an error occurs in one or more specific records. You can select
one of the following from the list:
Ignore record: The task continues and the error is ignored.
Log record to the exceptions table (default): The task continues and
the error is written to the exceptions table.
Suspend table: The task continues but data from the table with the error record
Log record to the exceptions table: The task continues and the error
is written to the exceptions table.
Suspend table (default): The task continues but data from the table with
the error record is moved into an error state and its data is not replicated.
Table Errors
Click the Table Errors sub-tab and then click Change to Task Policy to configure
the following:
When encountering a table error: Select one of the following from the drop-
down list:
Suspend table (default): The task continues but data from the table with
the error record is moved into an error state and its data is not replicated
Stop task: The task is stopped and manual intervention is required.
Escalate error handling when table errors reach (per table): Select this check
box to escalate error handling when the number of table errors (per table) reaches
the specified amount.
Escalation action: The escalation policy for table errors is set to Stop task
and cannot be changed.
Duplicate key when applying an INSERT: Click the triangle to open the list and
select what happens when there is a conflict with an INSERT operation. You can
select one of the following from the list:
Ignore record: The task continues and the error is ignored.
Log record to the exceptions table (default): The task continues and
the record is written to the exceptions table.
Suspend table: The task continues but data from the table with the error
record is moved into an error state and its data is not replicated.
Stop task: The task is stopped and manual intervention is required.
Update the existing target record: The target record with the same
primary key as the INSERTED source record is updated.
No record found for applying an UPDATE: Click the triangle to open the list and
select what happens when there is a conflict with an UPDATE operation. You can
select one of the following from the list:
Ignore record: The task continues and the error is ignored.
Log record to the exceptions table (default): The task continues and
the record is written to the exceptions table.
Suspend table: The task continues but data from the table with the error
record is moved into an error state and its data is not replicated
Stop task: The task is stopped and manual intervention is required.
Insert the missing target record: The missing target record will be inserted
Note When the INSERT the missing target record option is selected,
LOB columns in the source tables will not be replicated to the target (even if
the task’s Replicate LOB columns option is enabled).
Escalate handling when apply conflicts reach (per table): Select this check
box to escalate error handling when the number of apply conflicts (per table)
reaches the specified amount.
Notes
Apply Conflicts errors are counted for each table separately when a task
is run. When a task stops, the error count reverts to zero. When a task
is resumed/reloaded, the number of errors for all the tables are reset.
When working in Batch optimized apply Change Processing mode, the
calculation of the Apply Conflicts amount does not include DELETE and
UPDATE conflicts that were ignored (as a result of enabling the Ignore
Record option described above).
Escalation action: Choose what action Replicate should perform when handling is
escalated. Note that the available actions are dependent on the action selected in
the drop-down lists described above.
Log record to the exceptions table (default): The task continues and
the error is written to the task log and to the exceptions table.
Suspend table: The task continues but data from the table with the error
record is moved into an error state and its data is not replicated.
Stop task: The task is stopped and manual intervention is required.
Logging
You can set the logging level for task logs by selecting the Logging tab in the
Task Settings dialog box and then selecting the Logging Level sub-tab.The level
you set determines what information is written to the log.
For more information, see Monitor Mode and Setting the Task Logging Level.
The following are the available logging levels. The list is in order from the lowest level
to the highest level.
1. Error
2. Warning
3. Info
4. Trace
5. Verbose
The higher levels always include the messages from the lower levels. Therefore, if you
select Error, only error messages are written to the log. However, if you select Info,
informational messages, warnings, and error messages are included. Selecting
Verbose writes all possible messages to the log.
For information on how to set the logging level, see Setting the Task Logging Level.
Running in "Trace" or "Verbose" logging mode will quickly use up available disk
space (unless the logging settings have been configured to prevent this).
Continually writing large amounts of data to the logs will affect performance.
Notes
Invalid values will be indicated by a red triangle in the top right of the table cell.
Hovering your mouse cursor over the triangle will show the error message.
Use the Substitute or Delete Source Characters table to define replacements for
specific source characters. This may be useful, for example, when the Unicode
representation of a character is different on the source and target platforms. For
example, on Linux, the minus character in the Shift_JIS character set is represented as
U+2212, but on Windows it is represented as U+FF0D.
To Do This
Define substitution actions. 1. Click the Add Character button above the table.
Edit the specified source or Click anywhere in the relevant column and change the
target character character as required.
Delete entries from the Select the desired entry or entries and click the Delete
table button.
To Do This
Define or edit a substitution 1. Select a character set from the Character Set
action. drop-down list in the table.
Disable the substitution action. Select the blank entry from the Character Set
drop-down list.
File Uploads
Note that the File Uploads tab will only be shown if the task is defined with an
endpoint that supports this feature.
Click the Optimize File Uploads button to improve performance when replicating to
file-based targets such as Amazon S3 and Hadoop. When this feature is enabled, the
button text changes to Disable File Upload Optimization. Click the Disable File
Upload Optimization button to disable file upload optimization.
The upload mode depends on the task type:
Full Load - Multiple files created from the same table are transferred in parallel, in
no particular order.
Note that disabling this option after the task has already started will require you to do
one of the following:
If the task is in the Full Load stage, reload the target using the Reload Target
Run option.
If the task is in the Change Processing stage, resume the task using the
Start processing changes from Run option.
Notes
Notes
Section Names
Section names must start with the characters a-z, A-Z, or _ (an underscore) and
can then be followed by any of the following characters: a-z, A-Z, 0-9, _
With the exception of the Record name and Key name sections (that do not end
with a slash), removing the slash from section names will flatten the hierarchy of
the associated section (see Slashes below).
All section names except Record name and Key name can be deleted (see
Deletion below)
The Data name and Include record Before-data section names cannot both
be deleted
The Data name and Include record Before-data section names cannot be
the same
Some of the section names in the UI end with a slash (e.g. beforeData/). The purpose
of the slash is to maintain a hierarchy of the different sections within the message. If
the slash is removed, the following will occur:
The hierarchical structure of that section will be flattened, resulting in the
section name being removed from the message
The section name will be prefixed to the actual metadata, either directly or using
a separator character (e.g. an underscore) that you appended to the name
"message":{
"data":{
"COL1": "159",
"COL2": "159"
},
"beforeData": null,
"headers": {
"operation": "INSERT",
"changeSequence": "2018100811484900000000233",
Deletion
Deleting a section name from the message will flatten the hierarchical structure of
that section. This will result in all of that section's data appearing immediately below
the content of the preceding section.
"message":{
"data":{
"COL1": "159",
"COL2": "159"
},
"message":{
"data":{
"COL1": "159",
"COL2": "159"
},
"beforeData": null,
"operation": "INSERT",
"changeSequence": "2018100811484900000000233",
Variables
You can add variables to names by clicking the button at the end of the row.
The following variables are available:
SERVER_NAME - The host name of the Replicate
Server TARGET_TABLE_NAME - The name of the table
TARGET_TABLE_OWNER - The table owner
TASK_NAME - The name of the Replicate task
Note The TARGET_TABLE_OWNER variable is not available for the Record name
and Key name options (described in the table below).
Table 12.18 |
Message Format Options
Option Description
Message Format
Option Description
Include A unique identifier for the message. This should be a string, separated by
Namespace periods. Note that the Namespace will be included in both the message and
the message key.
Example:
mycompany.queue.msg
Default name:
com.attunity.queue.msg.{{TASK_NAME}}.{{TARGET_TABLE_OWNER}}.
{{TARGET_TABLE_NAME}}
DataRecord
Default name:
data/
Include A unique identifier for the header columns section. This should be a
headers string, separated by periods.
namespace Example:
headers.queue.msg
Default name:
com.attunity.queue.msg
Headers The name of the section containing the Replicate column headers.
name
Default name:
headers/
Option Description
Include the For a description of the available header columns, see Data Messages in
following Metadata and Data Messages.
headers
Default:
Include When this check box is selected (the default), both pre and post UPDATE
record data will be included in UPDATE messages.
Before-data To include only the post UPDATE data in messages, clear the check box.
Default name:
beforeData/
Key name The name of the section containing the message key.
Default name:
keyRecord
In this chapter:
▶ Running a Task
▶ Viewing the Task Status
▶ Reading Messages about a Task
Running a Task
After you design a task (see Defining Tasks), you can run and monitor its progress with
one click in Attunity Replicate. This simple Click-2-Replicate function is described in this
topic. In addition, the various types of run options available are also described. This topic
has the following sub-topics.
How to Run a Task
Using the Run Button Options
Note The task run buttons area available in the toolbar at the top of the console in
the following views:
Tasks View (in both Designer Mode and Monitor Mode)
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1030
Note When you click Run, the following occurs:
If this is the first time that a task is run, the Start Processing operation is run.
If the task has been started and stopped, the Resume Processing
operation described in Using Advanced Run Options is run.
If changes were made to the endpoint, change processing takes place after the
full load operation. If you do not want change processing to occur or if you want
to start change processing from a predetermined point, you must make the
appropriate Using Advanced Run Options selection.
In some cases, task replication may stop due to an error although the task process is
still running.
See Tasks View for information on the task status and how Attunity Replicate
displays information on the current task status.
The Run button is available in the following views:
The Tasks view when you select a task from the Task List.
For the individual task, both the Designer mode and Monitor mode have the Run
and Stop buttons available.
Note You must be in the Monitor mode to view the task progress.
Resume Processing: Resumes task execution from the point that it was
stopped. You can also resume processing by clicking the Run button if the task
has been stopped.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1031
Note If the schema or a filter was changed after the task stopped, the task
should be reloaded as opposed to resumed (see below).
Reload Target (Only available when the Full Load or Full Load and Apply
Changes replication options are enabled)
Using Advanced Run Options
Start Processing
This is available the first time you run the task only. This will execute the initial full load
operation. If Change Processing is also enabled for the task or if it is an Apply Changes
only task type, change processing will start as soon as any changes are made to the
source endpoint.
Reload Target
Starts the Full Load and Change Processing (if enabled) from the beginning. Tables
that have already been processed are handled according to the relevant "Target
table preparation" setting.
Note To replicate tables that were added to the local file channel task after the
initial full load, you need to reload both the local and the remote file channel tasks.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1032
Using Advanced Run Options
Advanced Run Options provide you with additional options for resuming and
restarting tasks.
Note Some of the advanced run options are not available in a Log Stream
Staging setup.
For information on the availability of advanced run options in a Log Stream
Staging setup, see Using the Log Stream.
To use Advanced Run Options, click the triangle next to the Run button and
select Advanced Run Options.
The Advanced Run Options dialog box opens.
The Advanced Run Options dialog box lets you do the following:
**Restart task and start processing changes from current time: This starts
the Apply Changes replication task from the beginning (as if the task has not run
before).
**Only available for Apply Changes replication tasks.
Note When resuming a task from MySQL, the Date and Time or Source
change position must always correspond to the beginning of a transaction.
Date and Time: Select the date and time from where you want to Replicate
to start processing changes.
Notes
When logs are deleted from the database (e.g. due to a purge policy),
a log matching the specified date and time may not exist. In this
case, Replicate will resume the task from the earliest point it can after
the specified date and time.
The timestamp uses the local time of the browser machine.
This option is not relevant for the File Source endpoint.
Source change position (e.g. SCN or LSN): Specify the position in the log
from where to resume change processing. The source change position format
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1033
differs according to your source endpoint. For instance, to resume processing
from a Microsoft SQL Server database, you would need to specify the LSN
(e.g. 000000c1:00001d6f:0004). However, if your source endpoint is Oracle,
you would need to specify the SCN (e.g. 10737419121).
Metadata Only:
Create empty tables on the target and then manually edit them.
Create tables during a task.
Enabling the options will also ensure that supplemental logging is set up correctly on
the source tables before starting the actual replication task.
Note The "Metadata only" feature is not supported when the task is defined with any
of the following task options:
Apply Changes only
Store Changes only
Apply Changes and Store Changes
Recreate all tables and stop: Select this option to recreate the target tables as
defined in the Full Load Settings tab. When "Store Changes" is enabled, the
Change tables/Audit table will be created as defined in the Store Changes
Settings tab. To use this option, stop the existing task, run the task with this
option enabled (the task will stop automatically) and finally, resume the task.
Create missing tables and stop: Select this option to create missing target tables
including Change Tables. You can use this option to create Change Tables on the
target after enabling the "Store Changes" option (in the Store Changes Settings
tab) for an existing task. To use this option, stop the existing task, run the task with
this option enabled (the task will stop automatically) and finally, resume the task.
The table below shows which tables are created in the target database when the Metadata
only option is enabled for a unidirectional task. As the table shows, when certain task options
are enabled, Control tables and the Audit table will not be created on the target.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1034
Enabled Task Options Tables Created on the Target
Full Load All tables except for Control tables
Full Load and Apply Changes All tables except for Control tables
Full Load, Apply Changes and All tables except for Control tables
Store Changes - When changes
are stored in Change tables
Full Load and Store Changes - All tables except for Control tables and the Audit table
When changes are stored in an
Audit table
Full Load, Apply Changes and All tables except for Control tables and the Audit table
Store Changes - When changes
are stored in an Audit table
Recovery:
Recover using locally stored checkpoint: Use this option if recovery is not
possible using the Resume Processing or Start process changes from options
(due to corrupt swap files, for example). When this option is selected, Replicate
uses the checkpoint data stored in <Data_Folder_Path>\data\tasks\<task_
name>\StateManager to recover the task.
MySQL
PostgresSQL
SAP HANA
Tasks can only be recovered during Change Processing (i.e. after Full
Load Completes)
With the exception of the File Channel endpoint, all target endpoints
are supported. The following limitations apply:
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1035
In Transactional apply Change Processing mode: All target
endpoints that support transactions are supported.
In Batch optimized apply Change Processing mode: Oracle
target endpoint only is supported. Also requires the Preserve
transactional integrity option to be enabled.
For all other target endpoints or Change Processing modes, recovery
is supported, but may cause duplicates on the target.
Recover using checkpoint stored on target: Select to recover a task using the
CHECKPOINT value from the attrep_txn_state table (created in the target database).
Transactional apply
This option will only be available if the Store task recovery data in target
database option was enabled in the Task Settings' Change Processing Tuning
tab before Change Processing completed.
Select this option (as opposed to the Recover using locally stored checkpoint
option) if the files in the Data folder are corrupted or if the storage device
containing the Data folder has failed.
For a detailed explanation of how to set up and implement recovery using the
attrep_ txn_state table, see Recovering from Data Folder Loss or Corruption.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1036
Recovering from Data Folder Loss or Corruption
During normal operation, Attunity Replicate maintains the replication state in the
following location:
<Data_Folder_Path>\data\tasks\<task_name>\StateManager
This enables tasks that cannot be resumed normally (due to corrupt swap files, for
example) to be recovered using the Recover using locally stored checkpoint
option described in Using Advanced Run Options.
However, if the files in the data folder become corrupted or if the storage device
containing the data folder fails, tasks must be recovered using the means
described below.
1. Design a task. Make sure to enable the Store task recovery data in target
database option in the Task Settings' Change Processing Tuning tab. This option
can be enabled at any time during Change Processing, although it must be enabled
before Change Processing completes.
2. Export the task definitions as described Exporting Tasks.
In addition to the selected source tables, the task will write the checkpoint data to
the following table in the target database (and automatically create the table if it
has not already been created by another task):
attrep_txn_state
To initiate recovery
1. Import the task definition exported when you set up the task.
2. Enter the passwords in the endpoint connection settings.
3. Access the attrep_txn_state table on the target database and locate the failed task in
the TASK_NAME column. If there are tasks with the same name running on multiple
Replicate Servers, you will also need to locate the appropriate server in the SERVER_
NAME column. After locating the relevant task, copy the value in the corresponding
CHECKPOINT column.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1037
4. Select the Recover using checkpoint stored on target option and then provide
the CHECKPOINT value (preferably by pasting) as described in Using Advanced
Run Options.
5. Click OK to start the recovery.
During recovery, Replicate does not write anything to the target database until it
identifies the commit event corresponding to the CHECKPOINT value. Once it identifies
the CHECKPOINT commit event, recovery is performed and the task reverts to standard
operation.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1038
Fatal Error: When a fatal error occurs, the task stops and
you must resolve the error manually. You cannot start the
task again until the error is resolved. Use the logs or the
messages in the Alerts pane to see the error type.
See also:
Viewing Notifications
Viewing Notifications
The Notifications tab displays notifications about the task. These messages alert you
to specific events encountered by a task, such as the task starting or stopping, a
specific error type, or information about latency and disk space.
The Notifications tab displays the time of a notification and a description of the
notification. You define the notifications that are sent for each task and a description
for each notification in the Settings area. For more information, see Creating a New
Notification.
Opening a Notification
When you open a notification, you can see the full message presented in a dialog box. The
dialog box contains a button to copy the text so that you can use it somewhere else for
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1039
troubleshooting and the timestamp for the notification.
To open a notification:
Clearing a Notification
You can clear notifications from the list to make sure that you are seeing only those
that are relevant to you.
To clear a notification:
Sorting Notifications
You can sort log messages according to Date and Time and Message.
2. Click the Date and Time or Message column according to how you want to sort
the messages.
An upward arrow indicates that the column is sorted in ascending order whereas
a downward arrow indicates that the column is sorted in descending order.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1040
warnings, or the total of number of errors and warnings depending on what you select
to view in this tab. The Log Messages tab is shown in the figure below.
Two types of log messages are displayed in the Log Messages List. You can view
Errors, Warnings, or both.
Select the check box or boxes for the type messages you want to view. The
check boxes are located at the top of the Log Messages List.
When you open a log message, you can see the full log text presented in a dialog box.
The dialog box contains a button to copy the text so that you can use it somewhere
else for trouble shooting and the timestamp for the log message.
1. In the Messages section of the console, click the Log Messages tab.
2. Select the log message you want to open from the list.
3. Double-click the log message or click Open from the toolbar at the top of the list.
1. In the Messages section of the console, click the Log Messages tab.
2. Select the log message you want to clear from the list.
3. Click Clear from the toolbar at the top of the list.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1041
Sorting Log Messages
You can sort log messages according to Date and Time, Level and Message.
1. In the Messages section of the console, click the Log Messages tab.
2. Click the Date and Time, Level or Message column according to how you want
to sort the messages.
An upward arrow indicates that the column is sorted in ascending order whereas
a downward arrow indicates that the column is sorted in descending order.
Copyright © Attunity Ltd. Chapter 13 | Working with Tasks at Runtime | Page 1042
14 | Monitoring and
Controlling Replication Tasks
When you monitor and run a task, you can use the Click-2-Replicate function to carry
out the replication task and view its functions in near real time. This section describes
how to run and monitor a replication task.
In this chapter:
▶ Viewing Information in the Monitor
▶ Monitoring Full-Load Operations
▶ Monitoring Change Processing Operations
▶ Viewing Messages
▶ Using the Monitor Tools
1. When Viewing Specific Tasks, select the task you want to monitor.
2. From the toolbar at the top of the console, click Open.
The Monitor opens. To view the information in real time, you need to run the task
(if the task has not already started). For information on running a task, see
Running a Task.
To make sure you are viewing the information for a full-load operation, select the
Full Load tab.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1043
You can view the following:
General Information for a Full Load Detailed
Information for the Full Load Monitoring
Throughput in a Full Load Operation
Completed: The number of tables that finished loading into the target endpoint.
Loading: The number of tables that are in the process of loading into the target
endpoint.
Queued: The number of tables that are waiting to load into the target endpoint.
Error: The number of tables that could not be loaded due to an error. See
Reading Messages about a Task for information about error messages.
Full-load total completion bar: Displays the progress of all records being loaded to the
target endpoint. The bar is located in the Full Load tab at the top of the graph section.
Throughput gauge: Displays the current throughput.Throughput displays the number
of events read in the task for a specified amount of time.
You can also view Detailed Information for the Full Load.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1044
Detailed Information for the Full Load
For each of the status bars displayed in the General Information for a Full Load
graphs, a table is displayed in the section below with specific information about the
current loading status. The following information is available:
General Information for a Completed Task
Information for Each Table in the Task
Information for Tables that have Completed Loading
Information for Tables that are Currently Loading
Information for Tables that are in the Loading Queue
Information for Tables with Errors
Table 14.1 |
Progress Details for all Tables in the Task
Total Completed Remaining Notes
Tables The total number of The total number of The total number Additional
tables that are tables that completed of tables waiting to information.
included in the task. loading at the current be loaded.
time.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1045
Table 14.1 | Progress Details for all Tables in the Task (Cont.)
Time The estimated time The total elapsed time. The estimated Additional
to load all of the amount of time to information.
selected tables in load the remaining
the task. tables.
Status: This is a statement that describes the status for the table. The following
are the statuses that can be displayed:
Queued: The table is in the queue waiting to be loaded to the target endpoint.
Loading: The table is being processed but is not finished loading.
Completed: All of the table records are loaded to the target.
Error: The table stopped loading due to an error. See Reading Messages
about a Task for more information about task errors.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1046
Estimated Count: The number of records that are loaded to the target.
Elapsed Time: The total elapsed time since the table records began processing.
Progress: The table status and the time the table entered that status.
Reload: To reload selected tables, select the tables you want to reload and then
click the Reload button above the table list. When prompted to confirm the
operation, click OK. The data in the selected tables will be reloaded to the target
endpoint. Note that this option is not available for Apply Changes Only tasks.
Note When exclude filters are applied to SAP Application tables, the number
shown in this column will reflect the total number of records before the exclude
filter was applied. Consequently, the "Transferred Count" number may be higher
than the actual number of records that were replicated to the target.
Transferred Volume: The volume of the records (in KB) loaded to the target.
Load Duration: The amount of time that it took for all records to load to the target.
Throughput Records: The average throughput rate for the table. Throughput
describes the number of records read per second. For more information on
throughput, see Monitoring Throughput in a Full Load Operation.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1047
Throughput Volume: The average throughput rate for the table. Throughput
describes the volume of records (in KB) read per second. For more information
on throughput, see Monitoring Throughput in a Full Load Operation.
Reload: Click the Reload icon to reload the data for selected tables and run the
full-load operation again.
Note When replicating to an Oracle database with a full disk and/or partition
where Oracle is trying to write archived redo log files, insert operations may fail. In
such as case, no error will be shown and the task will not progress past the loading
stage. To confirm that this is indeed an Oracle Archiver error, stop and attempt to
restart the task. The task will not start and an appropriate error should be shown.
Estimated Finish Time: The approximate time the task finished loading the tables.
The timestamp displayed indicates the date and time.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1048
Note There may sometimes be a discrepancy between the "Estimated
Finish Time" and the"Time Remaining (Estimated)" values.
The "Time Remaining (Estimated)" value is calculated by the combined
transfer rate of all the records of the task, while the "Estimated Finish Time" is
calculated per table.
The discrepancy arises when the table transfer rate at the beginning of the task
is very fast, but slows down towards the end of the task.
In this situation, the "Time Remaining (Estimated)" value will be greater and
less accurate than the "Estimated Finish Time" value.
Progress: The table status and the time the table entered that status.
Reload: Click the Reload icon to reload the data for selected tables and run the
full-load operation again.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1049
Information for Tables with Errors
This section displays a table with information for each of the tables that stopped loading
or suspended CDC due to an error. To view this table, click the Error bar, shown in the
figure below.
You can set the throughput measurement values either to the number of records
replicated per second, or to the number of kilobytes replicated per second. The display is
always based on the current load operation.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1050
To set the unit of throughput measurement:
Select either rec/sec or kbyte/sec from the drop-down menu above the
Throughput gauge.
Click the Throughput gauge to display a graph with the throughput details as shown in
the figure below. To view the graph only, click the expand/collapse arrow in right side of
the gray bar above the graph. Click the arrow again to restore the status bars and
throughput gauge.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1051
General Change Processing Information
General information about the Change Processing task is presented in a graphical format.
The following figure shows the graphical information displayed.
After the table is created, only INSERT operations will be supported. Other
operations (e.g. UPDATE) will fail since the data does not exist (unless it has
been inserted earlier).
If the table already exists, Replicate will behave according to the If target
table already exists setting in the task settings' Full Load Settings tab.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1052
The number of metadata changes (DDL) processed. DDL changes include
information about events like changes to table names or to column names.
Apply Throughput gauge: A gauge that describes the number of change events
read per second. For additional details, you can also view a graph with Information
about Change Processing Throughput.
Apply Latency gauge: A gauge that displays the latency information.
The latency values displayed in the Attunity Replicate Console measure the time
delay (latency) between the time when a change is visible to the source (and
committed), and the time when this same change is visible to the target. The
display is always based on the current change being applied.
You should take the following into consideration:
For example, when the most recent latency value was 10 seconds and now a
transaction of one million rows gets committed at the source endpoint, Attunity
Replicate starts to apply that transaction to the selected target and it will take
some time to write all the changes to the target (for example 60 seconds).
During the next 60 seconds, the latency value gradually grows to 70 seconds
for the last change in the transaction. Once the transaction is committed, the
latency drops back to the 'regular' latency (10 seconds in this case).
When a time period passes with no changes applied to the target, the latency
calculation is based on the time difference between the current time and the
timestamp of the last change event read from the transaction log. This could
happen if, for example, there is high activity on tables which are not selected
for replication in the current task.
For additional details, you can also view a graph with Information about Apply Latency.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1053
Information about Incoming Changes
This section displays two bar graphs with information about incoming changes.
Incoming changes displays a snapshot of the number of change records currently
being read from the source endpoint and written to the target endpoint. To view these
graphs, click the Incoming Changes bar, shown in the figure below.
Accumulating: These bars display the number of records currently being read
from the source endpoint. These records are accumulated in a queue until they are
applied to the target. The following is displayed:
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1054
In Memory: The number of accumulating records that are currently in
the computer memory.
On Disk: The number of accumulating records that are currently stored on disk.
Applying: The number of records currently being written to the target. These are
the applied changes. The following is displayed:
In Memory: The number of records being applied that are currently in
the computer memory.
On Disk: The number of records being applied that are currently stored on disk.
To view these tables, click the Applied Changes pie graph, shown in the figure below.
The following tables are available when you select Applied Changes:
Recent Activity
Aggregates
Recent Activity
Click Recent Activity at the top of the Applied Changes Details section to view information
about which changes occurred in each table. It has the following information:
Table Name: The names of the source tables that are included in the task.
Insert: The number of INSERT operations processed for the specific table.
Delete: The number of DELETE operations processed for the specific table.
Update: The number of UPDATE operations processed for the specific table.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1055
DDL: The number of metadata changes (DDL) processed. DDL changes
include information about events like changes to table names or to column
names. Total Applied: The total number of changes applied to the target.
Data Errors: The number of data processing errors for the specific table. Data
errors occur at the record level and include conversion errors, errors in
transformations, and bad data.
Resetting the Data Errors Count
After you have resolved the data errors it is recommended to reset the data errors
count. This is especially important if you have configured Replicate to perform an
escalation action when the number of errors reaches a certain amount.
Details about the errors can be found in the attrep_apply_exceptions control table.
To reset the error count for a specific table, select the table and then click the
Reset data errors button above the table list. Note that resetting the error count
does not delete the error information from the attrep_apply_exceptions table.
For information about setting a data error escalation policy, see Data Errors.
Note Reloading a table resets the data error count for that table.
Last Modified: The time the last change occurred for the specific table.
Reload: To reload selected tables, select the tables you want to reload and then
click the Reload button above the table list. When prompted to confirm the
operation, click OK. The data in the selected tables will be reloaded to the target
endpoint. Note that this option is not available for Apply Changes Only tasks.
Aggregates
Click Aggregates at the top of the Applied Changes Details section to view
information about total changes for each change type and transaction type.
The Aggregate table displays the total changes (for all tables) applied for each of
the following types of operations:
INSERT
UPDATE
DELETE
DDL
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1056
The Aggregate table also displays the information about transactions. It displays the
total number and volume of:
COMMITS
ROLLBACKS
You can set the Apply Throughput measurement values either to the number of
change records replicated per second, or to the number of kilobytes replicated per
second. The display is always based on the current load operation.
Select either rec/sec or kbyte/sec from the drop-down menu below the
Apply Throughput gauge.
Click the Apply Throughput gauge to display a graph with the throughput details as
shown in the figure below. To view the graph only, click the expand/collapse arrow in
right side of the gray bar above the graph. Click the arrow again to restore the progress
bars and Change Processing gauges.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1057
Figure 14.15 | Apply Throughput Details Graph
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1058
Figure 14.16 | Apply Latency
The latency values displayed in the Attunity Replicate Console measure the time delay
(latency) between the time when a change is visible to the source (and committed), and
the time when this same change is visible to the target. The display is always based on
the current change being applied. For more information about latency, see Apply
Latency gauge.
Select the Apply Latency gauge to display a graph with the latency details. To view the
graph only, click the expand/collapse arrow in right side of the gray bar above the graph.
Click the arrow again to restore the progress bars and Change Processing gauges.
Note During data capture, the target latency will always be equal to the source
latency (even though no data has reached the target yet). This is simply because
target latency is the sum of source latency + apply latency, and can therefore never
be less than the source latency.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1059
Figure 14.17 | Apply Latency Details Graph
Viewing Messages
You can see messages sent for the task while in the monitor view. For information
on viewing messages, see Reading Messages about a Task.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1060
Deleting Log Files
Downloading a Diagnostics Package
Downloading a Memory Report
You can double-click the description cell to view a window with the full message if
the entire description is not available.
The following figure shows the History window.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1061
Figure 14.18 | History Window
Note
The logging level can also be set in the Logging Level sub-tab in the
Task Settings dialog box. For more information, see Logging.
1. Open the task you are working with if it is not displayed in the Attunity Replicate
Console. For information on opening a task, see Editing a Replication Task.
2. Switch to Monitor view. Then, click the Tools toolbar button and select Log
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1062
Management.
3. At the top of the Log Management window, set the Component Logging Level
slider to the log level you want. This sets the logging level for all log modules. Note
that all of the sliders for the individual modules move to the same position that you
set in the main slider.
4. Make any changes to the sliders for the individual modules. This is optional. Note
that if you change the main slider, all of the individual sliders are reset to the new
position. If you want to maintain a different logging level for a specific module, you
need to reset it.
Running in "Trace" or "Verbose" logging mode will quickly use up available disk
space (unless the logging settings have been configured to prevent this).
Continually writing large amounts of data to the logs will affect performance.
Viewing the Task Log Files and Manually Rolling them Over
In the Log Viewer window, you can view the logs for the task you are
currently monitoring and manually roll them over if necessary.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1063
To open the Log Viewer window:
1. Open the task whose log files you want to view or download.
3. Either, click the Tools toolbar button and then select View
Logs. -OR-
Click the View Logs button in the Messages pane in the lower right of the
4. Select the log file you want to view or download from the list in the Log Files pane.
If you want to download the file, skip to Step 8.
5. The contents of the log file will be displayed in the right pane. When you select a
row in the log file, a tooltip will be display the full message of the selected row.
6. You can browse through the log file using the scroll bar on the right and the
navigation buttons at the top of the window.
7. To search for a specific string in the log file, enter the search string in the search
box at the top of the window.
Any terms that match the specified string will be highlighted blue.
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1064
Deleting Log Files
In the Delete Logs window, you can manually delete log files older than the
specified number of days.
4. Optionally change the default number of days (45) and then click the Delete
button. All log files older than the specified number of days will be deleted.
1. Open the task you are working with if it is not displayed in the Attunity Replicate
Console. For information on opening a task, see Editing a Replication Task.
2. Click the Tools toolbar button and then select Support > Download
Memory Report.
Depending on your browser settings, the following file will either be automatically
downloaded to your designated download folder or you will be prompted to
download it:
File name:
<task_name>__diagnostics__<timestamp>.memp
Example:
MyTask__diagnostics__20180109161333.memp
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1065
Downloading a Diagnostics Package
You can generate a task-specific diagnostics package for Support to review. The
diagnostics package contains the task log files and various debugging data that may
assist in troubleshooting task-related issues.
1. Open the task you are working with if it is not displayed in the Attunity Replicate
Console. For information on opening a task, see Editing a Replication Task.
2. Click the Tools toolbar button and then select Support > Download
Diagnostics Package.
Depending on your browser settings, the following file will either be automatically
downloaded to your designated download folder or you will be prompted to
download it:
File name:
<task_name>__diagnostics__<timestamp>.zip
Example:
MyTask__diagnostics__20180109161333.zip
Copyright © Attunity Ltd. Chapter 14 | Monitoring and Controlling Replication Tasks | Page 1066
15 | Attunity Replicate
Server Settings
This chapter describes how to configure the Attunity Replicate Server settings.
Server settings are managed in SERVER view.
From the drop-down list in the top left corner of the console (below the product
logo) select Server.
Note
Server settings affect all Tasks that are created in the Attunity Replicate
instance you are working with.
Changes to server settings will not affect running tasks.
In this chapter:
▶ Notifications Settings
▶ License Settings
▶ Global Error Handling
▶ Logging
▶ File Transfer Service
▶ Scheduling Jobs
▶ User Permissions
▶ Resource Control
Notifications Settings
The following can be defined in the Notifications settings:
Defining Notifications
Setting up Mail Parameters
Creating a Default Recipient List
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1067
To view and edit the Notification settings:
In Server view, click the Notifications tab on the left. Then click the
Notifications sub-tabs to enter your settings.
Defining Notifications
To configure and create notifications, click the Notifications sub-tab.
You use notifications to send messages about events that occur when running tasks in
Attunity Replicate. Notifications are sent to inform users of any change in the system
state, including:
A task is started or stopped
Latency is too high
Memory utilization is too high
Disk utilization is too high
An error or a specific type of error occurred
You can manage notifications that you create from the Notifications list. This list provides
you with information about each notification defined and lets you activate/deactivate a
notification. In addition, you can make changes to the definitions of existing notifications
or delete them.
The following topics describe how to define notifications in Attunity Replicate:
Creating a New Notification
Using the Notification List
Editing a Notification
Deleting a Notification
From the Server view, click Notifications from the menu list at the left.
The Notifications sub-tab is displayed.
Notifications are sent by:
An email message to the default list of users and/or to a custom list of users.
Writing an entry in the Windows Event Log.
Displaying a message in the Attunity Replicate Console.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1068
Creating a New Notification
Use the New Notification wizard to determine the notifications that are sent and
who receives them.
1. In Server view, click the Notifications tab on the left and then click the
New Notification toolbar button.
2. From the drop-down menu, select Task Events or Server Events according to
the notification you want to define.
3. The New Notification wizard opens displaying either the Task Events or
Server Events screen (according to your earlier selection).
4. Continue from Creating a Notification for a Task Event or Define the Recipients
as appropriate.
Use the New Notification wizard to create notifications for task-based events.
Note For changes to task notification settings to take effect, the task(s) needs to
be stopped and resumed.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1069
Define the Action that Triggers the Notification
In the Operator section of the Task Events page, you can determine the action that
triggers the notification. If the Operator section is not displayed, click on the header with
the word Operator to display the options for this section. Select one of the following:
Task was started: To send the notification when the task starts.
Task was stopped manually or scheduled: To send the notification when the task
is stopped manually or by the Scheduler.
Task was stopped after Full Load: Cached changes were not applied: To send
the notification when the task is stopped after Full Load completes but before
cached changes (changes to the source tables that occurred during Full Load) are
applied to the target.
Task was stopped after Full Load: Cached changes were applied: To send
the notification when the task is stopped after Full Load completes and cached
changes (changes to the source tables that occurred during Full Load) have been
applied to the target.
Full load started: To send the notification when the Full Load process starts.
Full load completed: To send the notification when the Full Load process completes.
Once you determine when to send the notification, you can decide whether specific
changes in status trigger the notification.
If you want to send a message about problems in latency, memory utilization, or disk
utilization, click Performance/Resources. See Define Which Changes of Status
Trigger the Notification for an explanation.
If you want to send the notification when certain errors occur, click Errors. See
Define Errors That Trigger the Notification for an explanation.
Or you can click Next to Define the Recipients.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1070
Latency is higher than [N] seconds.
Memory utilization exceeded [N] MB
Disk utilization exceeded [N] MB
3. Define the value for the option you select. See the table below for an explanation
on each of these options and how to set the value.
Note If you select one of these options, the notification is sent only when the
selected parameter is true. However, you must also Define the Action that Triggers
the Notification.
Table 15.1 |
Set Values for Latency, Disk Utilization, Memory Utilization
Notification Set Value Notes
Latency is Click [N] and enter a value in the field
higher than that is displayed.
Value
Latency is the time interval in seconds
seconds
between the time a change was
committed in the source system and
the time it is applied and committed in
the target system.
Clear Use this to set the value that When latency is below the value
notification determines when latency returns to entered in this field, it is
when latency "normal limits." considered to be in the "normal"
drops below range and the notification status
Click [N] and enter a value.
<n> ends.
seconds.
If selected, a notification is sent
to indicate that latency returned
to "normal" status.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1071
Table 15.1 | Set Values for Latency, Disk Utilization, Memory Utilization (Cont.)
Clear Use this to set the value that When memory utilization is
notification determines when memory utilization below the value entered in this
when returns to "normal limits." field, it is considered to be in the
memory "normal" range and the
Click [N] and enter a value.
utilization is notification status ends.
below <n>
For more information, see Define
MB
the Notification Message.
Clear Use this to set the value that When disk utilization is below the
notification determines when disk utilization value entered in this field, it is
when disk returns to "normal limits." considered to be in the "normal"
utilization is range and the notification status
Click [N] and enter a value.
below <n> ends.
MB
For more information, see Define
the Notification Message.
Once you determine the status changes that trigger a notification, you can decide
whether specific errors trigger a notification.
If you want to send the notification when certain errors occur, click Errors. See
Define Errors That Trigger the Notification for an explanation.
Or you can click Next to Define the Recipients.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1072
To set up notifications for errors:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1073
To determine the notification recipients:
The Name field in the first available row in the list is activated.
Type the name of the user that you want to receive the message.
Note If you click another part of the Attunity Replicate Console, the cell will
become inactive. You can double-click the cell to enter additional information.
Press the [tab] key or double click in the in the Email cell, then type the
email address for the user you entered in the Name cell.
Repeat the above to add more recipients.
When you have finished, click Next to Define the Notification Message.
1. In the New Notification Wizard, Message page, double-click in any of the table
cells to open the Edit Notification Message dialog box. See the table Creating a
Notification Message for an explanation of the information to enter in each field.
2. Click in the right pane of the dialog box and begin to type your message. In some
cases a default message is displayed based on the information you entered in the
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1074
previous pages of the New Notification Rule wizard. You can edit or delete the
message, or create a new message to be sent with the notification in this dialog box.
3. Add variables in messages and email headers you define for notifications,
if necessary. You can enter variables in one of two ways:
Type a variable into the message pane on the right using the following format:
{{<VARIABLE_NAME >}}
Use the variables from the left pane of the Edit Notification dialog box. To
add a variable to the notification message, you can:
Double-click the variable. The variable is inserted where your cursor is located
in the notification message in the right pane.
Select the variable you want to use and click the arrow key in the middle of
the Edit Notification Message dialog box. The variable is inserted where
your cursor is located in the notification message in the right pane.
Drag the variable from the left pane to the location you want to place it in
the notification message in the right pane.
For more information, see the Supported Notification Variables.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1075
Table 15.2 |
Creating a Notification Message
To where: Notification On Notification Off Message Message
For more information, conditions for the type of message is sent for
notification to be sent. notifications about latency, disk
see Define the
For more information, utilization, and memory utilization.
Recipients.
see Define the Action For more information, see
that Triggers the Define Which Changes of Status
Notification, Define Trigger the Notification.
Which Changes of Status
Trigger the Notification,
and Define Errors That
Trigger the Notification.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1076
Table 15.2 | Creating a Notification Message (Cont.)
information, see
Define the
Recipients.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1077
Table 15.2 | Creating a Notification Message (Cont.)
Email Subject: In this field, you can edit, In this field, you can edit, change
change or delete the or delete the subject line for an
This is the subject
subject line for an email email that is sent when the
of the email
that is sent when the replication task returns to the
messages sent for
replication task meets the normal range as you defined when
the notification.
conditions for the you Define Which Changes of
See Define the
notification to be sent. Status Trigger the Notification.
Recipients for
Example: This field is relevant only for
information about
[{{SERVER_NAME}}\ notifications about latency, disk
sending a notification
{{NOTIFICATION_NAME}}] utilization, and memory utilization.
as an email.
{{TASK_NAME}} high Example:
latency notification Replicate notification '
This is the subject for an {{NOTIFICATION_NAME}}' for
email message sent task '{{TASK_NAME}}'
when latency reaches a This is the subject for an email
value higher than the message sent when latency
value you defined.
returns to within its normal limits.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1078
Table 15.2 | Creating a Notification Message (Cont.)
Email Message: In this field, you can edit, In this field, you can edit, change,
change or delete the or delete the message that is sent
This is the body of the
message that is sent by by email when the replication
email message sent
email when the task returns to the normal range
for the notification.
replication task meets the as you defined when you Define
See Define the
conditions for the Which Changes of Status Trigger
Recipients for
notification to be sent. the Notification.
information about
Example: This field is relevant only for
sending a notification
The latency for notifications about latency, disk
as an email.
replication task utilization, and memory utilization.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1079
Table 15.2 | Creating a Notification Message (Cont.)
Example:
[{{SERVER_NAME}}\
{{NOTIFICATION_NAME}}]
{{TASK_NAME}} high
latency notification
After you define the message sent with the notification, click Next to Associate Tasks
with the Notification.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1080
Supported Notification Variables
For information on the variables that can be included in a notification, see Creating a
New Notification.
1. In the New Notification Wizard, Associate page, select one of the following:
All Tasks: To associate this notification with all tasks that are defined in the
Attunity Replicate instance you are working with. In this case all tasks that were
previously defined and any future task will be associated with this notification.
If you choose to associate this notification with All Tasks, then click Next
to Review the Notification Rule.
Selected Tasks: To associate this notification with one or more specific
tasks only. Continue with the next step.
2. Select the check box next to any of the tasks you want to associate with
this notification. You can select one or more tasks.
Note The Task check box at the top of the check-box column lets you select all
of the tasks that are displayed. When you select this check box it is as if you
select each of the tasks individually. Therefore, if you add tasks in the future
they will not be included.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1081
After you close the wizard, make sure to click Save at the top of the Settings page.
This will save the information for all settings, not only for the notification rule that you
created. If you made changes that you do not want to keep, click Discard to discard all
changes before you make changes to any of the other settings.
Use the New Notification wizard to create notifications for server-based events.
Note Server notifications are not written to the console (Task > Monitor
> Notifications).
Note Disk space is checked only for the drive where the data folder resides.
Disk space utilization reaches the high threshold: The notification will be
triggered when disk space utilization reaches the percentage defined for the
high threshold.
Disk space utilization reaches the critical threshold: The notification will be
triggered when disk space utilization reaches the percentage defined for the
critical threshold.
Disk space utilization returns to normal: The notification will be triggered
when disk space utilization returns to normal percentage (i.e. not high or critical).
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1082
Disk space utilization reaches any of the defined thresholds or returns to
normal: The notification will be triggered in any of the following scenarios:
Disk space utilization increases from normal to the high threshold
Disk space utilization increases from normal to the critical threshold
Disk space utilization increases from the high threshold to the critical threshold
Disk space utilization returns to normal from the high threshold
Disk space utilization returns to normal from the critical threshold
Disk space utilization returns to the high threshold from the critical threshold
In the System Memory section, you can determine the system memory utilization event
that triggers the notification.
Note Memory utilization check also takes into account the swap file space.
System memory utilization reaches the high threshold: The notification will be
triggered when system memory utilization reaches the percentage defined for the
high threshold.
System memory utilization reaches the critical threshold: The notification will
be triggered when system memory utilization reaches the percentage defined for
the critical threshold.
System memory utilization returns to normal: The notification will be
triggered when system memory utilization returns to normal percentage (i.e. not
high or critical).
System memory utilization reaches any of the defined thresholds or returns
to normal: The notification will be triggered in any of the following scenarios:
System memory utilization increases from normal to the high threshold
System memory utilization increases from normal to the critical threshold
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1083
Define the Recipients
You can create a custom message for your notification. By default, a standard message
is created based on your settings in the Server Events screen.
1. In the New Notification Wizard, Message page, click the message text to open the
Edit Notification Message dialog box. See the table Creating a Notification
Message for an explanation of the information to enter in each field.
2. Click in the right pane of the dialog box and begin to type your message. In some
cases a default message is displayed based on the information you entered in the
previous pages of the New Notification Rule wizard. You can edit the message or
create a new message to be sent with the notification in this dialog box.
3. Optionally, add variables in messages and email headers you define for
notifications. You can enter variables in one of two ways:
Type a variable into the message pane on the right using the following format:
{{<VARIABLE_NAME >}}
Use the variables from the left pane of the Edit Notification Message
dialog box. To add a variable to the notification message, you can:
Double-click the variable. The variable is inserted where your cursor is located
in the notification message in the right pane.
Select the variable you want to use and click the arrow key in the middle of
the Edit Notification Message dialog box. The variable is inserted where
your cursor is located in the notification message in the right pane.
Drag the variable from the left pane to the location you want to place it in
the notification message in the right pane.
For more information, see the Supported Notification Variables.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1084
Table 15.3 |
Creating a Notification Message
To where: Notification Message
This column describes where the message is Sent when the server meets the
sent. conditions for the notification to be sent.
This is the subject of the email messages line for an email that is sent when the
sent for the notification. server meets the conditions for the
notification to be sent.
See Define the Recipients for information
about sending a notification as an email.
This is the body of the email message sent that is sent by email when the server
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1085
changes, click Back and go to the page or pages you want to change.
When you are sure that the notification rule is defined in the way that you want, click
Finish to close the wizard and add the rule to the notification list (seeUsing the
Notification List).
After you close the wizard, make sure to click Save at the top of the Settings page.
This will save the information for all settings, not only for the notification rule that you
created. If you made changes that you do not want to keep, click Discard to discard all
changes before you make changes to any of the other settings.
Editing a Notification
You can make changes to any notification rule.
1. From the Notification List, select the notification you want to edit.
list). or
3. Make any changes you need in the wizard. For information on how to work with each
of the pages in the New Notification Rule wizard, see Creating a New Notification.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1086
Note You can only make changes to those sections that you defined when
Creating a New Notification.
You cannot change name of the notification.
If you defined a notification to let you know when the task or full load
started or stopped, this cannot be edited. For example, if you created a
notification rule for starting a task and you now also want to get notified
when the task stops, you must create a new notification rule.
In the Notify When? page, you can make changes to the data you defined in
the original notification rule. For example, if you defined a Memory utilization
message in the Notify when? page, Performance/Resources section, you
can only change this parameter. If you want to add information about
something that was not defined in the original notification rule, for example
you want to add errors to your notification or you want to get information
about latency, you must create a new notification rule.
Deleting a Notification
You can delete notification rules that you no longer want to use.
To delete a notification:
1. From the Notification List select the notification you want to delete.
2. Click Delete (at the top of the list).
1. Click the Mail Settings sub-tab and enter the following information:
Mail server: Type the outgoing mail server you are using to send
the notifications that you define in Attunity Replicate, for example,
smtp.example.com.
Port: Type the port number where the mail server is located. The default value
is 25.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1087
Use SSL: Select this check box if you want to use SSL security to connect to
the mail server for the notifications that are sent.
Anonymous login: Check this to allow an Attunity Replicate user to access the
mail server to receive messages without having to provide any user credentials.
User name: Type an email user name for the user account that is sending the
notifications. For SMTP authentication be sure to supply a valid user name.
Password: Type the password for the email user account that is sending the
notifications. For SMTP authentication be sure that password provided is valid.
Sender email address: Enter the email address that sends the email
notifications. This is the address that appears in the From field of the
email notification.
Send test email: Click to open the Send Test Email dialog box.
Email address for test email: Type an email address to receive a test
email message from the server you configured. Use this to determine that
the Mail Parameters you defined are valid.
2. Click Save at the top of the screen to save all of the changes you made, not only
for the recipient list.
Note If you made changes that you do not want to keep, click Discard to
discard all changes before you make changes to any of the other settings.
A default recipient list is a list of recipients that receive all of the notifications that you
define for task-based or server-based events. This allows you to use one list for all
email notifications without having to define the list each time you create a notification.
Note You can choose to send notifications to a different list or to additional users
for any specific notification. You define these exceptions when you create the
specific notification. For more information, see Creating a New Notification.
1. At the top of the Default Recipient List settings page, click Add
Recipient. The next row in the Recipient List table becomes available.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1088
2. Type the name of the user you want to add to the list of default recipients. Continue
to enter a name and email address for each recipient you want to include in the
default list.
3. Press the [tab] key or double click in the in the Email cell, then type the email
address for the user you entered in the Name cell.
4. Click Save at the top of the screen to save all of the changes you made.
Note Click Save to save the information for all settings, not only for the
recipient list. If you made changes that you do not want to keep, click Discard
to discard all changes before you make changes to any of the other settings.
Event ID Description
OTHER
261 Any error.
SERVER
300 Disk utilization has changed
TASK
400 Task has started.
403 Task has stopped after Full Load – cached changes were applied.
404 Task has stopped after Full Load – cached changes were not applied.
TABLE
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1089
500 Table load has started.
License Settings
You need to register the software before you can use Attunity Replicate. Your Attunity
vendor should provide you with a text file called license.txt. This file contains details
such as the product expiration date (if any).
Use the License settings page for:
Requesting a License
Registering a License
Viewing a License
From the Server view, click License from the menu list at the left. The License
sub-tab is displayed.
Requesting a License
You must have a valid license to work with Attunity Replicate. You can request a license
from the License settings page in the Attunity Replicate Console. In the License Request
dialog box, fill out the required information and submit the request by email. Once your
request is approved, the license file is sent to you by email. To use Attunity Replicate,
register the license by using the procedure described in Registering a License.
To request a license:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1090
3. Enter the requested information:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1091
Permanent: Select this if the license will always be valid.
Permanent licenses do not require an expiration date.
Evaluation: Select this if you are requesting a temporary license to
use Attunity Replicate for a trial period.
Term: Select this if you are requesting a license that is valid for a specific
period of time. In this case you must be sure to include an expiration date
in your request.
Expiration date: Click in this field to select the expiration date using the pop-
up calendar. This is required only if you selected Evaluation or Term in as the
License type.
Hosts: Type the name of the local computer where Attunity Replicate is
installed. By default the name of the local computer is displayed in this field.
You can change this or add additional computers if you are installing Attunity
Replicate in a different or an additional location.
Source Types: Click Edit to open the Edit Source Types dialog box. Check
the endpoint types you are working with as your replication sources. You can
select one or more endpoint endpoints as necessary. If you need to work with
all available endpoints, click All.
Target Types: Click Edit to open the Edit Target Types dialog box. Check
the endpoint types you are working with as your replication targets. You can
select one or more endpoint endpoints as necessary. If you need to work
with all available sources, click All.
4. Click Send by Mail to open an email request for the license. Send the email to
the address entered in the recipient field of your default email client.
Note If you have not registered a default email client, clicking the Send by
Mail button will not open your email client. For instructions on registering a
default email client, refer to your browser's or operating system's online help.
Click Copy to Clipboard to copy the information to the computer’s clipboard. You
can paste this information into the Advanced license request and edit it as
necessary. For more information, see Using the Advanced License Request Option.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1092
Request type:New License
Expiration date:
Hosts: bee01-xp.company.local
Registering a License
You must have a valid license to work with Attunity Replicate. If you did not receive a
license.txt file, you can request a license using the procedure described in Requesting a
License. Once you receive the license, you must register it to work with Attunity Replicate.
To register a license:
1. Copy the license.txt file to your computer or any computer in your network
you have access to.
2. From the Server page, click License.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1093
4. Click Load and browse to find and select the license file.
The license text is displayed in the dialog box as shown above. Check to be sure
that the details are correct.
5. Click Register License to register the license. A message indicating the license
was registered successfully is displayed.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1094
Note A message is displayed at the top of the Attunity Replicate Console that
indicates that you have a valid license and when it expires. If the license is
expired or invalid, the message indicates this.
You can also click on this message link to request, register, or view
license information.
Viewing a License
You can view the license information in the Attunity Replicate Console at any time.
The License tab is displayed. All of the license information is displayed in the
License tab.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1095
Global Error Handling
You can configure how Attunity Replicate responds to specific types of errors. You can
define error handling on the task level or the server level. The configurations you make
in the Server Settings affect all tasks created for this instance of Attunity Replicate
unless you define a task to use the definitions you create for that task. For information
on how to configure error handling for a specific task, see Error Handling in the
Customizing Tasks chapter.
From the Server view, click Global Error Handling from the menu list on the left.
The information you enter in this tab is the same as the information you enter in
the Environmental Errors tab for tasks. For information about the options
available in this tab, see Environmental Errors.
Data Error: An error related to data processing at the record level.
The information you enter in this tab is the same as the information you enter in
the Data Error tab for tasks. For information about the options available in this
tab, see Data Errors in the Customizing Tasks chapter.
Table Error: An error in processing data or metadata for a specific table. This
only includes general table data and not an error that relates to a specific record.
The information you enter in this tab is the same as the information you enter in
the Table Error tab for tasks. For information about the options available in this
tab, see Table Errors in the Customizing Tasks chapter.
Apply Conflicts: Errors that occur when the target endpoint is not synchronized
with the source endpoint when processing changes. This can cause duplicate key
errors on INSERT operations or zero rows affected on UPDATE/DELETE operations.
The information you enter in this tab is the same as the information you enter in
the Apply Conflicts tab for tasks. For information about the options available in
this tab, see Apply Conflicts in the Customizing Tasks chapter.
Logging
The following topics describe the server logging management options:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1096
Setting Logging Levels for the Server and File Transfer
Service Setting Automatic Roll Over and Cleanup Viewing
and Downloading Log Files Deleting Server, Task and FTS
Log Files
Setting Logging Levels for the Server and File Transfer Service
You set the logging level for the Replicate Serverlogs and File Transfer Service logs in
Server view. The level you set determines what information is written to the logs. The
Server logs provide information about the Attunity Replicate Server instance you are
working with as opposed to individual tasks. For information on configuring the task
logs, see Setting the Task Logging Level.
The following logging levels are available, ordered from the lowest level to the highest:
1. Errors
2. Warnings
3. Info
4. Trace
5. Verbose
The higher levels always include the messages from the lower levels. Therefore, if you
select Error, only error messages are written to the log. However, if you select Info,
informational messages, warnings, and error messages are included. Selecting
Verbose writes all possible messages to the log.
You can set a global logging level for all components or you can set a separate
logging level for each component.
1. On the left side of the Server view, click Logging and then click the Server
Logging Levels or File Transfer Service Logging Levels sub-tab as required.
The Component Logging Level sliders are displayed.
2. To set a global logging level, move the top slider (the slider with the labels) to the
log level you want. Note that all of the sliders for the individual modules move to
the same position that you set in the main slider.
3. Make any changes to the sliders for the individual modules. This is optional. Note
that if you change the main slider, all of the individual sliders are reset to the new
position. If you want to maintain a different logging level for a specific module, you
need to reset it.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1097
4. Click Save at the top of the screen. Changes to the logging level take place
immediately. There is no need to restart the Attunity Replicate Server.
Note Click Save to save the information for all settings, not only for the
logging settings. If you made changes that you do not want to keep, click
Discard Changes to discard all changes before you make changes to any of
the other settings.
Running in "Trace" or "Verbose" logging mode will quickly use up available disk
space (unless the logging settings have been configured to prevent this).
Continually writing large amounts of data to the logs will affect performance.
Automatic Rollover
You can determine when to stop logging to the current log file and begin to log to a new log
file. Rolled over log files are appended with a 12-digit timestamp. The current server log
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1098
file is named repserv while saved (older) server log files are named
repserv_ xxxxxxxxxxxx.
The current running task log file is named reptask_<task_name> while saved (older)
task log files are named reptask_<task_name>_xxxxxxxxxxxx.
For both server and task saved log file names, xxxxxxxxxxxx represents a 12-
digit timestamp.
Roll over the log if the log file is older than (days): Select the check box and
then specify the maximum number of days the current log file is allowed to exist
before being rolled over.
The default value is 7 days.
Roll over the log if the log file is larger than (MB): Select the check box and
then specify the maximum number of megabytes the current log file is allowed to
reach before being rolled over.
Notes
If you edit this setting while tasks are running, the new setting will not affect the
task log files until the tasks are stopped and then resumed. The server log files
are not affected by this limitation.
The scheduled process (LogFileCleanLogs) that checks the log file size runs
every five minutes. Consequently, the actual size/age of the rolled over log file
may deviate slightly from the specified value(s).
Automatic Cleanup
You can determine the maximum number of days old log files (i.e. log files that have
been rolled over) are retained before being deleted.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1099
Delete log files that are older than (days): Select the check box and then
specify the maximum number of days to retain a saved log file. Log files that are
older than the specified number of days will be automatically deleted from the
system. For example, if you specify 4, then on the fifth day, any log file older than 4
days will be deleted.
The default value is 45 days.
1. Select the Server Logging Levels or File Transfer Service Logging Level sub-
tab as required.
2. Click the Log Viewer toolbar button.
3. Continue from step 4 in Viewing and Downloading the Task Log Files.
3. Select the log file without a timestamp and then click the Roll Log File toolbar
button in the top right of the window.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1100
To delete the log files:
3. Select which logs to delete and, for each log, optionally change the default number
of days (45).
4. Click Delete.
Selected logs older than the specified number of days will be immediately deleted.
How it Works
A solution using FTS consists of two Attunity Replicate Servers: A local Attunity
Replicate Server installed on the source endpoint LAN and a remote Attunity Replicate
Server installed on the target endpoint LAN.
A local task on the local server is defined from the source endpoint to a File Channel
target. A remote task on the remote Attunity Replicate Server is defined from a File
Channel source to the target endpoint.
The FTS runs on the remote Attunity Replicate Server only and transfers the File
Channel files from the storage location defined in the local task to the storage location
defined in the remote task.
Upon file transfer, and before Compression and Encryption, large files are split into
smaller blocks which form recoverable transport units, and small files are merged into
bigger blocks to be sent at the same time. The blocks are then transferred and
reconstructed into File Channel files when received by the FTS server.
For information on setting up a File Channel source or target to use FTS, see Using
Advanced Properties for a File-Channel Source and Setting Advanced Connection
Properties respectively.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1101
Compression
File Channel files are compressed upon sending using GZIP. You can disable
the compression and control the compression level.
Encryption
After compression, File Channel files are encrypted using a randomly generated AES-
256 session key. The session key is exchanged between the client and server using the
Diffie-Hellman key exchange protocol which is authenticated using a secret key that is
shared between the client and the server.
Note The File Transfer Service should be configured on the remote Attunity
Replicate Server only.
2. In the left side of the Server view, click File Transfer Service.
4. Edit the values in the Name, Host and Port columns as follows:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1102
5. Click Save to save your settings.
2. Edit the values in the Name, Host and Port columns as follows:
Note When you edit a File Transfer Service, make sure that any File Channel
targets configured to use the File Transfer Service are also updated
accordingly. For more information on File Channel Targets, see Setting
General Connection Properties.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1103
To delete a File Transfer Service:
1. In the File Transfer Services List, select the item you want to delete.
2. Click the Remove toolbar button.
Scheduling Jobs
Use the Attunity Replicate Scheduler to schedule a one-time job or a recurrent job for
specific task operations. A job is essentially an operation that can be scheduled to
occur once, daily, weekly or monthly.
2. In the left side of the Server view, click the Scheduler tab.
The Scheduler tab consists of two sub-tabs: Scheduled Jobs and Executed
Jobs. The Scheduled Jobs tab contains a list of jobs that are scheduled to run
periodically or once only while the Executed Jobs tab contains a list of jobs that
have already run.
Note The Executed Jobs tab will only show executed jobs that were scheduled
to run once only. In other words, jobs scheduled to run periodically (e.g. Daily,
Weekly, Monthly) will not be shown.
4. Specify a Job Name and then, from the Select scheduled job type drop-down
list, select one of the following:
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1104
Run task to run or resume the task(s) at the scheduled time.
Note For Full Load only tasks, it is preferable to select Reload target rather
than Run task when the scheduling is set to Daily, Weekly or Monthly. This
will update the table’s data whereas Run task will replace the existing table.
Stop task
Reload target
Note Selecting Reload target will execute the task according to the task's
replication settings. For example, if the task's Full Load and Apply
Changes options are enabled, Reload target will reload the target tables
and apply any subsequent changes.
Use server local time - When this option is selected (the default), the job
will run when the specified time is reached in the server's location.
See also: Impact of DST Change on Attunity Replicate.
Use universal time (UTC) - When this option is selected, the job will run at
the specified UTC time. So, for example, if the server is located in a UTC + 2
timezone, a job scheduled to run at 13:00 UTC time will actually run at 15:00
local server time. Scheduling a job to run in UTC mode may be useful if you
need tasks on several Replicate servers (located in different timezones) to run
concurrently.
Note For reference, both the server local time and the UTC time are displayed
to the right of the Scheduled Time heading.
Once (Run the job once on the specified day and at the specified time)
Daily - (Run the job every day at the specified time)
Weekly - (Run the job on the specified days and at the specified time)
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1105
Note To run the job on the last day of evey month, select Last day of
every month from the Day of month drop-down list.
7. For the Apply to tasks option, select which tasks to schedule. Select either All
tasks to apply the job to all current and future tasks or Selected tasks to apply the
job to specific tasks. If you choose Selected tasks, a list of currently defined tasks
is displayed. Select which tasks to apply the job to.
8. Click OK to save your settings.
In the Scheduled Jobs tab, select or clear the check box in the Enabled column
as required.
User Permissions
You can grant Attunity Replicate users different permissions according to the tasks you
want them to perform. Four predefined "roles" are available: Admin, Designer,
Operator and Viewer. Each role has its own set of permissions, which are described in
the following table.
Default User Permissions According to Roles
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1106
Permissions Admin Designer Operator Viewer
View all of the endpoint settings in the Manage Yes Yes Yes No
Endpoint Connections window.
Note The user under whose account Attunity Replicate is installed will be
associated with the Admin role by default.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1107
You can set user permissions using Active Directory Groups or Local Groups. To set
user permissions using Active Directory groups, you can either create Active Directory
groups with the names listed in the following table or you can create Active Directory
groups with different names. Then, add users to the groups according to the role you
want them to perform.
Table 15.4 |
Roles and Their Corresponding Active Directory Names
Administrator AttunityReplicateAdmins
Designer AttunityReplicateDesigner
Operator AttunityReplicateOperator
Viewer AttunityReplicateViewer
For information on encrypting user permissions, see Encrypting the User Permissions File.
Note If you create Active Directory groups with different names, you need to add
them to the User Permissions window and set their permissions as described in
Managing User Permissions.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1108
4. Select User or Group as appropriate.
3. Select the user/group you want to remove and then click the Delete toolbar
Resource Control
You can set high and critical disk space and memory utilization thresholds.
Note Disk space is checked only for the drive where the data folder resides.
Thresholds are calculated as a percentage of total capacity. So, for example, a disk space
utilization threshold of 80% would mean that 20% of available disk space remains.
After setting the thresholds, you can click the New Notification button to define a
notification that will be sent whenever a given threshold is exceeded and/or returns
to normal.
Disk Space
In the High Disk Space Utilization Threshold section, specify the high disk space
utilization threshold (in terms of percentage). When the threshold is reached, a notification
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1109
will be sent (if defined).
In the Critical Disk Space Utilization Threshold section, specify the critical disk
space utilization threshold (in terms of percentage). When the threshold is reached, all
tasks will be stopped and a notification will be sent (if enabled). Replicate will resume
the tasks automatically when there is sufficient disk space to do so.
System Memory
Memory utilization is calculated using the following formula (note that “swap file” is used
generically to refer to both page file memory on Windows and swap file memory on Linux):
(used_swap_file + used_physical_memory) /
(total_swap_file + total_physical_memory) * 100
Example:
(5 GB + 5 GB) / (10 GB + 10 GB) * 100 = 50%
In the High System Memory Utilization Threshold section, specify the high
system memory utilization threshold (in terms of percentage). When the threshold is
reached, a notification will be sent (if defined).
In the Critical System Memory Utilization Threshold section, specify the critical
system memory utilization threshold (in terms of percentage). When the threshold is
reached, Replicate will start stopping tasks and a notification will be sent (if enabled).
The tasks will be resumed automatically when there is sufficient memory to do so.
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1110
Copyright © Attunity Ltd. Chapter 15 | Attunity Replicate Server Settings | Page 1111
A | Using Change Tables
You can use Attunity Replicate tasks to save the change events in change tables.
This section describes how to use Attunity Replicate with change tables.
In this appendix:
▶ Working with Change Tables
▶ Reading the Change Tables
▶ Use Example
To apply TRUNCATE operations to both the Change Table and the Target Table
(for sources that support TRUNCATE):
1. In the task settings' Store Changes Settings tab, make sure that Apply to
change table (the default) is selected from the DDL options drop-down list.
2. In the task settings' Apply Changes Settings tab, make sure that TRUNCATE target
table (the default) is selected from the When source table is truncated drop-
down list.
The target and change tables must be in the same endpoint, although they can have
different schemas. For example, the Change Tables will contain the metadata headers.
For further details about changing the schema, see Rename Change Table Schema .
Changes applied to the Change Table will be handled exactly the same as the
changes performed in the corresponding transaction in the source database.
Therefore, when using Transactional apply mode or Batch optimized apply
mode with the Preserve transaction consistency option selected, the changes
will be processed as a single transaction.
The exception to this is when an error is encountered and Replicate switches to
"one-by-one" apply mode in order to determine which of the Change operations is
responsible for the error.
The same data columns are both applied and stored with the exception of the
change header columns, which are only added to the stored change tables.
Change Tables
For every target table in the replication task, a change table with the corresponding name is
maintained by Attunity Replicate in the endpoint with the target tables. For more information,
see Working with Change Tables. A change table contains the original table columns, and
header columns. The header columns contain a prefix so that the name does not conflict with
the source table column names. The default prefix is header__. For information on how to
change this prefix, see the Change tables listing under Metadata in Task Settings. The
following table lists the default change table header columns.
YYYYMMDDHHmmSShhxxxxxxxxxxxxxxxxxxx
Where:
[header_ varchar The operation type. This can be one of the following:
_]change_ (1) I: INSERT
oper
D: DELETE
U: UPDATE
B: Before Image
[header__] varbinary The change mask indicates which data columns in the change
change_mask (128) table are associated with columns that changed in the source
table.
[header__] varchar The operation associated with the change record. It can be
operation (12) one of the following:
INSERT
UPDATE
DELETE
BEFOREIMAGE
[header__] varchar The ID of the transaction that the change record belongs to.
transaction_ (32) The value is a hex-string of the 128-bit transaction ID.
id
[header__] timestamp The original change UTC timestamp (the value may be
timestamp approximate).
Note: With PostgreSQL source, the timestamp is only known
after the commit occurs. Therefore, until the changes are
committed to the source tables, the default date will be
displayed (e.g. 1970-01-01).
[header__] string The name of the partition created on the target when Change
partition_ Data Partitioning is enabled. The partition name consists of
name the partition start and end time.
Example:
20170313T123000_20170313T170000
Use Example
The following SQL statement returns a range of changes that starts after the last
handled event (of change sequence "20120723144522110000000000000901203") and
until the event committed on 23-Jul-2012 at 23:00:00.00. For update we also get the
before image value (here for the salary before and after).
SELECT CHANGE.[header__change_seq]
,CHANGE.[header__stream_position]
,CHANGE.[header__transaction_id]
,CHANGE.[header__timestamp]
,CHANGE.[EMPLOYEE_ID]
,CHANGE.[FIRST_NAME]
,CHANGE.[LAST_NAME]
,CHANGE.[SALARY]
,BI.[SALARY]
JOIN [Replication].[HR].[EMPLOYEES_ct] BI ON
BI.[header__change_seq] = CHANGE.[header$__change_seq]
CHANGE.[header__stream_position] >
'20120723144522110000000000000901203' AND
CHANGE.[header__stream_position] <= '2012072323000000Z'
AND ORDER BY
CHANGE.[header__stream_position], CHANGE.[header$__stream_oper]
Table B.1 |
Audit Table Headers
Column Type Description
Name
task_name varchar The name of the Attunity Replicate replication task.
(128)
Where:
change_oper varchar The operation type. This can be one of the following:
(1) I: INSERT
D: DELETE
U: UPDATE
B: Before Image
operation varchar The operation associated with the change record. It can be
(12) one of the following:
INSERT
UPDATE
DELETE
BEFOREIMAGE
transaction_ varchar The ID of the transaction that the change record belongs to.
id (32) The value is a hex-string of the 128-bit transaction ID.
1. Create a new environment variable with the following name and value:
Variable: AREP_CRASH_DUMP_TYPE
In this appendix:
▶ Required Pivotal Greenplum Software Environments
▶ Required Pivotal Greenplum Configuration and Environment
▶ Troubleshooting gpfdist Issues
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1125
Greenplum Connectivity Tools 4.2.1.0. Unzip and install all of the components.
Install and configure UnixODBC.
Ensure that the following Pivotal Greenplum environment scripts are executed
when logging to the account where Attunity Replicate is run:
greenplum_connectivity_path.sh
greenplum_clients_path.sh
greenplum_loaders_path.sh
Make sure that port 8080 is open.
Because gpfdist is a Web server program and because it needs to be accessible from
the Pivotal Greenplum database segment nodes, there are some networking
configuration settings that must be in place to allow for this access. This is
documented in the EMC Greenplum database Administration Guide.
The following sections provide a simple test that verifies the proper configuration of the
Pivotal Greenplum database and the local software installation. You can run this test before
installing any Attunity software on your local computer. This test must be completed
successfully to ensure that Attunity Replicate can work with a Pivotal Greenplum database.
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1126
<<Pivotal Greenplum-host>>: The Pivotal Greenplum database host name (master).
Tiger,Woods
LOCATION ('gpfdist://<<gpfdist-host>>:<<gpfdist-
port>>/greenplum_ test123.txt')
FORMAT 'TEXT' (DELIMITER ',');
Start gpfdist
Open a command shell and change the current directory to the directory where you
created the Test Input File and the SQL Script File. Then start the gpfdist program with
the following command:
$ gpfdist -v -p <<gpfdist-port>> -d .
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1127
Run the SQL Script
Run the SQL script that you created with the following command:
$ psql -d <<Pivotal Greenplum-database>> -h <<Pivotal Greenplum-host>> -
p <<Pivotal Greenplum-port>> -U <<Pivotal Greenplum-user>> -f greenplum_
test123.sql
If the script runs successfully, the following is displayed:
CREATE EXTERNAL TABLE
name | descr
-------+-----------
Tiger | Woods
(2 rows)
If the script is not successful, you will get an output that is similar to the
following example:
CREATE EXTERNAL TABLE
In the example above, the problem was that a firewall on the local computer prevented
the Pivotal Greenplum database segment from reaching the local gpfdist instance.
Any error that occurs must be resolved before using Attunity Replicate with EMC
Pivotal Greenplum. Once you resolve the error, you should run this test again to
ensure that you can work with the Pivotal Greenplum database.
For information about what to do if this test fails, see Troubleshooting gpfdist Issues.
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1128
Troubleshooting gpfdist Issues
If the test described in Required Pivotal Greenplum Configuration and Environment
fails, you should carry out the following checks:
Did gpfdist start on the correct port or protocol?
Can Pivotal Greenplum reach gpfdist?
For Linux:
$ netstat -a -n | grep "<<gpfdist-port>>"
The following is the output that you should get if gpfdist started on port 8080 on Windows:
$ netstat -a -n | find "8080"
This indicates that gpfdist is listening on any network interface on both IPv4
(0.0.0.0:8080) and IPv6 ([::]:8080).
If only the IPv6 line is shown in most cases there is a local networking configuration
problem. Pivotal Greenplum's recommendation in this case is to disable IPv6 on the
local computer (see the Microsoft knowledge base article on how to carry this out at
http://support.microsoft.com/kb/929852).
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1129
This should return a status page with the following or similar content:
read_bytes 0
total_bytes 0
total_sessions 0
When carrying out the network checks from the Pivotal Greenplum internal master
node, there is usually a file called seg_host that contains a list of the Pivotal
Greenplum database segment node names. If this file exists, you can check access
from all segment nodes using a single Pivotal Greenplum gpssh command:
$ gpssh -f seg_host
If this check fails, a network or system manager must change the network or
system configuration so that the check succeeds.
Copyright © Attunity Ltd. D | Pivotal Greenplum Prerequisites for Attunity Replicate | Page 1130
E | Setting up Attunity Replicate in
a Cluster Environment
This section describes how to set up Attunity Replicate in Windows and Linux
clustering environments.
In this appendix:
▶ Setting up Attunity Replicate in a Windows Server Cluster (HA)
▶ Setting up Attunity Replicate in a Linux Cluster
C:\Program Files\Attunity\Replicate.
In the setup wizard, specify a shared storage location for the "data" folder;
for example, F:\Replicate\data.
For more information, see Installing or Upgrading Attunity Replicate on Windows.
Copyright © Attunity Ltd. E | Setting up Attunity Replicate in a Cluster Environment | Page 1131
2. Open a command prompt as an administrator and change the working directory to
the Replicate bin directory. Then run the following two commands, where
Command 1 should only be run on the first cluster node:
Command 1:
repctl -d "replicate_data_folder" setmasterkey new_master_key
master_ key_scope=1
Command 2:
RepUiCtl -d "replicate_data_folder" masterukey set -p
new_master_user_ key
where:
Important: The setmasterkey command overrides the mk.dat file, rendering all
stored secrets invalid (as they were encrypted using the old key). Therefore, after
changing the master key, you need to reenter the passwords in all of the relevant
places. For more information, see Changing and Protecting the Master Key.
Additionally, because the Replicate server password is also stored in the mk.dat
file, you need to reset it as described in Changing the Server Password.
4. Now move your cluster to the next cluster node and install Attunity Replicate as
described in Steps 1-3 above, making sure only to run the second command in Step 2.
Copyright © Attunity Ltd. E | Setting up Attunity Replicate in a Cluster Environment | Page 1132
Step 2: Add the Attunity Replicate Services
The two Attunity Replicate services must be added as resources to the service
(Windows Server 2008 Cluster) or role (Windows Server 2012\2016 Cluster).
The Attunity Replicate services are called Attunity Replicate UI Server and
Attunity Replicate Server.
Windows Server 2008 Cluster: Right-click the service you are working
with and point to Add a resource. Then select Generic Service.
Windows Server 2012\2016 Cluster: In the left pane of the Failover
Cluster Manager, select Roles. The available roles will be listed in the right
pane of the console. Right-click the role you are working with and point to
Add a resource. Then select Generic Service.
2. In the Select Service screen of the New Resource wizard, select Attunity
Replicate UI Server from the List.
3. Click Next and follow the directions in the wizard to create the resource.
For information on how to use this wizard, see the Microsoft online help.
4. Repeat the same steps for the Attunity Replicate Server.
Note Attunity Replicate must be installed on the computers where you defined
the service for the Attunity Replicate services to be available in the list.
Copyright © Attunity Ltd. E | Setting up Attunity Replicate in a Cluster Environment | Page 1133
Windows Server 2012\2016 Cluster: In the left pane of the Failover
Cluster Manager console, select Roles. The available roles will be listed in the
right pane of the console. Select the role you are working with and then, in the
bottom right pane, select the Resource tab. From the list of the available
roles, select Attunity Replicate UI Server.
2. Do one of the following (according to your Windows Server version):
9. Register the license. The license should contain all host names of the cluster.
http://cluster_name_ip/attunityreplicate/5.5.118/#
Copyright © Attunity Ltd. E | Setting up Attunity Replicate in a Cluster Environment | Page 1134
installation, you do not need to change the names of these files. However, in a High
Availability environment in which Attunity Replicate Console is installed on two different
machines, you may want each machine to have its own unique settings. In such a
situation, changing the name of these files to include the hostname of the Attunity
Replicate Console machine (as it appears in the output of the Windows hostname
command) will allow you to store a different configuration for each machine.
The file name should be formatted as follows: [Name]Configuration-[hostname].xml
For example, let’s assume that one Attunity Replicate Console is installed on a machine
called replicate-main and the other Attunity Replicate Console is installed on a machine
called replicate-failover. To set up a different configuration for each machine, simply
create two copies of the ServiceConfiguration.xml file and then rename them to
ServiceConfiguration-replicate-main.xml and ServiceConfiguration-
replicate-failover.xml accordingly. After renaming the files, edit each of them with
your desired settings.
Attunity Replicate data (the data folder, tasks folder, file channel, etc.) should be
stored in a SAN for shared access between the cluster nodes. To change the
default location of the Attunity Replicate data folder, run the following command
on the primary cluster node when the installation completes:
./repctl-d <shared_storage_path> service start
Only one instance of Replicate can be active at a given data location. The cluster
software should be set so that during failover, one Replicate instance is stopped
and the other is started.
Copyright © Attunity Ltd. E | Setting up Attunity Replicate in a Cluster Environment | Page 1135
F | Control Tables
This section describes the Attunity Replicate Control Tables that are created on the
target endpoint when the corresponding table is selected in the Control Tables tab.
Notes
Control tables are not created in Full Load only replication tasks.
Setting Replicate to DROP and CREATE the table if the target table already
exists will not affect Control Tables.
Replicate automatically creates the Control tables on the target during run-time.
If you prefer to manually create the tables, make sure that all column names
and table names are lower case, apart from the Apply Exceptions table (attrep_
apply_exceptions) which should be upper case.
In this appendix:
▶ Apply Exceptions
▶ Replication Status
▶ Suspended Tables
▶ Replication History
▶ Change Data Partitions
▶ DDL History
Apply Exceptions
Change Processing errors are recorded in the attrep_apply_exceptions table, which
is described below.
The data in this table is never deleted.
STATEMENT nvchar The statement that was being executed when the error
occurred.
Replication Status
The attrep_status table contains the current status of each replication task and the
target data. Although updates for the tasks in the attrep_status table are generated
every few seconds, Replicate will only apply the updates after it has applied changes to
the target tables. In some cases, this may take a few minutes.
DISK_ int The amount of disk space that is occupied by old or offloaded
SWAP_SIZE transactions.
SOURCE_ varchar The POSITION in the source endpoint that Attunity Replicate is
CURRENT_ currently reading from.
POSITION
SOURCE_ timestamp The TIMESTAMP in the source from which Attunity Replicate is
CURRENT_ currently reading.
TIMESTAMP
Note Replicate aligns the timestamp read from the source
with the timestamp on the Replicate Server machine
(according to the source latency). So, for example, if the
time on the source was 12:30 and the time on the Replicate
Server machine is 12:32, Replicate will add two minutes to
the original source time. This will be the value in the
SOURCE_CURRENT_TIMESTAMP column.
SOURCE_ The POSITION of the oldest start transaction that is still not
TAIL_ committed. This represents the newest position that you can
varchar
POSITION revert to, without losing any changes. There may, of course,
be duplicates.
SOURCE_ timestamp The TIMESTAMP of the oldest start transaction that is still not
TAIL_ committed. This represents the newest TIMESTAMP that you
TIMESTAMP can revert to, without losing any changes. There may, of
course, be duplicates.
Table F.3 |
attrep_suspended_tables Table
Column Type Description
SERVER_NAME nvchar The name of the machine on which Attunity Replicate is
installed.
SUSPEND_ timestamp The date and time the table was suspended.
TIMESTAMP
Replication History
The attrep_history table provides statistics about each task, such as the number
and volume of records processed during a particular timeslot.
A new record is appended to the table at the end of each TIMESLOT_DURATION. In
other words, the data in this table is never deleted.
TIMESLOT_ int The latency at the end of the time slot. This is only applicable to
LATENCY CDC time slots. Note that this value contains the value of the
target latency only.
TIMESLOT_ int The number of records processed during the time slot.
RECORDS
Table F.5 |
attrep_cdc_partitions Table
Column Type Description
SERVER_NAME STRING The name of the machine on which Attunity Replicate
is installed.
Example:
20170313T123000_20170313T170000
2017-03-13 12:30:00.000
PARTITION_END_ TIMESTAMP When the partition was closed:
TIME
Example:
2017-03-13 17:00:00.000
DDL History
The attrep_ddl_history table contains a history of DDL changes that occurred in
the source during replication to the target.
Note Currently, the DDL History table is only supported with the Hadoop
target endpoint.
A new record is inserted into the table whenever a supported DDL change occurs in the
source. Multiple ALTER TABLE statements that occur during a task may be represented
as a single row in the control table. The JSON buffer (see below) describes all the
changes that occurred (e.g. ADD COLUMN A, DROP COLUMN B, ALTER COLUMN C).
For information on enabling the DDL History Control Table as well as its limitations,
see Control Tables.
Note When the Apply Changes task option is enabled, an entry is created for the
base table (e.g. tblT1). If the Store Changes task option is also enabled, an
additional entry is created for the CT table (e.g. tblT1__CT).
TABLE_ STRING The source table name. If the table was renamed, this will be
NAME the table name before the change.
TABLE_ INTEGER Replicate assigns an internal version number to the table. The
VERSION version number increases whenever a DDL change occurs in
the source table.
Example:
{
"owner": "string",
"table": "string",
"tableNewName": "string",
"version": number,
"columns":
{
"col1":
{
"columnNewName": "string",
"action": "string",
"type":" string",
"length": number,
"precision": number,
"scale": number,
"primaryKeyPosition": number,
"nullable": boolean,
"ordinal": number
},
"col2":
{
…
}
}
}
where:
Note that when replicating to an HP NonStop SQL/MP target, you must select SQLMP
(ARC) as the provider type. For information on how to select a provider type as well as
a detailed explanation of how to define HP NonStop SQL/MP as an ODBC target in a
Replicate task, see Using ODBC to Connect to a Target.
In this appendix:
▶ Prerequisites
▶ Table Settings
▶ Task Setting Limitations
Prerequisites
Before designing a task with HP NonStop SQL/MP as an ODBC Target, the
following prerequisites must be met:
All source tables and their indexes must already exist on the target database.
The default name for the Replicate Apply Exceptions table (attrep_apply_
exceptions) is not compliant with HP NonStop SQL/MP table naming conventions
which dictate that table names cannot exceed 8 characters or contain non-
alphanumeric characters. To resolve this conflict, you need to add the following
entry to your NAVMAP file or create a new NAVMAP file (The NAVMAP file is located
in the subvolume where ARC is installed):
Syntax:
[SQLMP]
attrep_apply_exceptions=
\machine_name.$volume_name.subvolume_ name.atreapex
Copyright © Attunity Ltd. Appendix G | Using HP NonStop SQL/MP as an ODBC Target | Page 1145
You also need to manually create the Apply Exceptions table on the HP
NonStop SQL/MP target before starting the task. The table should be named
atreapex and should contain the following columns:
TABLE_OWNER Varchar(128)
TABLE_NAME Varchar(128)
STATEMENT Varchar(1000)
ERROR Varchar(1000)
If you intend to enable the Replicate Store Changes option, you need to manually
create the audit table on the HP NonStop SQL/MP target before starting the task.
Note This also requires you make the following changes in the Store
Changes Settings tab:
Select Audit table from the Store changes drop-down list.
*stream_position Varchar(128)
change_seq Varchar(35)
change_oper Varchar(1)
schema_name Varchar(128)
table_name Varchar(128)
operation Varchar(12)
Copyright © Attunity Ltd. Appendix G | Using HP NonStop SQL/MP as an ODBC Target | Page 1146
Column Name Data Type
transaction_id Varchar(32)
timestamp Timestamp
change_record Varchar(1000)
bu_change_record Varchar(1000)
Note : You also need to create a Unique Index consisting of the task_name
and stream_position columns (marked with an asterisk above).
Table Settings
When the source database in a Replicate task is not HP NonStop SQL/MP, you must
make sure that the selected source tables comply with HP NonStop SQL/MP
conventions and limitations. This section describes how to apply transformations to
source tables that are not HP NonStop SQL/MP-compliant.
The maximum size of a Unique Index in HP NonStop SQL/MP cannot exceed 240
bytes. In the Transform tab of the <Table_Name> Table Settings dialog box, you
can check which columns comprise the Unique Index and change them if necessary
(i.e. if their combined Type value exceeds 240 bytes). For an explanation of how to
do this, see Using the Transform Tab.
HP NonStop SQL/MP does not support UPDATE operations on Primary Key columns.
Therefore, if your application updates source tables columns that are part of a
Primary Key, you will need to create the target table with a Unique Index on those
columns instead of a Primary Key. For an explanation of how to do this, see Using
the Transform Tab.
Valid HP NonStop SQL/MP table names cannot exceed 8 characters or contain
non-alphanumeric characters. If any of the source table names are not HP
NonStop SQL/MP-compliant, you will need to map them to a valid name.
To do this:
Copyright © Attunity Ltd. Appendix G | Using HP NonStop SQL/MP as an ODBC Target | Page 1147
Task Setting Limitations
When defining your task settings, the following limitations apply:
Metadata:
Target Metadata: As HP NonStop SQL/MP does not support LOB data types,
the setting in this tab are not relevant.
Control Tables: None of the optional tables are supported. The Apply
Exceptions table, which is required, should be configured as described in
Prerequisites above.
Full Load:
Change Processing:
Store Changes Settings: Change tables are not supported. If you want
Replicate to store captured changes on HP NonStop SQL/MP, choose Audit
table from the Store changes in drop-down list. This also requires you to
manually create the Audit Table on HP NonStop SQL/MP before starting the
task, as described in Prerequisites above.
After creating the Audit table, specify its name in the Audit table name
field (instead of the default name).
Audit table creation options:
As the Audit Table is created manually, the DROP and CREATE audit
table option is not supported.
Change Processing Tuning: Only "Transactional apply" Change
processing mode is supported.
Error Handling: No limitations.
Logging: No limitations.
For a detailed description of Task Settings, see Task Settings.
Copyright © Attunity Ltd. Appendix G | Using HP NonStop SQL/MP as an ODBC Target | Page 1148
H | Impact of DST Change
on Attunity Replicate
This section describes how Attunity Replicate is affected by Daylight Saving Time
(DST) and provides guidelines for handling changes brought about by DST.
Running Attunity Replicate tasks do not depend on the timezone or DST for
correctly scanning and processing the transaction logs. Internally, Attunity Replicate
timers use UTC.
Still, there are several places where DST may have an effect:
1. Timestamps in logs and audit messages are in local time. As a result, when Winter
time starts, the logs will show the time going back an hour; conversely, when
Summer time starts, the logs may appear to be missing one hour.
2. Scheduled jobs as well as the global and table manipulation variables timestamp
and commit_timestamp use local time so these will also be affected. The impact of
this depends on the manipulation done and on the intended use of the timestamp
based data.
Copyright © Attunity Ltd. Appendix H | Impact of DST Change on Attunity Replicate | Page 1149
twice.
DST On (winter to summer): Do not schedule a task to start from the time
the clock changes until the following hour. For example, if DST starts at
02:00 am, do not schedule a task to run between 02:00 and 02:59 as this
hour does not exist.
If you have existing jobs scheduled to start at the overlap time and you do not want
to modify them, then you need to stop the Attunity Replicate Server. Going in to
Winter time, for example, if at 02:00 AM the clock is to be set back to 01:00 AM then
when the time is 00:55 AM the Attunity Replicate Server should be stopped and, after
an hour and ten minutes (at 01:05 AM), should be started again.
If you forget to do this, all scheduled jobs will run an hour earlier than
intended. You can rectify this by setting the desired scheduling time and then
restarting the Attunity Replicate Server service.
3. Statistics shown on the console are also sensitive to local time and thus may also
show confusing/inaccurate data in the overlap period (going in to Winter time) or for
the skipped period (going into Summer time).
4. If the clock on Attunity Replicate Server machine is one hour behind the clock on the
Attunity Replicate Console (UI) machine, the following issues are known to occur:
The Applied Changes circle graph will be updated as the changes are applied,
but the information in the Recent Activity tab will not be updated.
Scheduled jobs will start according to the Attunity Replicate Server time
(as expected), but will remain in the Active Jobs list after execution
instead of moving to the Expired Jobs tab.
For more information on scheduling jobs, see Scheduling Jobs.
In general, it is recommended to avoid non-critical task design changes during the first
overlap period (going in to Winter time) so as to prevent confusion about when the
changes took place.
In addition to Attunity Replicate, other components are also involved including:
The source endpoint system
The target endpoint system
The local operating system
The task design (specifically using timestamp based variables)
Copyright © Attunity Ltd. Appendix H | Impact of DST Change on Attunity Replicate | Page 1150
Given the complexity of the topic and the involvement of many independent
components and settings, Attunity generally recommends that customers first verify the
impact of DST changes in their test environment.
Copyright © Attunity Ltd. Appendix H | Impact of DST Change on Attunity Replicate | Page 1151
I | Metadata File Description
When the Create metadata files in the target folder option in the File target endpoint
or Amazon S3 target endpoint is selected, for each CSV/JSON file Replicate creates a
corresponding metadata file under the specified target folder.
The metadata file offers several benefits such as enabling custom batch processes to
perform better validation, supporting deeper automation, offering lineage information
and improving processing reliability.
The metadata files (which are in standard JSON format) are described in the table below.
Field Description
Task Information
operation If a target data file has been created, this field will contain
the following value: dataProduced
File Information
extension The extension of the data file (.csv or.json according to the
selected target file format).
Format Information
options The options for delimited file format. These options will not
be shown for json format as they are not relevant.
recordDelimiter The delimiter used to separate records (rows) in the target
files. The default is newline (\n).
fieldDelimiter The delimiter used to separate fields (columns) in the target
files. The default is a comma.
nullValue The string used to indicate a null value in the target file.
quoteChar The character used at the beginning and end of a column. The
default is the double-quote character (").
escapeChar The character used to escape a string when both the string
and the column containing the string are enclosed in double
quotes. Note that the string’s quotation marks will be
removed unless they are escaped.
Custom Information
customInfo This section contains any custom parameters that were set
using the dfmCustomProperties internal parameter.
Parameter1=Value1;Parameter2=Value2;Parameter3=Valu
e3
Example:
Color=Blue;Size=Large;Season=Spring
Data Information
targetSchema The name of the target table schema (if the source schema
name was changed).
targetTable The name of the target table (if the source table name was
changed).
type The column data type. See File Target Data Types or Amazon
S3 Target Data Types for more information.
width The maximum size of the data (in bytes) permitted for the
column.
In this appendix:
▶ Supported Platforms
▶ Supported Source Endpoints
▶ Supported Target Endpoints
▶ Endpoints Supported in Bidirectional Replication
▶ Supported Browsers
Supported Platforms
Supported Windows Platforms
Attunity Replicate can be installed on any of the following Windows platforms:
Windows Server 2008 R2 (64-bit)
Windows Server 2012 (64-bit)
Windows Server 2012 R2 (64-bit)
Windows Server 2016 (64-bit)
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1156
Supported Source Endpoints
The table below lists the source endpoint versions supported by Attunity Replicate.
Cloud-Based
ARC-Based
Itanium:
H06.22/J06.14
8.3
Itanium:
8.3
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1157
Endpoint Version Windows Red SUSE
Hat Linux
Linux
IMS:
13 and 14
File
File N/A
Hadoop
Relational Databases
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1158
Endpoint Version Windows Red SUSE
Hat Linux
Linux
DB2:
10, 11 and 12
Percona 5.6.28
Data Warehouses
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1159
Endpoint Version Windows Red SUSE
Hat Linux
Linux
ODBC
Other
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1160
Supported Target Endpoints
The table below lists the target endpoint versions supported by Attunity Replicate.
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1161
Endpoint Version Windows Red SUSE
Hat Linux
Linux
Amazon S3 N/A
Streaming
MemSQL 6.0.18
File-Based
File N/A
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1162
Endpoint Version Windows Red SUSE
Hat Linux
Linux
Hadoop
Data Warehouses
Relational Databases
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1163
Endpoint Version Windows Red SUSE
Hat Linux
Linux
ODBC
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1164
Endpoints Supported in Bidirectional Replication
Bidirectional tasks support the following endpoints:
Source Endpoints:
Oracle
Microsoft SQL Server
MySQL
PostgreSQL
All AIS sources
File Channel
SAP Sybase ASE
IBM DB2 for iSeries
Amazon RDS for MySQL
Amazon RDS for PostgreSQL
Amazon RDS for SQL Server
Target Endpoints:
Oracle
Microsoft SQL Server
MySQL
PostgreSQL
ODBC
File Channel
SAP Sybase ASE
Supported Browsers
The following browsers are supported:
Note Displaying the console in a window that spans multiple vertical windows is
not supported.
Copyright © Attunity Ltd. Appendix J | Supported Platforms and Endpoints | Page 1165
K | Best Practices when
Working with Oracle ASM
This appendix provides detailed guidelines for configuring Attunity Replicate to work
with the Oracle source endpoint when the redo logs are stored in Oracle ASM.
When the redo logs are stored in Oracle ASM, it is recommended to use the Copy
redo logs to temporary folder option available in the Advanced tab. This option
is only available when the Attunity Log Reader redo log access method is selected.
Attunity Log Reader is Attunity's proprietary high speed redo log parser for parsing the Oracle
redo logs. Attunity Log Reader provides greatly improved performance and reduces the load
on the Oracle server when compared with other methods such as Oracle LogMiner.
In this appendix:
▶ The "Copy redo logs to temporary folder" Method
▶ Oracle Permissions Required for the Attunity Log Reader and the "Copy redo logs
to temporary folder" Options
▶ Permissions for Deleting the Processed Redo logs from the Temporary Folder
▶ Oracle ASM Access Permissions
▶ Setting up the File Share if the "Direct Access" option was chosen
▶ Configuring the "Copy to Temp Folder" Option in Replicate
▶ Additional Considerations
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1166
After the redo logs have been copied to the temporary folder, Replicate reads them
using one of the following methods:
Method 1: Using Oracle BFILE to read from files or partial files from Oracle
directory objects.
Method 2: Directly from the temporary folder after providing Replicate with access
to the temporary folder. Using this method, Replicate reads the physical files directly
from the folder.
Both options are significantly faster than accessing the redo logs directly from Oracle
ASM. However, the "Direct Access" option using a shared network folder is the fastest
and has the least impact on Oracle Server resources. This is because Replicate reads the
files directly from the shared folder, thereby eliminating the need for Oracle to send
Replicate the file content. However, using the "Direct Access" option requires some
additional configuration (as described below).
To prevent old redo logs from accumulating in the temporary folder, Replicate should
be configured to delete the redo log files from the temporary folder once they have
been processed. The delete operation is performed using Oracle file groups and the
Oracle DBMS_FILE_GROUP package.
This section describes only those permissions required for the Attunity Log Reader
and Copy redo logs to temporary folder options.
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1167
corresponding directories manually. The directories should point to the archived
redo logs and temporary folder paths. Do not append the 'attrep_' prefix to the
directory names as Replicate ignores such object directories.
If you create the directories manually and the Oracle user specified in the Oracle
Source endpoint is not the user that created the Oracle directories, grant the READ
ON DIRECTORY privilege as well.
If the Oracle user specified in the Oracle source endpoint is not the user that
created the Oracle directories, the following additional permissions are required:
READ on the Oracle directory object specified as the source directory (i.e. the
ASM archived redo logs path and the temporary folder in the event that the
BFILE method is used to read from the temporary folder)
WRITE on the directory object specified as the destination directory in the
copy process (i.e. the temporary folder).
Example:
Example:
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1168
performed using a function from the Oracle DBMS_DISKGROUP package, which
requires the SYSASM or SYSADM privileges.
From Oracle 11g Release 2 (11.2.0.2), Attunity Replicate must be granted the
SYSASM privilege in order to access the ASM account. For older supported versions,
granting Attunity Replicate the SYSDBA privilege should be sufficient.
You can also validate ASM account access by opening a command prompt and issuing
the following statements:
sqlplus asmuser/asmpassword@+asmserver as sysdba
-OR-
sqlplus asmuser/asmpassword@+asmserver as sysasm
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1169
3. To access the temporary folder using BFILE, select the Copy redo log files to
temporary folder check box and then specify the full path of the temporary
folder path in the designated field (e.g. /mnt/share).
4. To access the temporary folder directly (using a shared network folder):
a. Select the Copy redo log files to temporary folder check box and then
specify the full path of the temporary folder path in the designated field
(e.g. /mnt/share).
b. Select the Replicate has file level access to temporary folder check box.
c. Select the Access Archived Redo logs in folder check box and the specify
the share pointing to the temporary folder in the designated field. For example,
if the temporary folder /mnt/share is shared with the Replicate Linux
machine as /storage/ora_share, enter /storage/ora_share.
5. Select the Delete processed archived redo log files check box.
Additional Considerations
This section describes additional factors that should be taken into consideration
when working with Oracle ASM.
If the Oracle source database is part of a RAC cluster, the temporary folder must be
located on a file share that is accessible to all of the nodes in the Oracle Server RAC.
If you chose the "BFILE" method for accessing the temporary folder, the folder does
not need to be a shared network folder. In such as case, only authorized Oracle users
can access it and only through Oracle.
If you choose the "Direct Access" option, only the Oracle user and the "Remote"
Replicate user (NFS or Simba user) that will read the redo log should be granted
permission to access the folder.
Also, as the archived redo logs are deleted from the temporary folder after processing,
the risk of unauthorized access to the redo logs is greatly diminished.
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1170
conflicts, only one Replicate task should access the same temporary folder at any given
time. You can, however, create a different subfolder under the same root folder for each
task. Then, for example, instead of specifying /mnt/share as the temporary folder for
both tasks, you can specify /mnt/share/task1 for one task, and /mnt/share/task2
for the other.
Copyright © Attunity Ltd. Appendix K | Best Practices when Working with Oracle ASM | Page 1171
L | Replicate Loggers
This appendix provides a description of the following Replicate loggers:
ADDONS
ASSERTION
COMMON
COMMUNICATION
DATA_RECORD
DATA_STRUCTURE
FILE_FACTORY
FILE_TRANSFER (AKA CIFTA)
INFRASTRUCTURE
IO
METADATA_CHANGES
METADATA_MANAGER
PERFORMANCE
REST_SERVER
SERVER
SORTER
SORTER_STORAGE
SOURCE_CAPTURE
SOURCE_LOG_DUMP
SOURCE_UNLOAD
STREAM
STREAM_COMPONENT
TABLES_MANAGER
TARGET_APPLY
TARGET_LOAD
TASK_MANAGER
TRANSFORMATION
UTILITIES
ASSERTION
Not implemented at present.
COMMON
Writes low level messages such as network activity.
Note Not recommended to set to "Trace" as it will write a huge amount of data to
the log.
COMMUNICATION
Provides additional information about the communication between Replicate and the
Source and Target components. For example, when using Hadoop, it will print the CURL
debug messages and the "Apply" of the actual files (Mainly CURL and HTTP Client).
DATA_RECORD
Only available for some endpoints and may be implemented differently for each
endpoint. It writes information about each change that occurs. While in Oracle it writes
only the header fields, for some endpoints, such as Sybase ASE and IBM DB2 for LUW
sources, it will also include the changed data. It records when a specific event was
captured as well as the event context.
The content will not be presented; all events will be logged, even when a record is not
in the task scope. (This is not relevant for Oracle LogMiner, as the list of objects are
propagated to the LogMiner session).
Example 16 - Example
Produce INSERT event: object id 93685 context
'0000000003A4649301000001000005190000821F001000010000000003A46427' xid
FILE_FACTORY
Relevant to Hadoop Target, Amazon Redshift and Microsoft Azure SQL Data
Warehouse. This component is responsible for moving the files from Replicate to the
target which, in the case of Hadoop, is the HDFS stage of the task.
INFRASTRUCTURE
Records infrastructure information related to the infrastructure layers of Replicate code:
ODBC infrastructure, logger infrastructure, PROTO_BUF, REPOSITORY, FILES USE,
opening a new thread, closing a thread, saving the task state, and so on.
IO
Logs all IO operations (i.e. file operations), such as checking directory size,
creating directories, deleting directories, and so on.
Example 17 - Example:
[IO ]T: scanning 'E:\Program Files\Attunity\Replicate\data\tasks\Task_
name/data_files' directory size (at_dir.c:827)
METADATA_CHANGES
Will show the actual DDL changes which are included in the scope (available for
specific endpoints).
PERFORMANCE
Currently used for latency only. Logs latency values for source and target endpoints
every 30 seconds.
REST_SERVER
Handles all REST requests (API and UI). Also shows the interaction between Replicate
and Attunity Enterprise Manager.
SERVER
The server thread in the task that communicates with the Replicate Server service on
task start, stop, etc. Includes init functions for the task and the task definition.
SORTER
The main component in CDC that routes the changes captured from the source to
the target.
Responsible for:
Storing the transactions that arrive from the source database until they are
committed, and sending them to the target database in the correct order (i.e.
by commit time).
Whenever there is a CDC issue such as missing events, events not applied, or
unacceptable CDC latency, it is recommended to enable "Verbose" for this logger.
SORTER_STORAGE
SORTER_STORAGE is the storage component of the Sorter which stores transactions (i.e.
Changes) in memory and offloads them to disk when the transactions are too large, or
SOURCE_CAPTURE
This is main CDC component on the source side. As such, it should be used to
troubleshoot any CDC source issue. Note that setting to "Verbose" is not recommended
unless absolutely necessary as it will record an enormous amount of data to the log.
Some target side components that use the source (e.g. LOB lookup) also use this
logger. Setting the logger to “Trace” may be useful for troubleshooting performance
issues and other source CDC issues such as bad data read from the source CDC.
SOURCE_LOG_DUMP
When using Attunity Log Reader, this component creates additional files with dumps of
the read changes. The logger will write the actual record as it's being captured from the
source.
Note that the data will be stored in a separate file and not in the log itself.
SOURCE_UNLOAD
Records source activity related to Full load operations and includes the SELECT
statement executed against the source tables prior to Full Load.
STREAM
The Stream is the buffer in memory where data and control commands are kept. There are
two types of stream: Data streams and Control streams. In the Data stream, source data is
passed to the target or to the Sorter using this memory buffer. In the Control stream,
Commands such as Start Full Load and Stop Full Load are passed to components.
As it records a large amount of data, this logger should only be set to "Trace" when a
specific stream issue is encountered. Examples of stream issues include poor
performance, issues with Control commands (e.g. commands not being performed),
issues when loading many tables that may overload the control stream with Control
commands, and so on.
Example 18 - Example
Force switch to Transactional Apply mode for Hadoop target
(endpointshell.c:1340)
TABLES_MANAGER
Manage the table status including whether they were loaded into the target, the
number of events, how the tables are partitioned, and so on.
TARGET_APPLY
Determines which changes are applied to the target during CDC and is relevant to both the
Batch Optimized Apply and Transactional Apply methods. It provides information about all
Apply issues including missing events, bad data, and so on. As it usually does not record a
lot of data, it can be safely set to "Verbose" in order to troubleshoot issues.
TARGET_LOAD
Provides information about Full Load operations on the target side. Depending on
the target, it may also print the metadata of the target table.
TASK_MANAGER
This is the parent task component that manages the other components in the task.
It is responsible for issuing commands to start loading or finish loading a table, create
the different components threads, start or stop tasks, and so on.
It is useful for troubleshooting situations such as tables not loading, tables stuck in
loading, one of the components not stopping or starting properly, and so on.
Example 19 - Example:
In the example below, a new column named "C" was added to the table. The
expression is $AR_H_STREAM_POSITION.
[TRANSFORMATION ]T: Transformation on table USER3.TEST1 exists
(manipulation_manager.c:511)[TRANSFORMATION ]T: Set transformation for
table 'USER3.TEST1' (manipulator.c:596)[TRANSFORMATION ]T: New column 'C',
type: 'kAR_DATA_TYPE_STR' (manipulator.c:828)[TRANSFORMATION ]T:
Transformation expression is '$AR_H_STREAM_POSITION'
(manipulator.c:551) [TRANSFORMATION ]T: Final expression is
'$AR_H_STREAM_POSITION' (expression_calc.c:822)
UTILITIES
In most cases, UTILITIES logs issues related to notifications.
AR_ADDONS_STATUS_FAILED = 1,
In this appendix:
▶ Memory Management Methods
▶ Log Management Methods
create_pool
Creates a new memory pool.
Syntax
Parameters
destroy_pool
Deletes the given pool and deallocates the memory associated with it.
Note Using the clear_pool method (described below) will provide much
better performance than using the destroy_pool and create_pool methods.
Parameters
clear_pool
Deallocates the memory associated with a given pool.
Syntax
Parameters
calloc
Performs the following operations:
Syntax
Parameters
get_ctx_pool
Returns the ctx (context) thread pool (and creates a new pool if needed). Using ctx pool
per thread minimizes the chances of a crash as each addon will have its own pool
allocator. The pool will be destroyed automatically if a thread already exists.
Syntax
Parameters
get_ctx
Gets the metadata from the thread pool.
Syntax
Parameters
set_ctx
Sets metadata on the thread pool.
Important: The data to be attached to the pool should have a life span at least as
long as the pool to which it is being attached.
Syntax
Parameters
Notes
To see messages in the log, you also need to set the "ADDONS" component (in
the task's Log Settings) to the relevant logging level.
log_error
Writes errors to the log.
Syntax
Parameters
log_warning
Writes warnings to the log.
Syntax
Parameters
Syntax
Parameters
F T
Full Load Target Endpoint
Creates all defined files or tables at A collection of files or tables managed
the target endpoint, automatically by an Endpoint Management System
defines the metadata that is required (DBMS), which may be different from
at the tar-get, and populates the the DBMS managing the source end-
tables with data from the source. point. It contains data that is derived
from the source. It may contain only a
subset of the tables, columns, or rows
L
that appear in the source. Its tables
Latency may contain columns that do not
Latency can be understood as follows: appear in the source but are trans-
- Source Latency: The gap in seconds formations or computations based on
between the original change in the the source data.
source endpoint and capturing it. -
Tar-get Latency: The gap in seconds
between the original change in the
source endpoint and applying it to the
target endpoint. - Apply Latency: The
gap in seconds between capturing the
change in the source endpoint and
applying it to the target endpoint.
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1187
Index
A add databases
AIS 203
Accessing the console 100
source 134
Actian Vector 476
target 134
account access 478
add tables 136
data types 478
advance license request options
limitations 477
1092 advanced properties
Linux prerequisites 477
Amazon RDS for PostregSQL
security requirements 478
source 194
Windows prerequisites 477
Amazon Redshift target 518,
Active Directory groups 1108 862, 869
add database 132 file channel source 891
Amazon RDS for MySQL source 177 Hadoop source 487-488, 584-
Amazon Redshift target 516, 585, 595, 602, 621, 629,
859, 868 759-760, 879, 881
IBM Netezza target 283, 454 Kafka target 502, 666, 685, 745
Microsoft APS PDW target 703-704 Microsoft SQL Server source 331
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1188
AIS sources prerequisites 512
adding 203 Amazon Redshift registration 512
alerts 1039 Amazon Redshift requirements 512
alternate backup folder 332- apply changes task settings 1004
334 Amazon RDS apply changes tuning settings 1010
Oracle 393 apply global transformation rule
Amazon RDS for MySQL 168 941 ARCHIVELOG (Oracle source)
add a Amazon RDS for 390 Attunity Studio
MySQL source 177 create AIS endpoint 202
limitations 172
source data types 173 B
Amazon RDS for MySQL source
backup
data types 173
Amazon RDS for SQL Server 163
Amazon RDS for PostregSQL 181
Microsoft SQL Server 292, 326
advanced properties (source) 194
Bidirectional Tab 1000
Amazon RDS for PostregSQL
source data types 186
C
Amazon RDS for SQL Server
CDC
151 backup and recovery
monitoring 1051
163 data types
monitoring tables 1053
not supported 159
processes 50
limitations 154
change processing 50
source data types 156
monitoring 1051
Amazon RDS for SQL Server source
monitoring tables 1053
database configuration 162
change processing, query based
Amazon RDS for SQL Server
source data types 156 Teradata Database source
285, 368, 469
Amazon Redshift 511, 855, 864
change tables 1114
advanced properties (target)
518, 862, 869 change table model 1113
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1189
concepts 45 Hadoop source data types 226,
concepts and design 45 485, 583, 593, 619, 757
IBM DB2 for iSeries source
configuration
data types 240
Amazon RDS for SQL
IBM DB2 for LUW source
Server source 162
data types 251
Microsoft SQL Server source
IBM DB2 for z/OS source
291, 326
data types 265
configure log files 1098
IBM Informix source data types 277
configuring tasks 994
Microsoft Azure SQL Data
control tables Ware-house 769
creating on target 1000 Microsoft SQL Server
create a notification message not supported 300, 323
1076, 1085
Microsoft SQL Server source
create notifications 1069, 1082 data types 296, 320
creating expressions for Microsoft SQL Server target
transforms 919 data types 778, 787
customizing tasks 908 MySQL source data types 342
MySQL target data types 554,
D 695, 723, 798
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1190
Data Types 94 PostgreSQL 408, 561, 729,
database configuration 833 SAP Sybase ASE 458, 844
Amazon RDS for SQL SAP Sybase IQ 850 Teradata
Server source 162 Database 465, 871 view
Microsoft SQL Server source
information 133 working with
291, 326
131
databases 93
default recipient list 1088
Actian Vector 476
default SSL certificates,
adding 132
replacing the 79
Amazon RDS for MySQL 168
define changes that trigger the
Amazon RDS for PostregSQL noti-fication 1070
181 Amazon RDS for SQL define errors that trigger
Server 151 Amazon Redshift a notification 1072
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1191
Amazon RDS for PostregSQL 181 date and time functions
Amazon RDS for SQL Server 151 983 functions 972, 980
Amazon Redshift 511, 855, 864 global transformations 972
File 210, 521, 538, 568, 706 header columns 973-974
file channel 883 input columns 972
files 883 inputs 972
Hadoop 225, 581, 590, 617, LOBs 981
754 HP Vertica 450, 638, 840 logical operators 975
IBM DB2 for LUW 236, mathematical operators 979
247 IBM DB2 for z/OS metadata
256 IBM Informix 275 IBM expression builder
Netezza 282, 644 using variables 972
Kafka 492, 650, 675-676, null check functions 982
736 Microsoft APS PDW 701 numeric functions 981
Microsoft Azure SQL Data operations 989
Ware-house 765
operators 972, 974
Microsoft SQL Server 288,
other functions 989-990,
305, 775, 784
992 strings 975, 980 test
MySQL 337, 552, 694, 720, 795
expression 971
ODBC 350, 803
using elements 972
Oracle 809
expression builder for transforms
Pivotal Greenplum 819
and filters 967
architecture 821
PostgreSQL 408, 561, 729, F
833 SAP Sybase ASE 458, 844
File 210, 521, 538, 568, 706 file-
SAP Sybase IQ 850 Teradata
channel directory structure 886
Database 465, 871
file-channel files, working with 886
environmental errors settings 1015
file channel 883
error handling 1096
directory structure 886
error handling settings 1015
distribution 884
expression builder 967
file-channel files 886
build expression 969
directory structure 886 file-
data enrichment 985
channel files, working with 886
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1192
installation requirements 888 delete transformation rule 966
limitations 889 local tasks edit transformation rule 966
883 expression builder 967
remote tasks 883-884 new rule wizard 939
source 889 transformation rule
target 892 add column 957
tasks 883-884 convert data type 958
file channel source drop column 958
advanced properties 891 rename column 954
file transfer service rename schema 945, 950
settings 1101 rename table 950, 958, 962
files 883 transformation rules list 966
filters 921 transformation type 939
expression builder 967 gpfdist
range builder 926 as a service 827
SQLite syntax 928 multiple gpfdist programs 829
full load 50 troubleshooting 1129
monitoring 1043
monitoring tables 1045 H
full load task settings Hadoop 225, 581, 590, 617, 754
1001 full load tuning 1003 add as source 227
function advanced properties (source) 487-
data enrichment 985 488, 584-585, 595, 602, 621,
629, 759-760, 879, 881
limitations 226, 484, 582, 592,
G
618, 755
general properties source data types 226, 485,
Kafka target 499, 656, 682, 583, 593, 619, 757
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1193
header columns 973-974 IBM Informix 275
headers 1114 advanced properties (source) 280
high availability limitations 276
different configurations source data types 277
1134 history 1061, 1065 IBM Informix data types
HP NonStop SQL/MP target unsupported 278
prerequisites 1145 table IBM Informix source data types
settings 1147 277 IBM Netezza 282, 644
task setting limitations add IBM Netezza target 283, 454
1148 HP Vertica 450, 638, 840 advanced properties (target)
add HP Vertica target 641-642, 842 285, 456
IBM Netezza target data types
HP Vertica target data types
640, 841 646 IBM Netezza target
HP Vertica target security requirements 453, 645
security requirements 639, 840 IBM Netezza target data types 646
HP Vertica target data types 640, input columns (for expressions)
841 HTTPS support 972 installation
Server 79 file channel 888
Web Console 78 Linux 65
uninstall 73
I Oracle 373
IBM DB2 for iSeries Windows 57
source data types 240 installation requirements Pivotal
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1194
L log file actions (task) 1064
log messages (task) 1040
license settings 1090
log viewer 1042
licensing 1090
logging
advanced license request
options 1092 task log settings 1019, 1021, 1023
registering 1093 logging settings (server) 1096
request a license 1090
M
viewing a license 1095
limitations mail parameters 1087
Actian Vector 477 manage databases 131-133
Amazon RDS for MySQL 172 master key
Amazon RDS for SQL Server changing and protecting 85
154 file channel 889 messages 1039, 1060
Hadoop 226, 484, 582, 592, log messages (task) 1040
618, 755 notifications 1039
IBM DB2 for LUW 250
Microsoft APS PDW 701
IBM Informix 276
add Microsoft APS PDW target
Microsoft SQL Server 294, 703-704
309, 777, 787 Microsoft APS PDW target
MYSQL 341, 554, 695, 722, 797 data types 702
ODBC 351, 361 Microsoft APS PDW target
Oracle 374, 810 security requirements 702
Pivotal Greenplum 825 Microsoft APS PDW target
SAP Sybase ASE 432, 446, 459, data types 702
Microsoft Azure SQL Data
845 Linux
Warehouse 765
required software 54
data types 769
Linux prereqisites
Microsoft Azure SQL Data Ware-
Pivotal Greenplum 823, house target, adding a
1125 Linux prerequisites 647-648, 771, 773
Actian Vector 477 prerequisites 767
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1195
Microsoft SQL Server 288, 305, tables 1053
775, 784 throughput 1057
advanced properties (source)
full load
331 backup and recovery 292,
all tables 1046
326 data types
completed loading 1047
not supported 300, 323
loading tables 1048
limitations 294, 309, 777, 787
queued 1049-1050
source data types 296, 320
tables 1045
supported versions 306, 784
througput 1050
target data types 778, 787
total completion 1045
Microsoft SQL Server file 883
messages 1060
Microsoft SQL Server source
run options 1031
database configuration 291, 326
running a task 1030
Microsoft SQL Server source
status tab
data types 296, 320
full load 1043
Microsoft SQL Server target
task status 1038
add Microsoft Microsoft
SQL Server 790 tools 1060
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1196
target data types 554, 695, define the notification
723, 798 message 1074, 1084
MYSQL define trigger 1070,
limitations 341, 554, 695, 722, 797 1082 deleting 1087
MySQL source data types 342 editing 1086
MySQL target notification list 1086
security 554, 695, 723, 798 review the notification rule
MySQL target data types 554, 1081, 1085
695, 723, 798 set values for latency, disk
util-ization, memory
N utilization 1071
using variables 1075, 1084
new task 126
notifications settings 1067
notification list 1086
notification rule wizard 1069, 1082 O
notification message page
ODBC
1074, 1084
advanced properties (target)
notify how/who? page 1073, 1084
807 limitations 351, 361
Notify When? page (changes
of status) 1070 ODBC data source 350, 803
Notify When? page (define ODBC prerequisites 350, 360
errors) 1072
ODBC source
Notify When? page (define
adding a 366, 468
when) 1070, 1082
data types 352, 362,
summary page 1081, 1085
467 ODBC target
notifications 1039, 1068, 1075, 1084
data types 803
create 1069, 1082
Oracle 809
create a notification message
1076, 1085 add Oracle target 815
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1197
limitations 374, 810 advanced properties (source)
multitenant support 395 421 data types 562, 731, 835
Oracle source data types PostgreSQL source
386 Oracle target data types data types 413
812 supplemental logging prerequisites
390 supported versions 373 AIS 197
Oracle file 883
Oracle on Amazon RDS R
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1198
SAP Sybase ASE source server log level 1097
security 459 server log settings 1096
SAP Sybase ASE source server password, changing the
data types 460 82 server settings
SAP Sybase ASE target
error handling 1096
security 845
server view 110
SAP Sybase ASE target data types
setting up a new task 126
845 SAP Sybase IQ 850
settings 1067
SAP Sybase IQ target data
configure log files 1098
types 851
default recipient list 1088
SAP Sybase IQ target
license 1090
security requirements 850
logging (server) 1096
SAP Sybase IQ target data types
mail parameters 1087
851 scheduler settings (server)
notifications 1068
1104 security
create 1069, 1082
Actian Vector 478
notificationsl 1067
AIS source 199
scheduler (server) 1104
Hadoop source 593, 619
server log level 1097
MySQL target 554, 695, 723,
software requirements 54
798 Pivotal Greenplum 824
source database
SAP Sybase ASE source
Amazon RDS for MySQL 168
459 SAP Sybase ASE target
Amazon RDS for SQL Server
845 Teradata Database 874
151 File 210, 521, 538, 568,
security requirements
706 file channel 883
HP Vertica target 639, 840
Hadoop 225, 581, 590, 617,
IBM Netezza target 453, 645
754 HP Vertica 450, 638, 840
Microsoft APS PDW target 702
IBM DB2 for LUW 236,
Oracle target 810
247 IBM DB2 for z/OS
Pivotal Greenplum target 825
256 IBM Informix 275 IBM
SAP Sybase IQ target 850
Netezza 282, 644
Teradata Database target 875
Microsoft SQL Server 288,
selecting tables 137 self-signed
305, 775, 784
certificate, using a 78 MySQL 337, 552, 694, 720, 795
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1199
Oracle 809 Microsoft SQL Server 288,
SAP Sybase ASE 458, 844 305, 775, 784
Pivotal Greenplum 819
SAP Sybase IQ 850
PostgreSQL 408, 561, 729, 833
source databases
Teradata Database 465, 871
adding 134
target databases
files 883
source endpoint
adding 134
Amazon RDS for MySQL
file channel 889
168 files 883
SQLite
Hadoop 225, 581, 590, 617,
for transforms 920
754 HP Vertica 450, 638, 840
SQLite syntax
IBM Netezza 282, 644
for filters 928
Microsoft APS PDW 701 MySQL
SSL certificate, checking if installed 78
337, 552, 694, 720, 795
supplemental logging 390 supported
Oracle 809
databases 93
SAP Sybase ASE 458,
supported Linux versions 54
844 SAP Sybase IQ 850
supported Windows versions
target endpoint
54 system architecture 49
file channel 892
T task
log 1042
table errors settings 1017
task configurations 994
table settings 908
task logging
filters 921
log file actions 1064
target database
task logging settings 1019, 1021,
Actian Vector 476
1023 task settings 994
Amazon RDS for PostregSQL
apply changes 1004, 1010
181 Amazon RDS for SQL
apply changes settings
Server 151 Amazon Redshift
sub-tab 1004
511, 855, 864 File 210, 521,
apply changes tuning sub-tab
538, 568, 706 file channel 883
1010 Control Tables tab 999
Microsoft Azure SQL Data
data error sub-tab 1016
Ware-house 765
environmental errors sub-tab 1015
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1200
error handling 1015-1017 Teradata Database 465, 871
error handling settings account access 874
sub-tab 1015 data types 876
full load 1001, 1003
installation requirements 872
full load settings sub-tab 1001
Load Options 872
full load tuning sub-tab 1003
security requirements 874
logging 1019, 1021, 1023 table
Teradata Database target
error sub-tab 1017 Target
security requirements 875
Metadata tab 995
Transform Tab
task status 1038
unique index, designating
tasks 104, 125, 908, 1030 on target 913
add filters 921 transforms
add tables 136 creating expressions 919
creating 126 expression builder 967
customizing 908 SQLite syntax 920
deleting 143 tutorial 112, 896
designing 125
editing 131, 142 U
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1201
Vertica 701
view database information 133
viewing a license 1095
Copyright © Attunity
Ltd. Replicate Add-ons API | Page 1202