Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
All Products
Search

This Product

    Data Transmission Service:RDS MySQL migration to AnalyticDB for PostgreSQLAnalyticDB for PostgreSQL

    Document Center

    Data Transmission Service:RDS MySQL migration to AnalyticDB for PostgreSQLAnalyticDB for PostgreSQL

    Last Updated:Apr 16, 2025

    Data Transmission Service (DTS) supports migrating RDS MySQL to AnalyticDB for PostgreSQL.

    Supported source databases

    MySQL migration to AnalyticDB for PostgreSQL supports the following types of MySQL source databases.

    • RDS MySQL instances

    • Self-managed MySQL databases:

      • Self-managed database with a public IP address

      • Self-managed databases hosted on ECS instances

      • Self-managed databases connected through Database Gateway

      • Self-managed databases connected through Cloud Enterprise Network (CEN)

      • Self-managed databases connected through Express Connect, VPN Gateway, or Smart Gateway

    Note

    This topic describes how to configure a flow by using an RDS MySQL instance as the source database. The configuration flows for other data sources are similar to this example.

    Before you begin

    • The destination AnalyticDB for PostgreSQL instance has been created. If not, see Create an instance.

    • The storage space of the destination AnalyticDB for PostgreSQL instance must be larger than the storage space occupied by the source RDS MySQL instance.

    Notes

    Note

    • During schema migration, DTS migrates foreign keys from the source database to the destination database.

    • During full data migration and incremental data migration, DTS temporarily disables the constraint check and cascade operations on foreign keys at the session level. If you perform the cascade update and delete operations on the source database during data migration, data inconsistency may occur.

    Type

    Description

    Limits on source database

    • The tables to be migrated must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

    • If you select tables as the objects to be migrated and you need to edit tables (such as renaming tables or columns) in the destination database, up to 1,000 tables can be migrated in a single data migration task. If you run a task to migrate more than 1,000 tables, a request error occurs. In this case, we recommend that you split the tables and configure multiple tasks to migrate the tables, or configure a task to migrate the entire database.

    • Binary logs:

      • Binary logging must be enabled. For more information, see Configure parameters. The value of the binlog_row_image parameter must be set to full. Otherwise, error messages appear during the precheck, and the data migration task cannot be started.

        Important

        • If the source database is a self-managed MySQL database, binary logging must be enabled, and the value of the binlog_format parameter must be set to row and the value of the binlog_row_image parameter must be set to full.

        • If the source database is a self-managed MySQL database in a dual-primary cluster (mutual master-slave replication), you must enable the log_slave_updates parameter to ensure that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

      • If you perform only incremental data migration, the binary logs of the source database must be stored for more than 24 hours. If you perform both full data migration and incremental data migration, the binary logs of the source database must be stored for at least seven days. Otherwise, DTS may fail to obtain the binary logs and the task may fail. In exceptional circumstances, data inconsistency or loss may occur. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the Service Level Agreement (SLA) of DTS does not ensure service reliability and performance. If the source database is an RDS MySQL instance, see Manage local binary logs.

    • During data migration, do not perform DDL operations to modify primary keys or add comments (such as ALTER TABLE table_name COMMENT='table comment';). Otherwise, the DDL operation fails during data migration.

    • If the data to be migrated from the source database contains date-type data 0000-00-00 00:00:00, the task may fail.

      Note

      When DTS migrates this date value to the destination database, the value is converted to null. You can temporarily modify the date value in the source database to 0001-01-01 00:00:00, or configure the corresponding field in the destination database to allow null values.

    • Limits on operations to be performed on the source database:

      • During schema migration and full data migration, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data migration task fails.

      • If you perform only full data migration, do not write data to the source database during data migration. Otherwise, data inconsistency between the source and destination databases occurs. To ensure data consistency, we recommend that you select schema migration, full data migration, and incremental data migration as the migration types.

    • The data generated by change operation of binary logs, such as data restored from a physical backup or data from a cascade operation, is not recorded and migrated to the destination database when the data migration instance is running.

      Note

      If the change data is not recorded and migrated to the destination database, you can migrate full data again on the premise that your business is not affected.

    • If the source database is MySQL database 8.0.23 or later, and the data to be migrated includes invisible columns, the data of the columns cannot be obtained and data loss occurs.

      Note

      • To make the columns visible, run the ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE; command. For more information, see Invisible Columns.

      • Tables that do not have primary keys automatically generate invisible primary keys. You need to make the invisible primary keys visible. For more information, see Generated Invisible Primary Keys.

    Others

    • Requirements for the objects to migrate:

      • Only tables can be selected as objects to migrate.

      • Migration of VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, TXID_SNAPSHOT, and POINT data types is not supported.

      • Migration of prefix indexes is not supported. If prefix indexes exist in the source database, data migration may fail.

    • If the table to be migrated has a primary key, the primary key columns of the destination table must be the same as those of the source table. If the table to be migrated does not have a primary key, the primary key columns of the destination table must be the same as the distribution key.

    • The unique key (including primary key columns) of the destination table must contain all columns of the distribution key.

    • Before you migrate data, evaluate the impact of data migration on the performance of the source and destination databases. We recommend that you migrate data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads on the database servers.

    • During full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

    • If you migrate data of a single table or multiple tables (not an entire database), do not use tools such as pt-online-schema-change to perform online DDL operations on the objects to be migrated during data migration. Otherwise, data migration fails.

      You can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables.

    • During data migration, do not write data to the destination database by using tools other than DTS. Otherwise, data inconsistency between the source and destination databases occurs. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

    • The destination table does not support append-optimized (AO) tables.

    • If column mapping is used for non-full table migration or if the schemas of the source and destination tables are inconsistent, the data in the columns of the source database that are not contained in the destination database is lost.

    • If the source database is an ApsaraDB RDS for MySQL instance for which the EncDB feature is enabled, full data migration cannot be performed.

      Note

      ApsaraDB RDS for MySQL instances that have the Transparent Data Encryption (TDE) feature enabled support schema migration, full data migration, and incremental data migration.

    • If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.

      Note

      Only the parameters of the task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.

    Special scenarios

    • If the source database is a self-managed MySQL database, take note of the following limits:

      • If you perform a primary/secondary switchover on the source database when the data migration task is running, the task fails.

      • DTS calculates migration latency based on the timestamp of the latest migrated data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for a long time, the migration latency may be inaccurate. If the latency of the data migration task is excessively high, you can perform a DML operation on the source database to update the latency.

        Note

        If you select an entire database as the object to be migrated, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

      • DTS executes the CREATE DATABASE IF NOT EXISTS `test` statement in the source database as scheduled to move forward the binary log file position.

    • If the source database is an Apsara RDS for MySQL instance, take note of the following limit:

      • In incremental data migration, an ApsaraDB RDS for MySQL instance that does not record transaction logs, such as a read-only ApsaraDB RDS for MySQL V5.6 instance, cannot be used as the source database.

      • DTS executes the CREATE DATABASE IF NOT EXISTS `test` statement in the source database as scheduled to move forward the binary log file position.

    Billing

    Migration type

    Instance configuration fee

    Internet traffic fee

    Schema migration and full data migration

    Free of charge.

    When the Access Method parameter of the destination database is set to Public IP Address, you are charged for Internet traffic. For more information, see Billing overview.

    Incremental data migration

    Charged. For more information, see Billing overview.

    Migration types

    • Schema migration

      DTS migrates the schemas of the objects to be migrated to the destination database.

      Note

      This scenario involves data migration between heterogeneous databases. DTS cannot completely map the schemas of the source and destination databases during schema migration. We recommend that you evaluate the impact of data type conversion on your business. For more information, see Data type mappings between heterogeneous databases.

    • Full migration

      DTS migrates all historical data of the objects to be migrated from the source database to the destination database.

    • Incremental migration

      After full data migration is complete, DTS migrates incremental data from the source database to the destination database. Incremental data migration allows you to ensure service continuity during data migration.

    SQL operations that support incremental migration

    Operation type

    SQL statement

    DML

    INSERT, UPDATE, DELETE

    Note

    When data is written to the destination AnalyticDB for PostgreSQL instance, UPDATE statements are automatically converted to REPLACE INTO statements (if the primary key is updated, they are converted to DELETE+INSERT statements).

    DDL

    DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ADD COLUMN, DROP COLUMN

    Warning

    If the data type of a field in the source table is changed during data migration, the migration task reports an error and stops. You can manually fix the issue by using the following method:

    1. When migrating to the destination database AnalyticDB for PostgreSQL, the migration task fails because the field type of the source table (such as customer) is changed.

    2. Create a new table (customer_new) in AnalyticDB for PostgreSQL with the same structure as the customer table.

    3. Use the INSERT INTO SELECT command to copy data from the customer table to the newly created customer_new table to ensure that the data in the two tables is consistent.

    4. Rename or delete the customer table that failed to be migrated, and then rename the customer_new table to customer.

    5. Restart the data migration task in the DTS console.

    Permissions required for database accounts

    Database

    Schema migration

    Full migration

    Incremental migration

    RDS MySQL

    SELECT permission

    SELECT permission

    REPLICATION SLAVE, REPLICATION CLIENT, and SELECT permissions on the objects to be migrated. DTS automatically grants these permissions.

    AnalyticDB for PostgreSQL

    Read and write permissions

    For information about how to create a database account and grant permissions to the database account, see the following topics:

    Procedure

    1. Use one of the following methods to go to the Data Migration page and select the region in which the data migration instance resides.

      DTS console

      1. Log on to the DTS console.

      2. In the left-side navigation pane, click Data Migration.

      3. In the upper-left corner of the page, select the region in which the data migration instance resides.

      DMS console

      Note

      The actual operation may vary based on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

      1. Log on to the DMS console.

      2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration .

      3. From the drop-down list to the right of Data Migration Tasks, select the region in which the data synchronization instance resides.

    2. Click Create Task to go to the task configuration page.

    3. Configure the source and destination databases. The following table describes the parameters.

      Category

      Parameter

      Description

      N/A

      Task Name

      The name of the DTS task. DTS automatically generates a task name. We recommend that you specify an informative name that makes it easy to identify the task. You do not need to specify a unique task name.

      Source Database

      Database Type

      Select MySQL.

      Access Method

      Select Cloud Instance.

      Instance Region

      Select the region to which the source RDS MySQL instance belongs.

      Replicate Data Across Alibaba Cloud Accounts

      In this example, the migration is performed within the same Alibaba Cloud account. Select No.

      RDS Instance ID

      Select the source RDS MySQL instance ID.

      Database Account

      Enter the database account of the source RDS MySQL instance. For permission requirements, see Permissions required for database accounts.

      Database Password

      The password that is used to access the database instance.

      Encryption

      Specifies whether to encrypt the connection to the source database instance. Select Non-encrypted or SSL-encrypted based on your business requirements. If you want to set this parameter to SSL-encrypted, you must enable SSL encryption for the ApsaraDB RDS for MySQL instance before you configure the DTS task. For more information, see Use a cloud certificate to enable SSL encryption.

      Destination Database

      Database Type

      Select AnalyticDB PostgreSQL.

      Access Method

      Select Cloud Instance.

      Instance Region

      Select the region to which the destination AnalyticDB for PostgreSQL instance belongs.

      Instance ID

      Select the target AnalyticDB for PostgreSQL instance ID.

      Database Name

      Enter the name of the database to which the migration objects belong in the destination AnalyticDB for PostgreSQL instance.

      Database Account

      Enter the initial account of the destination AnalyticDB for PostgreSQL instance.

      Note

      You can also enter an account that has the RDS_SUPERUSER permission. For more information about how to create such an account, see Manage permissions.

      Database Password

      The password that is used to access the database instance.

    4. In the lower part of the page, click Test Connectivity and Proceed.

      Note

      • Make sure that the CIDR blocks of DTS servers can be automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add the CIDR blocks of DTS servers.

      • If the source or destination database is a self-managed database and its Access Method is not set to Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

    5. Configure the objects to be migrated.

      1. On the Configure Objects page, configure the objects that you want to migrate.

        Parameter

        Description

        Migration Types

        • To perform only full data migration, select Schema Migration and Full Data Migration.

        • To ensure service continuity during data migration, select Schema Migration, Full Data Migration, and Incremental Data Migration.

        Note

        • If you do not select Schema Migration, make sure a database and a table are created in the destination database to receive data and the object name mapping feature is enabled in Selected Objects.

        • If you do not select Incremental Data Migration, we recommend that you do not write data to the source database during data migration. This ensures data consistency between the source and destination databases.

        DDL and DML Operations to Be Synchronized

        Select the SQL operations that you want to migrate at the instance level during incremental data migration. For more information about the supported operations, see SQL operations that support incremental migration.

        Note

        To select the SQL operations that you want to migrate at the database or table level, right-click the objects in the Selected Objects section and select the SQL operations in the dialog box that appears.

        Processing Mode of Conflicting Tables

        • Precheck and Report Errors: checks whether the destination database contains tables that use the same names as tables in the source database. If the source and destination databases do not contain tables that have identical table names, the precheck is passed. Otherwise, an error is returned during the precheck and the data migration task cannot be started.

          Note

          If the source and destination databases contain tables with identical names and the tables in the destination database cannot be deleted or renamed, you can use the object name mapping feature to rename the tables that are migrated to the destination database. For more information, see Map object names.

        • Ignore Errors and Proceed: skips the precheck for identical table names in the source and destination databases.

          Warning

          If you select Ignore Errors and Proceed, data inconsistency may occur and your business may be exposed to the following potential risks:

          • If the source and destination databases have the same schema, and a data record has the same primary key as an existing data record in the destination database, the following scenarios may occur:

            • During full data migration, DTS does not migrate the data record to the destination database. The existing data record in the destination database is retained.

            • During incremental data migration, DTS migrates the data record to the destination database. The existing data record in the destination database is overwritten.

          • If the source and destination databases have different schemas, only specific columns are migrated or the data migration task fails. Proceed with caution.

        Storage Engine Type

        Select the storage engine type of the destination table based on your business requirements. The default value is Beam.

        Note

        This configuration item is available only when the kernel version of the destination AnalyticDB for PostgreSQL instance is v7.0.6.6 or later, and Migration Types is set to Schema Migration.

        Source Objects

        Select one or more objects from the Source Objects section. Click the 向右小箭头 icon to add the objects to the Selected Objects section.

        Note

        You can select columns, tables, or schemas as the objects to be migrated. If you select tables or columns as the objects to be migrated, DTS does not migrate other objects, such as views, triggers, or stored procedures, to the destination database.

        Selected Objects

        • To rename an object that you want to migrate to the destination instance, right-click the object in the Selected Objects section. For more information, see Map the name of a single object.

        • To rename multiple objects at a time, click Batch Edit in the upper-right corner of the Selected Objects section. For more information, see Map multiple object names at a time.

        Note

        • To select the SQL operations that you want to migrate at the database or table level, right-click the object in the Selected Objects section and select the SQL operations in the dialog box that appears.

        • To configure a filter to filter data, right-click the table that you want to migrate in the Selected Objects section and configure a filter in the dialog box that appears. For more information, see Configure SQL conditions to filter data.

        • If you use the object name mapping feature to rename an object, other objects that are dependent on the object may fail to be migrated.

      2. Click Next: Advanced Settings to configure advanced settings.

        Parameter

        Description

        Dedicated Cluster for Task Scheduling

        By default, DTS schedules the data migration task to the shared cluster if you do not specify a dedicated cluster. If you want to improve the stability of data migration tasks, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster.

        Retry Time for Failed Connections

        The retry time range for failed connections. If the source or destination database fails to be connected after the data migration task is started, DTS immediately retries a connection within the retry time range. Valid values: 10 to 1,440. Unit: minutes. Default value: 720. We recommend that you set the parameter to a value greater than 30. If DTS is reconnected to the source and destination databases within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails.

        Note

        • If you specify different retry time ranges for multiple data migration tasks that share the same source or destination database, the value that is specified later takes precedence.

        • When DTS retries a connection, you are charged for the DTS instance. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at the earliest opportunity after the source database and destination instance are released.

        Retry Time for Other Issues

        The retry time range for other issues. For example, if DDL or DML operations fail to be performed after the data migration task is started, DTS immediately retries the operations within the retry time range. Valid values: 1 to 1440. Unit: minutes. Default value: 10. We recommend that you set the parameter to a value greater than 10. If the failed operations are successfully performed within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails.

        Important

        The value of the Retry Time for Other Issues parameter must be smaller than the value of the Retry Time for Failed Connections parameter.

        Enable Throttling for Full Data Migration

        Specifies whether to enable throttling for full data migration. During full data migration, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers. You can enable throttling for full data migration based on your business requirements. To configure throttling, you must configure the Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) parameters. This reduces the loads of the destination database server.

        Note

        You can configure this parameter only if you select Full Data Migration for the Migration Types parameter.

        Enable Throttling for Incremental Data Migration

        Specifies whether to enable throttling for incremental data migration. To configure throttling, you must configure the RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) parameters. This reduces the loads of the destination database server.

        Note

        You can configure this parameter only if you select Incremental Data Migration for the Migration Types parameter.

        Enclose Object Names in Quotation Marks

        Specify whether to add quotation marks to the names of destination objects.

        • Select Yes: If the source database has the following scenarios, DTS automatically adds single quotation marks (') or double quotation marks (") to the names of schemas, tables, or columns that meet the requirements during schema migration and incremental data migration.

          • The business environment of the source database is case-sensitive and the database name contains both uppercase and lowercase letters.

          • A source table name does not start with a letter and contains characters other than letters, digits, and specific special characters.

            Note

            Special characters include only underscores (_), number signs (#), and dollar signs ($).

          • The names of the schemas, tables, or columns that you want to migrate are keywords of the destination database, reserved keywords, or invalid characters.

        • Select No: DTS does not add quotation marks to the names of destination objects.

        Whether to delete SQL operations on heartbeat tables of forward and reverse tasks

        Specifies whether to write SQL operations on heartbeat tables to the source database while the DTS instance is running. Valid values:

        • Yes: does not write SQL operations on heartbeat tables. In this case, a latency of the DTS instance may be displayed.

        • No: writes SQL operations on heartbeat tables. In this case, features such as physical backup and cloning of the source database may be affected.

        Environment Tag

        The environment tag that is used to identify the DTS instance. You can select an environment tag based on your business requirements. In this example, no environment tag is selected.

        Configure ETL

        Specifies whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

        Monitoring and Alerting

        Specifies whether to configure alerting for the data migration task. If the task fails or the migration latency exceeds the specified threshold, the alert contacts receive notifications. Valid values:

      3. Click Next Step: Data Verification to configure the data verification task.

        For more information about how to use the data verification feature, see Configure a data verification task.

      4. Optional: Configure database and table fields. Set the AnalyticDB for PostgreSQL Primary Key Columns and Distribution Key information for the tables to be migrated in the destination AnalyticDB for PostgreSQL database. For more information, see CREATE TABLE.

        Note

        This step is available only when you select the Schema Migration option when you configure the objects to migrate.

    6. Save the task settings and run a precheck.

      • To view the parameters to be specified when you call the relevant API operation to configure the DTS task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

      • If you do not need to view or have viewed the parameters, click Next: Save Task Settings and Precheck in the lower part of the page.

      Note

      • Before you can start the data migration task, DTS performs a precheck. You can start the data migration task only after the task passes the precheck.

      • If the task fails to pass the precheck, click View Details next to each failed item. After you analyze the causes based on the check results, troubleshoot the issues. Then, run a precheck again.

      • If an alert is triggered for an item during the precheck:

        • If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.

        • If the alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur, and your business may be exposed to potential risks.

    7. Purchase an instance.

      1. Wait until Success Rate becomes 100%. Then, click Next: Purchase Instance.

      2. On the Purchase Instance page, configure the Instance Class parameter for the data migration instance. The following table describes the parameters.

        Section

        Parameter

        Description

        New Instance Class

        Resource Group

        The resource group to which the data migration instance belongs. Default value: default resource group. For more information, see What is Resource Management?

        Instance Class

        DTS provides instance classes that vary in the migration speed. You can select an instance class based on your business scenario. For more information, see Instance classes of data migration instances.

      3. Read and agree to Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.

      4. Click Buy and Start. In the message that appears, click OK.

        You can view the progress of the task on the Data Migration page.

        Note

        • If a data migration task cannot be used to migrate incremental data, the task automatically stops. The Completed is displayed in the Status section.

        • If a data migration task can be used to migrate incremental data, the task does not automatically stop. The incremental data migration task never stops or completes. The Running is displayed in the Status section.