Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Set Up & Operate Tungsten Replicator

•
•
Continuent
Continuent

Tungsten Replicator is an innovative and reliable tool that can solve your most complex replication problems. In this webinar we introduce Replicator installation and show you how to use key Replicator features effectively with MySQL. Course Topics: - Checking host and MySQL prerequisites - Downloading code from http://code.google.com/p/tungsten-replicator/ - Installation using the tpm utility - Transaction filtering using standard filters as well as customized filters you write yourself - Enabling and managing parallel replication - Configuring multi-master and fan-in using multiple replication services - Backup and restore integration - Troubleshooting replication problems - Logging bugs and participating in the Tungsten Replicator community Replication is a powerful technology that takes knowledge and planning to use effectively. This webinar gives you the background that makes replication easier to set up and allows you to take full advantage of the Tungsten Replicator benefits.

1 of 36
©Continuent 2014 Tungsten University:  Set Up and Operate Tungsten Replication Robert Hodges, CEO
©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
©Continuent 2014 Introducing Tungsten Replicator 3
©Continuent 2014 Tungsten Replicator Overview 4 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
©Continuent 2014 Master Replication Service 5 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
©Continuent 2014 Slave Replication Service 6 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
©Continuent 2014 EC2 Installation Using Github project 1.Set up Github vagrant project git clone https://github.com/continuent/ continuent-vagrant 2.Pick EC2 Vagrantfile and fill in keys 3.Copy examples/MasterSlave/default.pp to manifests/ 4.Run launch.sh 7 (Follow directions in README.md)
©Continuent 2014 Tpm installation overview 8 /opt/continuent/etc/tungsten/tungsten.ini check prereqs [copy code] configure topology [defaults]! user=tungsten! home-directory=/opt/continuent! datasource-user=tungsten! datasource-password=secret! skip-validation-check=MySQLPermissionsCheck! start-and-report=true! ! [university]! master=db1! slaves=db2,db3! topology=master-slave! tpm update (Installation)
©Continuent 2014 What Is Tungsten Doing? 9
©Continuent 2014 Commands to Check Replication • trepctl services -- All services • trepctl status -- Overall pipeline status • trepctl status -name stages -- Stage summary • trepctl status -name stores -- Store summary • trepctl status -name tasks -- Task detail • trepctl [-service name] heartbeat -- Is it Alive? 10
©Continuent 2014 Changing Replicator Configuration 11
©Continuent 2014 Reconfiguring a Replicator • Edit /etc/tungsten/tungsten.ini • No restart on local host trepctl offline! tpm update --no-restart! trepctl configure! trepctl online! ! • With automatic restart tpm update 12 Don’t change replicator configuration files directly!
©Continuent 2014 Enabling Parallel Replication 13
©Continuent 2014 Parallel Replication of Sharded Data 14 Extract Filter Apply Stage Extract Filter Apply Stage Stage Pipeline Transaction History Log Parallel Queue Slave DBMS Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Replicator tcp/ip
©Continuent 2014 Fixing Problems 15
©Continuent 2014 The Scene of the Crime 16 trepctl -host db2 status Processing status command... NAME VALUE ---- ----- ... pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on master pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000008:0000000461736666;0 pendingErrorSeqno : 1957469 pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */ ... Finished status command...
©Continuent 2014 Correcting problems 1.Look at the transaction (thl list -seqno NNN) 2.Look at the replicator message log 3.Fix the problem and go online OR 4.Skip over the problem and go online OR 5.Reprovision the slave 17
©Continuent 2014 Tungsten Logs 18
©Continuent 2014 Simple Facts about Logs • Replicator log = THL (Transaction History Log) • One THL per replication service • 2x the size of MySQL binlog • Default retention of 7 days • Take service offline to move or delete logs safely • The thl tool shows you log contents 19
©Continuent 2014 Resetting Replication 20
©Continuent 2014 Resetting/Reinstalling Automatically Reset 1.tpm reset Reinstall 1.tpm uninstall --i-am-sure 2.Rerun installation 21
©Continuent 2014 Resetting/Reinstalling Manually Reset 1.Stop replicator 2.Remove THL files 3.Remove tungsten_<svc> databases Reinstall 1.All of the above, plus delete /opt/continuent/* 2.Rerun installation 22
©Continuent 2014 Backup and Restore 23
©Continuent 2014 What Backup Tools Does MySQL Have? 24 MySQL Server Process File System DBMS Files mysqldump (dataset < 10Gb) xtrabackup (10Gb < dataset < 1Tb tar/rsync (dataset < 1Tb) file system snapshot (1Tb << dataset)
©Continuent 2014 Advanced Installation Patterns 25
©Continuent 2014 How Do I Install Fan-In Replication? 26 db1 Replicator db3 Service db1 db2 Replicator Service db2 Replicator Service db1 Service db2
©Continuent 2014 How Do I install Multi-Master? 27 db1 Replicator Service db1 Service db2 Service db3 db3 Service db1 Service db2 Service db3 db2 Replicator Service db1 Service db2 Service db3
©Continuent 2014 Tpm Understands Topologies! • Use --topology argument! 28 Fan-In StarAll-Masters --topology=fan-in --topology=all-masters --topology=star
©Continuent 2014 Tpm install from staging directory 29 db1 db2 db3 Staging copy of files check prereqs copy code configure topology tpm configure ... tpm update
©Continuent 2014 Commands to Perform Staging Install 30 $ wget http://downloads.tungsten-replicator.org/download.php? file=tungsten-replicator-2.2.1-403.tar.gz ! $ tar -xvzf tungsten-replicator-2.2.1-403.tar.gz ! $ ln -s tungsten-replicator-2.2.1-403 tungsten ! $ tungsten/tools/tpm configure defaults --reset --user=tungsten --home-directory=/opt/continuent --datasource-user=tungsten --datasource-password=secret --skip-validation-check=MySQLPermissionsCheck --start-and-report=true ! $ tungsten/tools/tpm configure fanin --members=db1,db2,db3 --masters=db1,db2 --master-services=db1,db2 --topology=fan-in ! $ tungsten/tools/tpm update
©Continuent 2014 Wrapping Up 31
©Continuent 2014 Tungsten Replicator is 100% Open Source 32 • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, find builds, post in discussion group • Documentation: https://docs.continuent.com • Install automation: https://github.com/continuent !
©Continuent 2014 Heterogeneous OracleMySQL Oracle MySQL MySQL star master-slave fan-in slave all-masters MySQLOracle Oracle
©Continuent 2014 We Do Clustering Too 34 Tungsten clusters combine off- the-shelf open source DBMS servers into data services with: ! • 24x7 data access • Scaling of load on replicas • Simple management commands ! ...without app changes or data migration Amazon US West apache /php GonzoPortal.com Connector Connector
©Continuent 2014 Conclusion 35 • Tungsten Replicator is an advanced, open source replicator • Use tpm to install either from staging or using batch commands • Use trepctl to manage replication • Tungsten Replicator has capabilities far beyond what we showed today Use your imagination!
©Continuent 2014 Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs http://flyingclusters.blogspot.com/ 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com

More Related Content

Set Up & Operate Tungsten Replicator

  • 1. ©Continuent 2014 Tungsten University:  Set Up and Operate Tungsten Replication Robert Hodges, CEO
  • 2. ©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
  • 4. ©Continuent 2014 Tungsten Replicator Overview 4 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
  • 5. ©Continuent 2014 Master Replication Service 5 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
  • 6. ©Continuent 2014 Slave Replication Service 6 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
  • 7. ©Continuent 2014 EC2 Installation Using Github project 1.Set up Github vagrant project git clone https://github.com/continuent/ continuent-vagrant 2.Pick EC2 Vagrantfile and fill in keys 3.Copy examples/MasterSlave/default.pp to manifests/ 4.Run launch.sh 7 (Follow directions in README.md)
  • 8. ©Continuent 2014 Tpm installation overview 8 /opt/continuent/etc/tungsten/tungsten.ini check prereqs [copy code] configure topology [defaults]! user=tungsten! home-directory=/opt/continuent! datasource-user=tungsten! datasource-password=secret! skip-validation-check=MySQLPermissionsCheck! start-and-report=true! ! [university]! master=db1! slaves=db2,db3! topology=master-slave! tpm update (Installation)
  • 9. ©Continuent 2014 What Is Tungsten Doing? 9
  • 10. ©Continuent 2014 Commands to Check Replication • trepctl services -- All services • trepctl status -- Overall pipeline status • trepctl status -name stages -- Stage summary • trepctl status -name stores -- Store summary • trepctl status -name tasks -- Task detail • trepctl [-service name] heartbeat -- Is it Alive? 10
  • 12. ©Continuent 2014 Reconfiguring a Replicator • Edit /etc/tungsten/tungsten.ini • No restart on local host trepctl offline! tpm update --no-restart! trepctl configure! trepctl online! ! • With automatic restart tpm update 12 Don’t change replicator configuration files directly!
  • 14. ©Continuent 2014 Parallel Replication of Sharded Data 14 Extract Filter Apply Stage Extract Filter Apply Stage Stage Pipeline Transaction History Log Parallel Queue Slave DBMS Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Replicator tcp/ip
  • 16. ©Continuent 2014 The Scene of the Crime 16 trepctl -host db2 status Processing status command... NAME VALUE ---- ----- ... pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on master pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000008:0000000461736666;0 pendingErrorSeqno : 1957469 pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */ ... Finished status command...
  • 17. ©Continuent 2014 Correcting problems 1.Look at the transaction (thl list -seqno NNN) 2.Look at the replicator message log 3.Fix the problem and go online OR 4.Skip over the problem and go online OR 5.Reprovision the slave 17
  • 19. ©Continuent 2014 Simple Facts about Logs • Replicator log = THL (Transaction History Log) • One THL per replication service • 2x the size of MySQL binlog • Default retention of 7 days • Take service offline to move or delete logs safely • The thl tool shows you log contents 19
  • 21. ©Continuent 2014 Resetting/Reinstalling Automatically Reset 1.tpm reset Reinstall 1.tpm uninstall --i-am-sure 2.Rerun installation 21
  • 22. ©Continuent 2014 Resetting/Reinstalling Manually Reset 1.Stop replicator 2.Remove THL files 3.Remove tungsten_<svc> databases Reinstall 1.All of the above, plus delete /opt/continuent/* 2.Rerun installation 22
  • 24. ©Continuent 2014 What Backup Tools Does MySQL Have? 24 MySQL Server Process File System DBMS Files mysqldump (dataset < 10Gb) xtrabackup (10Gb < dataset < 1Tb tar/rsync (dataset < 1Tb) file system snapshot (1Tb << dataset)
  • 26. ©Continuent 2014 How Do I Install Fan-In Replication? 26 db1 Replicator db3 Service db1 db2 Replicator Service db2 Replicator Service db1 Service db2
  • 27. ©Continuent 2014 How Do I install Multi-Master? 27 db1 Replicator Service db1 Service db2 Service db3 db3 Service db1 Service db2 Service db3 db2 Replicator Service db1 Service db2 Service db3
  • 28. ©Continuent 2014 Tpm Understands Topologies! • Use --topology argument! 28 Fan-In StarAll-Masters --topology=fan-in --topology=all-masters --topology=star
  • 29. ©Continuent 2014 Tpm install from staging directory 29 db1 db2 db3 Staging copy of files check prereqs copy code configure topology tpm configure ... tpm update
  • 30. ©Continuent 2014 Commands to Perform Staging Install 30 $ wget http://downloads.tungsten-replicator.org/download.php? file=tungsten-replicator-2.2.1-403.tar.gz ! $ tar -xvzf tungsten-replicator-2.2.1-403.tar.gz ! $ ln -s tungsten-replicator-2.2.1-403 tungsten ! $ tungsten/tools/tpm configure defaults --reset --user=tungsten --home-directory=/opt/continuent --datasource-user=tungsten --datasource-password=secret --skip-validation-check=MySQLPermissionsCheck --start-and-report=true ! $ tungsten/tools/tpm configure fanin --members=db1,db2,db3 --masters=db1,db2 --master-services=db1,db2 --topology=fan-in ! $ tungsten/tools/tpm update
  • 32. ©Continuent 2014 Tungsten Replicator is 100% Open Source 32 • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, find builds, post in discussion group • Documentation: https://docs.continuent.com • Install automation: https://github.com/continuent !
  • 33. ©Continuent 2014 Heterogeneous OracleMySQL Oracle MySQL MySQL star master-slave fan-in slave all-masters MySQLOracle Oracle
  • 34. ©Continuent 2014 We Do Clustering Too 34 Tungsten clusters combine off- the-shelf open source DBMS servers into data services with: ! • 24x7 data access • Scaling of load on replicas • Simple management commands ! ...without app changes or data migration Amazon US West apache /php GonzoPortal.com Connector Connector
  • 35. ©Continuent 2014 Conclusion 35 • Tungsten Replicator is an advanced, open source replicator • Use tpm to install either from staging or using batch commands • Use trepctl to manage replication • Tungsten Replicator has capabilities far beyond what we showed today Use your imagination!
  • 36. ©Continuent 2014 Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs http://flyingclusters.blogspot.com/ 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com