MySQL Multi-Master Single-Slave Replication (aka Saskia)

Oli Sennhauser ( GNU FDL)

Rebenweg 6 CH 8610 Uster Switzerland oli.sennhauser@bluewin.ch

Introduction

through simple FEDERATED TABLES...

MySQL provides its replication for High Availability (HA) and for read Scale-out. Generally it is known that in a MySQL replication you can only replicate from one Master to many slaves.

M1

M2

M S S1 S2 S3 SF

...

It is told, that it is NOT possible to replicate from several masters to a single slave. It is also told, that this will be solved in future releases of MySQL.

M1

M2

The federated Slave (SF) can be put on the same server as the real Slave (S). From the application point of view everything looks transparent. And from the server infrastructure it also does not look to complicated. This concept does not work, if you have a write I/O bottleneck on the slave server. Replication should not have a problem with this configuration and should work as usual. If somebody finds any error in this concept, please let me know.

M1 S

M2

X
S SF

But possibly some MySQL users cannot or do not want to wait. Thus we were thinking about a possible simple solution with MySQL's own tools. The idea behind it is quite simple (and thus matches to the KISS concept): One master replicates to a simple slave. The second master replicates to an other slave which is connected to the first slave

Prof of Concept (PoC)

This step can be checked as follows:

master1>SHOWVARIABLESLIKE'server%'; +++ |Variable_name|Value| +++ |server_id|1| +++ master1>SHOWMASTERSTATUS; +++ |File|Position| +++ |bin_log.000009|362| +++ master2>SHOWVARIABLESLIKE'server%'; +++ |Variable_name|Value| +++ |server_id|2| +++ master2>SHOWMASTERSTATUS; +++ |File|Position| +++ |bin_log.000001|98| +++

For proving, if this concept works, the following playground was built:

M1
name = 5.1.11 port = 3315 user = repl1 server_id = 1 tables = m_equal, m_t1 -

M2
name = 5.1.9 port = 3309 user = repl2 server_id = 2 tables = m_equal, m_t2

S
name = 5.1.14 port = 3312 user = repl3 server_id = 10 tables = m_equal, m_t1, m_t2 -

SF
name = 5.1.12 port = 3322 user = none server_id = 20 tables = m_equal, m_t2

The name is an internal name of my set-up (corresponds to the underlying MySQL version). Because all 4 databases (instances) are running on the same host different ports have to be chosen for each database. On both masters a replication account has to be created. The replication account on the Slave (S) is used for the FEDERATED tables to propagate the data. To prevent any conflicts four different server_id's are chosen as usual for replication. On both Masters M1 and M2 two tables are created. One table to simulate a table which exists in both masters (m_equal) and one table to simulate tables which only exist on one of both masters (m_t1 and m_t2). All three tables have to exist on the Slave (S). The tables on the federated Slave have to be created differently.

We skip the step to do a proper backup here because we only want to prove that the concept works. In fact it is a little bit trickier than setting up a simple replication (mysqldump --skip-add-drop-table, ...). In this area some more investigation has to be done. Now the tables have to be set up. On both masters (M 1 and M2) as well as on the Slave S this table has to be created:
mysql>CREATETABLEm_equal( idINTUNSIGNEDNOTNULLAUTO_INCREMENT PRIMARYKEY ,tsTIMESTAMPDEFAULT CURRENT_TIMESTAMP ,dataVARCHAR(32) );

Set-up

Let us start with the two masters: First the server_id and binary logging (log_bin) have to be set and the databases have to be restarted. Then the normal replication accounts have to created on the masters:
master1>GRANTREPLICATIONSLAVE ON*.* TO'repl1'@'%' IDENTIFIEDBY'repl1'; master2>GRANTREPLICATIONSLAVE ON*.* TO'repl2'@'%' IDENTIFIEDBY'repl2';

On Master 1 (M1) and the Slave (S) the following table was created:
mysql>CREATETABLEm_t1( idINTUNSIGNEDNOTNULLAUTO_INCREMENT PRIMARYKEY ,tsTIMESTAMPDEFAULT CURRENT_TIMESTAMP ,dataVARCHAR(32) );

On Master 1 (M1) and the Slave (S) the analog table was created:

mysql>CREATETABLEm_t2( idINTUNSIGNEDNOTNULLAUTO_INCREMENT PRIMARYKEY ,tsTIMESTAMPDEFAULT CURRENT_TIMESTAMP ,dataVARCHAR(32) );

Testing

When the set up is finished we can start with some tests:

master1>INSERTINTOm_t1 VALUES(1,NOW(),'Bla'); master2>INSERTINTOm_t2 VALUES(1,NOW(),'Bla'); slave>SELECT* FROMm_t1t1 JOINm_t2t2ONt1.id=t2.id; ++++ +++ |id|ts|data|id|ts |data| ++++ +++ |1|2006122020:40:48|Bla|1| 2006122020:41:02|Bla| ++++ +++

Now nearly everything is set up except our federated Slave (SF). For setting up the federated Slave we first need an account on the main Slave (S):
slave>GRANTINSERT,UPDATE,DELETE ON*.* TO'feed_user'@'%' IDENTIFIEDBY'feed_user';

Then the two tables which belong to the Master 2 (M 2) have to be created in the federated Slave (S F) as FEDERATED tables:
slave_f>CREATETABLEm_t2( idINTUNSIGNEDNOTNULLAUTO_INCREMENT PRIMARYKEY ,tsTIMESTAMPDEFAULT CURRENT_TIMESTAMP ,dataVARCHAR(32) )ENGINE=FEDERATED CONNECTION='mysql://repl3:repl3@master:3312/t est/m_t2'; slave_f>CREATETABLEm_equal( idINTUNSIGNEDNOTNULLAUTO_INCREMENT PRIMARYKEY ,tsTIMESTAMPDEFAULT CURRENT_TIMESTAMP ,dataVARCHAR(32) )ENGINE=FEDERATED CONNECTION='mysql://repl3:repl3@master:3312/t est/m_equal';

A simple INSERT seems to work. :-) No a little bit more difficult:

master1>INSERTINTOm_equal VALUES(1,now(),'Bla'); master2>UPDATEm_equal SETdata='Blabla' WHEREid=1; slave>SELECT* FROMm_equal; ++++ |id|ts|data| ++++ |1|2006122016:16:36|Blabla| ++++

Now the slaves can be started as usual:

slave>CHANGEMASTERTO master_host='master' ,master_port=3315 ,master_user='repl2' ,master_password='repl2' ,master_log_file='bin_log.000009' ,master_log_pos=362; slave>STARTSLAVE; slave_f>CHANGEMASTERTO master_host='master' ,master_port=3309 ,master_user='repl2' ,master_password='repl2' ,master_log_file='bin_log.000001' ,master_log_pos=98; STARTSLAVE;

Also this seems to work correctly (but should NEVER be done because master 2 should not know anything about row 1). So it looks like everything works like in a normal replication...

Transactions

MySQL supports transactions on FEDERATED tables since release 5.1. Even transactions seems to work (the tables have to be change to InnoDB before):
slave>SELECT*FROMm_t2; ++++ |id|ts|data| ++++ |1|2006122113:45:10|Trxtest1| ++++ master2>mysql>BEGIN; master2>INSERTINTOm_t2VALUES(2,'2006 122113:49:10','Trxtest1');

slave>SELECT*FROMm_t2; ++++ |id|ts|data| ++++ |1|2006122113:45:10|Trxtest1| ++++ master2>UPDATEm_t2SETdata='Trxtest2' WHEREid=1; master2>COMMIT; slave>SELECT*FROMm_t2; ++++ |id|ts|data| ++++ |1|2006122113:49:10|Trxtest2| |2|2006122113:49:10|Trxtest1| ++++

tions and limitations are applicable [1], [2]. SET statements are not inherited through the FEDERATED storage engine. This makes it impossible to use features like AUTO_INCREMENT or NOW()/CURRENT_TIMESTAMP. Row based replication, which is new in 5.1 does not work in the Master 2 chain.

Outlook

This kind of replication enables to design much more complex database architectures. It allows us to model some kind of data flows from several different databases to some others...

M1

M2

M3

Also a rollback works.

Conflicts

Additionally to the usual Master-Master conflicts we also have here some other conflict potential: AUTO_INCREMENT and NOW()/CURRENT_TIMESTAMP values are not propagated through the FEDERATED table. These values get lost and this causes corrupt data on the Slave (S). Auto_increment_increment and Auto_increment_offset do not help here. See also Bugs. Rows in the shared table can be deleted from the Master where they never have been inserted (see example above). This causes the other master to be out of sync and leads to possibly corrupt data.

S1

S2

S3

S4

If someone introduces such a construct in production I would very appreciate it to get some feedback from out there...

Recommendations

Accessing the same table from different masters (m_equal) is tricky. Errors can creep in which are difficult to detect later. So it is recommended to use a shared table with care. On the dedicated tables (m_t1 and m_t2) the AUTO_INCREMENT should work, if the values are in sync. But they can easily get out of sync which causes corrupt data. Using this kind of set-up is pretty dangerous and has to be tested very careful.

Thanks to ...

... Ralf Gebhardt, Saskia Schweitzer and Jens Bollmann (who still hangs on an European Airport) for the contribution.

References

Limitations

It looks like normal DML (INSERT, UPDATE and DELETE) works fine. DDL (CREATE TABLE, ALTER TABLE, etc.) will only work on the Master 1 (M 1) chain but not on the Master 2 (M2) chain! TRUNCATE TABLE on the other hand seems to work properly on both chains. Each FEDERATED table creates a permanent database connection to the data provider database (at least with release 5.1.12). This means, that this kind of replication will NOT work with too many tables. The limit will be around some dozens to some hundreds of federated tables. But this should be sufficient enough for most uses. For FEDERATED tables and replication the usual restric-

[1] Limitations of the FEDERATED Storage Engine: http://dev.mysql.com/doc/refman/5.1/en/federated-limitations.html [2] Replication Features and Known Problems: http://dev.mysql.com/doc/refman/5.1/en/replication-features.html

Bugs

#20724, FEDERATED does not honour SET INSERT_ID: http://bugs.mysql.com/bug.php?id=20724 #20026, FEDERATED lacks support for auto_increment_increment and auto_increment_offset: http://bugs.mysql.com/bug.php?id=20026