Tungsten Replicator 2.0.6 released - Multi-Master replication made easy and more

Tungsten Replicator version 2.0.6 was released today.

You can get both the binaries and the source code at the project's downloads page.

This release contains many bug fixes, and various improvements. All of them are listed in the Release Notes. The most interesting ones are the improvement in multi-master topologies. Using this release with star topologies you will get less traffic than before, because we have reduced some duplication of transaction history logs that were sent between servers.

And speaking of multi-master topologies, this release includes the cookbook recipes mentioned in this blog some time ago. Normally, these recipes are in their own repository, but for the 2.0.6 binaries we've made an exception and included them in the distribution. You can finally deploy a star or fan-in topology with ease.

Other improvements are a batch loader for data warehousing and more stability for heterogeneous replication. There are also more instrumentation that helps when you are using row-based replication. trepctl status -name tasks gives more detail than before, to let you know what's happening when dealing with huge transactions that were transferred in several chunks.

For an example of deploying a star topology, let's see a quick demo:

1. First, we get the binaries:

   $ mkdir someplace
   $ cd someplace
   $ wget http://tungsten-replicator.googlecode.com/files/tungsten-replicator-2.0.6.tar.gz
   

2. Then, we make sure that we have fulfilled the pre-requisites
3. Inside the cookbook directory you will find a script named simple_services. You should copy it in a directory that is in your $PATH (for example, $HOME/bin)
4. We can now set the cookbook recipes for our purpose. Inside the ./cookbook directory, there is a file named NODES_STAR.sh (There is such a file for every topology). The file looks like this:

   #!/bin/bash
   
   export NODE1=server1.mydomain.com
   export NODE2=server2.mydomain.com
   export NODE3=server3.mydomain.com
   export NODE4=server4.mydomain.com
   
   export ALL_NODES=($NODE1 $NODE2 $NODE3 $NODE4)
   # indicate which servers will be masters, and which ones will have a slave service
   # in case of all-masters topologies, these two arrays will be the same as $ALL_NODES
   # These values are used for automated testing
   
   # for all-masters and star replication
   export MASTERS=($NODE1 $NODE2 $NODE3 $NODE4)
   export SLAVES=($NODE1 $NODE2 $NODE3 $NODE4)
   export HUB=$NODE3
   
   # MMSERVICES are the names used for services when installing multiple masters
   export MM_SERVICES=(alpha bravo charlie delta echo foxtrot golf hotel)
   export HUB_SERVICE=charlie
   

5. If you have 4 nodes, all you need to do is fill in the nodes list, making sure that the server you want to use as a hub is the third one.
6. If you have more nodes, or less nodes, you also need to adjust the lists for ALL_NODES, MASTERS, and SLAVES.
7. Once the servers list is ready, you can start the installation.

   $ ./cookbook/install_star.sh
   

8. If you have all your pre-requisites right, you will only see the list of servers being installed and at the end the list of services being activated for each server.
9. To make sure that the cluster is installed correctly, you can run

   $ ./cookbook/show_star.sh
   $ ./cookbook/test_star.sh
   

10. If you need it, you can also remove the cluster using

    $ ./cookbook/clear_cluster_star.sh
    

Happy hacking!

Overwhelming response from the MySQL community in Barcelona

Within hours of my post about meeting the MySQL community in Barcelona, we got several offers to help, and within one day, an event was created and agreed upon.

Thanks!

Today the event was posted at Evenbrite. It will take place on Tuesday, November 13th, at 7pm. It will be a one hour talk about State of the art in MySQL high availability and replication, followed by one hour of Q&A, networking, beer, and snacks.

Registration is necessary, because the seats are limited. If you want to attend, you should register as soon as possible!

MySQL community in Barcelona - Let's meet in November

Update: With the enthusiastic responses that we have received from Barcelona, there is now an event defined at Evenbrite. If you are in Barcelona on November 13th, please register!

My company, Continuent, will have an engineering meeting in Barcelona, Spain, from November 11th to 16th.

We are meeting because, as we all work from home, we need to get in touch face-to-face at least a few times a year, and every time we try choosing a nice, inspiring place where we can both work and relax. This time, the choice went to Barcelona, which happens to be one of my favorite towns.

Now comes the community. We build software that is mainly directed to MySQL users (although we also dedicate much effort to replication to and from Oracle, Postgres, MongoDB). We use MySQL a lot, and we use many open source resources to build our software: Linux, Java, Ruby, Perl, Jenkins, Subversion, Eclipse, to mention just a few. We give back to the community in two ways: by releasing open source software (the foundation of our replication system, Tungsten Replicator, is open source) and by actively sponsoring and participating in open source events. Our company has frequent speakers at these events. In addition to myself, you may have seen on stage Robert Hodges, Neil Armitage, Ronald Bradford, Stephane Giron, Jeff Mace, Gilles Rayrat, and Linas Virbalas. We mostly talk about MySQL and replication, but we are also keen on touching other technology topics, such as virtualization, cloud computing, performance tuning, networking, and more.

We are a friendly bunch and we like to meet people. If there is a MySQL user group in Barcelona, we would like to organize an informal event with local MySQL users (and occasional traveling MySQLers who happen to be in town). Without being too formal, we would like to be contacted on this matter, and continue the discussion by chat or email. If you are in Barcelona in that period, please reach us at contributions AT continuent DOT com.

Tungsten Replicator cookbook. Advanced replication topologies made easy

I have been asked many times to provide an easy way of deploying fan-in and star schema replication schemas. So far, I have been delayed by more pressing duties.

Now the time has come. Since we are about to release a new version of Tungsten Replicator, I made the effort of putting together the steps for an easy deployment.

Recipes

The package (with downloads and svn code available at Tungsten-Replicator Toolbox) includes some juicy goodies. There are recipes to install.

• Master/slave, the classic replication topology. Nothing fancy, but with the tools mentioned in the next section, it becomes as valuable as the other topologies.
• All-masters. This is the Tungsten no-SPOF topology. Every node is a master, and every node has a direct slave service to every other node. A bit heavy on the network, but quite robust.
• Fan-in. The legendary multiple-source replication, where one slave can get instant updates from many masters.
• Star schema. The most efficient multiple-master deployment, where all the nodes are connected through a central hub. Here the trade-off is less traffic in exchange for a SPOF.

For each topology, there is a NODES_xxxx.sh file, which you need to edit, to add the list of your nodes. The nodes must be reachable by the O.S. account used for the installation, using ssh wit an authentication key (and no password).

Once you have set the list of nodes (the README file has more details), you can run the installer using the corresponding ./cookbook/install_xxxxx.sh.

Easier administration

In addition to the recipes, there are some tools that come with the package. For each topology, there is a script that shows the cluster, one that performs a simple replication test, checking that data generated in the masters reaches all the slaves, and a script that removes all replication in one go. Again, the README file has all the details.

All the scripts are written in (hopefully simple) Bash shell language. You can use them as they are, or use them as a basis to create additional administration tools.

Happy hacking!