MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup

Safe Harbor Statement
The following is intended to outline our generalproduct direction. It isintended for
information purpose only, and may not be incorporated into any contract. It isnot a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. Thedevelopment, release and timing of any features or
functionality described for Oracle´s product remains at thesole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
3 / 168

Who I am ?
4 / 168

Frédéric Descamps
@lefred
MySQLEvangelist
ManagingMySQLsince 3.23
devops believer
http://about.me/lefred

5 / 168

get more online
MySQL Group Replication
6 / 168

Blogs
http://lefred.be/
http://mysqlhighavailability.com/
https://thesubtlepath.com/blog/mysql/
7 / 168

Agenda
MySQLInnoDB Cluster & Group Replication concepts
Migration from Master-Slave to GR
How to monitor ?
Application interaction
8 / 168

LAB1: Current situation
9 / 168

launch
run_app.sh
onmysql1 into
a screen
session
verify that
mysql2 is a
running slave
LAB1: Current situation
10 / 168

Summary

+--------+----------+--------------+-----------------+
| | ROLE | SSH PORT | INTERNAL IP |
+--------+----------+--------------+-----------------+
| | | | |
| mysql1 | master | 8821 | 192.168.56.11 |
| | | | |
| mysql2 | slave | 8822 | 192.168.56.12 |
| | | | |
| mysql3 | n/a | 8823 | 192.168.56.13 |
| | | | |
| mysql4 | n/a | 8824 | 192.168.56.14 |
| | | | |
+--------+----------+--------------+-----------------+
11 / 168

Easy High Availability
MySQL InnoDB Cluster
12 / 168

InnoDB
cluster
Ease-of-Use
ExtremeScale-Out
Out-of-BoxSolution
Built-inHA
HighP erformance
EverythingIntegr ated
13 / 168

Our vision in 4 steps
MySQLDocumentStore
Relational&DocumentModels
MySQLHA
Out-Of-BoxHA
ReadScale-Out
AsyncReplication+A utoFailover
WriteScale-Out
Sharding
E1 E3
E2 E4
14 / 168

Step 2´s Architecture
Application
MySQLConnector
MySQLRouter
MySQLShell
InnoDB
cluster
Application
MySQLConnector
MySQLRouter
Mp
M
M
15 / 168

Application
MySQLConnector
MySQLRouter
MySQLShell
InnoDB
cluster
Application
MySQLConnector
MySQLRouter
Mp
M
M
S1 S2 S3 S4 S...
16 / 168

Application
MySQLConnector
MySQLRouter
MySQLShell
Application
MySQLConnector
MySQLRouter
InnoDB
cluster
Mp
M
M
S2 S3 S4 S...S1
InnoDB
cluster
Mp
M
M
S1 S2 S3 S4 S...
InnoDB
cluster
Mp
M
M
S1 S2 S3 S4 S...
Application
MySQLConnector
MySQLRouter
Application
MySQLConnector
MySQLRouter
Application
MySQLConnector
MySQLRouter
replicaset1
replicaset2
replicaset3
17 / 168

the magic explained
Group Replication Concept
18 / 168

Group Replication: heart of MySQL InnoDB
Cluster
19 / 168

Group Replication: heart of MySQL InnoDB
Cluster
20 / 168

MySQL Group Replication
but what is it ?!?
GR is a plugin forMySQL, made byMySQLand packaged withMySQL
GR is an implementation of Replicated Database State Machine theory
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
GR implements conflict detection and resolution
GR allows automatic distributed recovery
Supported onallMySQLplatforms!!
Linux, Windows, Solaris, OSX,FreeBSD
21 / 168

MySQL Group Communication System (GCS)
MySQLXcom protocol
Replicated Database State Machine
Paxos based protocol (similarto Mencius)
Its task: Deliver messages accross thedistributed system:
Atomically
in Total Order
MySQLGroup Replication receives theOrdered 'tickets' from thisGCS subsystem.
22 / 168

And for users ?
not longer necessary to handle server fail-over manually or with a complicated script
GR provides fault tolerance
GR enables update-everywhere setups
GR handles crashes, failures, re-connects automatically
Allows an easy setup of a highly available MySQLservice!
23 / 168

OK, but how does it work ?
it´s just ...
24 / 168

it´s just ...
25 / 168

it´s just ...
... no, in fact the writeset replication issynchronousand then certification and apply of
the changes are local to each nodes and happen asynchronous.
not that easy to understand... right ? As a picture isworth a 1000words, let´s illustrate
this...
26 / 168

MySQLGroup Replication (autocommit)
27 / 168

28 / 168

29 / 168

30 / 168

31 / 168

32 / 168

33 / 168

34 / 168

35 / 168

36 / 168

37 / 168

38 / 168

MySQLGroup Replication (full transaction)
39 / 168

40 / 168

41 / 168

42 / 168

43 / 168

44 / 168

45 / 168

46 / 168

47 / 168

Group Replication : Total Order Delivery - GTID
48 / 168

49 / 168

50 / 168

51 / 168

52 / 168

53 / 168

54 / 168

55 / 168

56 / 168

57 / 168

58 / 168

59 / 168

60 / 168

61 / 168

62 / 168

63 / 168

64 / 168

65 / 168

66 / 168

Group Replication: return commit
Asynchronous Replication:
67 / 168

Group Replication: return from commit (2)
Semi-Sync Replication:
68 / 168

Group Replication: return from commit (3)
Group Replication:
69 / 168

Group Replication : Optimistic Locking
Group Replication uses optimistic locking
during a transaction,local (InnoDB) lockinghappens
optimistically assumesthere will be no conflicts across nodes
(no communication between nodes necessary)
cluster-wide conflict resolution happens only at COMMIT, duringcertification
Let´s first have a look at the traditional locking to compare.
70 / 168

Traditional locking
71 / 168

Traditional locking
72 / 168

Traditional locking
73 / 168

Traditional locking
74 / 168

Traditional locking
75 / 168

Traditional locking
76 / 168

Optimistic Locking
77 / 168

Optimistic Locking
78 / 168

Optimistic Locking
79 / 168

Optimistic Locking
80 / 168

Optimistic Locking
81 / 168

Optimistic Locking
82 / 168

Optimistic Locking
The system returns error 149 ascertification failed:
ERROR 1180 (HY000): Got error 149 during COMMIT
83 / 168

Certification
Certification is the process that only needs to answer thefollowing unique question:
can the write (transaction) beapplied?
based on unapplied earlier transactions
such conflicts must come forother members/nodes
84 / 168

Certification (2)
certification is a deterministic operation
certification individually happens on every member/node
communication with other members isnot needed for certification
pass: enter in the apply queue
fail: drop the transaction
serialized by the total order in GCS/Xcom + GTID
first committer wins rule
85 / 168

Drawbacks of optimistic locking
having a first-committer-wins system means conflicts will more likely happen with:
large transactions
long running transactions
86 / 168

GTID
GTIDs are the same as those used by asynchronous replication.
mysql> SELECT * FROM performance_schema.replication_connection_statusG
************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
GROUP_NAME: afb80f36-2bff-11e6-84e0-0800277dd3bf
SOURCE_UUID: afb80f36-2bff-11e6-84e0-0800277dd3bf
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-57,
f037578b-46b1-11e6-8005-08002774c31b:1-48937
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
87 / 168

GTID
but transactions use the Group´s GTID
mysql> show master statusG
************************** 1. row ***************************
File: mysql4-bin.000001
Position: 1501
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-57,
f037578b-46b1-11e6-8005-08002774c31b:1-48937
88 / 168

Requirements
exclusively works with InnoDBtables
every table must have a PK defined
only IPV4 is supported
a good network with low latency isimportant
maximum of 9 members per group
log-bin must be enabled and onlybinlog_format=ROW is supported
89 / 168

Requirements (2)
enable GTIDs
replication meta-data must bestored insystem tables
--master-info-repository=TABLE
--relay-log-info-repository=TABLE
writesets extraction must be enabled TODO: I need to check on 5.7.19 and 8.0.1
--transaction-write-set-extraction=XXHASH64
log-slave-updates must alsobe enabled
90 / 168

Limitations
binlog checksum is not supported
--binlog-checksum=NONE
savepointswere not supported before 5.7.19& 8.0.1
SERIALIZABLEis not supported as transaction isolation level
http://lefred.be/content/mysql-group-replication-limitations-savepoints/
http://lefred.be/content/mysql-group-replication-and-table-design/
91 / 168

Is my workload ready for Group Replication ?
As the writesets (transactions) are replicated to all available nodes on commit, and as
they are certified on every node, a very large writeset could increase theamount of
certification errors.
Additionally, changing the samerecord on all thenodes (hotspot) concurrently will also
cause problems.
And finally, the certification uses theprimarykey of thetables, a table without PK isalso
a problem.
92 / 168

Is my workload ready for Group Replication ?
Therefore, when using Group Replication, we should pay attention to these points:
PK is mandatory (and a good one isbetter)
avoid large transactions
avoid hotspot
93 / 168

ready ?
Migration from Master-Slave to GR
94 / 168

The plan
95 / 168

1) We install and
setupMySQLInnoDB
Clusteron one of the
new servers
The plan
96 / 168

2) We restore a
backup
3) setup
asynchronous
replication on the new
server.
The plan
97 / 168

4) We add a new
instance to our group
The plan
98 / 168

5) We point the
application to one of
our new nodes.
6) We wait and check
that asynchronous
replication is caught
up
7) we stop those
asynchronous slaves
The plan
99 / 168

8) We attach the
mysql2 slave to the
group
The plan
100 / 168

9) Use MySQL Router
for directing traffic
The plan
101 / 168

LatestMySQL5.7is already installed on
mysql3.
Let´s take a backup onmysql1 usingmeb:
[mysql1 ~]# mysqlbackup
--host=127.0.0.1
--backup-dir=/tmp/backup
--user=root --password=X
backup-and-apply-log
LAB2: Prepare mysql3
Asynchronous slave
102 / 168

LAB2: Prepare mysql3 (2)
Asynchronous slave
Copy the backup frommysql1 to mysql3:
[mysql1 ~]# scp -r /tmp/backup mysql3:/tmp
And restore it:
[mysql3 ~]# mysqlbackup --backup-dir=/tmp/backup --force copy-back
[mysql3 ~]# rm /var/lib/mysql/mysql*-bin* # just some cleanup
[mysql3 ~]# chown -R mysql. /var/lib/mysql
103 / 168

LAB3: mysql3 as asynchronous slave (2)
Asynchronous slave
Configure /etc/my.cnf with the minimal requirements:
[mysqld]
...
server_id=3
enforce_gtid_consistency = on
gtid_mode = on
log_bin
log_slave_updates
104 / 168

LAB2: Prepare mysql3 (3)
Asynchronous slave
Let´s startMySQLonmysql3:
[mysql3 ~]# systemctl start mysqld
105 / 168

find the GTIDs purged
change MASTER
set the purged GTIDs
start replication

106 / 168

Find the latest purged GTIDs:
[mysql3 ~]# cat /tmp/backup/meta/backup_gtid_executed.sql
SET @@GLOBAL.GTID_PURGED='33351000-3fe8-11e7-80b3-08002718d305:1-1002';
Connect to mysql3 and setup replication:
mysql> CHANGE MASTER TO MASTER_HOST="mysql1",
MASTER_USER="repl_async", MASTER_PASSWORD='Xslave',
MASTER_AUTO_POSITION=1;
mysql> RESET MASTER;
mysql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";
mysql> START SLAVE;
Check that you receive the application´s traffic
107 / 168

Administration made easy and more...
MySQL-Shell
108 / 168

MySQL Shell
TheMySQLShell is an interactive Javascript, Python, or SQLinterface supporting
development and administration for theMySQLServer and is acomponent of theMySQL
Server. You can use theMySQLShell to perform data queriesand updates aswell as
various administration operations.
109 / 168

MySQL Shell (2)
TheMySQLShell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
MySQLStandard and X Protocols
and more...
110 / 168

LAB4: MySQL InnoDB Cluster
Create a single instance cluster
Time to use the newMySQLShell !
[mysql3 ~]# mysqlsh
Let´s verify if our server is ready to becomea member of a newcluster:
mysql-js> dba.checkInstanceCon guration('root@mysql3:3306')
Change the configuration !
mysql-js> dba.con gureLocalInstance()
111 / 168

LAB4: MySQL InnoDB Cluster (2)
Restart mysqld to use the new configuration:
[mysql3 ~]# systemctl restart mysqld
Create a single instance cluster
[mysql3 ~]# mysqlsh
mysql-js> c root@mysql3:3306
mysql-js> cluster = dba.createCluster('MyInnoDBCluster')
112 / 168

LAB4: Cluster Status
mysql-js> cluster.status()
{
"clusterName": "MyInnoDBCluster",
"defaultReplicaSet": {
"name": "default",
"primary": "mysql3:3306",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures.",
"topology": {
"mysql3:3306": {
"address": "mysql3:3306",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
}
}
}
}
113 / 168

Addmysql4 to the Group:
restore the backup
set the purged GTIDs
useMySQLshell
LAB5: add mysql4 to the cluster (1)

114 / 168

LAB5: add mysql4 to the cluster (2)
Copy the backup frommysql1 to mysql4:
[mysql1 ~]# scp -r /tmp/backup mysql4:/tmp
And restore it:
[mysql4 ~]# mysqlbackup --backup-dir=/tmp/backup --force copy-back
[mysql4 ~]# rm /var/lib/mysql/mysql*-bin* # just some cleanup
[mysql4 ~]# chown -R mysql. /var/lib/mysql
StartMySQLonmysql4:
[mysql4 ~]# systemctl start mysqld
115 / 168

LAB5: MySQL shell to add an instance (3)
[mysql4 ~]# mysqlsh
Let´s verify the config:
And change the configuration:
Restart the service to enablethechanges:
116 / 168

Group of 2 instances
Find the latest purged GTIDs:
[mysql4 ~]# cat /tmp/backup/meta/backup_gtid_executed.sql
SET @@GLOBAL.GTID_PURGED='33351000-3fe8-11e7-80b3-08002718d305:1-1002';
...
Connect to mysql4 and set GTID_PURGED
[mysql4 ~]# mysqlsh
mysql-js> sql
mysql-sql> RESET MASTER;
mysql-sql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";
117 / 168

mysql-sql> js
mysql-js> cluster = dba.getCluster()
mysql-js> cluster.checkInstanceState('root@mysql4:3306')
mysql-js> cluster.addInstance("root@mysql4:3306")
118 / 168

Cluster Status
{
"status": "Cluster is NOT tolerant to any failures.",
"topology": {
"mysql3:3306": {
"status": "ONLINE",
"role": "HA",
"mode": "R/W",
"leaves": {
"mysql4:3306": {
"status": "RECOVERING",
"role": "HA",
"mode": "R/O",
"leaves": {}
}
}
}
}
}
}
119 / 168

Recovering progress
On standard MySQL, monitor thegroup_replication_recovery channel to see
the progress:
mysql> show slave status for channel 'group_replication_recovery'G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: mysql3
Master_User: mysql_innodb_cluster_rpl_user
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Retrieved_Gtid_Set: 6e7d7848-860f-11e6-92e4-08002718d305:1-6,
7c1f0c2d-860d-11e6-9df7-08002718d305:1-15,
b346474c-8601-11e6-9b39-08002718d305:1964-77177,
e8c524df-860d-11e6-9df7-08002718d305:1-2
Executed_Gtid_Set: 7c1f0c2d-860d-11e6-9df7-08002718d305:1-7,
b346474c-8601-11e6-9b39-08002718d305:1-45408,
e8c524df-860d-11e6-9df7-08002718d305:1-2
...
120 / 168

point the application
to the cluster
Migrate the application
121 / 168

LAB6: Migrate the application
Now we need to point the application to mysql3, this is the only downtime !
...
[ 21257s] threads: 4, tps: 12.00, reads: 167.94, writes: 47.98, response time: 18
^C
[mysql1 ~]# run_app.sh mysql3
122 / 168

LAB6: Migrate the application
Stop asynchronous replicationon mysql2 andmysql3:
mysql2> stop slave;
mysql3> stop slave;
Make sure gtid_executed range onmysql2 is lower or equal than onmysql3
mysql[2-3]> show global variables like 'gtid_executed'G
mysql[2-3]> reset slave all;
123 / 168

previous slave
(mysql2) can now
be part of the cluster
Add a third instance
124 / 168

LAB7: Add mysql2 to the group
We first validate the instance using MySQLShell and we configure it.
[mysql2 ~]# mysqlsh
We also need to remove super_read_onlyfrommy.cnf to be able to use the shell to add
the node to the cluster.
125 / 168

LAB7: Add mysql2 to the group (2)
Back inMySQLshell we add the new instance:
[mysql2 ~]# mysqlsh
mysql-js> cluster.addInstance("root@mysql2:3306")
126 / 168

LAB7: Add mysql2 to the group (3)
{
"name": "default",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysql2:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysql3:3306": {
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE"
},
"mysql4:3306": {
"mode": "R/O",
"readReplicas": {},Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
127 / 168

writing to a single server
Single Primary Mode
128 / 168

Default = Single Primary Mode
By default, MySQLInnoDBClusterenablesSingle Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
| Variable_name | Value |
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
129 / 168

Default = Single Primary Mode
By default, MySQLInnoDBClusterenablesSingle Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
In Single Primary Mode, a single member acts asthewritable master (PRIMARY) and the
rest of the members act as hot-standbys (SECONDARY).
The group itself coordinates and configures itself automatically to determine which
member will act as thePRIMARY, through a leader election mechanism.
130 / 168

Who´s the Primary Master ?
As the Primary Master is elected, all nodes partof thegroup knows which one was
elected. This value is exposed instatus variables:
mysql> show status like 'group_replication_primary_member';
+----------------------------------+--------------------------------------+
+----------------------------------+--------------------------------------+
| group_replication_primary_member | 28a4e51f-860e-11e6-bdc4-08002718d305 |
+----------------------------------+--------------------------------------+
mysql> select member_host as "primary master"
from performance_schema.global_status
join performance_schema.replication_group_members
where variable_name = 'group_replication_primary_member'
and member_id=variable_value;
+---------------+
| primary master|
+---------------+
| mysql3 |
+---------------+
131 / 168

Create a Multi-Primary Cluster:
It´s also possible to create a Multi-PrimaryCluster using theShell:
mysql-js> cluster=dba.createCluster('perconalive',{multiMaster: true})
A new InnoDB cluster will be created on instance 'root@mysql3:3306'.
The MySQL InnoDB cluster is going to be setup in advanced Multi-Master Mode.
Before continuing you have to con rm that you understand the requirements and
limitations of Multi-Master Mode. Please read the manual before proceeding.
I have read the MySQL InnoDB cluster manual and I understand the requirements
and limitations of advanced Multi-Master Mode.
Con rm [y|N]:
Or you can force it to avoid interaction (for automation) :
js> cluster=dba.createCluster('perconalive',{multiMaster: true, force: true})
132 / 168

get more info
Monitoring
133 / 168

Performance Schema
Group Replication usesPerformance_Schema to expose status
mysql3> SELECT * FROM performance_schema.replication_group_membersG
*************************** 1. row ***************************
MEMBER_ID: 00db47c7-3e23-11e6-afd4-08002774c31b
MEMBER_HOST: mysql3.localdomain
MEMBER_PORT: 3306
MEMBER_STATE: ONLINE
mysql3> SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
THREAD_ID: NULL
SERVICE_STATE: ON
RECEIVED_TRANSACTION_SET: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-2
LAST_ERROR_MESSAGE:
134 / 168

Member State
These are the different possible state for a node member:
ONLINE
OFFLINE
RECOVERING
ERROR: when a node is leaving but the plugin wasnot instructed to stop
UNREACHABLE
135 / 168

Status information & metrics
Members
mysql> SELECT * FROM performance_schema.replication_group_membersG
*************************** 1. row ***************************
MEMBER_ID: 00db47c7-3e23-11e6-afd4-08002774c31b
MEMBER_HOST: mysql3.localdomain
MEMBER_PORT: 3306
*************************** 2. row ***************************
MEMBER_ID: e1544c9d-4451-11e6-9f5a-08002774c31b
MEMBER_HOST: mysql4.localdomain.localdomain
MEMBER_PORT: 3306
136 / 168

Connections
mysql> SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
THREAD_ID: NULL
SERVICE_STATE: ON
RECEIVED_TRANSACTION_SET: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089,
afb80f36-2bff-11e6-84e0-0800277dd3bf:1-2834
LAST_ERROR_MESSAGE:
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_recovery
GROUP_NAME:
SOURCE_UUID:
THREAD_ID: NULL
SERVICE_STATE: OFF
COUNT_RECEIVED_HEARTBEATS: 0Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
137 / 168

Local node status
mysql> select * from performance_schema.replication_group_member_statsG
*************************** 1. row ***************************
VIEW_ID: 14679667214442885:4
MEMBER_ID: e1544c9d-4451-11e6-9f5a-08002774c31b
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 5961
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089
afb80f36-2bff-11e6-84e0-0800277dd3bf:1-5718
LAST_CONFLICT_FREE_TRANSACTION: afb80f36-2bff-11e6-84e0-0800277dd3bf:5718
138 / 168

Performance_Schema
You can find GR information in thefollowing Performance_Schematables:
replication_applier_con guration
replication_applier_status
replication_applier_status_by_worker
replication_connection_con guration
replication_connection_status
replication_group_member_stats
replication_group_members
139 / 168

Status during recovery
mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_recovery'G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: <NULL>
Master_User: gr_repl
Master_Port: 0
...
Relay_Log_File: mysql4-relay-bin-group_replication_recovery.000001
...
Slave_IO_Running: No
Slave_SQL_Running: No
...
Executed_Gtid_Set: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089,
afb80f36-2bff-11e6-84e0-0800277dd3bf:1-5718
...
Channel_Name: group_replication_recovery
140 / 168

Sys Schema
The easiest way to detect if anode isa member of theprimarycomponent (when there
are partitioning of your nodes due to network issues for example) and therefore a valid
candidate for routing queriesto it, isto use thesys table.
Additional information forsys can be downloaded at
https://github.com/lefred/mysql_gr_routing_check/blob/master/addition_to_sys.sql
On the primary node:
[mysql? ~]# mysql < /tmp/addition_to_sys.sql
141 / 168

Sys Schema
Is this node part of PRIMARY Partition:
mysql3> SELECT sys.gr_member_in_primary_partition();
+------------------------------------+
| sys.gr_node_in_primary_partition() |
+------------------------------------+
| YES |
+------------------------------------+
To use as healthcheck:
mysql3> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
| viable_candidate | read_only | transactions_behind | transactions_to_cert |
+------------------+-----------+---------------------+----------------------+
| YES | YES | 0 | 0 |
+------------------+-----------+---------------------+----------------------+
142 / 168

LAB8: Sys Schema - Health Check
On one of the non Primary nodes, run thefollowing command:
mysql-sql> ush tables with read lock;
Now you can verify what the healthcheck exposes to you:
mysql-sql> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
+------------------+-----------+---------------------+----------------------+
| YES | YES | 950 | 0 |
+------------------+-----------+---------------------+----------------------+
mysql-sql> UNLOCK TABLES;
143 / 168

application interaction
MySQL Router
144 / 168

MySQL Router
MySQLRouter is lightweight middleware that provides transparent routing between your
application and backendMySQLServers. It can be used for awide variety of use cases,
such as providing high availability and scalability by effectively routing database trafficto
appropriate backendMySQLServers.
MySQLRouter doesn´t require any specific configuration. Itconfigures itself automatically
(bootstrap) usingMySQLInnoDB Cluster´s metadata.
145 / 168

LAB9: MySQL Router
We will now usemysqlrouter between our application and thecluster.
146 / 168

LAB9: MySQL Router (2)
Configure MySQLRouter that will run on the app server (mysql1).We bootstrap it using
thePrimary-Master:
[root@mysql1 ~]# mysqlrouter --bootstrap mysql3:3306 --user mysqlrouter
Please enter MySQL password for root:
WARNING: The MySQL server does not have SSL con gured and metadata used by
the router may be transmitted unencrypted.
Bootstrapping system MySQL Router instance...
MySQL Router has now been con gured for the InnoDB cluster 'MyInnoDBCluster'.
The following connection information can be used to connect to the cluster.
Classic MySQL protocol connections to cluster 'MyInnoDBCluster':
- Read/Write Connections: localhost:6446
- Read/Only Connections: localhost:6447
X protocol connections to cluster 'MyInnoDBCluster':
- Read/Write Connections: localhost:64460
- Read/Only Connections: localhost:64470
147 / 168

Now let´s modify the configuration file to listen on port 3306:
in/etc/mysqlrouter/mysqlrouter.conf:
[routing:MyInnoDBCluster_default_rw]
-bind_port=6446
+bind_port=3306
We can stopmysqld onmysql1 and startmysqlrouter into ascreensession:
[mysql1 ~]# systemctl stop mysqld
[mysql1 ~]# systemctl start mysqlrouter
148 / 168

Before killing a member we will change systemd´s default behavior that restarts
mysqld immediately:
in/usr/lib/systemd/system/mysqld.service add the following under
[Service]
RestartSec=30
[mysql3 ~]# systemctl daemon-reload
149 / 168

Now we can point the application to therouter (back to mysql1):
[mysql1 ~]# run_app.sh
Check app and killmysqld onmysql3(the Primary Master R/W node) !
[mysql3 ~]# kill -9 $(pidof mysqld)
mysql> select member_host as "primary" from performance_schema.global_status
join performance_schema.replication_group_members
where variable_name = 'group_replication_primary_member'
and member_id=variable_value;
+---------+
| primary |
+---------+
| mysql4 |
+---------+
150 / 168

ProxySQL / HA Proxy / F5 / ...
3rd party router/proxy
151 / 168

ProxySQLalso has native support for Group
Replication which makes it maybe thebest
choice for advanced users.
3rd party router/proxy
MySQLInnoDB Clustercan also work withthird party router / proxy.
If you need some specific features that are not yet available inMySQLRouter, like
transparent R/W splitting, then you canuse your software of choice.
The important part of such implementation isto use a good health check to verify if the
MySQLserver you plan to route the traffic isina valid state.
MySQLRouterimplements that natively, andit´s very easy to deploy.
152 / 168

operational tasks
Recovering Node
153 / 168

Recovering Nodes/Members
The old master (mysql3) got killed.
MySQLgot restarted automatically by systemd
Let´s addmysql3 back to the clsuter
154 / 168

LAB10: Recovering Nodes/Members
[mysql3 ~]# mysqlsh
mysql-js> c root@mysql4:3306 # The current master
mysql-js> cluster.rejoinInstance("root@mysql3:3306")
Rejoining the instance to the InnoDB cluster. Depending on the original
problem that made the instance unavailable, the rejoin operation might not be
successful and further manual steps will be needed to x the underlying
problem.
Please monitor the output of the rejoin operation and take necessary action if
the instance cannot rejoin.
Please provide the password for 'root@mysql3:3306':
Rejoining instance to the cluster ...
The instance 'root@mysql3:3306' was successfully rejoined on the cluster.
The instance 'mysql3:3306' was successfully added to the MySQL Cluster.
155 / 168

{ "clusterName": "MyInnoDBCluster",
"name": "default",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mysql2:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" },
"mysql3:3306": {
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" },
"mysql4:3306": {
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE" }
}
}
}
156 / 168

Recovering Nodes/Members (automatically)
This time before killing a member of thegroup, we will persistthe configuration ondisk in
my.cnf.
We will use again the sameMySQLcommand as previously
dba.con gureLocalInstance() but this time when all nodes are already
part of the Group.
157 / 168

LAB10: Recovering Nodes/Members (2)
Verify that all nodes are ONLINE.
...
Then on all nodes run:
158 / 168

LAB10: Recovering Nodes/Members (3)
Kill one node again:
[mysql3 ~]# kill -9 $(pidof mysqld)
systemd will restartmysqld and verify if the node joined.
159 / 168

understanding
Flow Control
160 / 168

Flow Control
When usingMySQLGroup Replication, it´s possible that somemembers are lagging behind
the group. Due to load, hardware limitation, etc... Thislagcan becomeproblematicto keep
good certification performance and keep the possible certification failures aslow as
possible.
More problems can occur in multi-primary/write clusters when theapplying queue grows,
the risk to have conflicts with those not yet applied transactions increases.
161 / 168

Flow Control (2)
WithinMySQLGroup Replication´s FC implementation :
the Group is never totally stalled
the node having issues doesn´t send flow control messages to therest of thegroup
asking to slow down
162 / 168

Flow Control (3)
Every member of the Group send somestatistics about its queues (applier queue and
certification queue) to the other members.
Then every node decide to slow down or not if they realize that one node reached the
threshold for one of the queue:
group_replication_ ow_control_applier_threshold (default
is 25000)
group_replication_ ow_control_certi er_threshold
(default is 25000)
163 / 168

Flow Control (4)
So whengroup_replication_ ow_control_mode is set to QUOTA on the
node seeing that one of the other members of thecluster is lagging behind (threshold
reached), it will throttle the write operations to thetheminimumquota.
This quota is calculated basedon thenumber of transactions applied inthelast second,
and then it is reduced below that by subtracting the"over the quota"messages from the
last period.
164 / 168

LAB10: Flow Control
During this last lab, we will reduce theflow control threshold on the Primary Master:
mysql> show global variables like '% ow%';
+----------------------------------------------------+-------+
+----------------------------------------------------+-------+
| group_replication_ ow_control_applier_threshold | 25000 |
| group_replication_ ow_control_certi er_threshold | 25000 |
| group_replication_ ow_control_mode | QUOTA |
+----------------------------------------------------+-------+
3 rows in set (0.08 sec)
mysql> set global group_replication_ ow_control_applier_threshold=100;
165 / 168

LAB10: Flow Control (1)
And now we block all writes onone of theSecondary-Masters:
mysql> ush tables with read lock;
And we check how the queueis growing:
mysql> SELECT * FROM sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
+------------------+-----------+---------------------+----------------------+
| YES | NO | 487 | 0 |
+------------------+-----------+---------------------+----------------------+
Did you notice something on theapplication when thethreshold was reached ?
166 / 168

LAB10: Flow Control (2)
If nothing happened, please increase thetrx rate:
[root@mysql1 ~]# run_app.sh mysql1 --tx-rate=500
When the application's writesare low, you canjust remove thelock and see thequeue and
the effect on theapplication:
mysql> UNLOCK TABLES;
Create flow control again and when you see theapplication writing just a few
transactions, on the Primary-Master, disable theflow control mode:
mysql> set global group_replication_ ow_control_mode='DISABLED';
167 / 168

Thank you !
Any Questions ?
168 / 168

MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup

Similar to MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup (20)

More from Frederic Descamps

More from Frederic Descamps (20)

Recently uploaded

Recently uploaded (20)

MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup