How to operate MySQL InnoDB Cluster with MySQL Shell

MySQL InnoDB Cluster
How to Operate the cluster with MySQL Shell
Frédéric Descamps
MySQL Community Manager
October 2019
Copyright @ 2019 Oracle and/or its affiliates.
2 / 94

Safe Harbor
The following is intended to outline our general product direction. It is intended for information purpose only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any
features or functionality described for Oracle´s product may change and remains at the sole discretion of Oracle
Corporation.
Statement in this presentation relating to Oracle´s future plans, expectations, beliefs, intentions and ptospects
are "forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of
these factors and other risks that a ect our business is contained in Oracle´s Securities and Exchange
Commission (SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading
"Risk Factors". These lings are available on the SEC´s website or on Oracle´s website at
h p://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle
undertakes no duty to update any statement in light of new information or future events.
3 / 94

about.me/lefred
Who am I ?
4 / 94

Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
watched Norsemen
h ps://lefred.be
5 / 94

"A single product — MySQL — with high availability and scaling features baked in;
providing an integrated end-to-end solution that is easy to use."
6 / 94

7 / 94

Components:
MySQL Server
MySQL Group Replication Plugin
MySQL Clone Plugin
MySQL Shell
MySQL Router
8 / 94

One Product: MySQL
All components created together
Tested together
Packaged together
MySQL InnoDB Cluster - Goals
9 / 94

One Product: MySQL
Tested together
Packaged together
Easy to Use
One client: MySQL Shell
Easy packaging
Integrated orchestration
Homogenous servers (*)
Secure automatic provisioning
10 / 94

One Product: MySQL
Tested together
Packaged together
Easy to Use
Easy packaging
Flexible and Modern
SQL and NoSQL together
Protocol Bu ers
Asynchronous API
Developer friendly
11 / 94

One Product: MySQL
Tested together
Packaged together
Easy to Use
Easy packaging
Flexible and Modern
SQL and NoSQL together
Protocol Bu ers
Asynchronous API
Developer friendly
Completely Open Source -- GPL!
No license required to have High Availability
12 / 94

Heart of MySQL InnoDB Cluster
MySQL Group Replication
13 / 94

14 / 94

High Available Distributed MySQL database
Open Source -- GPL
Distributed fault tolerance
Automatic failover
Active/Active update anywhere (limits apply)
Automatic membership con guration
Adding/removing members
Network partitions, failures
Con ict detection and resolution
Prevents data loss
15 / 94

Implementation of Replicated Database State Machine
Total Order - Writes
XCOM - Paxos implementation
Con gurable Consistency Guarantees
eventual consistency
8.0+: per session & global read/write consistency
Using MySQL replication framework by design
binary logs, relay logs
GTIDs: Global Transaction IDs
Generally Available since MySQL 5.7
Supported on all platforms: linux, windows, solaris, macosx, freebsd
16 / 94

MySQL Router
Transparent Routing To MySQL
17 / 94

provides transparent routing between your application
and back-end MySQL Servers
MySQL Router
18 / 94

provides transparent routing between your application
and back-end MySQL Servers
Transparent Access to Database Architecture
Open Source -- GPL
Transparent client connection routing
Load balancing
Application connection failover
Stateless design o ers easy HA client routing
Router as part of the application stack
Native support for InnoDB clusters
Understands Group Replication topology
Utilizes metadata schema on each member
Currently TCP Port each for PRIMARY and NON-
PRIMARY tra c
MySQL Router
19 / 94

MySQL Router
Con gure MySQL Router:
# mysqlrouter --bootstrap mysql1:3306
# systemctl start mysqlrouter
20 / 94

MySQL Router
Recommended to be installed on the application server.
21 / 94

MySQL Router
Recommended to be installed on the application server.
MySQL Router comes also with a REST API to monitor what's going on.
h ps://lefred.be/content/mysqlrouter-8-0-17-and-the-rest-api/
h ps://lefred.be/content/mysql-router-8-0-17s-rest-api-mysql-shell-extensions/
22 / 94

MySQL Shell
The MySQL DBA's Tool
23 / 94

MySQL Shell provides the developer and DBA with a
single intuitive, exible, and powerful interface for all
MySQL related tasks!
MySQL Shell
24 / 94

MySQL Shell provides the developer and DBA with a
single intuitive, exible, and powerful interface for all
MySQL related tasks!
Database Administration Interface
Open Source -- GPL
Multi-Language: JavaScript, Python, and SQL
Naturally scriptable
Supports Document and Relational models
Exposes full Development and Admin API
Classic MySQL protocol and X protocol
MySQL Shell
25 / 94

Ready for some action ?
Setup MySQL InnoDB Cluster
26 / 94

27 / 94

28 / 94

29 / 94

30 / 94

Operating the MySQL InnoDB Cluster
Maintenance & Troubleshooting
31 / 94

Node Provisioning
Before MySQL 8.0.17, any joining node had to be provisioned in advance, using physical or
logical backup for example.
Now, this is not required anymore and everything is automatic thanks to the MySQL
CLONE Plugin !
32 / 94

MySQL InnoDB Cluster Provisioning
binlog binlog
33 / 94

binlog binlog
34 / 94

binlog binlog
35 / 94

binlog binlog
36 / 94

binlog binlog
37 / 94

binlog binlog
hello, I would
like to join !
38 / 94

binlog binlog
I have no
data yet !
39 / 94

binlog binlog
40 / 94

binlog binlog
Option 1: Incremental
donor joiner
binlog
41 / 94

binlog binlog
donor joiner
binlog
42 / 94

binlog binlog
donor joiner
binlog
43 / 94

binlog binlog
binlog
44 / 94

45 / 94

binlog binlog
Option 2: Clone
binlog
donor joiner
clone
46 / 94

binlog binlog
Option 2: Clone
binlog
donor joiner
clone
47 / 94

binlog binlog
hello, I would
like to join !
binlog
48 / 94

binlog binlog
hello, I would
like to join !
binlog
And I have
no data yet !
49 / 94

binlog binlog
We are not
able to provide
you incremental data !
Use CLONE !
binlog
50 / 94

binlog binlog
donor joiner
clone
binlog
51 / 94

binlog binlog binlog
52 / 94

53 / 94

restored
backup
What kind of provisioning will happen ?
54 / 94

restored
backup
Incremental Recovery !
What kind of provisioning will happen ?
55 / 94

56 / 94

Observability
The Admin API's cluster object has a status() method that allows you to see the status of
your cluster as I showed you in the previous demo:
mysqlsh-js> cluster.status()
Provides general information about the cluster and each member status
57 / 94

By default the status()
method disables the
extended mode.
It's possible to have much
more information enabling
it.
Observability - status()
58 / 94

Observability - status({extended})
The extended option allows 4 values (or boolean):
0: disables the command verbosity (default)
1: includes information about the Group Protocol Version, Group name, cluster member
UUIDs, cluster member roles and states as reported by Group Replication and the list of
fenced system variables
2: includes information about transactions processed by connection and applier
3: includes more detailed stats about the replication machinery of each cluster member
59 / 94

Provides information of:
GR Group name
Group protocol version
Cluster members UUIDs
Cluster members roles
and states
Read-only se ings
Observability - status({extended:1})
60 / 94

Transactions processed
by connection and
applier
61 / 94

Provides information of
Detailed stats about the
replication machinery
of each cluster member
62 / 94

Observability - Performance_Schema
Other than the status() in the MySQL Shell, it's possible to get information from
Performance_Schema tables:
63 / 94

Observability - Performance_Schema (2)
64 / 94

65 / 94

66 / 94

More Info: h ps://lefred.be/content/using-the-new-mysql-
shell-reporting-framework-to-monitor-innodb-cluster/
Observability - MySQL Shell UDR
It's also possible to constantly monitor the status of your MySQL InnoDB Cluster using
User-De ned Reports in MySQL Shell:
67 / 94

cluster.options()
the cluster global con guration options
each cluster member se ings
Cluster Options: Check and Change
68 / 94

Cluster Options: Group Specific Settings
Within the MySQL Shell you have the possibility to de ne some Group speci c se ings via
the Admin API:
clusterName: string value to de ne the cluster name.
exitStateAction: string value indicating the group replication exit state action.
memberWeight: integer value with a percentage weight for automatic primary election
on failover.
consistency: string value indicating the consistency guarantees that the cluster
provides.
69 / 94

Cluster Options: Group Specific Settings
expelTimeout: integer value to de ne the time period in seconds that cluster members
should wait for a non-responding member before evicting it from the cluster.
autoRejoinTries: integer value to de ne the number of times an instance will a empt to
rejoin the cluster after being expelled.
disableClone: boolean value used to disable the clone usage on the cluster.
70 / 94

Cluster Options: Instance Specific Settings
Within the MySQL Shell you can de ne some Group Replication speci c se ings via the
Admin API that will only a ect the targeted instance:
exitStateAction: string value indicating the group replication exit state action.
(ABORT_SERVER, READ_ONLY and OFFLINE_MODE [8.0.18])
memberWeight: integer value with a percentage weight for automatic primary election
on failover.
autoRejoinTries: integer value to de ne the number of times an instance will a empt to
rejoin the cluster after being expelled.
71 / 94

Cluster Options: Instance Specific Settings
label: a string identi er of the instance.
72 / 94

What guarantees do we have that:
We always read the most up-to-date
data?
We do not read stale data of an
evicted server?
Dirty reads do not happen?
Consistency Levels
MySQL InnoDB Cluster operates, by default, in single-primary mode:
Apps and users write to the primary instance
73 / 94

mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
| Variable_name | Value |
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+

Consistency Levels
The default consistency level is EVENTUAL. This means that there is no synchronization point
for the transactions, when you perform a write on a node, if you immediately read the
same data on another node, it is eventually there.
74 / 94

mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
| Variable_name | Value |
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+

Consistency Levels
The default consistency level is EVENTUAL. This means that there is no synchronization point
for the transactions, when you perform a write on a node, if you immediately read the
same data on another node, it is eventually there.
Since MySQL 8.0.16, we have the possibility to
set the synchronization point at read or at
write or both (globally or for a session).
75 / 94

Consistency: BEFORE (READ)
As a DBA, I want to load balance my reads without deploying additional
restrictions on which server I read from to avoid reading stale data, my group
writes are much more than my group reads.
76 / 94

As a DBA, I want to load balance my reads without deploying additional
restrictions on which server I read from to avoid reading stale data, my group
writes are much more than my group reads.
As a developer, I want specific transactions in my workload to always read up-
to-date data from the group, so that whenever that sensitive data is updated, I
will enforce that reads shall read the most up to date value.
77 / 94

78 / 94

Consistency: AFTER (WRITE)
I want to load balance my reads without deploying additional restrictions on
which server I read from to avoid reading stale data, my group writes are
much less than my group reads.
79 / 94

I want to load balance my reads without deploying additional restrictions on
which server I read from to avoid reading stale data, my group writes are
much less than my group reads.
I have a group that mostly does reads-only, I want my read-write transactions
to be applied everywhere once they commit, so that subsequent reads are done on
up-to-date data that includes my latest write. Without paying at reads.
80 / 94

81 / 94

Consistency: BEFORE_AND_AFTER
I want that my application to replicate data as close as possible to synchronous.
And of course I'm OK to pay the required price !
82 / 94

Consistency: BEFORE_AND_AFTER
It's also possible and recommended for the most restrictive levels to de ne it at statement
session level:
83 / 94

84 / 94

Change Primary
The leader election happens when the current leader is missing from the the majority of
members. This is an automatic process but as a DBA, you can in uence on the result of
such election by se ing weight to nodes individually:
85 / 94

Change Primary
But you can also change the Primary master on demand:
86 / 94

87 / 94

Topology Modes
By Default, we use Single-Primary Mode (only one node gets the writes), but it's possible to
run in Multi-Primary Mode too:
88 / 94

Security
Internal recovery accounts have randomly secure generated passwords
But security policies may enforce passwords lifetime
Support to reset a cluster’s recovery accounts credentials added [8.0.18]
89 / 94

Logging levels:
1: none
2: internal
3: error
4: warning
5: info
6, 7, 8: debug
Logging
$ mysqlsh –-log-level=8
90 / 94

Verbose output level:
[1, 4]
enables diagnostic
messages output
Logging
$ mysqlsh –-verbose=4
91 / 94

DBA Log SQL level 8.0.18:
0 (default): no logging
1: log all queries except
SELECT and SHOW
2: log all queries
Logging AdminAPI queries (SQL)
$ mysqlsh --dba-log-sql=2
92 / 94

93 / 94

Thank you !
94 / 94

How to operate MySQL InnoDB Cluster with MySQL Shell

More Related Content

How to operate MySQL InnoDB Cluster with MySQL Shell