Lessons learned when managing MySQL in the Cloud

© Pythian Services Inc. 2025
Lessons learned when
managing MySQL in the
Cloud
March 2025

Pythian Services Inc. | Confidential | 3
About Me
󰞵 Principal Consultant: Pythian (OSDB Practice)
🎓 Education: MSc in Software Engineering
📜 Certifications
● MongoDB Certified DBA
● Oracle Professional MySQL 5.7
● Terraform Associate Certified
● GCP Professional Architect
💻 Expertise: Bash, Python, Hybrid Cloud
🌟 Personal: Husband and Father, Avid Traveler, Speaker
🌐 Social
@igorle doncovski in/igorle

🛠 Hands-On Experience
💡 No GenAI Content (except…)
☁ Cloud-Focused (DBaaS)
❌ Not a Training Session
🌱 Beginner Friendly
About

🎤 Introduction
⚙ Conﬁguration and Installation
🤖 Automation
📦 Version Control and Upgrades
📊 Performance Tuning
📈 Cost-Effective Scalability
🌍 High Availability
📉 Monitoring
♻ Disaster Recovery
🔒 Security
💬 Q&A
Agenda

Automation Scalability Cost Efficiency Collaboration
Resilience and DR
Security and Compliance
Innovation
Beneﬁts of Cloud Adoption

Infrastructure as Code
● Terraform (OpenTofu)
● Cloud Formation
● Cloud Development Kit
● Helm (Kubernetes)
● Ansible
● Chef
● Puppet
● Reverse Engineering

Infrastructure as Code
provider "aws" {
region = "us-west-1" # Change to your preferred region
}
resource "aws_instance" "example" {
ami = "ami-0c55b159cbfafe1f0" # Valid AMI ID for the region
instance_type = "t2.micro"
root_block_device {
volume_size = 20 # OS disk size in GB
}
ebs_block_device {
device_name = "/dev/sdb"
volume_size = 500 # MySQL partition size in GB
}
network_interface {
network_interface_id = aws_network_interface.example.id
device_index = 0
}
tags = {
Name = "MyEC2Instance"
}
}
resource "aws_network_interface" "example" {
subnet_id = "subnet-d744039d" # Replace with your subnet ID
private_ips = ["10.0.1.100"]
}
provider "google" {
project = "my-gcp-project" # Replace with valid GCP project ID
region = "us-west1"
}
resource "google_compute_instance" "example" {
name = "my-gcp-instance"
machine_type = "e2-medium"
zone = "us-west1-a"
boot_disk {
initialize_params {
image = "debian-cloud/debian-11" # Replace with a valid image
size = 20 # OS disk size in GB
}
}
attached_disk {
source = google_compute_disk.mysql_disk.id
device_name = "mysql-disk"
}
network_interface {
network = "default"
subnetwork = google_compute_subnetwork.example.id
network_ip = "10.0.1.100"
}
}
resource "google_compute_disk" "mysql_disk" {
name = "mysql-disk"
type = "pd-ssd"
size = 500 # MySQL partition size in GB
zone = "us-west1-a"
}
resource "google_compute_subnetwork" "example" {
name = "example-subnet"
ip_cidr_range = "10.0.1.0/24"
network = "default"
region = "us-west1"
}

Self managed MySQL
🛠 Flexibility & Customization
● Full control over MySQL version, storage, and conﬁgurations
● Ability to ﬁne-tune performance settings (OS and Database layer)
● Choose instance types based on workload needs
● Similar experience for all setups, making migration easier
📉 Management Overhead
● OS patching, MySQL upgrades, backups, and monitoring
● Scaling requires downtime or complex automation
● Increased complexity and lack of expertise
● No managed failover, needs custom HA solutions (e.g., Orchestrator)

Self Managed vs DBaaS
✅ Self Managed
● Workloads needing custom MySQL tuning
● Applications requiring speciﬁc MySQL versions
or extensions
● Teams with DBA expertise to handle operations
● Custom OS patching and tuning
🛠 Managed Database Service
● Automated backups with PITR, failover, and
scaling
● Your workload is highly dynamic and beneﬁts
from serverless options
● Fully managed security & compliance without
manual effort

● High Availability (Production, Dev/Test, Free Tier)
● Multi AZ deployment (standby, replicas)
● Cost and Pricing (On Demand, Reserved)
● Storage Autoscaling
● Parameter Group (Flags) for GLOBAL VARIABLES
● Primary instance, Secondary instances
● Data Migration (DMS, Logical Backup, xtrabackup …)
● Performance (Query) Insights (7 days free tier)
● Backup storage (up to 35 days with no extra cost)
DBaaS Provisioning

High Availability (AWS RDS)
● Multi AZ deployment (single standby or two readable standbys)
● Read replicas - asynchronous replication

High Availability (GCP CloudSQL)
● Multi AZ deployment - synchronous replication to a standby node
● Read replicas - asynchronous replication

● Storage and IO is not included in the Price (USD)
Pricing
Resource On-Demand Price 1-Year Commitment 1-Year Discount (%) 3-Year Commitment 3-Year Discount (%)
vCPUs $36.208 per vCPU $27.16 25% $17.38 52%
Memory $6.132 per GB $4.60 25% $2.94 52%
HA vCPUs $72.343 per vCPU $54.26 25% $34.72 52%
HA Memory $12.264 per GB $9.20 25% $5.89 52%
Resource On-Demand Hourly Rate 1-Year Reserved Instance 1-Year Savings (%) 3-Year Reserved Instance 3-Year Savings (%)
db.t3.micro $0.02 $0.01 29% $0.01 34%
db.m5.large $0.19 $0.13 30% $0.12 37%
db.r5.xlarge $0.48 $0.33 30% $0.30 37%

Parameter Group
Parameter Groups
● Default parameter group - non modiﬁable
● Custom parameter group (some options also non modiﬁable)
● Best practice, one parameter group per instance (or group of
instances)
● At least one parameter group per Writer and Reader
● Some changes are applied only after instance reboot
● Cluster parameter groups
● Compare parameter groups
Source
Analytics Dashboard
mysql8-c1-writer
mysql8-c1-dashboard
mysql8-c1-analytics
Replication

Database migration
● Cloud native DMS tools
○ Does not always work, hard to troubleshoot (Oracle, MSSQL → MySQL)
● Xtrabackup
○ Works with AWS, also hard to troubleshoot
● mydumper
○ Logical multi threaded, issues with JSON column type
● Mysqldump
○ Single threaded, takes long for large datasets
● MySQL Workbench
○ Single threaded, graphical interface for database migration. (Oracle,
MSSQL → MySQL)
● MySQL Shell
○ Multi threaded, logical backup

Database migration plan
● Create custom parameter group upfront
○ log_bin_trust_function_creators=ON (if importing stored procedures)
○ SUPER, SYSTEM_VARIABLES_ADMIN, FILE privileges are restricted
● Choose the proper instance size that matches your on-prem instance
(CPU, Memory, Disk, Network)
● Skip the multi-AZ initially if loading huge data
● Create an instance with the same DB minor version as your source
instance
● Backups and Performance Insights are free with 7 days retention period
● Maintenance - disable auto minor version upgrades
● Use Deletion protection to prevent accidental deletion of the instance

Database migration execution
● Mysql Shell (Migrating from Google CloudSQL to AWS RDS)
● Single command, util.copyInstance()
mysqlsh --uri mysql://<username>:<password>@<dbass-endpoint-gcp>:3306
util.copyInstance('mysql://<username>@<dbass-endpoint-rds>:3306', {checksum: true, compatibility:
["strip_definers"], users: false, dropExistingObjects: true, dryRun : true})
pt-show-grants -h<dbass-endpoint-gcp> -u<username> --ask-pass > users.sql
mysql -u<username> -h<dbaas-endpoint-rds> -p < users.sql

Replication
CHANGE MASTER to MASTER_HOST=<external_host>, MASTER_USERNAME=<username>, MASTER_PASSWORD=<password>,
MASTER_PORT=<port>, MASTER_LOG_FILE=<binlog>, MASTER_LOG_POS=<position>;
START SLAVE;
CHANGE REPLICATION SOURCE to SOURCE_HOST=<external_host>, SOURCE_USERNAME=<username>,
SOURCE_PASSWORD=<password>, SOURCE_PORT=<port>, SOURCE_LOG_FILE=<binlog>, SOURCE_LOG_POS=<position>;
START REPLICA;
MySQL < 8.4
MySQL >= 8.4

Replication
MySQL < 8.4
MySQL >= 8.4
CHANGE MASTER to MASTER_HOST=<external_host>, MASTER_USERNAME=<username>, MASTER_PASSWORD=<password>,
MASTER_PORT=<port>, MASTER_LOG_FILE=<binlog>, MASTER_LOG_POS=<position>;
START SLAVE;
CHANGE REPLICATION SOURCE to SOURCE_HOST=<external_host>, SOURCE_USERNAME=<username>,
SOURCE_PASSWORD=<password>, SOURCE_PORT=<port>, SOURCE_LOG_FILE=<binlog>, SOURCE_LOG_POS=<position>;
START REPLICA;

Replication
● CALL mysql.rds_set_external_master (host_name, host_port, replication_user_name, replication_user_password,
mysql_binary_log_file_name, mysql_binary_log_file_location, ssl_encryption);
● CALL mysql.rds_set_external_source_gtid_purged(server_uuid, start_pos, end_pos);
● CALL mysql.rds_set_external_master_with_auto_position (host_name, host_port, replication_user_name,
replication_user_password, ssl_encryption, delay);
● CALL mysql.rds_start_replication;
● CALL mysql.rds_reset_external_master;

Replication

● Create Aurora read replica
● Set RDS Primary as read-only
● Conﬁrm replication is in sync
● Promote Aurora as Standalone
● Point Apps to Aurora
Replication - switch to Aurora

Major version upgrade
● Latest major version might not be available as soon as the community version
● Upgrading to next major version might fail if:
○ The current version is not the latest minor version
○ The current instance class is not supported (only current and next
generation classes are supported)
○ Custom parameter group is not created for the next version
● Restore a test instance from snapshot
● Attempt an upgrade on the test instance
● The provider does pre-check for upgrade and report on issues
● Verify application compatibility and deprecated features
● Upgrade secondary nodes prior upgrading Primary
● Promote a Secondary node
● Extended support for EOL version ($$)

Blue - Green Deployments
● Blue - current environment
● Green - staging environment
● Green environment can be changed without affecting
the Blue environment
● Test the Green environment
● Switchover in less than a minute with no App changes
● Can be useful for schema changes on big tables
● Major version upgrades with failback option

● Create the blue-green deployment
○ Verify binary logging is ON
○ Create new parameter groups
○ Create a blue-green deployment
○ Change the binlog retention for the new cluster
● Switchover to next major MySQL
○ Switchover to the green environment
○ Capture the binary log ﬁle name and position
○ Delete the blue-green deployment
○ Set the previous (old) server to NOT writeable
○ Verify the old server has been set to read-only
● Setup replication from new MySQL to old
○ Create a replication user on the new writer
○ Set up reverse replication
BG Deployments - major upgrade

Percona toolkit
Collection of advanced open source command-line tools
pt-show-grants -h<endpoint.amazonaws.com> -u<username> --ask-pass > users.sql
pt-online-schema-change --alter 'ADD COLUMN id INT' h=<endpoint.amazonaws.com> D=<database>,t=<table>;
pt-duplicate-key-checker -h<endpoint.amazonaws.com> -u<username> --ask-pass > duplicate_indexes.txt
pt-query-digest --processlist h=<endpoint.amazonaws.com> -u<username> --run-time=300s --daemonize --ask-pass >
query_digest_$(date '+%Y-%m-%d').txt
pt-mysql-summary -h<endpoint.amazonaws.com> -u<username> --ask-pass > mysql_summary.txt
Bastion Host

Monitoring - Performance Insights

Monitoring - PMM

🖥 System Resources
● CPU Overload
When the database uses too much
CPU power, it struggles to process
queries efficiently
● Slow Disk Access
If the database has to read or write
a lot of data, it can cause delays
● Not Enough Memory
If there isn’t enough memory,
queries take longer reading data
from disk
📊 Workload
● Slow queries
Some queries take a long time to
ﬁnish, missing index is common
reason
● Complex queries
Queries that join many tables or
process large amounts of data take
longer to run. Partial indexes or
queries doing analytics
● High concurrency
When too many users are running
queries at the same time, it can
slow things down
Common Performance Issues
⚙ Conﬁguration
● Wrong Settings
Some database settings might not
be ideal for the workload, leading to
reduced performance
● Repeated Queries
Queries run 1000s of times without
adding caching layer

Horizontal
Vertical
us-west-1
Scaling
db.t3.micro
db.t4g.xlarge
us-west-1
db.t3.micro db.t3.micro
us-west-2
db.t3.micro db.t3.micro
Write traffic Read traffic Replication
Primary Replica Intermediate Primary

Vertical Scaling

Horizontal Scaling
● Scalability
● High Availability
● Shard Management
● Topology Management
● Monitoring
● Support for Transactions
● Query route

Autoscaling
● Aurora only feature
● Capacity
○ Min 0 replica
○ Max 15 replicas
● Target Metric
○ CPU Utilization
○ Average connections
● Cooldown period (default 5 min)

🔑 IAM for access, store credentials in Secrets Manager
🔒 Principle of least privilege granting minimal permissions
🏠 Deploy the database and application in the same VPC
🌍 Don’t expose the instance publicly, use private subnets and
VPNs where necessary
🔐 Encryption at REST. Storage encryption to protect sensitive
data. Use TLS/SSL for in-transit encryption
📲 Multi-factor authentication where possible
📅 Track CVEs and keep the database on the latest minor version
🔍 Enable and conﬁgure audit plugin where possible
Security

Prompt engineering
INSERT INTO `t1` (`p1`, `pp1`, `c1`, `v1`, `name`, `zzzz`, `tt`)
SELECT
FLOOR(RAND() * 1000), -- Random integer for p1
FLOOR(RAND() * 1000), -- Random integer for pp1
FLOOR(RAND() * 1000), -- Random integer for c1
FLOOR(RAND() * 5000) + 1, -- v1 must be NOT NULL
CONCAT('Name_', FLOOR(RAND() * 10000)), -- Random name
FLOOR(RAND() * 1000), -- Random integer for zzzz
FLOOR(RAND() * 1000) -- Random integer for tt
FROM (SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4
UNION ALL
SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8
UNION ALL
SELECT 9 UNION ALL SELECT 10) AS a
CROSS JOIN (SELECT 1 AS b FROM information_schema.tables LIMIT 1000) AS b;

Cloud Only
New MySQL Hypergraph Optimizer - OCI Heatwave
Vector Store and GEN AI - OCI Heatwave
MySQl Autopilot Indexing - OCI Heatwave
Javascript Stored Programs Support - OCI Heatwave
Aurora features (parallel query, ML) - AWS
Aurora Global Database - AWS
Vector Store - GCP
Adaptive Caching Layer - GCP

Lessons learned when managing MySQL in the Cloud

Recommended

More Related Content

Recently uploaded (20)

Featured (20)

Lessons learned when managing MySQL in the Cloud