Query Optimization With MySQL 8.0 and MariaDB 10.3 - The Basics - FileId - 160092 PDF

Query Optimization with MySQL 8.0 and MariaDB 10.
3: The Basics
Query Optimization With MySQL

8.0 and MariaDB 10.3:
The Basics
Jaime Crespo
Percona Live Europe 2018
-Frankfurt, 5 Nov 2018-
dbahire.com/pleu18
© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0

Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Agenda
1. Introduction 6. Joins
2. Query Profiling 7. Subqueries
3. Access Types and Basic 8. FULLTEXT, Geodata Search

Indexing Techniques and JSON support
4. Multi-Column Indexing 9. Conclusions
5. Break
© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 2

Query Optimization with MySQL 8.0 and MariaDB

10.3: The Basics
INTRODUCTION

This is me fighting bad query

performance
●
Sr. Database Administrator
at Wikimedia Foundation
●
Used to work as a trainer
for Oracle (MySQL), as a
Consultant (Percona) and
as a Freelance
administrator
(DBAHire.com)

Example Application (I)
• Wiktionary (and all Wikimedia projects’ data) is

licensed under the Creative Commons BY-SA-3.0
License and is Copyright its Contributors
Example Application (II)
• OSM Database is licensed under the Open

DataBase License and is Copyright OpenStreetMap
Contributors
Install the example databases

●
Downloads and instructions at:
http://dbahire.com/pleu18
– Requirements: a MySQL 8.0 or MariaDB 10.3
installation
– The Wiktionary and OSM extracts
●
Import them by doing:
$ tar xf <file> && cd <dir> && myloader -d .

Alternative: Query the Live

Database
●
https://quarry.wmflabs.org/


10.3: The Basics
QUERY PROFILING

Which Queries Should I Optimize

First?
●
Tools:
– Application-level profiling
– General monitoring metrics (CPU, IO, etc.)
– Aggregated PROCESSLIST
– pt-query-digest
– PERFORMANCE_SCHEMA
●
They are not exclusive to each other
Application-Level profiling

General Monitoring Metrics

New index added
at 15:30h

pt-query-digest
• It is a 3rd party tool written in Perl, originally
created by Baron Schwartz
• It requires activation of the slow log: Be careful
with extra
– SET GLOBAL slow_query_log = 1; IO and
– SET long_query_time = 0; latency!
• In Percona Server and MariaDB it can provide extra
information:
– SHOW GLOBAL VARIABLES like 'log_slow_verbosity';

pt-query-digest Execution (I)

# 1094.7s user time, 9.4s system time, 141.22M rss, 205.21M vsz
# Current date: Wed Jul 1 07:32:28 2015
# Hostname: db1018
# Files: STDIN
# Overall: 4.66M total, 640 unique, 53.47 QPS, 0.02x concurrency _________
# Time range: 2015-06-30 07:00:10 to 2015-07-01 07:11:37
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 1320s 1us 3s 283us 332us 3ms 152us
# Lock time 238s 0 13ms 51us 93us 39us 52us
# Rows sent 5.02M 0 4.16k 1.13 1.96 8.69 0.99
# Rows examine 9.50M 0 232.93k 2.14 3.89 261.15 0.99
# Merge passes 0 0 0 0 0 0 0
# Query size 1.06G 17 67.89k 243.89 511.45 368.99 192.76
# Boolean:
# Filesort 8% yes, 91% no Actual execution
# Full scan 94% yes, 5% no
# Priority que 3% yes, 96% no on Wikipedia
# Tmp table 29% yes, 70% no production
# Tmp table on 1% yes, 98% no servers
servers
pt-query-digest Execution (II)

# Profile
# Rank Query ID Response time Calls R/Call V/M Item
# ==== ================== ============== ======= ====== ===== ============
# 1 0xSANITIZED 242.2765 18.4% 691005 0.0004 0.00 SELECT revision page user
# 3 0xSANITIZED 162.8476 12.3% 1025179 0.0002 0.00 SELECT page
# 5 0xSANITIZED 66.8302 5.1% 354562 0.0002 0.00 SELECT page revision
# 6 0xSANITIZED 57.0374 4.3% 211631 0.0003 0.00 SELECT page revision
# 7 0xSANITIZED 44.0751 3.3% 6925 0.0064 0.07 SELECT page categorylinks
category
# 8 0xSANITIZED 35.0655 2.7% 9689 0.0036 0.00 SELECT text
# 9 0xSANITIZED 29.4363 2.2% 152259 0.0002 0.00 SELECT page
# 10 0xSANITIZED 24.1864 1.8% 176927 0.0001 0.00 SELECT msg_resource
# 11 0xSANITIZED 23.7016 1.8% 144807 0.0002 0.00 SELECT page_restrictions
# 12 0xSANITIZED 16.6547 1.3% 10135 0.0016 0.03 SELECT revision
# 13 0xSANITIZED 15.0564 1.1% 263809 0.0001 0.00 SET

pt-query-digest Execution (III)

# Query 1: 7.93 QPS, 0.00x concurrency, ID 0xSANITIZED at byte 1553864032
# This item is included in the report because it matches –limit.
# Scores: V/M = 0.00
# Time range: 2015-06-30 07:00:10 to 2015-07-01 07:11:37
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 14 691005
# Exec time 18 242s 163us 91ms 350us 348us 563us 301us
# Lock time 26 63s 47us 7ms 91us 103us 14us 84us
# Rows sent 12 657.18k 0 1 0.97 0.99 0.16 0.99
# Rows examine 6 657.18k 0 1 0.97 0.99 0.16 0.99
# Query size 31 345.42M 501 749 524.16 537.02 9.22 511.45
# String:
# Databases itwiki (225976/32%), enwiktiona... (219461/31%)... 15 more
# Hosts
# Users wikiuser
# Query_time distribution
# 1us
# 10us
# 100us ################################################################
# 1ms #
# 10ms #
# 100ms
# 1s
# 10s+
# Tables
# SHOW TABLE STATUS FROM ènwiktionary` LIKE 'revision'\G
# SHOW CREATE TABLE ènwiktionary`.`revision`\G
# SHOW TABLE STATUS FROM ènwiktionary` LIKE 'page'\G
# SHOW CREATE TABLE ènwiktionary`.`page`\G
# SHOW TABLE STATUS FROM ènwiktionary` LIKE 'user'\G
# SHOW CREATE TABLE ènwiktionary`.ùser`\G
# EXPLAIN /*!50100 PARTITIONS*/
SELECT /* Revision::fetchFromConds SANITIZED */ * FROM `revision`
INNER JOIN `page` ON ((page_id = rev_page)) LEFT JOIN ùser` ON
((rev_user != 0) AND (user_id = rev_user)) WHERE page_namespace = '0' AND
page_title = 'SANITIZED' AND (rev_id=page_latest) LIMIT 1\G

PERFORMANCE_SCHEMA
• Monitoring schema (engine) enabled by
default since MySQL 5.6
– performance_schema = 1 (it is not dynamic)
• Deprecates the old query profiling
• It is way more user-friendly when combined
with the SYS schema/ps_helper (a set of
views and stored procedures created by Mark
Leith)
– Included by default since 5.7.7
Installation of the SYS Schema

<5.7/MariaDB
$ git clone https://github.com/MarkLeith/mysql-sys.git
Cloning into 'mysql-sys'...
remote: Counting objects: 926, done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 926 (delta 35), reused 6 (delta 2)
Receiving objects: 100% (926/926), 452.19 KiB | 225.00
KiB/s, done.
Resolving deltas: 100% (584/584), done.
$ cd mysql-sys/
$ mysql < sys_56.sql
Example Usage: Discovering Unused

Indexes
mysql (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5;
+---------------+-------------------+---------------------------------+
| object_schema | object_name | index_name |
+---------------+-------------------+---------------------------------+
| osm | acls | acls_k_idx |
| osm | changeset_tags | changeset_tags_id_idx |
| osm | current_nodes | current_nodes_timestamp_idx |
| osm | current_nodes | current_nodes_tile_idx |
| osm | current_relations | current_relations_timestamp_idx |
+---------------+-------------------+---------------------------------+
5 rows in set (0.04 sec)
mysql-5.7.8 (osm) > SELECT * FROM current_nodes WHERE tile = 100;
...
mysql-5.7.8 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5;

+---------------+-------------------+---------------------------------+
| object_schema | object_name | index_name |
+---------------+-------------------+---------------------------------+
With enough
| osm | acls | acls_k_idx | activity, it can
| osm | changeset_tags | changeset_tags_id_idx |
| osm | current_nodes | current_nodes_timestamp_idx | help us clean
| osm | current_relations | current_relations_timestamp_idx |
| osm | current_relations | changeset_id | up our schema
+---------------+-------------------+---------------------------------+
Example Usage: Slow Queries

(ordered by server time)
mysql (osm) > SELECT * FROM sys.statement_analysis LIMIT 10\G
*************** 1. row ***************
query: SELECT `way_id` AS rows_examined: 20152155
ìd` , `v` FROM `way_tags` WHERE `v` LIKE ? rows_examined_avg: 1343477
db: osm rows_affected: 0
full_scan: * rows_affected_avg: 0
exec_count: 15 tmp_tables: 0
err_count: 0 tmp_disk_tables: 0
warn_count: 0 rows_sorted: 0
total_latency: 7.83 s sort_merge_passes: 0
max_latency: 1.33 s digest:
avg_latency: 521.84 ms 21f90695b1ebf20a5f4d4c1e5e860f58
lock_latency: 17.94 ms first_seen: 2014-11-01 17:04:51
rows_sent: 6779 last_seen: 2014-11-01 17:05:22
rows_sent_avg: 452

Example Usage: Top Queries

Creating Temporary Tables
mysql (osm) > SELECT * FROM sys.statements_with_temp_tables
WHERE db = 'osm' LIMIT 10\G
*************************** 1. row ***************************
query: SELECT ? AS TYPE , `node_id` A ... gs` WHERE `k` = ? AND `v` = ?
db: osm
exec_count: 11
total_latency: 7.57 s
memory_tmp_tables: 11
disk_tmp_tables: 0
avg_tmp_tables_per_query: 1
tmp_tables_to_disk_pct: 0
first_seen: 2014-11-01 17:33:55
last_seen: 2014-11-01 17:34:45
digest: 5e6e82799b7c7c0e5c57cfe63eb98d5d

Example Usage: Top Queries

Creating Temporary Tables (cont.)
mysql (osm) > SELECT DIGEST_TEXT FROM performance_schema.events_statements_summary_by_digest WHERE
digest = '5e6e82799b7c7c0e5c57cfe63eb98d5d'\G
*************************** 1. row ***************************
DIGEST_TEXT: SELECT ? AS TYPE , `node_id` AS ìd` FROM `node_tags` WHERE `k` = ? AND `v` = ? UNION
SELECT ? AS TYPE , `way_id` AS ìd` FROM `way_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS
TYPE , `relation_id` AS ìd` FROM `relation_tags` WHERE `k` = ? AND `v` = ?
1 row in set (0.00 sec)
mysql-5.7.8 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity'
and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe'
UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and
v='cafe';
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+
| 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where |
| 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where |
| 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where |
| NULL | UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary |
+----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+
4 rows in set, 1 warning (0.01 sec)


10.3: The Basics
ACCESS TYPES AND BASIC
INDEXING TECHNIQUES
EXPLAIN
●
Essential to understand the execution plan of our
queries
– Works on SELECTs, INSERTs, UPDATEs,
REPLACEs, DELETEs and connections
– Fully documented on:
https://dev.mysql.com/doc/refman/8.0/en/expl
ain-output.html

EXPLAIN Example
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM page
WHERE page_title = 'German';
+------+-------------+-------+------+---------------
+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys |
key | key_len | ref | rows | Extra |
+------+-------------+-------+------+---------------
+------+---------+------+--------+-------------+
| 1 | SIMPLE | page | ALL | NULL |
NULL | NULL | NULL | 778885 | Using where |
+------+-------------+-------+------+---------------
Difficult to see
+------+---------+------+--------+-------------+
something
EXPLAIN Example (vertical

format)
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM page WHERE
page_title = 'German'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 778885 Use \G for
Extra: Using where vertical
1 row in set (0.00 sec) formatting

EXPLAIN Example (id)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE Indicates nesting level,
table: page not execution order
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 778885
Extra: Using where

EXPLAIN Example (select_type)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE Not a subquery or a
table: page UNION
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 778885
Extra: Using where

EXPLAIN Example (table)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page Table scanned for
type: ALL this step
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 778885
Extra: Using where

EXPLAIN Example (type)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ALL All rows are read for
possible_keys: NULL this table (FULL
key: NULL TABLE SCAN)
key_len: NULL
ref: NULL
rows: 778885
Extra: Using where

EXPLAIN Example (rows)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 778885 Estimated number
Extra: Using where of rows to be read
1 row in set (0.00 sec) (all table rows)

How to improve performance?

MariaDB [dewiktionary]> SELECT * FROM page
●
Let's add an index on page.page_title:
MariaDB [dewiktionary]> ALTER TABLE page ADD INDEX

page_title (page_title);
Query OK, 0 rows affected (22.70 sec)
Records: 0 Duplicates: 0 Warnings: 0

Index creation results (type)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page type: ref means that an
type: ref equality comparison will
possible_keys: page_title be checked against an
key: page_title index and several results
key_len: 257
ref: const could be returned
rows: 1
Extra: Using index condition

Index creation results

(possible_keys and key)
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ref
possible_keys: page_title index(es) that the
key: page_title optimizer considered
key_len: 257 potentially useful, and
ref: const final index chosen
rows: 1

Index creation results (ref)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ref
possible_keys: page_title
key: page_title
key_len: 257
ref: const Index is compared
rows: 1 with a constant, not
with another table

Index creation results (rows)

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ref
possible_keys: page_title
key: page_title
key_len: 257
ref: const
rows: 1 Only 1 row read. In this
case, estimation is exact
1 row in set (0.00 sec) (thanks to index dive)

Index creation results (query

time)
MariaDB [dewiktionary]> SELECT * FROM page
1 row in set (0.00 sec) Query time has been
reduced substantially

Types of indexes
●
BTREE
– B-TREE in MyISAM, B+TREE in InnoDB
●
HASH
– Only available for MEMORY and NDB
●
FULLTEXT
– Inverted indexes in MyISAM and InnoDB
●
SPATIAL
– RTREEs in MyISAM and InnoDB
Finding “German” with a BTREE

Avast
Boss
German
Goal
Golf
Etch
Harlem
TABLE
Iceberg
Landscape
Leek
Maelstrom
Nasty
Rover
School
Walrus

Finding “German” with a BTREE

Avast
Boss
German
Goal
Golf
Etch
Harlem
TABLE
Iceberg
Landscape
Leek
Maelstrom
Nasty
Rover
School
Walrus

Do indexes always work? (1/2)

●
Can we use an index to make this query
faster?
SELECT * FROM page WHERE page_title like 'Spa%';

It is a range
page_title like 'Spa%'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: range Despite not being an
possible_keys: page_title equality, we can use the
key: page_title index to find the values
key_len: 257
ref: NULL quickly
rows: 563

BTREE Indexes can be used for

ranges, too
Avast
Boss
German
Goal
Golf
Etch
Harlem
TABLE
Iceberg
Landscape
Leek
Maelstrom
Nasty
Rover
School
Walrus

Do indexes always work? (2/2)

●
What about this other query?
SELECT * FROM page WHERE page_title like '%erman';

Let's check with EXPLAIN

page_title like '%erman'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page No index can be used
type: ALL for filtering. A full
possible_keys: NULL table scan is
key: NULL
key_len: NULL performed.
ref: NULL
rows: 778885
Extra: Using where

BTREE Index
Avast
Boss
German
Goal
Golf
Etch
Harlem
TABLE
Iceberg
Landscape
Leek
Maelstrom
? Nasty
Rover
School
Walrus

Btree indexes usage (I)

●
Filtering (“WHERE”)
– Equality (operator '=')
– Ranges (BETWEEN … AND, >, <, >=, <=, like
'prefix%')
– “EXISTS” operators: IN, OR on the same column
●
Ordering
– ORDER BY (indexed columns)
– GROUP BY (indexed columns)

Btree indexes usage (II)

●
Returning values directly from the index
– Covering index
– Functions like max(), min(), etc.
●
Reduced contention (writes)
– Locking less rows, increasing concurrency

type: const
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE
rev_id = 2\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision 'const' is a special case of 'ref',
type: const when the index can assure
possible_keys: PRIMARY that only 1 result can be
key: PRIMARY returned (equality + primary
key_len: 4
ref: const key or unique key). It is faster.
rows: 1
Extra:

type: NULL
rev_id = -1\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: NULL 'NULL' is not really a plan,
type: NULL just an optimization that
possible_keys: NULL allows immediately
key: NULL discarding impossible
key_len: NULL
ref: NULL conditions
rows: NULL
Extra: Impossible WHERE noticed after reading const
tables
type: ref_or_null
MariaDB [osm]> EXPLAIN SELECT * FROM nodes WHERE tile = 1 or
tile is null\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: nodes Equivalent to 'ref', but
type: ref_or_null also takes into account
possible_keys: nodes_tile_idx NULL values
key: nodes_tile_idx
key_len: 5
ref: const
rows: 2
Extra: Using index condition; Using where

type: range (using IN / OR)

page_title IN ('German', 'English', 'Spanish')\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page Despite being a range, its
type: range execution is very
possible_keys: page_title different from ranges
key: page_title using like, between or
key_len: 257
ref: NULL inequality operators
rows: 4

Is this a bug? (1/2)

page_namespace > 200\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ref
possible_keys: name_title An index is used to
key: name_title return pages with
key_len: 4 namespace>200
ref: const
rows: 50
Extra:

Is this a bug? (2/2)

page_namespace < 200\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ALL
possible_keys: name_title The index is not used
key: NULL with ns=0
key_len: NULL
ref: NULL
rows: 778885
Extra: Using where

Using an index is sometimes

suboptimal
This point is not

fixed, and
depends on the
hardware and
buffer state

What index should we add to

make this query faster?
left(rev_timestamp, 6) = '201809'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1376651
Extra: Using where

The table has already an index on

rev_timestamp
MariaDB [dewiktionary]> SHOW CREATE TABLE revision\G
*************************** 1. row ***************************
Table: revision
Create Table: CREATE TABLE `revision` (
...
KEY `rev_timestamp` (`rev_timestamp`),

We need to rewrite the query

rev_timestamp >= '201509' and rev_timestamp < '201510'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys: rev_timestamp
key: rev_timestamp
key_len: 14
ref: NULL
rows: 24588

This transformation is not trivial

or even possible in all cases
substr(rev_timestamp, 5, 2) = '09'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys: NULL Can you think of a way
key: NULL to improve this query?
key_len: NULL
ref: NULL
Rows: 1376651
Extra: Using where

Indexes for Ordering

MariaDB [dewiktionary]> EXPLAIN SELECT * FROM page ORDER BY
page_touched DESC LIMIT 10\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL "Using filesort" indicates that
ref: NULL
rows: 778885 an ordering is needed before
Extra: Using filesort returning the results

If that is frequent, we can create

an index on page_touched...
[dewiktionary]> ALTER TABLE page ADD INDEX page_page_touched(page_touched);
[dewiktionary]> EXPLAIN SELECT * FROM page ORDER BY page_touched DESC LIMIT

10\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page The index does not produce any
type: index
possible_keys: NULL advantage for filtering
key: page_page_touched
key_len: 16
ref: NULL However, it is very effective by
rows: 10
Extra: helping to avoid the sort phase

It can return data in index order

faster
Avast
Boss
German
Goal
Golf
Etch
Harlem
TABLE
Iceberg
Landscape
Leek
Maelstrom
Nasty
Rover
School
Walrus

Trick: ORDER BY NULL avoids

filesort
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision GROUP BY substr(rev_timestamp, 5,
2) = '09'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
There is no good index in this
type: ALL case
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1376651
Extra: Using temporary; Using filesort
MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM revision GROUP BY substr(rev_timestamp, 5,

2) = '09' ORDER BY NULL\G
*************************** 1. row ***************************
...
rows: 1376651 The advantage is not too big, but
Extra: Using temporary
1 row in set (0.00 sec) it avoids the filesort
Does LIMIT improve the

performance? (I)
MariaDB [dewiktionary]> ALTER TABLE page ADD INDEX page_touched (page_touched); […]
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM page ORDER BY page_touched\G
*************************** 1. row ***************************
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 783231
Extra: Using filesort
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM page ORDER BY page_touched LIMIT 10\G
*************************** 1. row ***************************
type: index
possible_keys: NULL In some cases it can be
key: page_page_touched
key_len: 16 essential to allow
ref: NULL
rows: 10 effective usage of the
Extra:
indexes

performance? (II)
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision ORDER BY rev_comment\G
*************************** 1. row ***************************
[...] type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 817636
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision ORDER BY rev_comment LIMIT 10\G
*************************** 1. row ***************************
[…] table: revision
type: ALL
possible_keys: NULL
In other cases, it has no
key: NULL effect on the scanned
key_len: NULL
ref: NULL rows (just on the
rows: 817636
Extra: Using filesort returned ones)

performance? (III)
MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page ORDER BY page_title LIMIT 100\G
*************************** 1. row ***************************
type: index
possible_keys: NULL
key: page_title
key_len: 257
ref: NULL
rows: 100
Extra:
MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page ORDER BY page_title LIMIT 10000, 100\G
*************************** 1. row ***************************
type: ALL
possible_keys: NULL
key: NULL
In this case,
key_len: NULL performance will vary
ref: NULL
rows: 90956 depending on the
offset (not ideal)
Indexes and GROUP BY (no

indexes)
MariaDB [dewiktionary]> EXPLAIN SELECT rev_page, count(*) FROM
revision IGNORE INDEX(rev_page_id, page_timestamp,
page_user_timestamp) GROUP BY rev_page\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL Without indexes, a temporary
ref: NULL table is created to order results
rows: 1376651
Extra: Using temporary; Using filesort
Indexes and GROUP BY

(rev_page_id)
MariaDB [dewiktionary]> EXPLAIN SELECT rev_page, count(*) FROM
revision GROUP BY rev_page\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE The index does not produce any
table: revision
type: index advantage for filtering (there is
possible_keys: NULL no WHERE clause)
key: rev_page_id
key_len: 8
ref: NULL
rows: 1376651 However, thanks to it we avoid a
Extra: Using index sort and a temporary table

type: index, loose index scan and

covering index (1/3)
node_id version k v
490018321N 234234344545 1 leaves deciduous

Accurate as
of 2010 Big
Big Ben
Ben 234234344546 3 name Northolt Road
Carlsberg
deciduous 234234344548 5 gps:latitude 490018321N
Gwynne
Place
London JOSM 234234344549 6 access uncontrolled
Borough of
Southwark
Southwark London Plane
234234344550 1 editor JOSM
Manor
Manor Place
Place
Northolt Road
recycling 234234344551 9 name Big Ben
survey
survey
tree 234234344552 1 source survey
tree
uncontrolled
234234344557 1 name London Plane
With 'type:index', all rows are

read in index order (full index
scan)

node_id version k v
Accurate as
of 2010 Big
Big Ben
Carlsberg
Gwynne
London
Place
JOSM 234234344549 6 access uncontrolled
Borough of
Southwark
Southwark London Plane 234234344550 1 editor JOSM
Manor
Manor Place
Place
Northolt Road 234234344551 9 name Big Ben
recycling
survey
survey 234234344552 1 source survey
tree
tree
uncontrolled 234234344557 1 name London Plane
If we have in addition 'Using

index for group-by' we have the
loose index scan optimization
Loose Index Scan Example

MariaDB [dewiktionary]> EXPLAIN SELECT rev_page,
max(rev_timestamp) FROM revision GROUP BY rev_page\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys: NULL
key: page_timestamp
key_len: 4
ref: NULL
rows: 80980
Extra: Using index for group-by


node_id version k v
Accurate as
of 2010 Big
Big Ben
Carlsberg
Gwynne
Place
London JOSM 234234344549 6 access uncontrolled
Borough of
Southwark
Southwark London Plane 234234344550 1 editor JOSM
Manor
Manor Place
Place
Northolt Road
234234344551 9 name Big Ben
recycling
survey
survey 234234344552 1 source survey
tree
tree
uncontrolled 234234344557 1 name London Plane
If we have in addition 'Using

index' we have the covering index
optimization
Covering Index Example (1/3)

MariaDB [dewiktionary]> ALTER TABLE revision DROP INDEX rev_page_id, drop index page_timestamp,
drop index page_user_timestamp;
Let's start with no indexes
MariaDB [dewiktionary]> EXPLAIN SELECT count(DISTINCT rev_user) FROM revision WHERE rev_page =
31579\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1376651
Extra: Using where
MariaDB [dewiktionary]> SELECT count(DISTINCT rev_user) FROM revision WHERE rev_page = 31579\G
*************************** 1. row ***************************
count(DISTINCT rev_user): 1


MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX revision_rev_page(rev_page);
MariaDB [dewiktionary]> EXPLAIN SELECT count(DISTINCT rev_user) FROM revision WHERE rev_page =
31579\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
Adding an index on rev_page
type: ref increases the speed due to
possible_keys: revision_rev_page
key: revision_rev_page improved filtering
key_len: 4
ref: const
rows: 4038
Extra:
*************************** 1. row ***************************


MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX revision_rev_page_rev_user(rev_page,
rev_user);
MariaDB [dewiktionary]> EXPLAIN SELECT count(DISTINCT rev_user) FROM revision WHERE rev_page = 31579\
G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE rev_page, rev_user does not
table: revision
type: ref increase the index
possible_keys: revision_rev_page,revision_rev_page_rev_user
key: revision_rev_page_rev_user selectiveness, but allows
key_len: 4
ref: const returning results directly
rows: 4038
Extra: Using index from the index
*************************** 1. row ***************************
1 row in set (0.00 sec) The speed difference can be huge

10.3: The Basics
MULTI-COLUMN INDEXES

In many cases, conditions are

applied on more than one column
rev_page = 31579 and rev_timestamp < '2008'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL Assuming there were no previously
possible_keys: NULL created indexes, which would the
key: NULL optimal one be?
key_len: NULL
ref: NULL
rows: 1376651
Extra: Using where

Options for indexes

●
1 index on column (rev_page)
●
1 index on column (rev_timestamp)
●
2 indexes, 1 on (rev_page) and another on
(rev_timestamp)
●
1 multi-column index on (rev_page, rev_timestamp)
●
1 multi-column index on (rev_timestamp,rev_page)
Are these last 2 different from each other?

Would it depend on the query order?
A brief reminder about query parsing

SELECT * FROM revision WHERE rev_page = 31579 and rev_timestamp < '2008'
Order in the query is not

SELECT
important
Fields FROM WHERE
revision. = <
... revision
rev_id
revision. (string
revision. (int constant)
rev_timest constant)
rev_page 31579
amp '2008'

Index on (rev_page)
MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX rev_page (rev_page);
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_page = 31579 and
rev_timestamp < '2008'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision Query time improves
type: ref significantly with this
possible_keys: rev_page index
key: rev_page
key_len: 4
ref: const
rows: 4038
Fewer rows are
Extra: Using where scanned
Adding (rev_timestamp)
MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX rev_timestamp
(rev_timestamp);
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_page =

31579 and rev_timestamp < '2008'\G
*************************** 1. row ***************************
id: 1 In general, only one index can
select_type: SIMPLE
table: revision be used per table access
type: ref
possible_keys: rev_page,rev_timestamp rev_page is preferred
key: rev_page
key_len: 4 over rev_timestamp
ref: const
rows: 4038
Extra: Using where
Forcing the use of

(rev_timestamp)
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision FORCE
INDEX(rev_timestamp) WHERE rev_page = 31579 and rev_timestamp
< '2008'\G
Forcing
*************************** 1. the
row index is worse than type:ALL!
***************************
id: 1
select_type: SIMPLE
table: revision It is a range
type: range access
possible_keys: rev_timestamp
key: rev_timestamp
key_len: 14 A lot more accessed
ref: NULL rows
rows: 688325
Adding (rev_page,
rev_timestamp)
MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX
rev_page_rev_timestamp(rev_page, rev_timestamp);
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_page =

31579 and rev_timestamp < '2008'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys: rev_page,rev_timestamp,rev_page_rev_timestamp
key: rev_page_rev_timestamp
key_len: 18
ref: NULL Reduced number of
rows: 530 rows scanned
Is (rev_timestamp, rev_page) a
better option?
MariaDB [dewiktionary]> ALTER TABLE revision ADD INDEX rev_timestamp_rev_page
(rev_timestamp, rev_page);
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_page = 31579 and
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys: rev_page,rev_timestamp,rev_page_rev_timestamp,rev_timestamp_rev_page
key_len: 18
ref: NULL Previous index is still
rows: 530 preferred, why?

Forcing (rev_timestamp,
rev_page)
INDEX(rev_timestamp_rev_page) WHERE rev_page = 31579 and
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys: rev_timestamp_rev_page
key: rev_timestamp_rev_page
key_len: 18
ref: NULL Only the 1st column* is being
rows: 688325 used effectively for filtering
A compound index produces a single

tree ordered by the 2 values
rev_page = 31579 and (4611, '2015')
rev_timestamp > '2008' (8005,
'2016') (11771, '2013')
(12301,
'2018') (12702, '2014')
(15444,
'2011')
TABLE
(23305, '2006')
(23443,
'2016') (30024, '2003')
(31579,
'2003')
(31579, (31579, '2004')
'2005')
(31579, '2008')
(31579,
(rev_page, '2017')
(428105, '2015')
rev_timestamp)
The alternative index cannot be used

in such an effective way
rev_page = 31579 and ('2001', 355)
rev_timestamp > '2008'
('2004', 107) ('2008', 206)
('2008', 800)
('2008', 31579)
('2008',
55765)
TABLE
('2009', 201)
('2011',
13605) ('2012', 791)
('2014', 999)
('2015', 31579)
('2016', 24)
('2018', 1784)
('2018',
(rev_timestamp, 31579)
('2018', 79405)
rev_page)
Order and column selection

• Range access using >, <, >=, <=, BETWEEN
can only be filtered once effectively, at the
end of an index
• When selecting indexes, prefer columns with
high cardinality (very selective)
– The optimal index can depend on the constants used

Can we filter and sort at the same

time using indexes?
rev_comment='' ORDER BY rev_timestamp ASC\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision This query is slow
type: ALL
possible_keys: NULL because a) the full
key: NULL table scan
key_len: NULL
ref: NULL
rows: 817636 b) Required sort
Extra: Using where; Using filesort after filtering

Adding an index on
(rev_comment, rev_timestamp)
rev_comment_rev_timestamp (rev_comment, rev_timestamp);
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_comment=''
ORDER BY rev_timestamp ASC\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ref Both ‘type: ALL’
possible_keys: rev_comment_rev_timestampand ‘filesort’ have
key: rev_comment_rev_timestamp
key_len: 769 disappeared
ref: const
rows: 266462
This is not always possible

rev_len > 5500 ORDER BY rev_timestamp ASC\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision The range makes impossible
type: range to use the index optimally
possible_keys: rev_len_rev_timestamp
for the ORDER BY: either we
key: rev_len_rev_timestamp filter (rev_len) or sort
key_len: 5
ref: NULL (rev_timestamp)
rows: 38744
Extra: Using index condition; Using filesort

A Strange Game. The Only

Winning Move is Not to Play
mysql (osm) > SELECT * FROM nodes FORCE INDEX(version_idx)
WHERE version < 15 ORDER BY changeset_id;
/* type: range, Using filesort */
2859673 rows in set (30.58 sec)
mysql (osm) > SELECT * FROM nodes FORCE
INDEX(changeset_id_idx) WHERE version < 15 ORDER BY
changeset_id;
/* type: index */
2859673 rows in set (30.92 sec)
mysql (osm) > SELECT * FROM nodes WHERE version < 15 ORDER BY
changeset_id;
/* type: ALL, Using filesort */
2859673 rows in set (16.54 sec)
DESC Indexes (new in 8.0)

●
Regular indexes could also be used to scan
rows in inverse order
– It had a penalty
– It improves certain queries combining
DESC and ASC order
mysql-8.0 [dewiktionary]> ALTER TABLE page ADD INDEX
`page_redirect_namespace_len_DESC`
(`page_is_redirect`,`page_namespace`,`page_len` DESC);
Query OK, 0 rows affected, 1 warning (3.96 sec)
Descending Indexes in Action

mysql-8.0 [dewiktionary]> EXPLAIN SELECT * FROM page WHERE
page_is_redirect = 1 ORDER BY page_namespace, page_len DESC\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: ref
possible_keys:
page_redirect_namespace_len,page_redirect_namespace_len_DESC
key: page_redirect_namespace_len
key_len: 1
ref: const Filesort
rows: 2404
Extra: Using where avoided
Index Condition Pushdown

rev_len_rev_timestamp(rev_len, rev_timestamp);
MariaDB [dewiktionary]> SET optimizer_switch='index_condition_pushdown=on'; EXPLAIN
SELECT * FROM revision WHERE rev_timestamp < '2008' AND rev_len > 10000\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE Index condition pushdown
table: revision
type: range (ICP) enables the engines to
use extra parts of the index
possible_keys: rev_timestamp,rev_timestamp_rev_page,rev_len_rev_timestamp
key: rev_len_rev_timestamp
key_len: 5 while avoiding costly row
ref: NULL
rows: 180270 movements to and from the
Extra: Using index condition SQL layer

ICP optimizations
●
Differences in execution time are more significant
when the extra column condition is very selective
(getting 5x the original performance)
●
ICP is ignored when using covering Index,
potentially making the performance worse

Use of "Handler_*" statistics

• They are post-execution statistics at row level
– Unlike EXPLAIN's "rows" column, they are exact, not a
guess
– They allow comparison of query execution
performance in a deterministic way, independently of
the execution time

"Handler" Stats (indexed)

mysql> SHOW SESSION STATUS like 'Hand%';
+----------------------------+----------+
| Variable_name | Value | Number of times that the first entry
+----------------------------+----------+
| Handler_commit | 80 | of an index was read. It may indicate
| Handler_delete | 0 |
| Handler_discover | 0 | the number of full index scans
| Handler_external_lock | 166 |
| Handler_mrr_init | 0 |
| Handler_prepare | 0 |
| Handler_read_first | 23 | Number of times a row
| Handler_read_key | 736212 |
| Handler_read_last | 0 | has been retrieved
| Handler_read_next
| Handler_read_prev
| 22208001 |
| 0 |
using an index
| Handler_read_rnd | 665215 |
| Handler_read_rnd_next | 14223297 |
| Handler_rollback | 0 | Next row has been requested
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 | in index order (typical for
| Handler_update | 66970 |
| Handler_write | 2869409 | index scans or ranges)
+----------------------------+----------+
"Handler" Stats (unindexed)

mysql> SHOW SESSION STATUS like 'Hand%';
+----------------------------+----------+
| Variable_name | Value | A row has been requested in a
+----------------------------+----------+
| Handler_commit | 80 | specific position (typical for joins
| Handler_discover | 0 | or order by without indexes)
| Handler_external_lock | 166 |
| Handler_mrr_init | 0 |
| Handler_prepare | 0 |
| Handler_read_first | 23 | Request to read the
| Handler_read_last | 0 | next row in “table
| Handler_read_next | 22208001 |
| Handler_read_prev | 0 | order” (typical for full
| Handler_read_rnd
| Handler_read_rnd_next
| 665215 |
| 14223297 |
table scans)
| Handler_rollback | 0 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 66970 | Insertions in SELECTS may
| Handler_write | 2869409 |
+----------------------------+----------+ indicate temporary tables
Comparing statistics of the

previous indexes (no indexes)
MariaDB [dewiktionary]> FLUSH STATUS;
MariaDB [dewiktionary]> SELECT * FROM revision IGNORE INDEX(rev_page, rev_timestamp, rev_page_rev_timestamp,
rev_timestamp_rev_page) WHERE rev_page = 31579 and rev_timestamp < '2008';
MariaDB [dewiktionary]> SHOW STATUS like 'Hand%';
+----------------------------+---------+
| Variable_name | Value |
+----------------------------+---------+
| Handler_commit | 1 |
…
| Handler_read_first | 0 |
| Handler_read_last | 0 |
| Handler_read_prev | 0 |
| Handler_read_rnd_deleted | 0 |
| Handler_read_rnd_next
| Handler_rollback
| 1430043 |
| 0 |
Typical result for a
…
full table scan
+----------------------------+---------+

Index on (rev_page)
+----------------------------+-------+
+----------------------------+-------+
… Using the index, request the
| Handler_read_first | 0 | first row with rev_page=31579
| Handler_read_prev | 0 | Then, scan them one by one in
| Handler_read_rnd | 0 | index order
…
+----------------------------+-------+
Index on (rev_timestamp)
+----------------------------+---------+ ICP will be explained later,
+----------------------------+---------+ let's ignore it for now
… Using the index, request the first
| Handler_read_first | 0 | row where rev_timestamp<2008
| Handler_read_prev | 0 | Then, scan them one by one in
| Handler_read_rnd | 0 | index order (more are matched)
…
+----------------------------+---------+
Index on (rev_page,
rev_timestamp)
+----------------------------+-------+
+----------------------------+-------+ With both conditions covered, we
| Handler_delete | 0 | can find the actual first row that
… matches the condition using the
| Handler_read_first | 0 | index
| Handler_read_prev | 0 | Rows scanned == Rows returned
…
+----------------------------+-------+
Index on (rev_timestamp,
rev_page), no ICP
+----------------------------+---------+
+----------------------------+---------+
| Handler_commit | 1 | Assuming no ICP, exact same
| Handler_delete | 0 | results as with (rev_timestamp).
… The extra column does not help.
| Handler_read_key | 1 | Also, EXPLAIN's row count was
| Handler_read_last | 0 | very off.
…
+----------------------------+---------+
Index on (rev_timestamp,
rev_page), with ICP
+----------------------------+--------+
+----------------------------+--------+
…
| Handler_icp_attempts | 452539 | ICP reduces the number of
| Handler_icp_match | 530 | ‘ENGINE API calls’ significantly,
…
| Handler_read_first | 0 | although making it work more
| Handler_read_last | 0 | internally
…
+----------------------------+--------+
Redundant Indexes
●
Creating all 4 previous indexes in production is
not a great idea
– "Left-most index prefix" allows, for example
(rev_page, rev_timestamp) doing everything
you can do with (rev_page)
– If two indexes have equal selectivity, MySQL
chooses the shortest one

"Left-most index" Example

INDEX(rev_page_rev_timestamp) WHERE rev_page = 31579\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ref
possible_keys: rev_page_rev_timestamp
key_len: 4 Only the first column is used
ref: const
rows: 4038
Extra:

Duplicate Indexes
●
It is very easy to create indexes with the same
exact definition (same columns and ordering)
– Set a convention for index naming (e.g
tablename_column1_column2_idx) – MySQL does
not allow 2 indexes with the same identifier
– Since MySQL 5.6, an warning is thrown if a
duplicate index is created: Still a warning on 8.0
Duplicate index 'page_random2' defined on the table
'dewiktionary.page'. This is deprecated and will be disallowed
in a future release.

pt-duplicate-index-checker
$ pt-duplicate-key-checker h=localhost,u=root,D=dewiktionary
[…]
# rev_timestamp is a left-prefix of rev_timestamp_rev_page
# Key definitions: Simple tool to check
# KEY `rev_timestamp` (`rev_timestamp`),
# KEY `rev_timestamp_rev_page` (`rev_timestamp`,`rev_page`) redundant and
# Column types:
#
#
duplicate indexes
`rev_timestamp` binary(14) not null default '\0\0\0\0\0\0\0\0\0\0\0\0\0\0'
`rev_page` int(10) unsigned not null
# To remove this duplicate index, execute:
ALTER TABLE `dewiktionary`.`revision` DROP INDEX `rev_timestamp`;
# rev_page is a left-prefix of rev_page_rev_timestamp

# Key definitions:
# KEY `rev_page` (`rev_page`),
# KEY `rev_page_rev_timestamp` (`rev_page`,`rev_timestamp`),
# Column types:
# `rev_page` int(10) unsigned not null
# `rev_timestamp` binary(14) not null default '\0\0\0\0\0\0\0\0\0\0\0\0\0\0'
# To remove this duplicate index, execute:
ALTER TABLE `dewiktionary`.`revision` DROP INDEX `rev_page`;
# ############################################# Summary of indexes ########################################

# Size Duplicate Indexes 47836942
# Total Duplicate Indexes 4
# Total Indexes 285

Invisible Indexes (new in 8.0)

●
Before removing a (believed) redundant or
duplicate index, make it “invisible” to
prevent long-running outages:
mysql-8.0 [dewiktionary]> ALTER TABLE page ALTER INDEX

page_len INVISIBLE;
mysql-8.0 [dewiktionary]> SELECT * FROM page FORCE

INDEX(page_len);
ERROR 1176 (42000): Key 'page_len' doesn't exist in table
'page'
"OR"-style conditions over the

same column
rev_page = 790 OR rev_page = 795 OR rev_page = 1024\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE Equivalent to:
table: revision SELECT * FROM revision WHERE
type: range rev_page IN (790, 795, 1024)
possible_keys: rev_page,rev_page_rev_timestamp
key: rev_page
key_len: 4
ref: NULL
rows: 201

Handlers on "IN" / "OR"

conditions over the same column
+----------------------------+-------+
+----------------------------+-------+
| Handler_delete | 0 | Despite identifying themselves
…
| Handler_prepare | 0 | as “range”s, the execution is
| Handler_read_key | 3 | slightly different, one index dive
| Handler_read_last | 0 | (similar to a ref) is done per
| Handler_read_prev | 0 | value. This can be an issue in
| Handler_read_rnd | 0 | conditions with thousands of
| Handler_read_rnd_next | 0 | items.
…
+----------------------------+-------+
"OR"-style conditions over

different columns
●
We cannot use a single index efficiently for
both conditions
– We can scan both conditions separately
and mix the results, discarding duplicates
– Or use an index for one condition and not
for the other
– Index merge allows the use of two indexes
for a single table access simultaneously

Index Merge Example

rev_page = 790 or rev_timestamp < '2004'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: index_merge
possible_keys: rev_page,rev_timestamp,rev_page_rev_timestamp,
rev_timestamp_rev_page
key: rev_page,rev_timestamp Both indexes are used, then
key_len: 4,14
combined using the "union"
ref: NULL
rows: 190 operation
Extra: Using sort_union(rev_page,rev_timestamp); Using
where
INDEX_MERGE Issues
●
Sometimes it is faster to to execute the
sentence using UNION:
– This is especially true with (UNION ALL)
since MySQL 5.7, if you do not care or
expect duplicates
●
There are also intersection merges, but
multi-column indexes are preferred

Disabling optimizer features (I)

●
The optimizer_switch variable allows enabling and
disabling globally or per session many query
optimizer features:
MariaDB [nlwiktionary]> SHOW VARIABLES like 'optimizer_switch'\G
*************************** 1. row ***************************
Variable_name: optimizer_switch
Value:
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=
on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_push
down=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materializatio
n=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=
on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache
=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bk
a=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=o
n,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

Disabling optimizer features (II)

MariaDB [dewiktionary]> SET optimizer_switch='index_merge_sort_union=off';
MariaDB [dewiktionary]> EXPLAIN SELECT * FROM revision

This WHERE
will only rev_page
have = for
effect
790 or rev_timestamp < '2004'\G
the current session.
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys:
rev_page,rev_timestamp,rev_page_rev_timestamp,rev_timestamp_rev_page
key: NULL
key_len: NULL
ref: NULL
rows: 1376651
Extra: Using where
What happens if we have two

ranges?
●
As seen previously, we cannot use efficiently two
range types on the same table access. Alternatives:
– Use only one index for the most selective column
– Use index condition pushdown to get an
advantage
– Change one of the two ranges into a discrete "IN"
comparison/bucketize with a new column
– Use quadtrees or R-TREEs (spatial indexing)

Example of Bucketizing (I)

MariaDB [dewiktionary]> ALTER TABLE revision DROP INDEX
rev_len_rev_timestamp;
MariaDB [dewiktionary]> EXPLAIN SELECT count(*) FROM revision
WHERE rev_timestamp < '2008' AND rev_len > 5500\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: ALL
possible_keys: rev_timestamp,rev_timestamp_rev_page
key: NULL
key_len: NULL
ref: NULL Looks like only an index on
rows: 1376651 (rev_timestamp) or (rev_len)
Extra: Using where would be useful as we have 2
1 row in set (0.00 sec) ranges. ranges.
Example of Bucketizing (II)

MariaDB [dewiktionary]> ALTER TABLE revision ADD COLUMN
rev_len_cat int; 8.0/10.3
Query OK, 0 rows affected (0.01 sec) instant
Records: 0 Duplicates: 0 Warnings: 0 alter-nice!
MariaDB [dewiktionary]> UPDATE revision set rev_len_cat =

IF(rev_len < 10000, rev_len div 1000, 10);
Rows matched: 1430042 Changed: 1430042 Warnings: 0

rev_len_cat_rev_timestamp (rev_len_cat, rev_timestamp);
Example of Bucketizing (III)

MariaDB [dewiktionary]> EXPLAIN SELECT count(*) FROM revision WHERE
rev_timestamp < '2008' AND rev_len > 5500 AND rev_len_cat IN (5, 6,
7, 8, 9, 10)\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys:
rev_timestamp,rev_timestamp_rev_page,rev_len_cat_rev_timestamp
key: rev_len_cat_rev_timestamp
key_len: 19
ref: NULL We did some transformations
rows: 77601
Extra: Using where to both the structure and the
1 row in set (0.00 sec) query.
Even better: Functional Indexes

(I) Indexing
virtual
MariaDB [dewiktionary]> ALTER TABLE revision columns,
DROP INDEX rev_len_cat_rev_timestamp, new in 10.3
DROP COLUMN rev_len_cat,
ADD COLUMN rev_len_cat_virtual int as (IF(rev_len < 10000,
rev_len div 1000, 10)) VIRTUAL,
ADD INDEX rev_len_cat_virtual_rev_timestamp
(rev_len_cat_virtual, rev_timestamp);
Query OK, 1430042 rows affected (1 min 42.35 sec)
Storing the extra content

may be a waste of resources,
lets make it a virtual column
Even better: Functional Indexes

(II)
MariaDB [dewiktionary]> EXPLAIN SELECT count(*) FROM revision WHERE
rev_timestamp < '2008' AND rev_len > 5500 AND rev_len_cat_virtual IN
(5, 6, 7, 8, 9, 10)\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: revision
type: range
possible_keys:
rev_timestamp,rev_timestamp_rev_page,rev_len_cat_virtual_rev_timestamp
key: rev_len_cat_virtual_rev_timestamp
key_len: 19
ref: NULL
rows: 78649 We did some transformations
Extra: Using where to both the structure and the
query.
MyISAM Internals
Index (part of revision.MYI) Data (revision.MYD)
node_id versio k v
234234344545 1 leaves deciduous

490018321N
Accurate as (empty row)
of 2010 Big Ben
Carlsberg 234234344548 5 gps:latitude 490018321N
Carlsberg
deciduous
deciduous
Gwynne 234234344549 6 access uncontrolled
Place
Place
London
London JOSM
(empty row)
Borough of
Southwark London
London Plane
Plane 234234344551 9 name Big Ben
Manor Place
Northolt Road 234234344552 1 source survey
recycling
recycling
survey 234234344557 1 name London Plane
tree
uncontrolled
uncontrolled 234234344552 2 source survey

InnoDB Internals (PRIMARY)

version k v
771 3 name Northolt Road Data

clustered
702 5 gps:latitu 490018321N always
702
605 6 access uncontrolled using the
476 primary
476 1 editor JOSM key
244
356 9 name Big Ben
244 2 source survey
105 7 amenity pub

InnoDB Internals (Secondary)

Index PK
PK / Data
(version) 9 356
7 105 version k v
771 3 name Northolt Road
6 6 605
3 5 702 702 702 5 gps:latitu 490018321N
Secondary
605 6 access uncontrolled
1
476
indexes 3 771
244
476 1 editor JOSM
contain the 2 244

356 9 name Big Ben
primary key 244 2 source
1 name
survey
London Plane
208
value 1 811
105 7 amenity pub
1 476
1 208

Consequences of using InnoDB (I)

●
Every table should have a primary key
– If one is not defined, MySQL will choose an
available NOT NULL unique key
– If that is not possible, an internal 6-byte row
identifier will be generated (not user-accesible)

Consequences of using
InnoDB (II)
●
Inserting in primary key order is much faster
– Less fragmentation/page-split
– Usage of "batch" mode, improving insert speed
●
Using auto-increment keys as primary keys can be
a good idea for InnoDB

Consequences of using
InnoDB (III)
• A very long primary key may increment
substantially the size of secondary keys
– Int or bigint types are recommended instead of UUIDs
or other long strings

Differences in size
mysql (osm) > CREATE TABLE
pk_int (id int PRIMARY KEY .ibd size (no secondary indexes) .ibd size (with secondary indexes)
auto_increment,
42500000
a int,
b int,
c int,
d int); 34000000
Query OK, 0 rows affected 32,505,856
(0.16 sec)
25500000
bytes
mysql (osm) > CREATE TABLE

pk_uuid (id char(36) 20,971,520
PRIMARY KEY, 17000000
a int,
b int, 13,631,488
12,582,912
c int, 8500000
d int);
Query OK, 0 rows affected
(0.04 sec)
0
pk_int pk_uuid

Extended primary key

optimization
●
As the primary key is part of all secondary keys,
this can be used “for free”:
– For row filtering (since MySQL 5.6)
– To return results in primary key order
– To avoid reading data from the table (covering
index)

Extended Primary Key Example

mysql (osm) > EXPLAIN SELECT node_id FROM nodes WHERE
changeset_id = 24284 and node_id <> 146472942\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: nodes
type: range
possible_keys: PRIMARY,changeset_id_idx
key: changeset_id_idx
key_len: 16
ref: NULL
rows: 50
Extra: Using where; Using index


10.3: The Basics
JOINS

(Block) Nested Loop Join

●
Until MySQL 5.5 there was only one algorithm to
execute a JOIN:
node_id version k v
node_id version lat lon

1 1 name Big Benn
1 1 52 0.5
1 1 tourism attraction
1 2 52 0.5
1 2 name Big Ben
2 1 51 1
3 1 53 1.5
3 1 name London Eye

Extra Access type: eq_ref

mysql (osm) > EXPLAIN SELECT * FROM nodes JOIN node_tags
USING(node_id, version) WHERE node_tags.v= 'Big Ben'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: node_tags
type: ref
possible_keys: PRIMARY,v_idx
key: v_idx
key_len: 767
ref: const
rows: 1
eq_ref is similar to ref, but
*************************** 2. row ***************************
id: 1
select_type: SIMPLE allows faster JOINS
table: nodes
type: eq_ref because, by using a unique
possible_keys: PRIMARY,version_idx
key: PRIMARY key, it only has to search
key_len: 16
ref: osm.node_tags.node_id,osm.node_tags.version
one row for each previous
rows: 1
Extra: NULL
result

JOIN Optimization
●
Two main goals:
– Perform an effective filtering on each table
access, if possible using indexes
– Perform the access in the most efficient table
order
●
When joining 3 or more tables in a star schema,
the "covering index" strategy can have a huge
impact

Example: optimize this JOIN (I)

SELECT n.node_id, n.latitude, n.longitude
FROM way_nodes w_n
JOIN way_tags w_t
ON w_n.way_id = w_t.way_id and
w_n.version = w_t.version
JOIN nodes n
ON w_n.node_id = n.node_id
JOIN node_tags n_t
ON n.node_id = n_t.node_id and
n.version = n_t.version
WHERE w_t.k = 'building' and We start without
n_t.k = 'entrance' and secondary indexes
n_t.v = 'main';
Example: optimize this JOIN (II)

************************ 1. row ************************ ************************ 3. row ************************
id: 1 id: 1
select_type: SIMPLE select_type: SIMPLE
table: w_t table: n_t
type: index type: ref
possible_keys: PRIMARY possible_keys: PRIMARY
key: PRIMARY key: PRIMARY
key_len: 783 key_len: 8
ref: NULL ref: osm.w_n.node_id
rows: 1335702 rows: 1
Extra: Using where; Using index Extra: Using where
************************ 2. row ************************ ************************ 4. row ************************
id: 1 id: 1
table: w_n table: n
type: ref type: eq_ref
possible_keys: PRIMARY possible_keys: PRIMARY
key: PRIMARY key: PRIMARY
key_len: 16 key_len: 16
ref: osm.w_t.way_id,osm.w_t.version ref: osm.w_n.node_id,osm.n_t.version
rows: 3 rows: 1
Extra: NULL Extra: Using index
mysql (osm) > SELECT …
Example: optimize this JOIN (III)

mysql (osm) > ALTER TABLE way_tags
ADD INDEX k_idx(k);
Creating an index
on way_tags.k

Example: optimize this JOIN (IV)

********************** 1. row ********************** ********************** 3. row **********************
id: 1 id: 1
table: w_t table: n_t
type: ref type: ref
possible_keys: PRIMARY,k_idx possible_keys: PRIMARY
key: PRIMARY
key: k_idx
key_len: 8
key_len: 767 ref: osm.w_n.node_id
ref: const rows: 1
rows: 452274 Extra: Using where
Extra: Using where; Using index ********************** 4. row **********************
********************** 2. row ********************** id: 1
id: 1 select_type: SIMPLE
select_type: SIMPLE table: n
table: w_n type: eq_ref
type: ref possible_keys: PRIMARY
key: PRIMARY
possible_keys: PRIMARY
key_len: 16
key: PRIMARY ref: osm.w_n.node_id,osm.n_t.version
key_len: 16 rows: 1
ref: osm.w_t.way_id,osm.w_t.version Extra: NULL
rows: 3 4 rows in set (0.00 sec)
Extra: NULL It seems like the index
is not very useful
Example: optimize this JOIN (V)

mysql (osm) > ALTER TABLE node_tags
ADD INDEX k_idx(k);
The order does not seem

to be adequate, let's try
adding an index to start
by accessing node_tags

Example: optimize this JOIN (VI)

********************** 1. row ********************** ********************** 3. row **********************
id: 1 id: 1
table: w_t table: n_t It keeps using
type: ref
type: ref
possible_keys: PRIMARY,k_idx
possible_keys: PRIMARY,k_idx the wrong
key: PRIMARY
key: k_idx
key_len: 767
key_len: 8 order, even if
ref: osm.w_n.node_id
ref: const
rows: 452274
rows: 1 we delete the
Extra: Using where
Extra: Using where; Using index w_t.k_idx
********************** 4. row **********************
********************** 2. row ********************** id: 1
id: 1 select_type: SIMPLE index
select_type: SIMPLE table: n
table: w_n type: eq_ref
type: ref
key: PRIMARY
possible_keys: PRIMARY key_len: 16
key: PRIMARY ref: osm.w_n.node_id,osm.n_t.version
key_len: 16 rows: 1
ref: osm.w_t.way_id,osm.w_t.version Extra: NULL
Extra: NULL

Example: optimize this JOIN (VII)

SELECT STRAIGHT_JOIN n.node_id, n.latitude, n.longitude
FROM node_tags n_t
JOIN nodes n
ON n.node_id = n_t.node_id and
n.version = n_t.version
JOIN way_nodes w_n
ON w_n.node_id = n.node_id
JOIN way_tags w_t
ON w_n.way_id = w_t.way_id and
w_n.version = w_t.version
WHERE w_t.k = 'building' and
n_t.k = 'entrance' and Let's see why rewriting it
n_t.v = 'main'; into this query

Example: optimize this JOIN (VIII)

********************** 1. row ********************** ********************** 3. row **********************
id: 1 id: 1
select_type: SIMPLE
table: n_t
select_type: SIMPLE
table: w_n There is no index
type: ref
type: ALL
on w_n that would
key: k_idx
key_len: 767
key: NULL
key_len: NULL
allow efficient
ref: const ref: NULL access
rows: 2390 rows: 3597858
Extra: Using index condition; Using where Extra: Using where; Using join buffer (Block
********************** 2. row ********************** Nested Loop)
id: 1 ********************** 4. row **********************
select_type: SIMPLE id: 1
table: n select_type: SIMPLE
type: eq_ref table: w_t
possible_keys: PRIMARY type: eq_ref
key: PRIMARY possible_keys: PRIMARY
key_len: 16 key: PRIMARY
ref: osm.n_t.node_id,osm.n_t.version key_len: 783
rows: 1 ref: osm.w_n.way_id,osm.w_n.version,const
Extra: NULL rows: 1

Example: optimize this JOIN (IX)

mysql (osm) > ALTER TABLE way_nodes
ADD INDEX node_id_idx(node_id);

Example: optimize this JOIN (X)

********************** 1. row ********************** ********************** 3. row **********************
id: 1 id: 1
select_type: SIMPLE
select_type: SIMPLE
table: n_t Now it starts with table: w_n
type: ref
the right table type: ref
possible_keys: PRIMARY,node_id_idx
key: k_idx (without key: node_id_idx
key_len: 8
key_len: 767
ref: const STRAIGHT_JOIN) ref: osm.n_t.node_id
rows: 1
rows: 2390
Extra: Using index
********************** 4. row **********************
********************** 2. row **********************
id: 1
select_type: SIMPLE table: w_t
table: n type: eq_ref
type: eq_ref possible_keys: PRIMARY
possible_keys: PRIMARY key: PRIMARY
key: PRIMARY key_len: 783
key_len: 16 ref: osm.w_n.way_id,osm.w_n.version,const
ref: osm.n_t.node_id,osm.n_t.version rows: 1
rows: 1 Extra: Using where; Using index
Extra: NULL 4 rows in set (0.04 sec)

Example: optimize this JOIN (XI)

************************* 1. row ************************* ************************ 3. row ************************
id: 1 id: 1
table: n_t table: w_n
type: ref type: ref
possible_keys: PRIMARY,node_id_idx
possible_keys: PRIMARY,k_idx,k_v_idx
key: node_id_idx
key: k_v_idx
key_len: 1534
An index on (k,v) is key_len: 8
ref: osm.n_t.node_id
ref: const,const
rows: 900
even better rows: 1
Extra: Using index
Extra: Using where; Using index ************************ 4. row ************************
************************* 2. row ************************* id: 1
select_type: SIMPLE table: w_t
table: n type: eq_ref
type: eq_ref
key: PRIMARY
possible_keys: PRIMARY key_len: 783
key: PRIMARY ref: osm.w_n.way_id,osm.w_n.version,const
key_len: 16 rows: 1
ref: osm.n_t.node_id,osm.n_t.version Extra: Using where; Using index
Extra: NULL
mysql (osm) > SELECT ...

“New” JOIN methods

●
MySQL 5.6 added:
– Batch Key Access
●
MariaDB has since 5.3:
– Batch Key Access
– Hash Joins
– Slightly modified versions of the above ones
(with “incremental” buffers to join 3 or more
tables)
Multi-range read
• This optimization orders results obtained
from a secondary key in primary key/physical
order before accessing the rows
– It may help execution time of queries when disk-bound
– It requires tuning of the read_rnd_buffer_size (size of
the buffer used for ordering the results)
• BKA JOINs are based on the mrr optimization

MRR Example (I)

mysql (osm) > EXPLAIN SELECT * FROM nodes WHERE timestamp >= '2013-07-
01 00:00:00' AND timestamp < '2014-01-01 00:00:00'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: nodes
type: range
possible_keys: nodes_timestamp_idx
key: nodes_timestamp_idx
key_len: 5
ref: NULL
rows: 429684
Extra: Using index condition; Using MRR

MRR example (II)

[restart]
mysql> SET optimizer_switch='mrr=off';
mysql> SELECT * FROM nodes WHERE timestamp >= '2013-07-01 00:00:00' AND timestamp <
'2014-01-01 00:00:00';
'2014-01-01 00:00:00';
[restart]
mysql> SET read_rnd_buffer_size=50 * 1024 * 1024;
'2014-01-01 00:00:00';
205617 rows in set (2.39 sec) “Cold” results are
mysql> SELECT * FROM nodes WHERE timestamp >= '2013-07-01significantly
00:00:00' AND better
timestamp <
with
'2014-01-01 00:00:00';
mrr (but it can impact
negatively, too)
Batch Key Access

Buffer
sorted in
physical/PK node_id version k v
node_id version lat lon

order
1 1 name Big Benn
1 1 52 0.5 1 1 tourism attraction
1 2 52 0.5 1 2 name Big Ben
2 1 51 1
3 1 53 1.5
3 1 amenity cafe

Hash Joins
• Only work for equi-joins
Hash table
for faster
access to node_id version k v
node_id version lat lon rows 1 1 name Big Benn

1 1 52 0.5 1 1 tourism attraction
1 2 52 0.5 1 2 name Big Ben
2 1 51 1 1 2 tourism attraction
3 1 53 1.5 3 1 amenity cafe

MySQL Configuration
• BKA requires changes of default optimizer
configuration:
mysql-8.0 (osm) > SET optimizer_switch= 'mrr=on';
mysql-8.0 (osm) > SET optimizer_switch= 'mrr_cost_based=off';
mysql-8.0 (osm) > SET optimizer_switch= 'batch_key_access=on';
– Additionally, configuring the join_buffer_size adequately

MariaDB configuration
mariadb-10.3 (osm) > SET optimizer_switch = 'join_cache_incremental=on';
mariadb-10.3 (osm) > SET optimizer_switch = 'join_cache_hashed=on';
mariadb-10.3 (osm) > SET optimizer_switch = 'join_cache_bka=on';
- Enabled by default
mariadb-10.3 (osm) > SET join_cache_level = 3 (for hash joins)

mariadb-10.3 (osm) > SET join_cache_level = 5 (for BKA)
- Also, configure join_buffer_size appropriately.

- Hash joins, like BKA, are highly dependent on disk-bound DBs to be
effective due to the extra overhead

Nested Loop Join (cold buffers

buffer_pool=100MB, join_buffer=4M)
mariadb-10.3 (osm) > EXPLAIN SELECT changeset_id, ******************** 2. row ********************
count(*) FROM changesets JOIN nodes on id: 1
select_type: SIMPLE
changesets.id = nodes.changeset_id GROUP BY table: nodes
visible\G type: ref
******************** 1. row ******************** possible_keys: changeset_id
id: 1 key: changeset_id
key_len: 8
select_type: SIMPLE ref: osm.changesets.id
table: changesets rows: 19
Extra:
type: index 2 rows in set (0.00 sec)
key: PRIMARY mariadb-10.3 (osm) > SELECT visible, count(*) FROM
changesets JOIN nodes on changesets.id =
key_len: 8 nodes.changeset_id GROUP BY visible;
ref: NULL +---------+----------+
rows: 69115 | visible | count(*) |
+---------+----------+
Extra: Using index; Using temporary; Using | 1 | 2865312 |
filesort +---------+----------+

Hash Join (cold buffers,

buffer_pool=100M, join_buffer=4M)
mariadb-10.3 (osm) > EXPLAIN SELECT ******************** 2. row ********************
changeset_id, count(*) FROM changesets JOIN id: 1
select_type: SIMPLE
nodes on changesets.id = nodes.changeset_id table: nodes
GROUP BY visible\G type: hash_ALL
******************* 1. row ******************* possible_keys: changeset_id
id: 1 key: #hash#changeset_id
key_len: 8
select_type: SIMPLE ref: osm.changesets.id
table: changesets rows: 2781732
type: index Extra: Using join buffer (flat, BNLH join)
key: PRIMARY mariadb-10.3 (osm) > SELECT visible, count(*) FROM
key_len: 8 changesets JOIN nodes on changesets.id =
ref: NULL nodes.changeset_id GROUP BY visible;
+---------+----------+
rows: 69115 | visible | count(*) |
Extra: Using index; Using temporary; +---------+----------+
Using filesort | 1 | 2865312 |
+---------+----------+


10.3: The Basics
SUBQUERIES

Access types:
unique_subquery/index_subquery
mysql-8.0 (osm) > EXPLAIN SELECT * ************** 2. row **************
FROM node_tags WHERE v = 'Big Ben' id: 2
and node_id NOT IN (SELECT node_id select_type: DEPENDENT SUBQUERY
FROM nodes WHERE tile < 100000000)\G table: nodes
************** 1. row ************** type: index_subquery
id: 1
select_type: PRIMARY possible_keys: PRIMARY,nodes_tile_idx
table: node_tags key: PRIMARY
type: ref key_len: 8 Unique
possible_keys: v_idx ref: func subquery
key: v_idx rows: 1 is similar,
key_len: 767 Extra: Using where but using a
ref: const 2 rows in set (0.00 sec)
rows: 1 unique or
Extra: Using where; Using primary
index key

Subqueries in MySQL
●
MySQL versions traditionally had very bad press
regarding subquries
– It was common to recommend rewriting them
(when possible) into JOINS
●
Since MySQL 5.6, its query execution plans have
improved significantly

Lazy Materialization of derived

tables
●
Option available since MySQL 5.6
– Improves the execution time of EXPLAIN (it no
longer needs to execute subqueries)
– Derived tables can be indexed automatically at
execution time to improve its performance

Derived Table Example

mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON
n.changeset_id = changesets.id;
+----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2865312 | |
| 1 | PRIMARY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | n.changeset_id | 1 | Using index |
| 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where |
+----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+
mysql-8.0 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON
n.changeset_id = changesets.id;
+----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------
+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
|
+----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------
+-------------+
| 1 | PRIMARY | changesets | index | PRIMARY | PRIMARY | 8 | NULL | 70917 | Using
index |
| 1 | PRIMARY
|
| <derived2> | ref | <auto_key0> | <auto_key0> | 8
Subquery is Auto-| osm.changesets.id | 40 | NULL
generated
| 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2853846 | Using
where |
not
+----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------
+-------------+
3 rows in set (0.00 sec) executed index
A Common 5.5 Performance

Problem
mysql-5.5.40 (osm) > EXPLAIN SELECT * FROM nodes
WHERE nodes.changeset_id IN (
SELECT changesets.id
FROM changesets
JOIN users
ON changesets.user_id = users.id and users.display_name = 'Steve');
+----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+
+----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+
| 1 | PRIMARY | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where |
| 2 | DEPENDENT SUBQUERY | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index |
| 2 | DEPENDENT SUBQUERY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | func | 1 | Using where |
+----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+
mysql-5.5.40 (osm) > SELECT ...;

This means that
+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+
| node_id | latitude | longitude | changeset_id | visible | timestamp
the subquery is
| tile | version |
+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+
|
|
99890 | 515276425 | -1497621 |
109174 | 515364532 | -1457329 |
552 |
1875 |
executed almost
1 | 2005-10-25 00:35:24 | 2062268512 |
1 | 2006-01-20 00:01:27 | 2062268519 |
1 |
1 |
|
|
276538 | 515324296 | -2094688 |
442987 | 515449207 | -1275650 |
810 |
1941 |
3 million times
1 | 2005-11-25 21:42:53 | 2062267078 |
1 | 2006-01-22 23:50:28 | 2062268628 |
1 |
1 |
| 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 |
| 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 |
| 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 |
+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+
Semijoin Optimization
●
The only way to execute certain IN subqueries was to
execute them with poor strategy
– This forced rewriting of certain queries into JOINS or scalar subqueries,
when possible
●
There are now several additional options (many
automatic):
– Convert to a JOIN
– Materialization (including index creation)
– FirstMatch
– LooseScan
– Duplicate Weedout
The Previous Query is Not a Problem

in 5.6+/MariaDB 5.3+
mysql-8.0 (osm) > EXPLAIN SELECT * FROM nodes
WHERE nodes.changeset_id IN (
SELECT changesets.id
FROM change sets
JOIN users
ON changesets.user_id = users.id and users.display_name = 'Steve');
+----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+
+----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+
| 1 | SIMPLE | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index |
| 1 | SIMPLE | changesets | ALL | PRIMARY | NULL | NULL | NULL | 70917 | Using where |
| 1 | SIMPLE | nodes | ref | changeset_id | changeset_id | 8 | osm.changesets.id | 21 | NULL |
+----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+
mysql-8.0 (osm) > SELECT ...;

+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+
| node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version |
Executed as
+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+
| 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 |
a regular
...
| 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 |
JOIN
+-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+

First Match Strategy

mysql-8.0 (osm) > EXPLAIN SELECT * FROM changesets WHERE id IN (SELECT changeset_id FROM nodes)\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: changesets
partitions: NULL
type: ALL
key: NULL
key_len: NULL
ref: NULL
rows: 70917
filtered: 100.00
Extra: NULL
*************************** 2. row *************************** It is converting
id: 1
select_type: SIMPLE the ref into an
table: nodes
partitions: NULL eq_ref, shot-
type: ref
possible_keys: changeset_id
key: changeset_id
circuiting the
key_len: 8
ref: osm.changesets.id
execution
rows: 33
filtered: 100.00
Extra: Using index; FirstMatch(changesets)
Enabling and disabling

materialization, semijoin, etc
mysql-8.0 (osm) > SHOW VARIABLES like 'optimizer_switch'\G
*************************** 1. row ***************************
Variable_name: optimizer_switch
Value:
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_in
tersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=o
n,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materializat
ion=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_m
aterialization_cost_based=on,use_index_extensions=on,condition_fanout_filter
=on,derived_merge=on


10.3: The Basics
FULLTEXT, GEODATA
SEARCH AND JSON SUPPORT
Fuzzy Search of “hotel/Grand

Hotel/Hotel X/etc”
●
“Typical” way to solve this:
mysql (osm) >

SELECT way_id as id, v
FROM way_tags
WHERE v like '%otel%';
Too slow

Let’s Add an Index

mysql (osm) > ALTER TABLE way_tags ADD INDEX(v);
Still slow, why?
mysql (osm) > SELECT ...;
mysql (osm) > EXPLAIN SELECT way_id as id, v FROM way_tags
WHERE v like '%otel%';
+----+.+-------+---------------+------+---------+------+---------+----------+--------------------------+
| id |.| type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+.+-------+---------------+------+---------+------+---------+----------+--------------------------+
| 1 |.| index | NULL | v | 769 | NULL | 8964000 | 11.11 | Using where; Using index |
+----+.+-------+---------------+------+---------+------+---------+----------+--------------------------+
1 row in set, 1 warning (0.01 sec)

Fulltext Index
mysql (osm) > ALTER TABLE way_tags add FULLTEXT index(v);
mysql (osm) > SELECT 'way' as type, way_id as id, v FROM

way_tags WHERE MATCH(v) AGAINST ('+hotel*' IN BOOLEAN MODE);
mysql (osm) > EXPLAIN …;

+----+.+----------+---------------+------+---------+-----+------+----------+-----------------------------------+
| id |.| type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+.+----------+---------------+------+---------+-----+------+----------+-----------------------------------+
| 1 |.| fulltext | v | v | 0 | | 1 | 100.00 | Using where; Ft_hints: no_ranking |
+----+.+----------+---------------+------+---------+-----+------+----------+-----------------------------------+
Some fields are

useless for FULTEXT
‘Newer’ Fulltext Optimizations

mysql-5.5 (osm) > EXPLAIN SELECT count(*) FROM way_tags_myisam WHERE MATCH(v) AGAINST('hotel');
+----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+
+----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | way_tags_myisam | fulltext | v | v | 0 | | 1 | Using where |
+----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+
mysql-5.5 (osm) > SHOW STATUS like 'Hand%';

+----------------------------+-------+
+----------------------------+-------+
...
...
+----------------------------+-------+

Newer Fulltext Optimizations

(cont.)
mysql-8.0 (osm) > EXPLAIN SELECT count(*) FROM way_tags WHERE MATCH(v) AGAINST('hotel');
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
mysql-8.0 (osm) > SHOW STATUS like 'Hand%';

+----------------------------+-------+
+----------------------------+-------+
| Handler_commit | 1 | It’s counting
...
directly from
| Handler_read_key
| Handler_read_last
| 0
| 0
|
|
the
| Handler_read_next
| Handler_read_prev
| 0
| 0
|
| FULLTEXT
index
...
+----------------------------+-------+

Open Issues and Limitations

●
No postfix support (wildcards)
●
Simple Ranking (and different from MyISAM)
●
No stemming support
●
Some multi-language limitations
More on FULLTEXT InnoDB support:

http://www.drdobbs.com/database/full-text-search-with-innodb/231902587
Alternatives
●
Apache Lucene
– Solr
– Elasticsearch
●
Sphinx
– SphinxSE

Find the Closest Starbucks

(Spatial Search)
mysql (osm) > SET @lat:=50.11745057; SET @lon:= 8.62699463;
mysql (osm) > SELECT n.node_id,
n.longitude/10000000 as longitude,
n.latitude/10000000 as latitude,
sqrt(pow((latitude/10000000 - @lat) * 111231.29, 2) + You are here
pow((longitude/10000000 - @lon) * 71520.91, 2))
as `distance in metres`
FROM nodes n
JOIN node_tags n_t1
ON n.node_id = n_t1.node_id
JOIN node_tags n_t2
WHERE
n_t1.k = 'amenity' and
n_t1.v = 'cafe' and
n_t2.k = 'name' and
n_t2.v like 'Starbucks%'
ORDER BY `distance in metres` ASC
LIMIT 1;
+------------+-----------+----------+--------------------+
| node_id | longitude | latitude | distance in metres |
+------------+-----------+----------+--------------------+
| 3136353405 | 8.6523 | 50.1093 | 2023.6465372574328 |
+------------+-----------+----------+--------------------+
1 row in set, 36 warnings (0.81 sec)

This Query is Slow

mysql [osm]> EXPLAIN SELECT ...\G
*********************** 2. row ***********************
**************** 1. row *************** id: 1
select_type: SIMPLE
table: n_t2
Id: 1 type: ref
select_type: SIMPLE key: PRIMARY
table: n_t1 key_len: 8
ref: osm.n_t1.node_id
type: ALL rows: 1
Extra: Using where
possible_keys: PRIMARY *********************** 3. row ***********************
key: NULL id: 1
select_type: SIMPLE
key_len: NULL table: n
type: ref
ref: NULL possible_keys: PRIMARY
key: PRIMARY
rows: 2210950 key_len: 8
Extra: Using where; ref: osm.n_t1.node_id
rows: 1
Using temporary; Using filesort Extra:

Can We Optimize it?

• We could add a bounding box:
mysql (osm) > ALTER TABLE nodes ADD INDEX (latitude, longitude);
mysql (osm) > SELECT n.node_id,
n.longitude/10000000 as longitude,
n.latitude/10000000 as latitude,
sqrt(pow((latitude/10000000 - @lat) * 111231.29, 2) +
pow((longitude/10000000 - @lon) * 71520.91, 2))
as `distance in metres`
FROM nodes n
JOIN node_tags n_t1
JOIN node_tags n_t2
WHERE
This is not a
n_t1.v = 'cafe' and square, only an
n_t2.k = 'name' and
n_t2.v like 'Starbucks%' and
approximation
n.latitude BETWEEN ((@lat - 1000/111231.29) * 10000000)
AND ((@lat + 1000/111231.29) * 10000000) and
n.longitude BETWEEN ((@lon - 1000/71520.91) * 10000000)
AND ((@lon + 1000/71520.91) * 10000000)
ORDER BY `distance in metres` ASC
LIMIT 1;

We Create an Index… and Force It

mysql-8.0 (osm) > EXPLAIN SELECT ...;
MySQL ignores the
+----+-------------+-------+------------+------+----------+---------+------+------------------+---------+----------+------------------+
| id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra |
| | | | | | _keys | newly| created
len | index, | | | |
+----+-------------+-------+------------+------+----------+---------+------+------------------+---------+----------+------------------+
| 1 | SIMPLE | n_t1 | NULL | ALL | PRIMARY | NULL
why?
| NULL | NULL | 2210950 | 0.00 | Using where; |
| | | | | | | | | | | | Using temporary; |
| | | | | | | | | | | | Using filesort |
| 1 | SIMPLE | n | NULL | ref | PRIMARY, | PRIMARY | 8 | osm.n_t1.node_id | 1 | 5.00 | Using where |
| | | | | | latitude | | | | | | |
| 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n_t1.node_id | 1 | 1.41 | Using where |
+----+-------------+-------+------------+------+----------+---------+------+------------------+---------+----------+------------------+
mysql-8.0 (osm) > EXPLAIN SELECT ... FROM nodes n FORCE INDEX(latitude) ...;
+----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+
| id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra |
| | | | | | _keys | | len | | | | |
+----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+
| 1 | SIMPLE | n | NULL | range | latitude | latitude | 8 | NULL | 949370 | 11.11 | Using where; |
| | | | | | | | | | | | Using index; |
| | | | | | | | | | | | Using filesort |
| 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 1 | 1.41 | Using where |
| 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 1 | 1.41 | Using where |
Performance Most of the gain
+----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+
3 rows in set, 1 warning (0.00 sec) Still many comes from the
comes from the
improvement is
Mysql-8.0 (osm) > SELECT ... FROM nodes n FORCE INDEX(latitude) ...;
not great
rows are covering index, not
0 rows in set (0.26 sec) the filtering
examined
Multiple Range Scans Cannot Be

Optimized with BTREE Indexes
●
We need quadtrees or R-TREE Indexes for
indexing in multiple dimensions
– The latter are implemented in MySQL with
the name “SPATIAL indexes”, as they only
apply to GIS types
●
Spatial indexing is available on MySQL since
5.7.5

Creating a Spatial Index

mysql (osm) > ALTER TABLE nodes
ADD COLUMN coord GEOMETRY NOT NULL SRID 4326;
Records: 0 Duplicates: 0 Warnings: 0 Instant, yay!
(or create it
VIRTUAL)
mysql (osm) > UPDATE nodes
SET coord = point(longitude/10000000,
latitude/10000000);
Query OK, 20216490 rows affected (7 min 10.28 sec)
Rows matched: 20216490 Changed: 20216490 Warnings: 0
mysql (osm) > ALTER TABLE nodes add SPATIAL index(coord);
Query OK, 0 rows affected (1 hour 7 min 13.29 sec)
This is new since
Records: 0 Duplicates: 0 Warnings: 0 5.7
New Query
mysql> SET @area := ST_Envelope(linestring(POINT(@lon - 500/71520.91, @lat - 500/111231.29), POINT(@lon +
500/71520.91, @lat + 500/111231.29)));
mysql> SELECT n.node_id,
x(n.coord) as longitude, We can use any
y(n.coord) as latitude,
st_distance(POINT(@lon, @lat), coord) as distance
shape we want
FROM nodes n thanks to 5.6
JOIN node_tags n_t1 improvements
JOIN node_tags n_t2
WHERE Also substitute
functions like
n_t1.v = 'cafe' and
n_t2.k = 'name' and Envelope with
n_t2.v like 'Starbucks%' and ST_Envelope
st_within(coord, @area)
since 5.7
ORDER BY st_distance(POINT(@lon, @lat), coord) ASC
LIMIT 1;

Better Performance
mysql (osm) > SELECT ...;
+-----------+------------+------------+-----------------------+
| node_id | longitude | latitude | distance |
+-----------+------------+------------+-----------------------+
| XXXXXXXXX | XXXXXXXXXX | XXXXXXXXXX | 0.0014631428672541478 |
+-----------+------------+------------+-----------------------+
This field used to
Empty set (0.11 sec)
be almost useless
(wait for it)
mysql (osm) > EXPLAIN SELECT ...;
+----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+
| id | select | table | parti | type | possible | key | key | ref | rows | filtered | Extra |
| | _type | | tions | | _keys | | _len | | | | |
+----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+
| 1 | SIMPLE | n | NULL | range | PRIMARY | coord | 34 | NULL | 2 | 100.00 | Using where; |
| | | | | | ,coord | | | | | | Using filesort |
| 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where |
| | | | | | | | | node_id | | | |
| 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where |
| | | | | | | | | node_id | | | |
+----+--------+-------+-------+-------+----------+---------+------+---------+------+---------------------------+

Better Filtering
mysql-8.0 (osm) > SHOW STATUS LIKE 'Hand%';
Not using the index: Using the BTREE index: Using the SPATIAL index:
+----------------------------+--------+ +----------------------------+--------+ +----------------------------+-------+
| Variable_name | Value | | Variable_name | Value | | Variable_name | Value |
+----------------------------+--------+ +----------------------------+--------+ +----------------------------+-------+
| Handler_commit | 1 | | Handler_commit | 1 |
| Handler_commit | 1 | | Handler_delete | 0 | | Handler_delete | 0 |
| Handler_delete | 0 | | Handler_discover | 0 | | Handler_discover | 0 |
| Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_external_lock | 6 |
| Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_mrr_init | 0 |
| Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_prepare | 0 |
| Handler_read_first | 0 | | Handler_read_first | 0 |
| Handler_prepare | 0 | | Handler_read_key | 274 | | Handler_read_key | 522 |
| Handler_read_first | 1 | | Handler_read_last | 0 | | Handler_read_last | 0 |
| Handler_read_key | 1914 | | Handler_read_next | 246540 | | Handler_read_next | 5254 |
| Handler_read_last | 0 | | Handler_read_prev | 0 | | Handler_read_prev | 0 |
| Handler_read_next | 1954 | | Handler_read_rnd | 0 | | Handler_read_rnd | 259 |
| Handler_read_rnd_next | 0 | | Handler_read_rnd_next | 0 |
| Handler_read_prev | 0 | | Handler_rollback | 0 | | Handler_rollback | 0 |
| Handler_read_rnd | 1 | | Handler_savepoint | 0 | | Handler_savepoint | 0 |
| Handler_read_rnd_next | 833426 | | Handler_savepoint_rollback | 0 | | Handler_savepoint_rollback | 0 |
| Handler_rollback | 0 | | Handler_update | 0 | | Handler_update | 0 |
| Handler_write | 0 | | Handler_write | 0 |
| Handler_savepoint | 0 | +----------------------------+--------+ +----------------------------+-------+
| Handler_savepoint_rollback | 0 | 18 rows in set (0.00 sec) 18 rows in set (0.00 sec)
+----------------------------+--------+
Geohash Functions
mysql (osm) > SELECT ST_GeoHash(@lon, @lat, 10);
+----------------------------+
| ST_GeoHash(@lon, @lat, 10) |
+----------------------------+
| u0yjh79gr9 |
+----------------------------+
• Useful to index coordinates with a BTREE
– It could be specially useful combined with indexed STORED
columns (emulating quadtrees)
More on Geohashing: http://mysqlserverteam.com/geohash-functions/

GeoJSON Functions
mysql (osm) > SELECT nm.v, ST_AsGeoJson(n.coord)
FROM node_tags n_t
JOIN nodes n USING (node_id, version)
JOIN node_tags nm USING (node_id, version)
WHERE n_t.k='tourism' AND
n_t.v='attraction' AND
nm.k='name';
+-------------------------+------------------------------------------------------------+
| v | ST_AsGeoJson(n.coord) |
+-------------------------+------------------------------------------------------------+
| Schießbuckel | {"type": "Point", "coordinates": [8.4938556, 49.6351754]} |
| AKW Informationszentrum | {"type": "Point", "coordinates": [8.4175585, 49.7060227]} |
| Burg Gleiberg | {"type": "Point", "coordinates": [8.6344314, 50.6150329]} |
| Brüderkirche | {"type": "Point", "coordinates": [9.5042975, 51.3149351]} |
| Römer | {"type": "Point", "coordinates": [8.6816587, 50.1104684]} |
.
.

GeoJSON Functions (cont.)

$ mysql osm -B -e "SET NAMES utf8; SET SESSION group_concat_max_len = 10000; SELECT
CONCAT('{\"type\":\"FeatureCollection\", \"features\":[ ',
GROUP_CONCAT(CONCAT('{\"type\":\"Feature\", \"geometry\":', ST_AsGeoJson(n.coord),
',\"properties\":{\"name\":\"',nm.v,'\"}}')), ' ]}') FROM node_tags n_t JOIN nodes n USING
(node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' and
n_t.v='attraction' AND nm.k='name'"
http://geojsonlint.com/

Older Issues
• Before MySQL 8.0, SRID could be set and retrieved, but all
operations were done in squared euclidean coordinates:
mysql-5.7.5 (osm) > SET @p1 := GeomFromText('POINT(8 50)', 4326); mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p2);
Query OK, 0 rows affected (0.00 sec) +-----------------------+
| st_distance(@p1, @p2) |
+-----------------------+
mysql-5.7.5 (osm) > SET @p2 := GeomFromText('POINT(7 50)', 4326); | 1 |
Query OK, 0 rows affected (0.00 sec) +-----------------------+
mysql-5.7.5 (osm) > SET @p3 := GeomFromText('POINT(8 51)', 4326);
Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p3);
+-----------------------+
mysql-5.7.5 (osm) > SELECT srid(@p1); | st_distance(@p1, @p3) |
+-----------------------+
+-----------+ | 1 |
| srid(@p1) | +-----------------------+
+-----------+ 1 row in set (0.00 sec)
| 4326 |
+-----------+

New 8.0 SRID Support !

mysql-8.0 (osm) > SET @p1 := ST_GeomFromText('POINT(8 50)', mysql-8.0 (osm) > SELECT st_distance(@p1, @p2);
4326); +-----------------------+
Query OK, 0 rows affected (0.00 sec) | st_distance(@p1, @p2) |
+-----------------------+
| 110243.2689139409 |
mysql-8.0 (osm) > SET @p2 := ST_GeomFromText('POINT(7 50)', +-----------------------+
4326); 1 row in set (0.00 sec)
Query OK, 0 rows affected (0.00 sec) Answers in
meters- The
mysql-8.0 (osm) > SELECT st_distance(@p1, @p3);
mysql-8.0 (osm) > SET @p3 := ST_GeomFromText('POINT(8 51)', +-----------------------+
| st_distance(@p1, @p3) | world is no
4326);
+-----------------------+
| 110592.12831394037 |
longer flat!
+-----------------------+
mysql-8.0 (osm) > SELECT ST_srid(@p1); 1 row in set (0.01 sec)
+--------------+
| ST_srid(@p1) | I expected
+--------------+ something around
| 4326 |
+--------------+
71-72 Km, did I do
1 row in set (0.00 sec) something wrong?

JSON Native Data Type

●
Since 5.7.8, MySQL allows columns defined with
the JSON data type: MariaDB
supports it
mysql-8.0 > CREATE TABLE json_test(id int PRIMARY KEY auto_increment,
only as a
content JSON); BLOB/LONGT
Query OK, 0 rows affected (0.03 sec) EXT alias
mysql-8.0 > INSERT INTO json_test (content) VALUES ('{"type":
"correct_json"}'); They get
Query OK, 1 row affected (0.00 sec) validated on
insert
mysql-8.0 > INSERT INTO json_test (content) VALUES ('{"type": insert
"incorrect_json}');
ERROR 3140 (22032): Invalid JSON text: "Missing a closing quotation
mark in string." at position 24 in value (or column) '{"type":
"incorrect_json}'.
JSON functions
●
MySQL includes almost all functions to
manipulate JSON that you may think of:
– Validation test: JSON_TYPE
– Object creation: JSON_ARRAY, JSON_MERGE, ...
– Searching: JSON_EXTRACT
– Modifying: JSON_SET, JSON_INSERT, ...

Indexing JSON
●
JSON Columns cannot be indexed:
mysql [localhost] {msandbox} (test) > ALTER TABLE
json_test ADD INDEX(content);
ERROR 3152 (42000): JSON column 'content' cannot be
used in key specification.
●
However, they can be compared with regular
fields and use indexes thanks to virtual columns


10.3: The Basics
CONCLUSIONS

8.0/10.3 Recently released

• They are mature enough to be used in production
• Latest release doesn’t have as many new features
related to query optimization
– But they have a lot of other features and quality of
life improvements
– They also fix long outstanding query performance
bugs
– MariaDB also catches up with MySQL in some areas

MySQL 8.0 New Features

• MySQL 5.6 seemed Percona Server-inspired
• MySQL 5.7 seemed MariaDB-inspired
• MySQL 8.0 seems Galera and Oracle DB-inspired
– Competition is always good for consumer

Modern SQL is now part of

MySQL!
●
Window functions, recursive CTEs:
WITH RECURSIVE cte (cl_from, cl_type) AS
(
SELECT cl_from, cl_type FROM categorylinks WHERE cl_to = 'Database_management_systems'
-- starting category
UNION
SELECT categorylinks.cl_from, categorylinks.cl_type FROM cte JOIN page ON cl_from =
page_id JOIN categorylinks ON page_title = cl_to WHERE cte.cl_type = 'subcat' --
subcat addition on each iteration
)
SELECT page_title FROM cte JOIN page ON cl_from = page_id WHERE page_namespace = 0
ORDER BY page_title; -- printing only articles in the end, ordered by title
Scan Wikipedia
categories,
page links
recursively!

Many Optimizer Advantages Have to

Be Manually Enabled
●
Modifying on a per-query basis:
SET optimizer_switch='batched_key_access=on';
SET join_cache_level=8; # for MariaDB
in order to take advantage of them, making

some of features useless unless you are fine-
tuning
– I expect that to change in the future

I Herby Declare MyISAM as Dead

• All major MyISAM-only features are
now on MySQL 5.7/8.0
– FULLTEXT
– GIS
– Transportable tables
• Previous blockers such as grant tables (new
InnoDB data dictionary) and temporary tables
are now in InnoDB format

Benchmarks
●
Do not trust first party
benchmarks
– In fact, do not trust 3rd
party benchmarks either
●
Only care about the
performance of your
application running on your hardware

Q&A

Not to Miss
●
Operations track:
TLS for MySQL at Large Scale:
How we do relational data on-
the-wire encryption at the
Wikimedia Foundation
●
Do you want to do query
optimization for a website with
20 Billion views per month?
https://wikimediafoundation.org/about/jobs/

Thank You for Attending!

●
Do not forget, after the session finishes,
to please login with your Percona Live
app and “Rate This Session”
●
Special thanks to in order by rand() to: Ariel
Glenn, Manuel Arostegui, Morgan Tocker, Sean
Pringle, David Hildebrandt, Bill Karwin, Domas
Mituzas, Mark Callaghan, Shlomi Noach, Valerii
Kravchuk, Miguel Ángel Nieto, Dimitri Kravtchuk,
Olav Sandstå and the whole Wikimedia Team,
and all people at the MariaDB, Percona and
MySQL/Oracle teams, and the Percona Live
Organization and Sponsors

Query Optimization With MySQL 8.0 and MariaDB 10.3 - The Basics - FileId - 160092 PDF

Uploaded by

Copyright:

Available Formats

Query Optimization With MySQL 8.0 and MariaDB 10.3 - The Basics - FileId - 160092 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Query Optimization With MySQL 8.0 and MariaDB 10.3 - The Basics - FileId - 160092 PDF

Uploaded by

Copyright:

Available Formats

Query Optimization with MySQL 8.0 and MariaDB 10.

Query Optimization With MySQL

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0

2. Query Profiling 7. Subqueries

3. Access Types and Basic 8. FULLTEXT, Geodata Search

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 2

Query Optimization with MySQL 8.0 and MariaDB

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 3

This is me fighting bad query

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 4

Example Application (I)

• Wiktionary (and all Wikimedia projects’ data) is

Example Application (II)

• OSM Database is licensed under the Open

Install the example databases

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 7

Alternative: Query the Live

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 8

Query Optimization with MySQL 8.0 and MariaDB

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 9

Which Queries Should I Optimize

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 11

General Monitoring Metrics

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 12

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 13

pt-query-digest Execution (I)

pt-query-digest Execution (II)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 15

pt-query-digest Execution (III)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 16

Installation of the SYS Schema

Example Usage: Discovering Unused

mysql-5.7.8 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5;

Example Usage: Slow Queries

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 20

Example Usage: Top Queries

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 21

Example Usage: Top Queries

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 22

Query Optimization with MySQL 8.0 and MariaDB

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 24

EXPLAIN Example (vertical

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 26

EXPLAIN Example (id)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 27

EXPLAIN Example (select_type)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 28

EXPLAIN Example (table)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 29

EXPLAIN Example (type)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 30

EXPLAIN Example (rows)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 31

How to improve performance?

MariaDB [dewiktionary]> ALTER TABLE page ADD INDEX

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 32

Index creation results (type)

© 2018 Jaime Crespo. http://jynus.com. License: CC-BY-SA-4.0 33

Index creation results