SQL Antipatterns

Tables and Queries That Don’t Work

Bill Karwin

1
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

2
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

3
Logical Database Antipatterns

1. Comma-Separated Lists
2. Multi-Column Attributes
3. Entity-Attribute-Value
4. Metadata Tribbles

4
Comma-Separated Lists

5
 Comma-Separated Lists
• Example: table BUGS references table
PRODUCTS using a foreign key

( * .. 1 )
BUGS PRODUCTS

6
 Comma-Separated Lists
CREATE TABLE bugs ( CREATE TABLE products (
bug_id SERIAL PRIMARY KEY, product_id SERIAL PRIMARY KEY,
description VARCHAR(200), product_name VARCHAR(50)
product_id BIGINT )
REFERENCES products
)

bug_id description product_id product_id product_name

1234 crashes while saving 1 1 Open RoundFile
3456 fix performance 2 2 Visual TurboBuilder
5678 support XML 3 3 ReConsider

7
 Comma-Separated Lists
• Objective: Allow a bug to reference
multiple products

( * .. * )
BUGS PRODUCTS

8
 Comma-Separated Lists
• Antipattern: reference multiple
products in a comma-separated list
CREATE TABLE bugs (
bug_id SERIAL PRIMARY KEY,
description VARCHAR(200),
product_id VARCHAR(50)
)

bug_id description product_id

1234 crashes while saving 1
3456 increase performance 1, 2, 3
5678 support XML 3

9
 Comma-Separated Lists

• Difficult to add a value to the list:

UPDATE bugs
SET product_id = CONCAT(product_id, ‘,’, 2)
WHERE bug_id = 3456

• Even more difficult to remove a value

10
 Comma-Separated Lists

• Difficult to count products per bug:

SELECT LENGTH(product_id)
– LENGTH(REPLACE(product_id, ‘,’, ‘’))+1
AS num_products
FROM bugs

11
 Comma-Separated Lists

• Difficult to count bugs per product:

SELECT SUM( FIND_IN_SET($id, product_id) > 0 )
AS num_bugs
FROM bugs

12
 Comma-Separated Lists

• Difficult to search for bugs for a specific

product:
SELECT *
FROM bugs
WHERE product_id RLIKE ‘[[:<:]]2[[:>:]]’

13
 Comma-Separated Lists

• Difficult to join BUGS to PRODUCTS:

SELECT *
FROM bugs b JOIN products p ON b.product_id
RLIKE CONCAT(‘[[:<:]]’, p.product_id, ‘[[:>:]]’)

14
 Comma-Separated Lists

• Can’t enforce referential integrity

or data type integrity:
do these values exist?
INSERT INTO bugs (product_id)
VALUES ( ‘23, 57, 94’ )
INSERT INTO bugs (product_id)
VALUES ( ‘1, 2, banana, 3’ )

• How does your application detect and

correct bogus data?

15
 Comma-Separated Lists
• Solution: a many-to-many relationship
always requires an intersection table.

(1 .. *) (* .. 1)
BUGS BUGS_PRODS PRODUCTS

16
 Comma-Separated Lists

• The intersection table references both tables

CREATE TABLE bugs_prods (
bug_id BIGINT REFERENCES bugs,
product_id BIGINT REFERENCES products,
PRIMARY KEY (bug_id, product_id)
)

17
 Comma-Separated Lists

• The intersection table references both tables

bug_id product_id
bug_id description 1234 1 product_id product_name
1234 crashes while saving 3456 1 1 Open RoundFile
3456 fix performance 3456 2 2 Visual TurboBuilder
5678 support XML 3456 3 3 ReConsider
5678 3

18
 Comma-Separated Lists
Add a product: INSERT INTO bugs_prods
VALUES (1234, 2)

Delete a product: DELETE FROM bugs_prods

WHERE bug_id = 1234
AND product_id = 2

Query bugs for a product: SELECT * FROM bugs

JOIN bugs_prods USING (bug_id)
WHERE product_id = 2

19
 Comma-Separated Lists
Join bugs to products: SELECT * FROM bugs
JOIN bugs_prods USING (bug_id)
JOIN products USING (product_id)
WHERE bug_id = 1234
Count bugs per product: SELECT product_id, COUNT(*)
FROM bugs_prods
GROUP BY product_id

Count products per bug: SELECT bug_id, COUNT(*)

FROM bugs_prods
GROUP BY bug_id

20
Multi-Column
Attributes

21
 Multi-Column Attributes
• Objective: support an attribute with
multiple values

bug_id description product_id

1234 crashes while saving 1
3456 fix performance 2
5678 support XML 3

22
 Multi-Column Attributes
• Antipattern: add more columns
CREATE TABLE bugs (
bug_id SERIAL PRIMARY KEY,
description VARCHAR(200),
product_id1 BIGINT REFERENCES products,
product_id2 BIGINT REFERENCES products,
product_id3 BIGINT REFERENCES products
)

bug_id description product_id1 product_id2 product_id3

1234 crashes while saving 1 NULL NULL
3456 increase performance 1 2 3
5678 support XML 3 NULL NULL

23
 Multi-Column Attributes

• Search must reference all columns:

SELECT * FROM bugs
WHERE product_id1 = 1
OR product_id2 = 1
OR product_id3 = 1

24
 Multi-Column Attributes
• Search for multiple products (two forms):
SELECT * FROM bugs
WHERE ( product_id1 = 1
OR product_id2 = 1
OR product_id3 = 1 )
AND ( product_id1 = 3
OR product_id2 = 3
OR product_id3 = 3 )
SELECT * FROM bugs
WHERE 1 IN (product_id1, product_id2, product_id3)
AND 3 IN (product_id2, product_id3, product_id4)

25
 Multi-Column Attributes
• Remove product 2 from a row:
bug_id product_id1 product_id2 product_id3

BEFORE 1234 1 3 2
AFTER 1234 1 3 NULL

UPDATE bugs
SET product_id1 = NULLIF(product_id1, 2),
product_id2 = NULLIF(product_id2, 2),
product_id3 = NULLIF(product_id3, 2)
WHERE bug_id = 1234

26
 Multi-Column Attributes
• Add product 3 to a row:
bug_id product_id1 product_id2 product_id3

BEFORE 1234 1 NULL 2

AFTER 1234 1 3 2

UPDATE bugs
SET product_id1 = CASE
WHEN 3 IN (product_id2, product_id3) THEN product_id1
ELSE COALESCE(product_id1, 3) END,
product_id2 = CASE
WHEN 3 IN (product_id1, product_id3) THEN product_id2
ELSE COALESCE(product_id2, 3) END,
product_id3 = CASE
WHEN 3 IN (product_id1, product_id2) THEN product_id3
ELSE COALESCE(product_id3, 3) END
WHERE bug_id = 1234

27
 Multi-Column Attributes
• How many columns are enough?
• Add a new column for fourth product:
ALTER TABLE bugs ADD COLUMN product_id4
BIGINT REFERENCES products

• Rewrite all queries searching for products:

SELECT * FROM bugs
WHERE product_id1 = 2
OR product_id2 = 2
OR product_id3 = 2
OR product_id4 = 2

28
 Multi-Column Attributes

• Solution: use an intersection table

Query for two products: SELECT * FROM bugs

JOIN bugs_prods AS p1 USING (bug_id)
JOIN bugs_prods AS p2 USING (bug_id)
WHERE p1.product_id = 1
AND p2.product_id = 3

29
Entity-Attribute-Value

30
 Entity-Attribute-Value
• Objective: make a table with a variable
set of attributes
bug_id bug_type priority description severity sponsor

crashes when loss of

1234 BUG high
saving functionality

3456 FEATURE low support XML Acme Corp.

31
 Entity-Attribute-Value

• Antipattern: store all attributes in a

second table, one attribute per row
CREATE TABLE eav (
bug_id BIGINT REFERENCES bugs,
attr_name VARCHAR(20) NOT NULL,
attr_value VARCHAR(100),
PRIMARY KEY (bug_id, attr_name)
)
mixing data
with metadata

32
 Entity-Attribute-Value

bug_id attr_name attr_value

1234 priority high
1234 description crashes when saving
1234 severity loss of functionality
3456 priority low
3456 description support XML
3456 sponsor Acme Corp.

33
 Entity-Attribute-Value
• Difficult to ensure consistent attribute
names

bug_id attr_name attr_value

1234 created 2008-04-01
3456 created_date 2008-04-01

34
 Entity-Attribute-Value
• Difficult to enforce data type integrity

bug_id attr_name attr_value

1234 created_date 2008-02-31
3456 created_date banana

35
 Entity-Attribute-Value

• Difficult to enforce mandatory attributes

(i.e. NOT NULL)
• SQL constraints apply to columns, not rows

• No way to declare that a row must exist with a

certain attr_name value (‘created_date’)

• Maybe create a trigger on INSERT for bugs?

36
 Entity-Attribute-Value
• Difficult to enforce referential integrity for
attribute values
bug_id attr_name attr_value
1234 priority new
3456 priority fixed
5678 priority banana

• Constraints apply to all rows in the column, not

selected rows depending on value in attr_name

37
 Entity-Attribute-Value
• Difficult to reconstruct a row of attributes:
SELECT b.bug_id,
e1.attr_value AS `created_date`, need one JOIN
e2.attr_value AS `priority` per attribute
FROM bugs b
LEFT JOIN eav e1 ON (b.bug_id = e1.bug_id
AND e1.attr_name = ‘created_date’)
LEFT JOIN eav e2 ON (b.bug_id = e2.bug_id
AND e2.attr_name = ‘priority’)

bug_id created_date priority

1234 2008-04-01 high

38
 Entity-Attribute-Value

• Solution: use metadata for metadata

• Define attributes in columns

• ALTER TABLE to add attribute columns

• Define related tables for related types

39
 Entity-Attribute-Value
• Define similar tables for similar types:
“Sister Tables”

CREATE TABLE bugs ( CREATE TABLE features (

bug_id SERIAL PRIMARY KEY, bug_id SERIAL PRIMARY KEY,
created_date DATE NOT NULL, created_date DATE NOT NULL,
priority VARCHAR(20), priority VARCHAR(20),
description TEXT, description TEXT,
severity VARCHAR(20) sponsor VARCHAR(100)
) )

40
 Entity-Attribute-Value
• Define related tables for related types:
“Inherited Tables”
CREATE TABLE issues (
issue_id SERIAL PRIMARY KEY,
created_date DATE NOT NULL
priority VARCHAR(20),
description TEXT
)

CREATE TABLE bugs ( CREATE TABLE features (

issue_id BIGINT PRIMARY KEY issue_id BIGINT PRIMARY KEY
REFERENCES issues, REFERENCES issues,
severity VARCHAR(20) sponsor VARCHAR(100)
) )

41
 Entity-Attribute-Value

• Appropriate usage of EAV

• When attributes must be truly dynamic

• Enforce constraints in application code

• Don’t try to fetch one object in a single row

• Consider non-relational solutions

for semi-structured data, e.g. RDF/XML

42
Metadata Tribbles

43
 Metadata Tribbles
My database has
fifty... thousand... tables.

44
 Metadata Tribbles

• Objective: improve performance

of a very large table

45
 Metadata Tribbles

• Antipattern: separate into many tables

with similar structure
• Separate tables per distinct value in atttribute

• e.g., per year, per month, per user,

per postal code, etc.

46
 Metadata Tribbles

• Must create a new table for each new value

CREATE TABLE bugs_2005 ( . . . );
CREATE TABLE bugs_2006 ( . . . );
CREATE TABLE bugs_2007 ( . . . );
CREATE TABLE bugs_2008 ( . . . );
... mixing data
with metadata

47
 Metadata Tribbles

• Automatic primary keys cause conflicts

CREATE TABLE bugs_2005 (bug_id SERIAL . . . );
CREATE TABLE bugs_2006 (bug_id SERIAL . . . );
CREATE TABLE bugs_2007 (bug_id SERIAL . . . );
CREATE TABLE bugs_2008 (bug_id SERIAL . . . );
...

48
 Metadata Tribbles

• Difficult to query across tables

SELECT b.status, COUNT(*) AS count_per_status
FROM (
SELECT * FROM bugs_2008
UNION
SELECT * FROM bugs_2007
UNION
SELECT * FROM bugs_2006 ) AS b
GROUP BY b.status

49
 Metadata Tribbles

• Table structures are not kept in sync

ALTER TABLE bugs_2008
ADD COLUMN work_estimate_hrs NUMERIC

• Prior tables don’t contain new column

• Dissimilar tables can’t be combined with UNION

50
 Metadata Tribbles
• Solution #1: use horizontal partitioning
• Physically split, while logically whole

• MySQL 5.1 supports partitioning

BUGS
(2006)

BUGS
BUGS (2007)

BUGS
(2008)

51
 Metadata Tribbles
• Solution #2: use vertical partitioning
• Move bulky and seldom-used columns to a
second table in one-to-one relationship

( 1 .. 1 )
PRODUCTS INSTALLERS

52
 Metadata Tribbles

• Columns can also be tribbles:

CREATE TABLE bugs (
bug_id SERIAL PRIMARY KEY,
...
product_id1 BIGINT,
product_id2 BIGINT,
product_id3 BIGINT
)

53
 Metadata Tribbles
• Solution #3: add a dependent table
CREATE TABLE bugs_prods (
bug_id BIGINT REFERENCES bugs
product_id BIGINT REFERENCES products,
PRIMARY KEY (bug_id, product_id)
)

(1 .. *)
BUGS BUGS_PRODS

54
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

55
Physical Database Antipatterns

5. ID Required
6. Phantom Files
7. FLOAT Antipattern
8. ENUM Antipattern
9. Readable Passwords

56
ID Required

57
 ID Required

• Antipattern: mandatory “id” column as

primary key
CREATE TABLE bugs (
id BIGINT AUTO_INCREMENT
PRIMARY KEY,
description VARCHAR(200),
...
)

58
 ID Required

• Even stranger: another candidate key

already exists
CREATE TABLE bugs (
id BIGINT AUTO_INCREMENT
PRIMARY KEY,
bug_id VARCHAR(10) UNIQUE,
description VARCHAR(200),
...
)

59
 ID Required

• Issues:
• Natural keys

• Compound keys

• Duplicate rows

• Obscure meaning

• JOIN ... USING

60
 ID Required

• Natural keys
• Not auto-generated

• Might not be integers

• Have domain meaning

CREATE TABLE bugs (
bug_id VARCHAR(10) PRIMARY KEY,
description VARCHAR(200),
...
)

61
 ID Required

• Compound keys
• More than one column

• Might not be auto-generated

• Might not be integers intersection tables

have compound keys
CREATE TABLE bugs_prods (
bug_id BIGINT NOT NULL REFERENCES bugs,
product_id BIGINT NOT NULL REFERENCES products,
PRIMARY KEY (bug_id, product_id)
)

62
 ID Required

• “id” allows duplicate rows

CREATE TABLE bugs_prods (
id BIGINT AUTO_INCREMENT
PRIMARY KEY,
bug_id BIGINT NOT NULL,
product_id BIGINT NOT NULL
)
compound candidate key

63
 ID Required

• “id” obscures meaning “software project

SELECT * bugs & issues
FROM sp_bi_mt main table”
WHERE id = 5678 of course!

64
 ID Required

• “bug_id” clarifies meaning

SELECT * must be something
FROM sp_bi_mt about bugs
WHERE bug_id = 5678

65
 ID Required
• “id” prevents JOIN...USING
SELECT *
FROM bugs
JOIN bugs_prods USING (bug_id)

BUGS. BUGS_PRODS.
bug_id bug_id

66
 ID Required
• “id” prevents JOIN...USING
SELECT *
FROM bugs b
JOIN bugs_prods p ON (b.id = p.bug_id)

BUGS. BUGS_PRODS.
id bug_id

67
 ID Required

• Solution:
• Use natural keys when needed
• Use compound keys when needed
• Choose sensible names
• Use same name in foreign keys
68
Phantom Files

69
 Phantom Files
• Objective: store screenshot images

BUGS SCREENSHOTS

70
 Phantom Files
• Antipattern: store path to image in
database, image on filesystem
image.png

BUGS SCREENSHOTS
A
CREATE TABLE screenshots (
bug_id BIGINT REFERENCES bugs,
image_path VARCHAR(200),
comment VARCHAR(200)
)

71
 Phantom Files
• Files don’t obey DELETE

bug_id image_path image.png

A
insert
row create
file

72
 Phantom Files
• Files don’t obey DELETE

bug_id image_path image.png

A
delete
row file is
orphaned

73
 Phantom Files
• Files don’t obey UPDATE
1. Client #1 updates row, acquires row lock

bug_id image_path image.png

A
B
update
row replace
file

74
 Phantom Files
• Files don’t obey UPDATE
2. Client #2 replaces image file, but not row

bug_id image_path image.png

B
C
2nd
update gets replaces file
conflict error anyway

75
 Phantom Files
• Files don’t obey ROLLBACK
1. Start transaction and INSERT

bug_id image_path image.png

A
insert
row create
file

76
 Phantom Files
• Files don’t obey ROLLBACK
2. ROLLBACK

bug_id image_path image.png

A
row is file is
discarded orphaned

77
 Phantom Files
• Files don’t obey ROLLBACK
1. Start transaction and UPDATE

bug_id image_path image.png

A
B
update
row replace
file

78
 Phantom Files
• Files don’t obey ROLLBACK
2. ROLLBACK

bug_id image_path image.png

B
new
new
row reverts
file doesn’t
to old
revert

79
 Phantom Files
• Files don’t obey transaction isolation
1. Client #1 starts transaction and UPDATE

bug_id image_path image.png

A
B
update
row replace
file

80
 Phantom Files
• Files don’t obey transaction isolation
2. Client #2 queries before #1 COMMITs

bug_id image_path image.png

B
query reads ...but reads
old row... new image file

81
 Phantom Files
• Files don’t obey database backup tools

bug_id image_path image.png

A
included in excluded
backup from backup

82
 Phantom Files
• Files don’t obey SQL access privileges

bug_id image_path image.png

A
you
but
can’t read
you can read
this row
this file

83
 Phantom Files
• Solution: consider storing images inside
the database, in a BLOB column
CREATE TABLE screenshots (
bug_id BIGINT REFERENCES bugs,
image_data BLOB,
comment VARCHAR(200)
)

BUGS SCREENSHOTS

84
 Phantom Files

• Legitimate advantages of filesystem storage:

• Database is much leaner without images

• Database backups are smaller, faster too

• Easier to manage & preview images

• Make your own informed decision

85
FLOAT Antipattern

86
 FLOAT Antipattern

• Objective: store real numbers exactly

• Especially money

• Work estimate hours

87
 FLOAT Antipattern

• Antipattern: use FLOAT data type

ALTER TABLE bugs
ADD COLUMN work_estimate_hrs FLOAT
INSERT INTO bugs (bug_id, work_estimate_hrs)
VALUES (1234, 3.3)

88
 FLOAT Antipattern

• FLOAT is inexact
SELECT work_estimate_hrs
FROM bugs WHERE bug_id = 1234

‣ 3.3
SELECT work_estimate_hrs * 1000000000
FROM bugs WHERE bug_id = 1234

‣ 3299999952.3163

89
 FLOAT Antipattern

• Inexact decimals assuming infinite precision

• 1/3 + 1/3 + 1/3 = 1.0

• 0.333 + 0.333 + 0.333 = 0.999 finite precision

90
 FLOAT Antipattern

• IEEE 754 standard for representing

floating-point numbers in base-2
• Some numbers round off, aren’t stored exactly

• Comparisons to original value fail

SELECT * FROM bugs
WHERE work_estimate_hrs = 3.3 comparison
fails

91
 FLOAT Antipattern

• Solution: use NUMERIC data type

ALTER TABLE bugs
ADD COLUMN work_estimate_hrs NUMERIC(9,2)
INSERT INTO bugs (bug_id, work_estimate_hrs)
VALUES (1234, 3.3)
SELECT * FROM bugs comparison
WHERE work_estimate_hrs = 3.3 succeeds

92
ENUM Antipattern

93
 ENUM Antipattern

• Objective: restrict a column’s values to a

fixed set of values
OK
INSERT INTO bugs (status)
VALUES (‘new’)
INSERT INTO bugs (status)
VALUES (‘banana’)
FAIL

94
 ENUM Antipattern

• Antipattern: use ENUM data type,

when the set of values may change
CREATE TABLE bugs (
status ENUM(‘new’, ‘open’, ‘fixed’)
)

95
 ENUM Antipattern

• Changing the set of values is a metadata

alteration
• You must know the current set of values
ALTER TABLE bugs MODIFY COLUMN
status ENUM(‘new’, ‘open’, ‘fixed’, ‘duplicate’)

96
 ENUM Antipattern

• Difficult to get a list of possible values

SELECT column_type
FROM information_schema.columns
WHERE table_schema = ‘bugtracker_schema’
AND table_name = ‘bugs’
AND column_name = ‘status’

• You must parse the LONGTEXT value

“ENUM(‘new’, ‘open’, ‘fixed’)”

97
 ENUM Antipattern

• Solution: use ENUM only if values are

set in stone
CREATE TABLE bugs (
...
bug_type ENUM(‘defect’, ‘feature’)
...
)

98
 ENUM Antipattern
• Use a lookup table if values may change
CREATE TABLE bug_status (
status VARCHAR(10) PRIMARY KEY
)
INSERT INTO bug_status (status)
VALUES (‘new’), (‘open’), (‘fixed’)

BUGS BUG_STATUS

99
 ENUM Antipattern

• Adding/removing a value is a data

operation, not a metadata operation
• You don’t need to know the current values
INSERT INTO bug_status (status) VALUES (‘duplicate’)

100
 ENUM Antipattern

• Use an attribute to retire values, not

DELETE
CREATE TABLE bug_status (
status VARCHAR(10) PRIMARY KEY,
active TINYINT NOT NULL DEFAULT 1
)
UPDATE bug_status
SET active = 0
WHERE status = ‘duplicate’

101
Readable Passwords

102
 Readable Passwords

• Objective:
help users who forget their password

103
 Readable Passwords

• Antipattern: store password in plain

text (i.e. not encoded)
CREATE TABLE accounts (
acct_name VARCHAR(20) PRIMARY KEY,
email VARCHAR(100) NOT NULL,
password VARCHAR(30) NOT NULL
)
INSERT INTO accounts
VALUES (‘bill’, ‘bill@example.com’, ‘xyzzy’)

104
 Readable Passwords

• Antipattern: compare user input

directly against stored password
SELECT (password = ‘xyzzy’) AS is_correct
FROM accounts
WHERE acct_name = ‘bill’

105
 Readable Passwords
• Antipattern: send password in plain
text in email to user upon request
From: daemon
To: bill@example.com
Subject: password request

Your account is “bill” and your

password is “xyzzy”

Click the link below to log in.

...

106
 Readable Passwords

• Solution: store password using a

one-way message digest function
CREATE TABLE accounts (
acct_name VARCHAR(20) PRIMARY KEY,
email VARCHAR(100) NOT NULL,
password_hash CHAR(32) NOT NULL
)
INSERT INTO accounts (acct_name, password_hash)
VALUES (‘bill’, MD5(‘xyzzy’))

107
 Readable Passwords

• Good: compare hash of user input against

stored hash
SELECT (a.password_hash = MD5(‘xyzzy’))
AS is_correct
FROM accounts a
WHERE a.acct_name = ‘bill’

108
 Readable Passwords
• Better: concatentate a salt before using
the hash function
CREATE TABLE accounts (
...
password_hash CHAR(32) NOT NULL,
salt BINARY(4) NOT NULL
)
SELECT (a.password_hash = MD5(
CONCAT(‘xyzzy’, a.salt))) AS is_correct
FROM accounts a
WHERE a.acct_name = ‘bill’

109
 Readable Passwords

• Best: create hash of user input in your

application, before sending to database
$salt = SELECT salt FROM accounts
WHERE acct_name = ‘bill’;
$hash = md5(‘xyzzy’ . $salt);
SELECT (a.password_hash = $hash) AS is_correct
FROM accounts a
WHERE a.acct_name = ‘bill’

110
 Readable Passwords
• Reset password to a temporary random
string, require user to change it
From: daemon
To: bill@example.com
Subject: password reset

Your password has been reset.

The password “p0trz3bie” will work

for one hour.

Click the link below to log in and

change your password.
...

111
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

112
 Query Antipatterns

10. Ambiguous GROUP BY

11. HAVING antipattern
12. Poor Man's Search Engine
13. Implicit columns in SELECT and INSERT

113
Ambiguous
GROUP BY

114
 Ambiguous GROUP BY

• Objective: perform grouping queries, and

include some attributes in the result
SELECT prod_name, bug_id,
MAX(created_date) as latest
FROM bugs
GROUP BY prod_name

115
 Ambiguous GROUP BY
• Antipattern: bug_id isn’t that of the
latest per product
product_name bug_id created_date
Open RoundFile 1234 2007-12-19
Open RoundFile 2248 2008-04-01 product_name bug_id latest
Visual TurboBuilder 3456 2008-02-16 Open RoundFile 1234 2008-04-01
Visual TurboBuilder 4077 2008-02-10 Visual TurboBuilder 3456 2008-02-16
ReConsider 5678 2008-01-01 ReConsider 5678 2008-01-01
ReConsider 8063 2007-11-09

116
 Ambiguous GROUP BY

• The Single-Value Rule: all columns in

the select-list must be either:
• Part of an aggregate expression
• Referenced in the GROUP BY clause
• A functional dependency of a column
named in the GROUP BY clause

117
 Ambiguous GROUP BY
• Functional dependency:
• For a given product_name, there is guaranteed to
be one value in a functionally dependent attribute
product_name bug_id created_date
Open RoundFile 1234 2007-12-19
multiple values per
Open RoundFile 2248 2008-04-01 product name

Visual TurboBuilder 3456 2008-02-16 bug_id is not

functionally dependent
Visual TurboBuilder 4077 2008-02-10
ReConsider 5678 2008-01-01
ReConsider 8063 2007-11-09

118
 Ambiguous GROUP BY
• Solution #1: use only functionally
dependent attributes in select-list
SELECT prod_name, bug_id,
MAX(created_date) as latest
FROM bugs
GROUP BY prod_name

product_name latest
Open RoundFile 2008-04-01
Visual TurboBuilder 2008-02-16
ReConsider 2008-01-01

119
 Ambiguous GROUP BY
• Solution #2: use GROUP_CONCAT()
function in MySQL to get all values
SELECT prod_name,
GROUP_CONCAT(bug_id) as bug_id_list,
MAX(created_date) as latest
FROM bugs
GROUP BY prod_name

product_name bug_id_list latest

Open RoundFile 1234, 2248 2008-04-01
Visual TurboBuilder 3456, 4077 2008-02-16
ReConsider 5678, 8063 2008-01-01

120
 Ambiguous GROUP BY
• Solution #3: use this OUTER JOIN
query instead of GROUP BY
SELECT b.prod_name, b.bug_id,
b.created_date AS latest
FROM bugs b LEFT OUTER JOIN bugs b2
ON (b.prod_name = b2.prod_name AND
b.created_date < b2.created_date)
WHERE b2.bug_id IS NULL

product_name bug_id latest

Open RoundFile 2248 2008-04-01
Visual TurboBuilder 3456 2008-02-16
ReConsider 5678 2008-01-01

121
HAVING Antipattern

122
 HAVING Antipattern

• Objective: use column aliases in query

criteria (the WHERE clause)
SELECT bug_id, YEAR(created_date) AS yr
FROM bugs
WHERE yr = 2008

not
allowed!

123
 HAVING Antipattern

• Antipattern: use a HAVING clause

instead of WHERE clause
SELECT bug_id, YEAR(created_date) AS yr
FROM bugs
HAVING yr = 2008

124
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions

2. SELECT expressions

3. Define column aliases

4. GROUP BY

5. HAVING expressions

6. ORDER BY

125
 HAVING Antipattern

• Sequence of query execution:

filter rows
1. WHERE expressions

2. SELECT expressions

3. Define column aliases

4. GROUP BY

5. HAVING expressions

6. ORDER BY

126
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions execute select-list
2. SELECT expressions for filtered rows
3. Define column aliases

4. GROUP BY

5. HAVING expressions

6. ORDER BY

127
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions

2. SELECT expressions

3. Define column aliases after expressions

4. GROUP BY

5. HAVING expressions

6. ORDER BY

128
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions

2. SELECT expressions

3. Define column aliases

4. GROUP BY reduce rows by groups

5. HAVING expressions

6. ORDER BY

129
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions

2. SELECT expressions

3. Define column aliases

4. GROUP BY

5. HAVING expressions filter groups

6. ORDER BY

130
 HAVING Antipattern

• Sequence of query execution:

1. WHERE expressions

2. SELECT expressions
postponing row filtering
3. Define column aliases causes the SELECT
4. GROUP BY expressions to execute
for all rows
5. HAVING expressions

6. ORDER BY

131
 HAVING Antipattern

• Other functions are run on rows that will

be filtered
SELECT bug_id,
YEAR(created_date) AS yr,
expensive_func(b.description)
FROM bugs
HAVING yr = 2008

132
 HAVING Antipattern

• Solution 1: repeat the expression in the

WHERE clause, instead of the column alias
SELECT bug_id,
YEAR(created_date) AS yr,
expensive_func(description)
FROM bugs
WHERE YEAR(created_date) = 2008

133
 HAVING Antipattern

• Solution 2: use a derived table

SELECT b.bug_id, b.yr,
expensive_func(b.description)
FROM
(SELECT bug_id, description,
YEAR(created_date) AS yr
FROM bugs) AS b
WHERE b.yr = 2008

134
 Poor Man’s
Search Engine

135
 Poor Man’s Search Engine

• Objective: search for specific words

in larger text
SELECT * FROM bugs
WHERE MATCH(description) AGAINST (‘crash’)

136
 Poor Man’s Search Engine

• Antipattern: use LIKE with wildcards

SELECT * FROM bugs
WHERE description LIKE ‘%crash%’

• Or regular expressions
SELECT * FROM bugs
WHERE description RLIKE ‘crash’

137
 Poor Man’s Search Engine

• Performance issues
• Indexes don’t benefit from substring searches

• String-matching + full table scan = expensive

138
 Poor Man’s Search Engine

• Telephone book analogy:

CREATE TABLE telephone_book (
full_name VARCHAR(40),
KEY (full_name)
)
INSERT INTO telephone_book VALUES
(‘Riddle, Thomas’),
(‘Thomas, Dean’)

139
 Poor Man’s Search Engine

• Search for all with last name “Thomas”

SELECT * FROM telephone_book uses
index
WHERE full_name LIKE ‘Thomas%’

• Search for all with first name “Thomas”

SELECT * FROM telephone_book
WHERE full_name LIKE ‘%Thomas’
doesn’t
use index

140
 Poor Man’s Search Engine

• Accuracy issues
• Substrings find irrelevant or false matches

• LIKE ‘%one%’ matches words ‘one’, ‘money’,

‘prone’, etc.

141
 Poor Man’s Search Engine

• Solution: use a real full-text search

engine
• MySQL FULLTEXT index (MyISAM only)

• http://lucene.apache.org/

• http://www.sphinxsearch.com/

142
Implicit Columns

143
 Implicit Columns

• Objective: write shorter, more general

SQL queries
wildcard column list
SELECT *
FROM bugs
INSERT INTO bugs (status, priority, description)
VALUES ( . . . )
implicit column list

144
 Implicit Columns

• Difficult to detect errors

• Add/remove a column

• Reposition a column

• Rename a column

145
 Implicit Columns

• SQL has no wildcard syntax for “all but one”

imaginary syntax;
SELECT * EXCEPT description doesn’t exist
FROM bugs

146
 Implicit Columns

• Solution: always name columns explicitly

SELECT bug_id, status, priority, description, . . .
FROM bugs
INSERT INTO bugs
(status, priority, description, . . .)
VALUES ( . . . )

147
 Implicit Columns

• Get informative errors immediately

• if you use a removed column

• if you use a renamed column

• Specify the order of columns in query

• even if columns are repositioned

148
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

149
 Application Antipatterns

14. User-supplied SQL

15. SQL Injection
16. Parameter Façade
17. Pseudokey Neat Freak
18. Session Coupling
19. Phantom Side Effects

150
User-Supplied SQL

151
 User-supplied SQL

• Objective: satisfy demand for flexibility

to query database

152
 User-supplied SQL

• Antipattern:
• Let users write their own SQL expressions

• Run user-supplied SQL verbatim

153
 User-supplied SQL

• Crash database server with one SELECT:

SELECT *
FROM bugs JOIN bugs JOIN bugs
JOIN bugs JOIN bugs

• 100 bugs = 10,000,000,000 row result set

154
 User-supplied SQL

• Filesort uses temporary disk space

SELECT *
FROM bugs JOIN bugs JOIN bugs
JOIN bugs JOIN bugs
ORDER BY 1

155
 User-supplied SQL

• Solution: users should specify values,

but not code
• You write SQL queries with parameter
placeholders
SELECT * plug in user’s
value here
FROM bugs
WHERE bug_id = ?

156
SQL Injection

157
 SQL Injection

• Antipattern: interpolating untrusted

data into SQL expressions
http://bug-server/?id=1234
<?php
$id = $_GET[‘id’];
$sql = “SELECT * FROM bugs
WHERE bug_id = $id”;
mysql_query($sql);

158
 SQL Injection

• Exposes your application to mischief

http://bug-server/?id=1234 or 1=1
SELECT * FROM bugs
WHERE bug_id = 1234 or 1=1

159
 SQL Injection

• Exposes your application to mischief

http://bug-server/set-passwd?user=bill’ or ‘x’=‘x
UPDATE accounts
SET password_hash = MD5($pass)
WHERE account_name = ‘bill’ or ‘x’=‘x’

160
 SQL Injection

• Solution #1: filter input

<?php
$id = intval( $_GET[‘id’] );
$sql = “SELECT * FROM bugs
WHERE bug_id = $id”;
mysql_query($sql);

• For strings, use mysql_real_escape_string().

161
 SQL Injection

• Solution #2: parameterize query values

<?php parameter
$query = “SELECT * FROM bugs placeholder
WHERE bug_id = ? ”;
$stmt = mysqli_prepare($query);
$stmt->bind_param(‘i’, $_GET[‘id’]);
$stmt->execute();

162
Parameter Façade

163
 Parameter Façade

• Objective: include application variables

in SQL statements
SELECT * FROM bugs
WHERE bug_id = $id

164
 Parameter Façade

• Antipatterns:
• Trying to use parameters to change syntax

• Interpolating variables for simple values

165
 Parameter Façade

• Interpolation can modify syntax

$list = ‘1234, 3456, 5678’
SELECT * FROM bugs three values
WHERE bug_id IN ( $list ) separated by commas

SELECT * FROM bugs

WHERE bug_id IN ( 1234, 3456, 5678 )

166
 Parameter Façade

• A parameter is always a single value

$list = ‘1234, 3456, 5678’
SELECT * FROM bugs
WHERE bug_id IN ( ? )
one string value
EXECUTE USING $list
SELECT * FROM bugs
WHERE bug_id IN ( ‘1234, 3456, 5678’ )

167
 Parameter Façade

• Interpolation can specify identifiers

$column = ‘bug_id’
SELECT * FROM bugs
WHERE $column = 1234
column identifier
SELECT * FROM bugs
WHERE bug_id = 1234

168
 Parameter Façade

• A parameter is always a single value

$column = ‘bug_id’;
SELECT * FROM bugs
WHERE ? = 1234
EXECUTE USING $column
one string value
SELECT * FROM bugs
WHERE ‘bug_id’ = 1234

169
 Parameter Façade

• Interpolation risks SQL injection

$id = ‘1234 or 1=1’
SELECT * FROM bugs
WHERE bug_id = $id logical
expression
SELECT * FROM bugs
WHERE bug_id = 1234 or 1=1

170
 Parameter Façade

• A parameter is always a single value

$id = ‘1234 or 1=1’
SELECT * FROM bugs
WHERE bug_id = ?
one string value
EXECUTE USING $id
SELECT * FROM bugs
WHERE bug_id = ‘1234 or 1=1’

171
 Parameter Façade

• Preparing a SQL statement

• Parses SQL syntax

• Optimizes execution plan

• Retains parameter placeholders

172
 Parameter Façade
SELECT expr-list *

query FROM simple-table bugs

bug_id

WHERE expr equality =

parameter
placeholder

173
 Parameter Façade

• Executing a prepared statement

• Combines a supplied value for each parameter

• Doesn’t modify syntax, tables, or columns

• Runs query
could invalidate
optimization plan

174
 Parameter Façade
SELECT expr-list *

query FROM simple-table bugs

bug_id

WHERE expr equality =

1234

supplied
value

175
 Parameter Façade
SELECT expr-list *

query FROM simple-table bugs

bug_id

WHERE expr equality =

1234
or 1=1

supplied
value

176
 Parameter Façade

• Interpolating into a query string

• Occurs in the application, before SQL is parsed

• Database server can’t tell what part is dynamic

177
 Parameter Façade
SELECT expr-list *

query FROM simple-table bugs

bug_id

WHERE expr equality =

1234

intended
value

178
 Parameter Façade
SELECT expr-list *

bug_id

query FROM simple-table bugs

equality =

1234
WHERE expr OR
1

equality =

SQL injection 1

179
 Parameter Façade

• The Bottom Line:

• Interpolation may change the shape of the tree

• Parameters cannot change the tree

• Parameter nodes may only be values

180
 Parameter Façade

• Example: IN predicate may supply

only one value
SELECT * FROM bugs
WHERE bug_id IN ( ? )
SELECT * FROM bugs
WHERE bug_id IN ( ?, ?, ?, ? )

must supply
exactly four values

181
 Parameter Façade
Scenario Value Interpolation Parameter
single value ‘1234’ SELECT * FROM bugs SELECT * FROM bugs
WHERE bug_id = $id WHERE bug_id = ?

multiple values ‘1234, 3456, 5678’ SELECT * FROM bugs SELECT * FROM bugs
WHERE bug_id IN ($list) WHERE bug_id IN ( ?, ?, ? )

column name ‘bug_id’ SELECT * FROM bugs

WHERE $column = 1234 NO

table name ‘bugs’ SELECT * FROM $table

WHERE bug_id = 1234 NO

other syntax ‘bug_id = 1234’ SELECT * FROM bugs

WHERE $expr NO

182
 Parameter Façade

• Solution:
• Use parameters only for individual values

• Use interpolation for dynamic SQL syntax

• Be careful to prevent SQL injection

183
Pseudokey Neat Freak

184
 Pseudokey Neat Freak
• Objective: address the discomfort over
the presence of “gaps” in the primary key

bug_id bug_status prod_name

1 NEW Open RoundFile
2 FIXED ReConsider
5 DUPLICATE ReConsider

185
 Pseudokey Neat Freak
• AntiPattern #1:
changing values to close the gaps:

bug_id bug_status prod_name

1 NEW Open RoundFile
2 FIXED ReConsider
3 DUPLICATE ReConsider
update
5 to 3
186
 Pseudokey Neat Freak
• AntiPattern #2:
recycling primary key values

bug_id bug_status prod_name

1 NEW Open RoundFile
2 FIXED ReConsider
5 DUPLICATE ReConsider
3 NEW Visual TurboBuilder

187
 Pseudokey Neat Freak

• Finding an unused value is complex:

SELECT b1.bug_id + 1
FROM bugs b1
LEFT JOIN bugs b2
ON (b1.bug_id + 1 = b2.bug_id)
WHERE b2.bug_id IS NULL
ORDER BY b1.bug_id LIMIT 1

188
 Pseudokey Neat Freak

• Finding an unused value: concurrency

• Two clients can find the same value

• Same problem as using MAX(bug_id)+1

to generate key values

• Resolved only with a table lock

189
 Pseudokey Neat Freak
• Reusing primary key values exposes data

bug_id description

insert row
bug_id = 1234

190
 Pseudokey Neat Freak
• Reusing primary key values exposes data

bug_id description

notify of
bug_id 1234

191
 Pseudokey Neat Freak
• Reusing primary key values exposes data
Zzz...

bug_id description

delete row
bug_id = 1234

192
 Pseudokey Neat Freak
• Reusing primary key values exposes data
no bug
1234 ?

bug_id description

193
 Pseudokey Neat Freak
• Reusing primary key values exposes data

bug_id description

re-insert row
bug_id = 1234

194
 Pseudokey Neat Freak
• Reusing primary key values exposes data
bug 1234
is not mine!

bug_id description

195
 Pseudokey Neat Freak

• Solution:
• Don’t re-use primary key values

• Keys are unique, but not contiguous

• You will not run out of integers!

• UNSIGNED BIGINT has a maximum value of
18,446,744,073,709,551,615

• Create 1,000 rows per second 24x7 for 584 million years

196
Session Coupling

197
 Session Coupling

• Objective: decrease the overhead of

making a database connection
• PHP “share nothing” architecture

• CGI applications

198
 Session Coupling
• Antipattern: persistent connections

User Apache Database

browser httpd server

199
 Session Coupling
• Scalability
• One database connection per Apache thread

• Most requests don’t need database access

Apache Database
httpd server

• Database server must maintains many idle

connections, may exhaust resources

200
 Session Coupling

• Uniform authentication
• All apps must use identical user & password

• Or else each Apache thread must have multiple

persistent database connections

201
 Session Coupling

• Connections have state:

Uncommitted Transactions
• Request #1 starts transaction, but doesn’t
commit

• Request #2 acquires connection, executes

changes, but rolls back current transaction

202
 Session Coupling

• Connections have state: Character Set

• Request #1:
SET NAMES cp1251_koi8

• Request #2:
mysteriously inherits connection character set,
uses Russian 8-bit encoding and collation

203
 Session Coupling

• Connections have state: Session Variables

• Request #1:
SET SESSION @@autocommit = 1

• Request #2:
mysteriously inherits autocommit behavior,
ROLLBACK ineffective

204
 Session Coupling

• Connections have state: Last Insert ID

• Request #1:
INSERT INTO bugs ( . . . )

• Request #2:
LAST_INSERT_ID() returns bug_id
generated in previous request

205
 Session Coupling

• Solution: reduce need for database

connections
• Create database connections lazily
(when needed)

• Redirect requests for static content

(images, CSS, Javascript, downloads)
to another web server, e.g. lighttpd

206
 Session Coupling

• Wishlist:
• Method to reset connection to default state

• Database resident connection pooling

(e.g. Oracle)

207
Phantom Side Effects

208
 Phantom Side Effects

• Objective: execute application tasks

with database operations
INSERT INTO bugs ( . . . )
...and send email to notify me

209
 Phantom Side Effects

• Antipattern: execute external effects in

database triggers, stored procedures, and
functions

210
 Phantom Side Effects
• External effects don’t obey ROLLBACK
1. Start transaction and INSERT

bug_id description

notify of
bug_id 1234

insert row
bug_id = 1234

211
 Phantom Side Effects
• External effects don’t obey ROLLBACK
2. ROLLBACK

bug_id description

I got email,
discard but no row
row 1234?

212
 Phantom Side Effects
• External effects don’t obey transaction
isolation
1. Start transaction and INSERT
bug_id description

notify of
bug_id 1234

insert row
bug_id = 1234

213
 Phantom Side Effects
• External effects don’t obey transaction
isolation
2. Email is received before row is visible
bug_id description

I got email,
row but no row
pending 1234?
commit

214
 Phantom Side Effects

• External effects run as database server user

• Possible security risk SQL injection

SELECT * FROM bugs

WHERE bug_id = 1234
or send_email(‘Buy cheap Rolex watch!’)

• Auditing/logging confusion

215
 Phantom Side Effects

• Functions may crash missing file

causes fatal error
SELECT pk_encrypt(description,
‘/nonexistant/private.ppk’)
FROM bugs
WHERE bug_id = 1234

216
 Phantom Side Effects

• Long-running functions delay query

• Accessing remote resources

• Unbounded execution time unresponsive

website
SELECT libcurl_post(description,
‘http://myblog.org/ . . .’)
FROM bugs
WHERE bug_id = 1234

217
 Phantom Side Effects

• Solution:
• Operate only on database in triggers, stored
procedures, database functions

• Wait for transaction to commit

• Perform external actions in application code

218
 Antipattern Categories
Logical Database Physical Database
Antipatterns Antipatterns
CREATE TABLE BugsProducts (
bug_id INTEGER REFERENCES Bugs,
product VARCHAR(100) REFERENCES Products,
PRIMARY KEY (bug_id, product)
);

Query Application
Antipatterns Antipatterns
SELECT b.product, COUNT(*) $dbHandle = new PDO(‘mysql:dbname=test’);
FROM BugsProducts AS b $stmt = $dbHandle->prepare($sql);
GROUP BY b.product; $result = $stmt->fetchAll();

219
Thank You
Bill Karwin

220