Sphinx

Presented by,
MySQL AB® & O’Reilly Media, Inc. High-performance full-text search for
MySQL

Andrew Aksyonoff, shodan@shodan.ru

Peter Zaitsev, pz@percona.com
What’s Sphinx?
 FOSS full-text search engine
 Specially designed for indexing databases
 Integrates well with MySQL
 Provides greatly improved full-text search
 Sometimes, can improve non-full-text queries
 By more efficient processing (in some cases)
 By distributed processing on a cluster (in all)
 Details later in this talk
Why Sphinx?
 Major reasons
 Better indexing speed
 Better searching speed
 Better relevance
 Better scalability
 “Minor” reasons
 Many other features
 Like fixed RAM usage, “faceted” searching, geo-
distance, built-in HTML stripper, morphology support,
1-grams, snippets highlighting, etc.
The meaning of “better”
 Better indexing speed
 50-100 times faster than MySQL FULLTEXT
 4-10 times faster than other external engines
 Better searching speed
 Heavily depends on the mode (boolean vs. phrase)
and additional processing (WHERE, ORDER BY, etc)
 Up to 1000 (!) times faster than MySQL FULLTEXT in
extreme cases (eg. large result set with GROUP BY)
 Up to 2-10 times faster than other external engines
The meaning of “better” 2.0
 Better relevancy
 Sphinx phrase-based ranking in addition to classic
statistical BM25
 Sample query – “To be or not to be”
 Optional, can be turned off for performance
 Better scalability
 Vertical – can utilize many CPU cores, many HDDs
 Horizontal – can utilize many servers
 Out of the box support
 Transparent to app, matter of server config changes
How does it scale?
 Distributed searching with several machines
 Fully transparent to calling application
 Biggest known Sphinx cluster
 1,200,000,000+ documents (yes, that’s a billion)
 1.5 terabytes
 1+ million searches/day
 7 boxes x 2 dual-core CPUs = 28 cores
 Busiest known Sphinx cluster
 30+ million searches/day using 15 boxes
How does it work?
 Two standalone programs
 indexer – pulls data from DB, builds indexes
 searchd – uses indexes, answers queries
 Client programs talk to searchd over TCP
 Via native APIs (PHP, Perl, Python, Ruby, Java)...
 Via SphinxSE, pluggable MySQL engine
 indexer periodically rebuilds the indexes
 Typically, using cron jobs
 Searching works OK during rebuilds
Indexing workflow
 Data sources – “where to get the data?”
 MySQL, Postgres, XML pipe…
 Local indexes – “how to index the data?”
 Also storage location, valid characters list, stop words,
stemming, word forms dictionaries, tokenizing
exceptions, substring indexing, N-grams, HTML
stripping…
 Arbitrary number of indexes
 Arbitrary number of sources per index
 Can pull data from different DB boxes in a shard
Sample – all eggs in one basket
Combining sharded database data for the ease of use

MySQL Source SRC01

Host DB01 Host SEARCHER
Index FULLINDEX
Host SEARCHER
MySQL Source SRC02
Host DB02 Host SEARCHER
Distributed indexes
 Essentially, lists of local and remote indexes
index dist1
{
type = distributed
local = chunk1
agent = box02:3312:chunk02
agent = box03:3312:chunk03
agent = box04:3312:chunk03
}

 All local indexes are searched sequentially

 All remote indexes are searched in parallel
 All results are merged
Sample – divide and conquer
Sharding full-text indexes to improve searching latency
Source CHUNK01
Index CHUNK01
Host GREP01

MySQL Source CHUNK02 Distributed index DIST01

Host UBERDB Index CHUNK02 Host WEB01
Host GREP02

. . . . .
.
Source CHUNK10
Index CHUNK10
Host GREP10
Searching 101 – the client side
 Create a client object
 Set up the options
 Fire the query
<?php
include ( “sphinxapi.php” );
$cl = new SphinxClient ();
$cl->SetMatchMode ( SPH_MATCH_PHRASE );
$cl->SetSortMode ( SPH_SORT_EXTENDED, “price desc” );
$res = $cl->Query ( “ipod nano”, “products” );
var_dump ( $res );
?>
Searching 102 – match contents
 Matches will always have document ID, weight
 Matches can also have numeric attributes
 No string attributes yet (pull ‘em from MySQL)
print_r ( $result[“matches”][0] ):

Array (
[id] => 123
[weight] => 101421
[attrs] => Array (
[group_id] => 12345678901
[added] => 1207261463 ) )
Searching 103 – why attributes
 Short answer – efficiency
 Long answer – efficient filtering, sorting, and
grouping for big result sets (over 1,000 matches)
 Real-world example:
 Using Sphinx for searching only and then sorting just
1000 matches using MySQL – up to 2-3 seconds
 Using Sphinx for both searching and sorting –
improves that to under 0.1 second
 Random row IO in MySQL, no row IO in Sphinx
 Now imagine there’s 1,000,000 matches… 
Moving parts
 SQL query parts that can be moved to Sphinx
 Filtering – WHERE vs. SetFilter() or fake keyword
 Sorting – ORDER BY vs. SetSortMode()
 Grouping – GROUP BY vs. SetGroupBy()
 Up to 100x (!) improvement vs. “naïve” approach
 Rule of thumb – move everything you can from
MySQL to Sphinx
 Rule of thumb 2.0 – apply sacred knowledge of
Sphinx pipeline (and then move everything)
Searching pipeline in 30 seconds
 Search, WHERE, rank, ORDER/GROUP
 “Cheap” boolean searching first
 Then filters (WHERE clause)
 Then “expensive” relevance ranking
 Then sorting (ORDER BY clause) and/or grouping
(GROUP BY clause)
Searching pipeline details
 Query is evaluated as a boolean query
 CPU and IO, O(sum(docs_per_keyword))
 Candidates are filtered
 based on their attribute values
 CPU only, O(sum(docs_per_keyword))
 Relevance rank (weight) is computed
 CPU and IO, O(sum(hits_per_keyword))
 Matches are sorted and grouped
 CPU only, O(filtered_matches_count)
Filters vs. fake keywords
 The key idea – instead of using an attribute,
inject a fake keyword when indexing
sql_query = SELECT id, title, vendor ...

$sphinxClient->SetFilter ( “vendor”, 123 );

$sphinxClient->Query ( “laptop”, “products” );

vs.

sql_query = SELECT id, title, CONCAT(‘_vnd’,vendor) ...

$sphinxClient->Query ( “laptop _vnd123”, “products” );

Filters vs. fake keywords
 Filters
 Will eat extra CPU
 Linear by pre-filtered candidates count
 Fake keywords
 Will eat extra CPU and IO
 Linear by per-keyword matching documents count
 That is strictly equal (!) to post-filter matches count
 Conclusion
 Everything depends on selectivity
 For selective values, keywords are better
Sorting
 Always optimizes for the “limit”
 Fixed RAM requirements, never an IO
 Controlled by max_matches setting
 Both server-side and client-side
 Defaults to 1000
 Processes all matching rows
 Keeps at most N best rows in RAM, at all times
 MySQL currently does not optimize that well
 MySQL sorts everything, then picks up N best
Grouping
 Also in fixed RAM, also IO-less
 Comes at the cost of COUNT(*) precision
 Fixed RAM usage can cause underestimates
 Aggregates-only transmission via distributed agents
can cause overestimates
 Frequently that’s OK anyway
 Consider 10-year per-day report – it will be precise
 Consider “choose top-10 destination domains from
100-million links graph” query – 10 to 100 times
speedup at the cost of 0.5% error might be acceptable
More optimization possibilities
 Using query statistics
 Using multi-query interface
 Choosing proper ranking mode
 Distributing the CPU/HDD load
 Adding stopwords
 etc.
Query statistics
 Applies to migrating from MySQL FULLTEXT
 Total match counts are immediately available –
no need to run 2nd query
 Per-keyword match counts are also available –
can be used not just as minor addition to search
results – but also for automatic query rewriting
Multi-query interface
 Send many independent queries in one batch,
allow Sphinx optimize them internally
 Always saves on network roundtrip
 Sometimes saves on expensive operations
 Most frequent example – same full-text query,
different result set “views”
Multi-query sample
$client = new SphinxClient ();
$q = “laptop”; // coming from website user

$client->SetSortMode ( SPH_SORT_EXTENDED, “@weight desc”);

$client->AddQuery ( $q, “products” );

$client->SetGroupBy ( SPH_GROUPBY_ATTR, “vendor_id” );

$client->AddQuery ( $q, “products” );

$client->ResetGroupBy ();
$client->SetSortMode ( SPH_SORT_EXTENDED, “price asc” );
$client->SetLimit ( 0, 10 );

$result = $client->RunQueries ();

Offloading non-full-text queries
 Basic “general” SQL queries can be rewritten to
“full-text” form – and run by Sphinx
SELECT * FROM table WHERE a=1 AND b=2
ORDER BY c DESC LIMIT 60,20

$client->SetFilter ( “a”, array(1) );

$client->SetFilter ( “b”, array(2) );
$client->SetSortBy ( SPH_SORT_ATTR_DESC, “c” );
$client->SetLimit ( 60, 20 );
$result = $client->Query ( “”, “table” );

 Syntax disclaimer – we are a full-text engine!

 SphinxQL coming at some point in the future
Why do that?
 Sometimes Sphinx reads outperform MySQL
 Sphinx always does RAM based “full scan”
 MySQL index read with bad selectivity can be slower
 MySQL full-scan will most likely be slower
 MySQL can’t index every column combination
 Also, Sphinx queries are easier to distribute
 But Sphinx indexes are essentially read-only
 well, almost (attribute update is possible)
 Complementary to MySQL, not a replacement
SELECT war story
 Searches on Sahibinden.com
 Both full-text and not
 “Show all auctioned items in laptops category with
sellers from Ankara in $1000 to $2000 range”
 “Show matches for ‘ipod nano’ and sort by price”
 Many columns, no way to build covering indexes
 Sphinx full scans turned out being 1.5-3x better
than MySQL full scans or 1-column index reads
 Also, code for full-text and non-full-text queries
was unified
GROUPBY war story
 Domain cross-links report on BoardReader.com
 “Show top 100 destination domains for last month”
 “Show top 100 domains that link to YouTube”
 ~200 million rows overall
 Key features of the report queries
 They always group by domain, and sort by counts
 The result sets are small
 Approximate results are acceptable – don’t care
whether there were exactly 813,719 or 814,101 links
from domain X
GROUPBY war story
 MySQL prototype took up to 300 seconds/query
 Sphinx queries were much easier to distribute
 URLs are preprocessed, then full-text indexed
 http://test.com/path/doc.html → test$com,
test$com$path, test$com$path$doc, …
 Queries are distributed over 7-machine cluster
 Now takes within 1-2 seconds in the worst case
 This is not the main cluster load
 Main load is searching 1.2B documents
Summary
 Discussed Sphinx full-text engine
 Discussed its pipeline internals – helps to
optimize queries
 Discussed how it can be used to offload and/or
optimize “general” SQL queries
 Got full-text queries? Try Sphinx

 Got questions? Now is the time!