Scalable relational database with SQL Azure

Building a scalable
database solution
with SQL Azure

About me
• Shy Engelberg, CTO @
• Email : Shy@Valinor.co.il
• Phone : 054-771-711-5
• Twitter : @ShyEngelberg

Agenda
• SQL Azure Database
• Intro
• Technical details
• Pricing tiers and performance
• Business continuity
• Elastic Database client library
• Shard management
• data-dependent routing
• cross-shard query capabilities

Microsoft Azure SQL Database
• SQL Database is a relational database service in the cloud based on
the market-leading Microsoft SQL Server engine, with mission-critical
capabilities.
SQL Database delivers predictable performance, scalability, business
continuity, data protection, and near-zero administration to cloud
developers and solution architects.
You can focus on rapid app development and accelerating your time
to market, rather than managing virtual machines and infrastructure.

• Because it’s based on the SQL Server engine,
Azure SQL Database supports existing SQL Server tools, libraries and
APIs, which makes it easier for you to move and extend to the cloud.
( Using the same TDS interface as regular SQL database)

• Based on SQL server technology. (A subset of the product)
• Fully managed service (Microsoft manage it completely!)
• We Don't to think and worry about:
 VMs
 Resources (IO, CPU, Memory)
 Installations, upgrades, patches
 Services
 Files placement
 Transaction log
Availability solutions

Microsoft Azure SQL Database – Server
• The basic unit in Azure that holds DBs is a “server”
• A server is a logical entity used for logical grouping of DBs and security
bounding.
It looks and smell like a server but it's basically only a TDS proxy endpoint.
(For the application it looks like a server (instance))
• When creating it, you are required to provide a unique server name-
(xxxx.database.windows.net).
• The server Initially contains only a master database.

Microsoft Azure SQL Database –DB
Microsoft Azure SQL Database has the following properties:
• Name – of the DB
• Service Tier (Performance Level)
• Max Size
• Collation – for all the tables are columns.
• “Server” – in which group to place the DB

Microsoft Azure SQL Database – service tiers
Azure SQL Database provides multiple service tiers to handle different types of
workloads.
• Performance level – measured by “DTU” (a combination of CPU, IO and log usage)
• Max size
• Point-in-time restore retention
• Parallel connections
• Size – does not include the log file size
You can change service tiers at any time with zero downtime to your application.
(Scale up)

Basic, Standard, and Premium service tiers all have an uptime SLA of 99.99%
• Basic: Best suited for a small size database, supporting typically one single active
operation at a given time.
(predictable performance hour over hour)
• Standard: The go-to option for most cloud applications, supporting multiple
concurrent queries.
(predictable performance minute over minute)
• Premium: Designed for high transactional volume, supporting a large number of
concurrent users and requiring the highest level of business continuity
capabilities.
(predictable performance second over second)

• The Database Transaction Unit (DTU) is the unit of measure in SQL
Database that represents the relative power of databases based on a
real-world measure: the database transaction. We took a set of
operations that are typical for an online transaction processing (OLTP)
request, and then measured how many transactions could be
completed per second under fully loaded conditions (that’s the short
version, you can read the gory details in the Benchmark overview).
• A Basic database has 5 DTUs, which means it can complete 5
transactions per second, while a Premium P11 database has 1750
DTUs.

• Create and manage Azure SQL database
DEMO

Microsoft Azure SQL Database – Connection
• The Microsoft Azure SQL Database service is only available with TCP
port 1433. (ensure your firewall allows outgoing TCP communication
on TCP port 1433)
• The protocol is TDS (Tabular Data Stream) protocol over TCP/IP.
• All connections are encrypted - SSL is required - if encryption is not
defined in the connection string, connection will fail.
• Use firewall rules to connect from outside Microsoft data center

Microsoft Azure SQL Database – Connection
• Connect using the server name : xxxx.database.windows.net
• Connection Security is managed by SQL Database Firewall, an IP
Address-based access control for SQL Database
 Rules can be defined at the server level
 No IP authorized by default
 Option to disable/enable access from
applications hosted in Windows Azure:

Azure SQL Database
• Firewall and connection
DEMO

Microsoft Azure SQL Database – Business continuity
• Let’s start from the bottom line:
• SLA is built- in 99.9%.
In opposed to traditional Always-On and failover cluster which
requires a lot of work and money, this solution is cheap and out-of-
the-box.

• Business continuity can be affected by one or more of the following
three major categories of issues:
Failure of individual servers, devices or network connectivity
(the disk that my data is on died)
Corruption, unwanted modification or deletion of data
(someone dropped a table or deleted a row)
Widespread loss of datacenter facilities
(Amsterdam is gone)

• Protection from Failure of Individual Servers and Devices - High
Availability through Infrastructure Redundancy:
Maintaining multiple (3) copies of all data in different physical nodes
located across fully independent physical sub-systems such as server
racks and network routers.

Point-in-time restore, Programmatic “oops recovery” of data deletion
or alteration:
• Auto backups, transactional logs every 5 min
• Backups in Azure Storage and geo-replicated
• Backups retention policy:
Basic, up to 7 days
Standard, up to 14 days
Premium, up to 35 days

Microsoft Azure SQL database - Scaling

Microsoft Azure SQL Database – Elastic tools
• Elastic Database features enables you to use the virtually unlimited
database resources of Azure SQL Database to create solutions for
transactional workloads, and especially Software as a Service (SaaS)
applications.
• Enabling this functionality is facilitated by features such as the elastic
database client library and split-merge tool, known together as
elastic database tools, which simplify building applications that rely
on sharding.

• Elastic Database tools:
simplify development and management of sharded database
solutions. The tools are: the Elastic Database client library and the
Elastic Database split-merge tool.

• Elastic Database pools (preview): A pool is a collection of databases to
which you can add or remove databases at any time. The databases in
the pool share a fixed amount of resources (DTUs).
• Elastic Database jobs (preview): Use jobs to manage large numbers of
Azure SQL databases. Easily perform administrative operations such
as schema changes, credentials management, reference data updates
and more
• Elastic Database query (preview): Enables you to run a Transact-SQL
query that spans multiple databases. This enables connection to
reporting tools such as Excel, PowerBI, Tableau, etc.

Microsoft Azure SQL database – sharding
“A database shard is a horizontal partition of data in a database.”
• Sharding is a technique to distribute large amounts of identically-
structured data across a number of independent databases. It is
especially popular with cloud developers who are creating Software
as a Service (SAAS) offerings for end customers or businesses. These
end customers are often referred to as “Tenants”.
• Sharding works best when every transaction in an application can be
restricted to a single value of a sharding key. That ensures that all
transactions will be local to a specific database.

Microsoft Azure SQL database – scaling out

Elastic Database client library
• The Elastic Database client library helps you easily develop sharded
applications using hundreds—or even thousands—of Azure SQL
databases hosted on Microsoft Azure
• Elastic Database client library is now available as open source
software on GitHub.
• Elastic database client library supports:
Shard management
Data-dependent routing
Multi-shard queries (MSQ)

Elastic Database library – Shard Map Management
• To manage a collection of shards, a special database called the "shard map
manager" is created.
It maintains information allowing an application to connect to the correct
database based upon the value of the sharding key.
• Shard map management is the ability for an application to manage various
metadata about its shards.
• Developers can use this functionality to register databases as shards,
describe mappings of individual sharding keys or key ranges to those
databases.
• Without the elastic database client library, you would need to spend a lot
of time writing the management code when implementing sharding.

• use the ShardMapManager class, found in the Elastic Database client
library to manage shard maps.
• Elastic Scale support the following .Net Framework types as sharding
keys:
integer
long
guid
byte[]
datetime
timespan
datetimeoffset

• Shard maps can be constructed using lists of individual sharding key
values, or they can be constructed using ranges of sharding key
values.
• The data managed by a ShardMapManager instance is kept in three
places:
Global Shard Map (GSM): You specify a database to serve as the repository
for all of its shard maps and mappings.
Local Shard Map (LSM): Every database that you specify to be a shard is
modified to contain shard map information.
Application cache

Elastic Database library – Data dependent routing
• Data dependent routing is the ability to use the data in a query to
route the request to an appropriate database. This is a fundamental
pattern when working with sharded databases.
• Each specific query or transaction in an application using data
dependent routing is restricted to accessing a single database per
request.
• the Shard Map Manager opens connections to the correct databases
when needed, based on the data in the shard map and the value of
the sharding key that is the target of the application’s request.

Elastic Database library – Data dependent routing
• The ShardMap.OpenConnectionForKey method returns an ADO.Net
connection ready for issuing commands to the appropriate database
based on the value of the key parameter.
• The OpenConnectionForKey method returns a new already-open
connection to the correct database. Connections utilized in this way
still take full advantage of ADO.Net connection pooling.

Elastic Database library – Multi-shard querying
• Multi-shard querying is used for tasks such as data
collection/reporting that require running a query that stretches
across several shards.
• The main entry point into multi-shard querying is the
MultiShardConnection class.
• myShardMap.GetShards() method retrieves all shards from the shard
map and provides an easy way to run a query across all relevant
databases.
• The collection of shards for a multi-shard query can be refined further
by performing a LINQ query over the collection returned from the call
to myShardMap.GetShards()

Elastic Database library
Working with Elastic database client library
DEMO

Scalable relational database with SQL Azure

More Related Content

Scalable relational database with SQL Azure

Editor's Notes