HA/DR options with SQL Server in Azure and hybrid

Editor's Notes

1. HA/DR options with SQL Server in Azure and hybrid What are all the high availability (HA) and disaster recovery (DR) options for SQL Server in a Azure VM (IaaS)?  Which of these options can be used in a hybrid combination (Azure VM and on-prem)?  I will cover features such as Always On AG, Failover cluster, Azure SQL Data Sync, Log Shipping, SQL Server data files in Azure, Mirroring, Azure Site Recovery, and Azure Backup.
2. https://blogs.msdn.microsoft.com/mast/2013/12/06/understanding-the-temporary-drive-on-windows-azure-virtual-machines/ SSD/HDD storage included in A-series, D-series, and Dv2-series VMs is local temporary storage. DS-series, G-series, GS-series SSD's have less local temporary storage due to storage used for caching purposes to ensure predictable levels of performance associated with premium storage. DS-series and GS-series support premium storage disks, which means you can attach SSD's to the VM (the other series support only standard storage disks). The pricing and billing meters for the DS sizes are the same as D-series and the GS sizes are the same as G-series. When you create an Azure virtual machine, it has a disk for the operating system mapped to drive C (size is 127GB) that is on Blob storage and a local temporary disk mapped to drive D. You can choose standard disk type or premium (if DS-series or GS-series) for your local temporary disk - the size of which is based on the series you choose (i.e. A0 is 20GB). You can also attach new disks - specify standard or premium, for standard: specify size (1GB-1023GB), for premium: specify P10, P20, or P30. the disks are .vhd files that reside in an Azure storage account
3. Because Azure stores three copies of your SQL databases in Azure Blob Storage, and Azure has redundant hardware (99.9% SLA for VMs), most companies don’t need to do anything more for HA (DR is a different story and Always On Availability Groups in different regions is one option for that). If the customer still feels like they need something more for HA, then Always On FCI or Always On Availability Groups in the same region would be the way to go. I lay out all the options in my deck at https://www.slideshare.net/jamserra/hadr-options-with-sql-server-in-azure-and-hybrid .
4. http://www.jamesserra.com/archive/2015/11/redundancy-options-in-azure-blob-storage/ Geo-replication in Azure disks does not support the data file and log file of the same database to be stored on separate disks. GRS replicates changes on each disk independently and asynchronously. This mechanism guarantees the write order within a single disk on the geo-replicated copy, but not across geo-replicated copies of multiple disks. If you configure a database to store its data file and its log file on separate disks, the recovered disks after a disaster may contain a more up-to-date copy of the data file than the log file, which breaks the write-ahead log in SQL Server and the ACID properties of transactions. If you do not have the option to disable geo-replication on the storage account, you should keep all data and log files for a given database on the same disk. If you must use more than one disk due to the size of the database, you need to deploy one of the disaster recovery solutions listed above to ensure data redundancy. From https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sql-high-availability-dr/
5. https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-classic-sql-dr/ It is up to you to ensure that your database system possesses the HADR capabilities that the service-level agreement (SLA) requires. The fact that Azure provides high availability mechanisms, such as service healing for cloud services and failure recovery detection for the Virtual Machines (https://azure.microsoft.com/en-us/blog/service-healing-auto-recovery-of-virtual-machines), does not itself guarantee you can meet the desired SLA. These mechanisms protect the high availability of the VMs but not the high availability of SQL Server running inside the VMs. It is possible for the SQL Server instance to fail while the VM is online and healthy. Moreover, even the high availability mechanisms provided by Azure allow for downtime of the VMs due to events such as recovery from software or hardware failures and operating system upgrades. I do not consider transactional replication as a HA/DR solution. Yes, data modifications are being pushed to subscribers but we're talking at the publication/article level. This is going to be a subset of the data (could include all the data, but that won't be enforced. I.e. you create a new table in the publisher database, and that will not automatically be pushed to the subscribers). Plus, tables without a primary key cannot be replicated. Can do SQL Server on-prem replication to Azure SQL Database. The intent of replication was very different than HA/DR.  "I want this table pushed to a data mart" or something like that. However, replication is the de facto solution for customers requiring subsets of data to be available.
6. http://www.netmagicsolutions.com/blog/defining-high-availability-and-disaster-recovery https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-high-availability-dr
7. https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sql-high-availability-dr/ https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-sql-server-alwayson-availability-groups-gui-arm/ https://blogs.msdn.microsoft.com/igorpag/2013/09/02/sql-server-2012-alwayson-availability-group-and-listener-in-azure-vms-notes-details-and-recommendations/ https://blogs.msdn.microsoft.com/igorpag/2014/07/03/deep-dive-sql-server-alwayson-availability-groups-and-cross-region-virtual-networks-in-azure/ https://blogs.technet.microsoft.com/dataplatforminsider/2014/06/19/sql-server-alwayson-availability-groups-supported-between-microsoft-azure-regions/ http://sqlha.com/2012/04/13/allans-alwayson-availability-groups-faq/ From Luis Carlos Vargas Herring: https://blogs.msdn.microsoft.com/clustering/2014/11/13/introducing-cloud-witness/ Both SQL FCI and AGs depend on Windows Cluster to handle quorum. Yes, in Windows Server 2016 they can leverage its Cloud Witness. This removes the need for a separate Azure VM to host the witness. We plan to enhance our AlwaysOn template to use this in H1CY17. File Share Witness: https://www.derekseaman.com/tag/file-share-witness, https://technet.microsoft.com/en-us/library/cc731739.aspx Availability Group listeners: https://msdn.microsoft.com/en-us/library/hh213417.aspx, https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-portal-sql-alwayson-int-listener Does an always on availability group listener require an additional VM? No, the listener is merely a networking concept (a way for client connections to be routed to the primary replica). It needs an Azure Load Balancer (https://azure.microsoft.com/en-us/services/load-balancer/, https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-portal-sql-ps-alwayson-int-listener)
8. https://msdn.microsoft.com/en-us/library/ff877884.aspx https://msdn.microsoft.com/en-us/library/hh510230.aspx https://www.brentozar.com/sql/sql-server-alwayson-availability-groups/
9. https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/distributed-availability-groups-always-on-availability-groups https://www.mssqltips.com/sqlservertip/5053/setup-and-implement-sql-server-2016-always-on-distributed-availability-groups/
10. https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/failover-and-failover-modes-always-on-availability-groups
11. An availability group has been configured with 2 replicas (primary P and secondary S1) for automatic failover and a Listener within the virtual network VNET1 in Region 1 (e.g. West US). This guarantees high availability of SQL Server in case of failures within the region.  A secure tunnel has been configured between VNET1 and another virtual network VNET2 in Region 2 (e.g. Central US). The availability group has been expanded with a third replica (S2) configured for manual failover in this VNET to enable disaster recovery in case of failures impacting Region1. Finally, the Listener has been configured to route connections to the primary replica, irrespective of which region hosts it. This allows client applications connect to the primary replica, with the same connection string, after failing over between Azure regions. https://blogs.technet.microsoft.com/dataplatforminsider/2014/06/19/sql-server-alwayson-availability-groups-supported-between-microsoft-azure-regions/ https://blogs.msdn.microsoft.com/igorpag/2014/12/22/sql-server-2014-high-availability-and-multi-datacenter-disaster-recovery-with-multiple-azure-ilbs/
12. https://azure.microsoft.com/en-us/blog/high-availability-for-a-file-share-using-wsfc-ilb-and-3rd-party-software-sios-datakeeper/ Replaces SQL Server failover cluster Requires shared storage
13. https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-portal-sql-create-failover-cluster
14. The only cost advantage for FCI is if a customer must use SQL Standard and an older version of SQL Server. AGs were only supported on SQL Enterprise before SQL16. In SQL16 they’re supported also on Standard edition.
15. https://msdn.microsoft.com/en-us/library/mt614935.aspx Basic availability groups use a subset of features compared to advanced availability groups on SQL Server 2016 Enterprise Edition. Basic availability groups include the following limitations: Limit of two replicas (primary and secondary). No read access on secondary replica. No backups on secondary replica. No support for replicas hosted on servers running a version of SQL Server prior to SQL Server 2016 Community Technology Preview 3 (CTP3). No support for adding or removing a replica to an existing basic availability group. Support for one availability database. Basic availability groups cannot be upgraded to advanced availability groups. The group must be dropped and re-added to a group that contains servers running only SQL Server 2016 Enterprise Edition. Basic availability groups are only supported for Standard Edition servers. https://www.concurrency.com/blog/july-2016/sql-server-2016-basic-availability-groups
16. https://msdn.microsoft.com/en-us/library/mt614935.aspx
17. What’s new in SQL Server 2016: http://searchsqlserver.techtarget.com/feature/Whats-new-in-2016s-SQL-Server-AlwaysOn-Availability-Groups
18. https://msdn.microsoft.com/en-us/library/ms187103.aspx https://msdn.microsoft.com/en-us/library/ms151224.aspx http://www.sqlservercentral.com/articles/Replication/logshippingvsreplication/1399/ http://gokasdba.blogspot.com/2014/06/log-shipping-vs-mirroring-vs-replication.html http://dba.stackexchange.com/questions/53815/clustering-vs-transactional-replication-vs-availability-groups
19. https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-sql-server-backup-and-restore/ https://blogs.technet.microsoft.com/dataplatforminsider/2016/04/19/sql-server-2016-cloud-backup-and-restore-enhancements/
20. https://msdn.microsoft.com/en-us/library/dn385720.aspx https://msdn.microsoft.com/en-us/library/dn673536.aspx
21. https://azure.microsoft.com/en-us/documentation/articles/site-recovery-overview/
22. https://blogs.technet.microsoft.com/uspartner_ts2team/2016/11/22/azure-single-instance-virtual-machine-sla/ https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-manage-availability/ https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-classic-configure-availability/ In ARM: You can’t change the VM’s Availability Set once the VM is created You can’t add an Azure VM to an Availability Set once the VM is created You can’t remove a VM from an Availability Set Instead use Powershell: https://buildwindows.wordpress.com/2016/02/25/add-or-change-an-arm-virtual-machines-availability-set/ From Luis Carlos Vargas Herring: “Availability Set” is an Azure concept (not a SQL technology). The Availability Set merely tells Azure to host VMs in different failure domains (racks) and upgrade domains. This ensures that a failure/upgrade only brings down one VM at a time. This works well for stateless apps (like web servers), but it’s not enough for stateful apps that need replicas with an exact copy of the data (like SQL Server). SQL Availability Groups solve this by replicating data from a primary SQL instance to one or more secondary SQL instances and orchestrating a failover in case of a failure. An Azure Availability Set is a deployment constraint. It tells Azure to allocate those VMs inside of it into different fault and upgrade domains. This ensures that a rack failure or a Host maintenance operation doesn’t impact more than one VM at the same time. This is a core measure for replicated components (e.g. web servers or SQL Server replicas). To configure the SQL Server replicas, you need AlwaysOn Availability Groups.
23. https://azure.microsoft.com/en-us/blog/azure-sql-data-sync-refresh/?v=17.23h https://azure.microsoft.com/en-us/documentation/articles/sql-database-get-started-sql-data-sync/ https://www.mssqltips.com/sqlservertip/3062/understanding-sql-data-sync-for-sql-server/ https://msdn.microsoft.com/en-us/library/hh868047.aspx Compare SQL Data Sync to Active Geo-Replication
24. When it comes to hybrid solution we are introducing new industry first concepts of stretch tables into Azure for operational databases. So think about it We are making migrating SQL Server from on-prem to Azure VMs much easier. Currently when you move SQL Server we only migrated the schema and data. With SQL v-next you will able to migrate systems objects, SQL Settings so that migration is literally point and click type of migration. We will provide a wizard drive experience that will provide gallery image, vm size recommendations.
25. Notes Now let’s talk about some of these innovative hybrid scenarios that can compliment your on-premises SQL Server investments. Stretch database, is one of these unique hybrid scenarios that only Microsoft provides and can be extremely valuable in your data strategy. We know your OLTP databases are growing rapidly and you need to think about how to cost effectively manage your data. More importantly, you need to think about what you’re doing with historical data and whether you want it to go offline onto tape. When it goes to tape, it’s not queryable anymore. What if you wanted the historical data at your fingertips but didn’t want to have it reside on premium storage with your hot data because it’s too costly. We can solve that problem with this stretch database hybrid scenario. Without modifying your app, you can stretch your database to Azure. You don’t even have to do any of the heavy lifting to make this work. You just have to set the policy you want to apply on the historical data and run the query as you normally would. SQL Server determines if the table has been stretched and retrieves the data from Azure. The other thing to note is that this technology does not impact the performance of writes on the same table being stretched as part of the engineering design point for this solution. A question you might ask is I am stretching historical customer data which has sensitive data, what about data security. The best part is that stretch database works with our new Always Encrypted technology that protects the columns you desire at rest and in motion, even in the memory buffer pool, so sensitive customer information is secure. Stretch database offers a great value prop in terms of saving you money and providing easy access to historical customer data so you can make improve customer experiences without requiring application modifications. This is just one example of how cloud-first innovation allows us to deliver new hybrid scenarios that our competitor simply can’t deliver or haven’t thought of.  
26. Source: https://msdn.microsoft.com/en-us/library/dn935011(v=sql.130).aspx Stretch Database lets you archive your historical data transparently and securely. In SQL Server 2016 Community Technology Preview 2 (CTP2), Stretch Database stores your historical data in the Microsoft Azure cloud. After you enable Stretch Database, it silently migrates your historical data to an Azure SQL Database. You don't have to change existing queries and client apps. You continue to have seamless access to both local and remote data. Your local queries and database operations against current data typically run faster. You typically enjoy reduced cost and complexity.
27. Source: https://msdn.microsoft.com/en-us/library/mt169378(v=sql.130).aspx Concepts and architecture for Stretch Database Terms Local database. The on-premises SQL Server 2016 Community Technology Preview 2 (CTP2) database. Remote endpoint. The location in Microsoft Azure that contains the database’s remote data. In SQL Server 2016 Community Technology Preview 2 (CTP2), this is an Azure SQL Database server. This is subject to change in the future. Local data. Data in a database with Stretch Database enabled that will not be moved to Azure based on the Stretch Database configuration of the tables in the database. Eligible data. Data in a database with Stretch Database enabled that has not yet been moved, but will be moved to Azure based on the Stretch Database configuration of the tables in the database. Remote data. Data in a database with Stretch Database enabled that has already been moved to Azure. Architecture Stretch Database leverages the resources in Microsoft Azure to offload archival data storage and query processing. When you enable Stretch Database on a database, it creates a secure linked server definition in the on-premises SQL Server. This linked server definition has the remote endpoint as the target. When you enable Stretch Database on a table in the database, it provisions remote resources and begins to migrate eligible data, if migration is enabled. Queries against tables with Stretch Database enabled automatically run against both the local database and the remote endpoint. Stretch Database leverages processing power in Azure to run queries against remote data by rewriting the query. You can see this rewriting as a "remote query" operator in the new query plan.
28. Source: https://msdn.microsoft.com/en-us/library/mt163698(v=sql.130).aspx