This document provides an overview of Microsoft Azure Data Services and Azure SQL Database. It discusses Infrastructure as a Service (IaaS) versus Platform as a Service (PaaS), and highlights the opportunities in the Linux database market. It also discusses Microsoft's commitment to customer choice and partnerships with companies like Red Hat. The remainder of the document focuses on features of Azure SQL Database, including an overview of the DTU and vCore purchasing models, managed instances, backup and recovery, high availability options, elastic scalability, and data sync capabilities.
ETL Made Easy with Azure Data Factory and Azure DatabricksDatabricks
This document summarizes Mark Kromer's presentation on using Azure Data Factory and Azure Databricks for ETL. It discusses using ADF for nightly data loads, slowly changing dimensions, and loading star schemas into data warehouses. It also covers using ADF for data science scenarios with data lakes. The presentation describes ADF mapping data flows for code-free data transformations at scale in the cloud without needing expertise in Spark, Scala, Python or Java. It highlights how mapping data flows allow users to focus on business logic and data transformations through an expression language and provides debugging and monitoring of data flows.
Modernizing to a Cloud Data ArchitectureDatabricks
Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.
Azure Data Factory ETL Patterns in the CloudMark Kromer
This document discusses ETL patterns in the cloud using Azure Data Factory. It covers topics like ETL vs ELT, the importance of scale and flexible schemas in cloud ETL, and how Azure Data Factory supports workflows, templates, and integration with on-premises and cloud data. It also provides examples of nightly ETL data flows, handling schema drift, loading dimensional models, and data science scenarios using Azure data services.
Microsoft Data Platform - What's includedJames Serra
This document provides an overview of a speaker and their upcoming presentation on Microsoft's data platform. The speaker is a 30-year IT veteran who has worked in various roles including BI architect, developer, and consultant. Their presentation will cover collecting and managing data, transforming and analyzing data, and visualizing and making decisions from data. It will also discuss Microsoft's various product offerings for data warehousing and big data solutions.
This document provides an overview of Azure SQL DB environments. It discusses the different types of cloud platforms including IaaS, PaaS and DBaaS. It summarizes the key features and benefits of Azure SQL DB including automatic backups, geo-replication for disaster recovery, and elastic pools for reducing costs. The document also covers pricing models, performance monitoring, automatic tuning capabilities, and security features of Azure SQL DB.
This is part 1 of the Azure storage series, where we will build our understanding of Azure Storage, and will also learn about the storage data services, and the types of Azure Storage. Last but not least, we will also touch base on securing storage accounts
In the second part, we will continue with our demo on creating and utilizing the Azure Storage.
Microsoft Data Integration Pipelines: Azure Data Factory and SSISMark Kromer
The document discusses tools for building ETL pipelines to consume hybrid data sources and load data into analytics systems at scale. It describes how Azure Data Factory and SQL Server Integration Services can be used to automate pipelines that extract, transform, and load data from both on-premises and cloud data stores into data warehouses and data lakes for analytics. Specific patterns shown include analyzing blog comments, sentiment analysis with machine learning, and loading a modern data warehouse.
Should I move my database to the cloud?James Serra
So you have been running on-prem SQL Server for a while now. Maybe you have taken the step to move it from bare metal to a VM, and have seen some nice benefits. Ready to see a TON more benefits? If you said “YES!”, then this is the session for you as I will go over the many benefits gained by moving your on-prem SQL Server to an Azure VM (IaaS). Then I will really blow your mind by showing you even more benefits by moving to Azure SQL Database (PaaS/DBaaS). And for those of you with a large data warehouse, I also got you covered with Azure SQL Data Warehouse. Along the way I will talk about the many hybrid approaches so you can take a gradual approve to moving to the cloud. If you are interested in cost savings, additional features, ease of use, quick scaling, improved reliability and ending the days of upgrading hardware, this is the session for you!
A closer look at the MySQL and PostgreSQL compatible relational database built for the cloud that combines the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. We’ll explore how Aurora uses the AWS cloud to provide high reliability, high durability, and high throughput.
Speakers:
Steve Abraham - Principal Database Specialist Solutions Architect, AWS
Peter Dachnowicz - Sr. Technical Account Manager, AWS
The document discusses Azure Data Factory V2 data flows. It will provide an introduction to Azure Data Factory, discuss data flows, and have attendees build a simple data flow to demonstrate how they work. The speaker will introduce Azure Data Factory and data flows, explain concepts like pipelines, linked services, and data flows, and guide a hands-on demo where attendees build a data flow to join customer data to postal district data to add matching postal towns.
The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.
This document is a training presentation on Databricks fundamentals and the data lakehouse concept by Dalibor Wijas from November 2022. It introduces Wijas and his experience. It then discusses what Databricks is, why it is needed, what a data lakehouse is, how Databricks enables the data lakehouse concept using Apache Spark and Delta Lake. It also covers how Databricks supports data engineering, data warehousing, and offers tools for data ingestion, transformation, pipelines and more.
Azure SQL Database is a managed cloud database service that makes building and maintaining applications easier. It provides continuous learning of app patterns to optimize performance, reliability, and data protection. The service takes care of scalability, backup, and high availability. It provides recommendations to optimize database performance and fix issues. Azure SQL Database offers pricing tiers for different performance levels and capabilities for security, monitoring, and compliance. It can be used for a variety of workloads including web, mobile, and multi-tenant apps.
Azure SQL Database is a cloud-based relational database service built on the Microsoft SQL Server engine. It provides predictable performance and scalability with minimal downtime and administration. Key features include elastic pools for cost-effective scaling, built-in backups and disaster recovery, security features like encryption and auditing, and tools for management and monitoring performance. The document provides an overview of Azure SQL Database capabilities and service tiers for databases and elastic pools.
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. I’ll include use cases so you can see what approach will work best for your big data needs.
Data Migration to Azure SQL and Azure SQL Managed Instance - June 19 2020Timothy McAliley
- This document provides information about upcoming webinars on migrating databases to Azure SQL services from June 19th through October 30th. It also lists resources for assessing databases and migrating them to Azure SQL Database or Managed Instance using tools like Azure Database Migration Service, Data Migration Assistant, and SQL Server Management Studio. Contact information is provided to RSVP or find more details on migration strategies and tools.
Microsoft Azure BI Solutions in the CloudMark Kromer
This document provides an overview of several Microsoft Azure cloud data and analytics services:
- Azure Data Factory is a data integration service that can move and transform data between cloud and on-premises data stores as part of scheduled or event-driven workflows.
- Azure SQL Data Warehouse is a cloud data warehouse that provides elastic scaling for large BI and analytics workloads. It can scale compute resources on demand.
- Azure Machine Learning enables building, training, and deploying machine learning models and creating APIs for predictive analytics.
- Power BI provides interactive reports, visualizations, and dashboards that can combine multiple datasets and be embedded in applications.
Azure SQL DB Managed Instances Built to easily modernize application data layerMicrosoft Tech Community
The document discusses Azure SQL Database Managed Instance, a new fully managed database service that provides SQL Server compatibility. It offers seamless migration of SQL Server workloads to the cloud with full compatibility, isolation, security and manageability. Customers can realize up to a 406% ROI over on-premises solutions through lower TCO, automatic management and scaling capabilities.
This document provides an overview of a course on implementing a modern data platform architecture using Azure services. The course objectives are to understand cloud and big data concepts, the role of Azure data services in a modern data platform, and how to implement a reference architecture using Azure data services. The course will provide an ARM template for a data platform solution that can address most data challenges.
This document provides an agenda and summary for a Data Analytics Meetup (DAM) on March 27, 2018. The agenda covers topics such as disruption opportunities in a changing data landscape, transitioning from traditional to modern BI architectures using Azure, Azure SQL Database vs Data Warehouse, data integration with Azure Data Factory and SSIS, Analysis Services, Power BI reporting, and a wrap-up. The document discusses challenges around data growth, digital transformation, and the shrinking time for companies to adapt to disruption. It provides overviews and comparisons of Azure SQL Database, Data Warehouse, and related Azure services to help modernize analytics architectures.
Ralph Kemperdick – IT-Tage 2015 – Microsoft Azure als DatenplattformInformatik Aktuell
In dieser Session möchten wir eine Orientierung geben, welche Daten-Services auf Azure die geeignete Plattform für eine App bzw. eine Anwendung sein können. Die Session konzentriert sich auf die Platform as a Service (PaaS) mit einem SQL Interface. Es wird Azure SQL Server, Azure SQL DW, DocumentDB, Stream Analytics, Spark/Scala/Hive und Data Lake Analytics betrachtet und Unterschiede herausgearbeitet. Live Demos begleiten die einzelnen Themen in der Session. Ferner werden Argumente für und gegen Cloud basierte Services diskutiert.
The document discusses new features and enhancements in Microsoft SQL Server 2016 including operational analytics and in-memory performance improvements, security upgrades like Always Encrypted, higher availability with AlwaysOn, improved scalability, access to any data with PolyBase and JSON support, powerful insights on any device with mobile BI, advanced analytics at massive scale, and breakthrough hybrid scenarios with SQL Server in Azure like Stretch Database. It also provides hardware and software requirements.
What are the features of SQL server standard editions.pdfDirect Deals, LLC
SQL Server Standard edition delivers core data management and business intelligence database for agencies and small organizations. It can help to process their applications and assists common advanced tools for on-premises and cloud-enabling effective database management with lesser IT resources. Visit Here: - https://www.directdeals.com/
The document summarizes new features and enhancements in SQL Server 2016 including operational analytics and in-memory performance improvements, security upgrades like Always Encrypted, higher availability with AlwaysOn, improved scalability, hybrid cloud solutions like Stretch Database, and built-in advanced analytics at massive scale. It also covers new reporting, mobile BI, and consistency between on-premises and Microsoft Azure environments.
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)Trivadis
In dieser Session stellen wir ein Projekt vor, in welchem wir ein umfassendes BI-System mit Hilfe von Azure Blob Storage, Azure SQL, Azure Logic Apps und Azure Analysis Services für und in der Azure Cloud aufgebaut haben. Wir berichten über die Herausforderungen, wie wir diese gelöst haben und welche Learnings und Best Practices wir mitgenommen haben.
Microsoft® SQL Server® 2012 is a cloud-ready information platform that will help organizations unlock breakthrough insights across the organization and quickly build solutions to extend data across on-premises and public cloud, backed by mission critical confidence.
Simplify and Accelerate SQL Server Migration to AzureDelphix
Migrating data and applications to the cloud are highly iterative and require repeated test cycles and rapid provisioning to ensure business continuity and smooth operations. Thousands of organizations are faced with the upcoming SQL Server 2008 end of service in July 2019 and have an immediate need to upgrade or migrate while maintaining data security without affecting their business-critical operations.
What is in a modern BI architecture? In this presentation, we explore PaaS, Azure Active Directory and Storage options including SQL Database and SQL Datawarehouse.
Microsoft Azure zmienia się. Jego częśc poświęcona bazie danych (Windows Azure SQL Database) zmienia się jeszcze szybciej. Podczas tej sesji chciałbym pokazac tym, którzy nie widzieli, oraz przypomniec tym, którzy już coś wiedzą - o co chodzi z WASD, jakie zmiany nastapiły i czego możemy po tej bazie oczekiwać. Dla odważnych będzie okazja podłączenia się do konta w chmurze i przetestowania ych rozwiązań samemu.
Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.
An Overview of All The Different Databases in Google CloudFibonalabs
Google cloud platform (GCP) is a high-performance infrastructure for cloud computing, data analytics, and machine learning. Google Cloud runs on the same infrastructure that Google uses for its end-user products like Google Search, Gmail, Google Drive, Google Photos, etc.
CirrusDB provides cloud database and business intelligence services that help companies reduce costs and improve flexibility. Their offerings include managed database services, cloud databases, pre-configured appliances, and professional services. CirrusDB integrates multiple cloud platforms through their Cirrus Enterprise Manager product and claims advantages in scalability, virtualization, and clustering.
This document provides an introduction to Azure SQL Database. It describes Azure SQL Database as a fully managed relational database service. It notes that Azure SQL Database differs from SQL Server in some ways, such as not supporting certain T-SQL constructs and commands. The document also discusses server provisioning, database deployment, monitoring, and new service tiers for Azure SQL Database that offer different levels of scalability, performance, and business continuity features.
The document discusses Microsoft's data platform and cloud services. It highlights:
1) Microsoft's data platform provides intelligence over all data with SQL and Apache Spark, enabling AI and machine learning over any data.
2) Microsoft offers data modernization solutions for migrating to the cloud or managing data on-premises and in hybrid environments.
3) Migrating databases to Azure provides cost savings, security, high performance, and intelligent capabilities through services like Azure SQL Database and Azure Cosmos DB.
Similar to Azure SQL Database & Azure SQL Data Warehouse (20)
Microsoft Azure Cosmos DB is a multi-model database that supports document, key-value, wide-column and graph data models. It provides high throughput, low latency and global distribution across multiple regions. Cosmos DB supports multiple APIs including SQL, MongoDB, Cassandra and Gremlin to allow developers to use their preferred API based on their application needs and skills. It also provides automatic scaling of throughput and storage across all data partitions.
Designing big data analytics solutions on azureMohamed Tawfik
This document discusses designing big data analytics solutions on Azure. It provides an overview of Azure's data landscape and common architectural patterns and scenarios for building analytics solutions using various Azure data and analytics services. These include Azure SQL Data Warehouse, Azure Data Lake Store, Azure Data Factory, Azure Machine Learning, and Power BI for reporting and visualization. The document also discusses using these services to build solutions for scenarios like data warehousing, data lakes, ETL/ELT, machine learning, streaming analytics and more.
Microsoft Azure Offerings and New Services Mohamed Tawfik
Microsoft Azure offers a wide range of computing services including networking, compute, storage, databases, developer tools, and analytics services. It provides benefits such as pay-as-you-go pricing, quick setup, scalability, redundancy, and high availability. Microsoft has seen incredible growth in Azure due to its ability to convert its large enterprise customer base into Azure customers and build hybrid cloud solutions. The presentation highlights several new Azure services and features in networking, compute, storage, databases, and security.
This document discusses setting up System Center Configuration Manager (SCCM) on Microsoft Azure. It begins with an overview of cloud computing benefits and Microsoft Azure features. It then reviews the System Center suite and describes the SCCM on Azure architecture with a SQL database, IIS, and load balancer. Steps are provided for deploying the base configuration in Azure. The document demonstrates SCCM functionality and concludes with notes on additional configuration topics.
This document provides an overview of IBM Watson including:
- A brief history of Watson and how it defeated human opponents on Jeopardy in 2011.
- Technical specifications of Watson including its architecture using 90 IBM Power 750 servers with 2,880 POWER7 processor threads and 16 terabytes of RAM.
- Key technologies that Watson utilizes including Apache UIMA, Hadoop, and DeepQA for natural language processing and question answering.
- Commercial applications of Watson that have been developed for industries like healthcare, finance, and customer service.
- Related cognitive computing technologies like Microsoft Azure Machine Learning and HPE HAVEn OnDemand.
Upcoming Challenges in E-Learning & Online Learning EnvironmentsMohamed Tawfik
Upcoming challenges in e-learning and online learning environments include:
1) Transitioning to blended learning models that combine online and in-person instruction.
2) Integrating learning management systems with remote laboratories and services like Web 2.0 tools.
3) Developing mobile learning capabilities that incorporate location-based and user-based interactions in a new framework deployable on smart devices.
FINTDI 2011 - Remote Laboratories for Electrical & Electronic Subjects in New...Mohamed Tawfik
Este documento describe una plataforma para laboratorios remotos de ingeniería eléctrica y electrónica. La plataforma utiliza una matriz de conmutación para conectar componentes como osciloscopios, fuentes de alimentación y generadores de funciones a través de una plataforma PXI controlada por software LabVIEW. Los estudiantes pueden acceder a los experimentos a través de una página web y realizar mediciones en tiempo real de forma remota. El objetivo es expandir el alcance de la plataforma y integrarla con sistemas de
This document discusses remote laboratories and their implementation in engineering education. It notes that remote labs help bridge the gap between educational curricula and real-world industry by allowing experimentation without constraints of location or time. Several challenges in developing remote labs are outlined, including selecting lab server software and integrating labs with learning management systems. Examples of remote lab architectures and systems like iLab, Labshare, and WebLab Deusto are provided. The benefits of standards-based integration of remote labs into online education are discussed.
Here are the key points about the PXI platform components:
- The PXI platform consists of instrument modules, a controller card, and a chassis to hold the cards.
- The modules (NI PXI instruments) substitute the traditional standalone instruments. They plug into the chassis.
- The controller card is an embedded PC that controls the entire system. It plugs into the chassis.
- The chassis provides power and communication connections for the modules and controller.
- At UNED, the specific models installed include the PXI-1031 chassis, modules like the PXI-4072 DMM and PXI-5114 oscilloscope, and the PXI-8105 controller.
-
GOLC 2012 - On Standardizing the Management of LabVIEW-based Remote Laborator...Mohamed Tawfik
This document discusses standardizing the management of remote laboratories built using LabVIEW through remote laboratory management systems (RLMSs). It outlines the need for a standard application programming interface (API) layer to wrap LabVIEW-based remote labs and make them compatible with different RLMSs. The layer would define a common set of communication tools from LabVIEW, such as VI server and web services, to connect remote labs to RLMSs while addressing factors like simultaneous access, security, and session management. Developing such an API layer could help share and manage the many existing LabVIEW-based remote labs across various university platforms.
REV 2011 - A New Node in the VISIR CommunityMohamed Tawfik
The document discusses developments in the VISIR remote laboratory project. VISIR allows students to perform measurements and experiments on electric and electronic circuits remotely. Several universities have implemented VISIR nodes. Efforts are underway to integrate VISIR with learning management systems and online engineering frameworks to expand access and sharing of laboratory resources between institutions.
The document discusses integrating remote laboratories into management systems. It describes challenges in integrating diverse lab interfaces and technologies like LabVIEW. The authors propose creating standard APIs to facilitate integrating remote labs, especially LabVIEW-based ones, into remote laboratory management systems like Sahara. This would allow labs developed across universities to be more easily shared and managed through a common system.
TAEE 2011- State-of-the-Art Remote Laboratories for Industrial Electronics Ap...Mohamed Tawfik
This document summarizes a study on state-of-the-art remote laboratories for industrial electronics applications. It discusses how remote labs address gaps in engineering education by providing ubiquitous experimentation. Common architectures use LabVIEW or MATLAB for the lab server software and technologies like AJAX or LabVIEW's web interface for client-server communication. The document also provides examples of remote lab systems and outlines challenges in selecting server and communication technologies.
This document provides information about the Institute of Electrical and Electronics Engineers (IEEE). IEEE is the world's largest technical professional organization dedicated to advancing technology for humanity. It has over 400,000 members across over 160 countries. IEEE was formed in 1963 by the merger of the Institute of Radio Engineers and the American Institute of Electrical Engineers. It consists of various societies, councils, sections and branches focused on different technical areas.
Copec ICECE 2011- DESIGN OF PRACTICAL ACTIVITIES IN ELECTRONICSMohamed Tawfik
VISIR is a remote laboratory for wiring and measuring electric circuits. It uses a PXI platform and relay switching matrix to connect various instruments. Several universities have implemented VISIR labs. Efforts are underway to standardize VISIR using LXI instruments, reduce costs, and integrate VISIR into learning management systems and online lab frameworks to enable broader access and sharing of lab resources.
TAEE 2012- Shareable Educational Architectures for Remote LaboratoriesMohamed Tawfik
This document discusses shareable educational architectures for remote laboratories in engineering education. It introduces remote laboratories, which allow students to control and administer online experiments interacting with physical instruments anywhere and anytime. Several existing remote laboratory systems are described that aim to integrate labs across learning management systems and universities through standard APIs. The document promotes the Global Online Laboratory Consortium which works to develop shared remote labs and interoperability between different remote lab systems to improve engineering education.
The PAC project aims to develop adaptive master's degree programs in engineering fields to better meet the needs of the labor market. It involves partnerships between universities and businesses to input employment needs into curriculum design. The programs will focus on skills and competencies required by industries, include virtual and practical learning components, and integrate work experience through internships. The goal is to transform engineering education from a traditional model to a performance-centered, employment-oriented approach.
Educon 2012- On the Design of Remote LaboratoriesMohamed Tawfik
This document discusses remote laboratory architectures and technologies for developing lab server software. It compares LabVIEW and MATLAB, the most common technologies used. Both LabVIEW and MATLAB possess rich features for data exchange, instrument control, and database connectivity. LabVIEW is most popular for remote labs due to its graphical interface while MATLAB is powerful for algorithms. Hybrid methods using both are common, with LabVIEW for signals/GUI and MATLAB for calculations. The document was presented by researchers from the Spanish University for Distance Education.
ASEE 2012 - Common Multidisciplinary Prototypes of Remote Laboratories in the...Mohamed Tawfik
This document summarizes common types of remote laboratories used in electrical and computer engineering education. It describes three main types: 1) Embedded systems using microcontrollers and programmable logic devices, 2) Instrumentation and measurements of electronic circuits and control systems using data acquisition cards, and 3) Programmable logic controllers for automation control. It also compares the popular remote lab development platforms of LabVIEW and MATLAB and describes a hybrid approach. In conclusion, more information about remote labs can be found on the UNED engineering department website.
TAEE2012-Putting Fundmentals of Electronic Circuits Practices onlineMohamed Tawfik
This document discusses putting fundamentals of electronic circuits practices online through remote laboratories. It presents several solutions for remote labs, including NetLab, Virtual Instrument Systems in Reality (VISIR), and labs based on National Instruments' ELVIS platform. These solutions allow students to perform circuit design, construction, and measurement experiments remotely. Schools implementing VISIR have seen pleasant results applying it to teach concepts like rectifiers, regulators, and transistor circuits. Remote labs provide ubiquitous access to improve engineering education when in-person labs have limitations.
Transcript: Details of description part II: Describing images in practice - T...BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and slides: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Navigating Post-Quantum Blockchain: Resilient Cryptography in Quantum Threatsanupriti
In the rapidly evolving landscape of blockchain technology, the advent of quantum computing poses unprecedented challenges to traditional cryptographic methods. As quantum computing capabilities advance, the vulnerabilities of current cryptographic standards become increasingly apparent.
This presentation, "Navigating Post-Quantum Blockchain: Resilient Cryptography in Quantum Threats," explores the intersection of blockchain technology and quantum computing. It delves into the urgent need for resilient cryptographic solutions that can withstand the computational power of quantum adversaries.
Key topics covered include:
An overview of quantum computing and its implications for blockchain security.
Current cryptographic standards and their vulnerabilities in the face of quantum threats.
Emerging post-quantum cryptographic algorithms and their applicability to blockchain systems.
Case studies and real-world implications of quantum-resistant blockchain implementations.
Strategies for integrating post-quantum cryptography into existing blockchain frameworks.
Join us as we navigate the complexities of securing blockchain networks in a quantum-enabled future. Gain insights into the latest advancements and best practices for safeguarding data integrity and privacy in the era of quantum threats.
GDG Cloud Southlake #34: Neatsun Ziv: Automating AppsecJames Anderson
The lecture titled "Automating AppSec" delves into the critical challenges associated with manual application security (AppSec) processes and outlines strategic approaches for incorporating automation to enhance efficiency, accuracy, and scalability. The lecture is structured to highlight the inherent difficulties in traditional AppSec practices, emphasizing the labor-intensive triage of issues, the complexity of identifying responsible owners for security flaws, and the challenges of implementing security checks within CI/CD pipelines. Furthermore, it provides actionable insights on automating these processes to not only mitigate these pains but also to enable a more proactive and scalable security posture within development cycles.
The Pains of Manual AppSec:
This section will explore the time-consuming and error-prone nature of manually triaging security issues, including the difficulty of prioritizing vulnerabilities based on their actual risk to the organization. It will also discuss the challenges in determining ownership for remediation tasks, a process often complicated by cross-functional teams and microservices architectures. Additionally, the inefficiencies of manual checks within CI/CD gates will be examined, highlighting how they can delay deployments and introduce security risks.
Automating CI/CD Gates:
Here, the focus shifts to the automation of security within the CI/CD pipelines. The lecture will cover methods to seamlessly integrate security tools that automatically scan for vulnerabilities as part of the build process, thereby ensuring that security is a core component of the development lifecycle. Strategies for configuring automated gates that can block or flag builds based on the severity of detected issues will be discussed, ensuring that only secure code progresses through the pipeline.
Triaging Issues with Automation:
This segment addresses how automation can be leveraged to intelligently triage and prioritize security issues. It will cover technologies and methodologies for automatically assessing the context and potential impact of vulnerabilities, facilitating quicker and more accurate decision-making. The use of automated alerting and reporting mechanisms to ensure the right stakeholders are informed in a timely manner will also be discussed.
Identifying Ownership Automatically:
Automating the process of identifying who owns the responsibility for fixing specific security issues is critical for efficient remediation. This part of the lecture will explore tools and practices for mapping vulnerabilities to code owners, leveraging version control and project management tools.
Three Tips to Scale the Shift Left Program:
Finally, the lecture will offer three practical tips for organizations looking to scale their Shift Left security programs. These will include recommendations on fostering a security culture within development teams, employing DevSecOps principles to integrate security throughout the development
The DealBook is our annual overview of the Ukrainian tech investment industry. This edition comprehensively covers the full year 2023 and the first deals of 2024.
AC Atlassian Coimbatore Session Slides( 22/06/2024)apoorva2579
This is the combined Sessions of ACE Atlassian Coimbatore event happened on 22nd June 2024
The session order is as follows:
1.AI and future of help desk by Rajesh Shanmugam
2. Harnessing the power of GenAI for your business by Siddharth
3. Fallacies of GenAI by Raju Kandaswamy
How RPA Help in the Transportation and Logistics Industry.pptxSynapseIndia
Revolutionize your transportation processes with our cutting-edge RPA software. Automate repetitive tasks, reduce costs, and enhance efficiency in the logistics sector with our advanced solutions.
How to Avoid Learning the Linux-Kernel Memory ModelScyllaDB
The Linux-kernel memory model (LKMM) is a powerful tool for developing highly concurrent Linux-kernel code, but it also has a steep learning curve. Wouldn't it be great to get most of LKMM's benefits without the learning curve?
This talk will describe how to do exactly that by using the standard Linux-kernel APIs (locking, reference counting, RCU) along with a simple rules of thumb, thus gaining most of LKMM's power with less learning. And the full LKMM is always there when you need it!
What Not to Document and Why_ (North Bay Python 2024)Margaret Fero
We’re hopefully all on board with writing documentation for our projects. However, especially with the rise of supply-chain attacks, there are some aspects of our projects that we really shouldn’t document, and should instead remediate as vulnerabilities. If we do document these aspects of a project, it may help someone compromise the project itself or our users. In this talk, you will learn why some aspects of documentation may help attackers more than users, how to recognize those aspects in your own projects, and what to do when you encounter such an issue.
These are slides as presented at North Bay Python 2024, with one minor modification to add the URL of a tweet screenshotted in the presentation.
Details of description part II: Describing images in practice - Tech Forum 2024BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and transcript: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsMydbops
This presentation, delivered at the Postgres Bangalore (PGBLR) Meetup-2 on June 29th, 2024, dives deep into connection pooling for PostgreSQL databases. Aakash M, a PostgreSQL Tech Lead at Mydbops, explores the challenges of managing numerous connections and explains how connection pooling optimizes performance and resource utilization.
Key Takeaways:
* Understand why connection pooling is essential for high-traffic applications
* Explore various connection poolers available for PostgreSQL, including pgbouncer
* Learn the configuration options and functionalities of pgbouncer
* Discover best practices for monitoring and troubleshooting connection pooling setups
* Gain insights into real-world use cases and considerations for production environments
This presentation is ideal for:
* Database administrators (DBAs)
* Developers working with PostgreSQL
* DevOps engineers
* Anyone interested in optimizing PostgreSQL performance
Contact info@mydbops.com for PostgreSQL Managed, Consulting and Remote DBA Services
this resume for sadika shaikh bca studentSadikaShaikh7
I am a dedicated BCA student with a strong foundation in web technologies, including PHP and MySQL. I have hands-on experience in Java and Python, and a solid understanding of data structures. My technical skills are complemented by my ability to learn quickly and adapt to new challenges in the ever-evolving field of computer science.
How Netflix Builds High Performance Applications at Global ScaleScyllaDB
We all want to build applications that are blazingly fast. We also want to scale them to users all over the world. Can the two happen together? Can users in the slowest of environments also get a fast experience? Learn how we do this at Netflix: how we understand every user's needs and preferences and build high performance applications that work for every user, every time.
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Chris Swan
Have you noticed the OpenSSF Scorecard badges on the official Dart and Flutter repos? It's Google's way of showing that they care about security. Practices such as pinning dependencies, branch protection, required reviews, continuous integration tests etc. are measured to provide a score and accompanying badge.
You can do the same for your projects, and this presentation will show you how, with an emphasis on the unique challenges that come up when working with Dart and Flutter.
The session will provide a walkthrough of the steps involved in securing a first repository, and then what it takes to repeat that process across an organization with multiple repos. It will also look at the ongoing maintenance involved once scorecards have been implemented, and how aspects of that maintenance can be better automated to minimize toil.
In this follow-up session on knowledge and prompt engineering, we will explore structured prompting, chain of thought prompting, iterative prompting, prompt optimization, emotional language prompts, and the inclusion of user signals and industry-specific data to enhance LLM performance.
Join EIS Founder & CEO Seth Earley and special guest Nick Usborne, Copywriter, Trainer, and Speaker, as they delve into these methodologies to improve AI-driven knowledge processes for employees and customers alike.
Interaction Latency: Square's User-Centric Mobile Performance MetricScyllaDB
Mobile performance metrics often take inspiration from the backend world and measure resource usage (CPU usage, memory usage, etc) and workload durations (how long a piece of code takes to run).
However, mobile apps are used by humans and the app performance directly impacts their experience, so we should primarily track user-centric mobile performance metrics. Following the lead of tech giants, the mobile industry at large is now adopting the tracking of app launch time and smoothness (jank during motion).
At Square, our customers spend most of their time in the app long after it's launched, and they don't scroll much, so app launch time and smoothness aren't critical metrics. What should we track instead?
This talk will introduce you to Interaction Latency, a user-centric mobile performance metric inspired from the Web Vital metric Interaction to Next Paint"" (web.dev/inp). We'll go over why apps need to track this, how to properly implement its tracking (it's tricky!), how to aggregate this metric and what thresholds you should target.
7. There’s big
opportunity
$15B+
Linux DB
market by 2019
Source: Cloud Market Intelligence, FY16 H1 LRF (Nov 2015)
Windows
Linux
Relational DB market
growth through 2019
New server shipments of Linux
expected to be 2.4xthat of
Windows by FY 2021
6.6%
per year
Microsoft is the only
Gartner RDBMS
Magic Quadrant
vendor without
support for Linux
8. Committed
to choice
Azure and Red Hat partnership
HDInsight for Linux
R Server on Linux
SQL Server on Linux
So for the first time
now, we have the
ability to go to an
enterprise and talk
about that entire data
estate across Windows
and Linux.
11. Windows Linux
Developer, Express, Web, Standard, Enterprise
Database Engine, Integration Services
R Services, Analysis Services, Reporting Services, MDS, DQS
Maximum number of cores Unlimited Unlimited
Maximum memory utilized per instance 24 TB 12 TB
Maximum database size 524 PB 524 PB
Basic OLTP (Basic In-Memory OLTP, Basic operational analytics)
Advanced OLTP (Advanced In-Memory OLTP, Advanced operational analytics)
Basic high availability (2-node single database failover, non-readable secondary)
Advanced HA (Always On - multi-node, multi-db failover, readable secondaries)
Security
Basic security (Basic auditing, Row-level security, Data masking, Always Encrypted)
Advanced security (Transparent Data Encryption)
Data
warehousing
PolyBase2
Basic data warehousing/data marts (Basic In-Memory ColumnStore, Partitioning, Compression)
Advanced data warehousing (Advanced In-Memory ColumnStore)
Advanced data integration (Fuzzy grouping and look ups)
Tools
Windows ecosystem: Full-fidelity Management & Dev Tool (SSMS & SSDT), command line tools
Linux/OSX/Windows ecosystem: Dev tools (VS Code), DB Admin GUI tool, command line tools
Developer
Programmability (T-SQL, CLR, Data Types, JSON)
Windows Filesystem Integration - FileTable
Business
intelligence
Basic reporting, analytics & data integration
Basic Corporate Business Intelligence (Multi-dimensional models, Basic tabular model)
Advanced Corporate Business Intelligence (Advanced tabular model, DirectQuery, advanced data mining)
Mobile BI (Datazen)
Advanced analytics
Basic “R” integration (Connectivity to R Open, Limited parallelism for ScaleR)
Advanced “R” integration (Full parallelism for ScaleR)
Hybrid cloud Stretch Database
What’s coming in
SQL Server on
Linux
12. 12
Azure SQL Database (PaaS)
Fully managed database-as-a-service that lets you focus on your business
Database provisioning on-demand
Scalable and elastic performance for all workloads
99.99% availability, zero maintenance
Intelligent: learns and adapts to optimize performance
Secure and compliant to protect sensitive data
Geo-replication and restore-from-backup for data protection
Compatible with SQL Server 2014, 2016
14. Seamless and compatibleIntelligent DBaaS Competitive TCO
( 2 0 1 7 ) A Z U R E S Q L DATA B A S E
Privacy and Trust
OPERATIONAL ANALYTICS
Columnstore
Hekaton (in-memory
OLTP)
PREDICTABLE PERFORMANCE
Query Store
Index Optimization
AUTOMATIC TUNING
AUTO QUERY PLAN
CORRECTION
PERFORMANCE INSIGHT IN
OMS
ADAPTIVE QUERY
PROCESSING
SQL GRAPH
ADVANCED ANALYTICS
NATIVE PREDICT
R SERVICES
ACTIVITY MONITORING
Engine Audit
Threat Detection (NEW
SCENARIOS)
CENTRALIZED DASHBOARD
OMS INTEGRATION
ACCESS CONTROL
SQL Firewall
RLS, Dyn. Data Masking
AAD WITH MFA
DATA PROTECTION
Encrypt in motion (TLS)
TDE & BYK
Always Encrypted (S/W)
SERVICE ENDPOINT
ALWAYS ENCRYPTED (SECURE
H/W)
DISCOVERY & ASSESSMENT
VULNERABILITY ASSESSMENT
HA-DR BUILT-IN
99.99% SLA
Geo-restore
ACTIVE GEO REPLICAS (4)
MULTI-AZ
BACKUP AND RESTORE
Backup with health
check
35 days PITR
10 YEARS DATA RETENTION
DISTRIBUTED APPLICATION
Change Tracking
TRANSACTION REPLICATION
DATA SYNC
SSIS SERVICE
BIZ MODEL & SKUS
DTU/eDTU
<=1TB
BIGGER STD: S4-S12
SEPARATE COMPUTE AND
STORAGE
AZURE HYBRID BENEFIT
COST OPTIMIZATION
INTELLIGENT PAAS
16. 16
Azure SQL Database (PaaS)
You need to use a logical server prior to creating your first database.A logical server is the entry point
for the databases and controls logins, firewall rules, auditing rules, thread detection policies and
failover groups.You should not confuse an Azure SQL Database logical server with an on-premises SQL
Server.The logical server is a logical structure that doesn’t provide any way for connecting to instance
or feature level.
Because of how Azure provides high availability to the databases, there is no need for the Logical server
to be on the same region as the databases it manages.Azure SQL Database does not guarantee that
the logical server and its related databases will be on the same region.
This first account is a SQL login account.You can only use SQL login andAzure Active Directory login
accounts.Windows authentication is not supported with SQL logical server.
21. 21
vCore-based model
Each 100 DTU in Standard tier requires at least 1 vCore in General Purpose tier; each
125 DTU in Premium tier requires at least 1 vCore in Business Critical tier.
In the vCore-based purchasing model, you can exchange your existing licenses for
discounted rates on SQL Database using the Azure Hybrid Use Benefit for SQL Server.
This Azure benefit allows you to use your on-premises SQL Server licenses to save
more than 40% on Azure SQL Database using your on-premises SQL Server licenses
with Software Assurance.
If your database or elastic pool consumes more than 300 DTU conversion to vCore
may reduce your cost.
28. 28
Elastic pools
You can configure resources for the pool based
either on the DTU-based purchasing model or the
vCore-based purchasing model.The resource
requirement for a pool is determined by the
aggregate utilization of its databases.The
amount of resources available to the pool is
controlled by the developer budget.
The user adds databases to the pool, sets the
minimum and maximum eDTUS for each
database, and sets the eDTU limit of the pool
based on their budget.This means that within the
pool, each database is given the ability to auto-
scale in a set range.
30. 30
Managed Instance
• Are your customers
interested in moving to
cloud?
• Want to close your data center
• Current hosting solution is high
maintenance
• You’re asked to do more with less
• Want to expand your reach globally
Managed Instance brings
PaaS closer to you!
??
?
• Do your customer want to
avoid app rewrites but still
benefit from PaaS?
34. 34
Backup
Configuring and performing point in time recovery Azure SQL Database does a full backup every week, a differential
backup each day, and an incremental log backup every five minutes. If you want to extend the default retention period,
you need to configure long-term retention.This feature depends on Azure Recovery Services, and you can extend the
retention time up to 10 years.
SQL Database automatically creates database backups and uses Azure read-access geo-redundant storage (RA-GRS) to
provide geo-redundancy.These backups are created automatically and at no additional charge.
If you delete the Azure SQL server that hosts SQL databases, all elastic pools and databases that belong to the server
are also deleted and cannot be recovered.You cannot restore a deleted server. But if you configured long-term
retention, the backups for the databases with LTR will not be deleted and these databases can be restored.
If your database is encrypted withTDE, the backups are automatically encrypted at rest, including LTR backups
Backup storage up to 100% of the maximum database size is included, beyond which you will be billed in GB/month
consumed.
35. 35
Backup
When you need to recover a database from an automatic backup you can
restore it to:
A new database in the same logical server from a point-in-time within
the retention period.
A database in the same logical server from a deleted database.
A new database from the most recent daily backup to any logical server
in any region.
37. 37
Backup
*If you need faster recovery, use active geo-replication. If you need to be able to recover data
from a period older than 35 days, use Long-term retention.
43. 43
Business Continuity
Every Azure SQL Database subscription has built-in redundancy.Three copies of your
data are stored across fault domains in the datacenter to protect against server and
hardware failure.This is built in to the subscription price and is not configurable.
Standard/general purpose model that provides 99.99% of availability but with some
potential performance degradation during maintenance activities.
Premium/business critical model that provides also provides 99.99% availability with
minimal performance impact on your workload even during maintenance activities.
Although high availability is a great feature, it does not protect against a catastrophic
failure of the entire Azure region. For those cases, you need to put in place a disaster
recovery plan. Azure SQL Database provides you with two features that makes it easier
to implement these type of plans: active geo-replication and auto-failover groups.
44. 44
Failover groups and active geo-replication
Active geo-replication has the following benefits:
Database-level disaster recovery goes quickly when you’ve replicated transactions to
databases on different SQL Database servers in the same or different regions.
You can fail over to a different data center in the event of a natural disaster or other
intentionally malicious act.
Online secondary databases are readable, and they can be used as load balancers for
read-only workloads such as reporting.
With automatic asynchronous replication, after an online secondary database has
been seeded, updates to the primary database are automatically copied to the
secondary database.
45. 45
Failover groups and active geo-replication
With active geo-replication you can configure up to four readable
secondary databases in the same or different regions. In case of a region
outage, your application needs to manually failover the database. If you
require that the failover happens automatically performance, then you
need to use auto-failover groups.
Secondary active geo-replication databases are priced at 100 percent of
primary database prices.The cost of geo-replication traffic between the
primary and the online secondary is included in the cost of the online
secondary. Active geo-replication is available for all database tiers.
46. 46
Failover groups and active geo-replication
Before you create an online secondary, the following requirements must be
met:
The secondary database must have the same name as the primary.
They must be on separate servers.
They both must be on the same subscription.
The secondary server cannot be a lower performance tier than the
primary.
51. 51
Elastic scalability
If you reach 80% of your performance metrics, it’s time to consider
increasing your service tier or performance level. If you’re consistently
below 10 percent of the DTU, you might consider decreasing your service
tier or performance level.
we can scale-up.This means that we will add CPU, memory, and better
disk i/o to handle the load. In Azure SQL Database, scaling up is very
simple: we just move the slider bar over to the right or choose a new
pricing tier.This will give us the ability to handle more DTUs.
52. 52
Elastic scalability
In some cases, even the highest performance tiers and performance optimizations might
not handle your workload on successful and cost-effective way. we might even not be able
to scale-up much further. In that cases you have other options to scale your database:
Read scale-out is a feature available in where you are getting one read-only replica of
your data where you can execute demanding read-only queries such as reports. Read-
only replica will handle your read-only workload without affecting resource usage on your
primary database.
Database sharding is a set of techniques that enables you to split your data into several
databases and scale them independently.
53. 53
Read scale-out
Each database in the Premium tier (DTU-based purchasing model)
or in the Business Critical tier (vCore-based purchasing model) is
automatically provisioned with severalAlwaysON replicas to
support the availability SLA.
These replicas are provisioned with the same performance level as
the read-write replica used by the regular database connections.
The Read Scale-Out feature allows you to load balance SQL
Database read-only workloads using the capacity of one of the
read-only replicas instead of sharing the read-write replica.
54. 54
Sharding
We may shard a database because:
It is too large to be stored in a single Azure SQL Database.
It is too much data to backup and restore in a reasonable amount of time.
Our customers require that their data is stored away from other customers
Sharding involves rewriting a significant portion of our applications to
handle multiple databases.
Sharding is easily implemented in AzureTable Storage and Azure Cosmos
DB, but is significantly more difficult in a relational database like Azure SQL
Database.The complexity comes from being transactionally consistent while
having data available and spread throughout several databases.
55. 55
Sharding
Microsoft has released a set of tools called Elastic DatabaseTools that
are compatible with Azure SQL Database.This client library can be used in
your application to create sharded databases.
The main power of the Elastic DatabaseTools is the ability to fan-out
queries across multiple shards without a lot of code changes.
56. 56
Sharding
When you use the Elastic client library, you deal with
shards, which is conceptually equivalent to a database.
This client library helps you with:
Shard map management creates a shard map
database for storing metadata about the mapping of
each tenant with its database, allowing you to register
each database as a shard
Data dependent routing allows you to select the
correct database based on the information that you
provide on the query for accessing the tenant’s data.
Multi-shard queries (MSQ) executes the sameT-SQL
on all shards that participate with the query and returns
the resultant data as the result of a UNION ALL.
57. 57
Azure SQL Data Sync
Synchronize data across multipleAzure SQL databases and
SQL Server instances, in uni-direction or bi-direction.
Keep data up-to-date across all SQL databases Distributed
Applications
Cloud
App
Cloud
App
Cloud
App
On-prem
App
58. 58
Azure SQL Data Sync
SQL Data Sync is a new service for Azure SQL Database. It allows you to bi-directionally
replicate data between two Azure SQL Databases or between an Azure SQL Database and
an on-premise SQL Server.
A Sync Group is a group of databases that you want to synchronize using Azure SQL Data
Sync.
A Sync Schema is the data you want to synchronize.
Sync Direction allows you to synchronize data in either one direction or bi-directionally.
Sync Interval controls how often synchronization occurs.
Finally, a Conflict Resolution Policy determines who wins if data conflicts with one another.
The hub database must always be an Azure SQL Database. A member database can either
be Azure SQL Database or an on-premise SQL Server.
This can be used to populate a read-only version of the database for reporting, but only if
the schema will be 100% consistent.
59. 59
Azure SQL Data Sync
• All SQL databases supported
(SQL Server, SQL IaaS & Azure SQL
Database)
• Zero code required to enable data
synchronization among SQL databases
• Hub-and-Spoke Synchronization
technology
• Both One-way or Bi-
directional synchronization
• Table-level synchronization with
Column Filter
• Minute-level latency
62. 62
Azure SQL Data Sync
Data Sync Active Geo Replication
Pros • Active-active support
• Sync selected tables and
columns
• Sync between on-prem and
Azure SQL Database
• Seconds level latency
• Transactional consistency
• Auto failover with failover
group
• Designed for DR or read-only
scaling
Cons • 5 min or more latency
• No transactional consistency
• Higher performance impact
• Non-Writeable secondaries
• Replicates the entire database
• Secondary must use same
edition
63. 63
Azure SQL Data Sync
Data Sync Transactional Replication
Pros • Active-active support
• Bi-directional between on-
prem and Azure SQL Database
• Lower latency
• Transactional consistency
• Designed for on-prem to
Azure DB replication or
migration
Cons • 5 min or more latency
• No transactional consistency
• Higher performance impact
• On-prem/Azure SQLVM to
Azure SQL Database only
• High maintenance cost
64. 64
Azure SQL Data Sync
Data Sync SSIS
Pros • Easy configuration • Support transformation
• Support more types of
sources and destinations
• Designed for ETL
Cons • Transformation is not
supported
• Domain knowledge required
• Need extra hosted services
(VM or SSIS PaaS)
• Need additional change
tracking technologies
65. 65
SQL Server Stretch Database
SQL Server Stretch Database migrates your cool data securely and
transparently to Azure.
The main advantage of this solution is that your data is always online, and
you not need to change any query or any configuration or code line in
your application to work with SQL Server Stretch Database.
Since you are moving your cool data to the cloud, you reduce your need
for high performance storage for the on-premises database servers.
You can migrate full tables or just parts of online tables by using a filtering
function.
66. 66
SQL Server Stretch Database
Creates a secure connection between the
Source SQL Server andAzure
Provisions remote instance and begins
migration
Apps and Queries continue to run for both
the local database and remote endpoint
Security controls and maintenance remain
local
Available in all versions of SQL Server 2016
SQL
Stretch
Database
SQL
2016 Cold DataHot data
Cold data
On-premises network Azure PaaS
67. 67
SQL Server Stretch Database
Compute billed as DU, storage billed as Standard Disk rates.
71. 71
Migration to Azure SQL Database
Migration with downtime during the migration
*Rather than using DMA, you can also use a BACPAC file.
See Import a BACPAC file to a new Azure SQL Database.
78. S E A M L E S S C LO U D
I N T E G R AT I O N
Easy lift-and-shift, integrate and
distribute
Active Geo-replicas “data CDN” for your edge
deployments
SQL Azure Data Sync v2 synchronize data
across distributed and occasionally connected
applications
Azure SQL Database Managed Instance
facilitates lift and shift migration from on-
premises SQL Server to cloud
Azure Hybrid Benefit for SQL Server
maximizes current on-premises license
investments to facilitate migration
Database Migration Service (DMS)
provides seamless and reliable migration at scale
with minimal downtime
Most consistent data platform
Database Migration
Ser vice (DMS)
Azure SQL Database
Managed Instance
Azure Hybrid Benefit
(AHB) for SQL Ser ver
SQL Ser ver
Managed SSIS in Azure
Azure SQL Database
79. 79
Graph Database
SQL Server 2017 introduces a new graph database feature.
Graph databases are yet another NoSQL solution.
Graph database introduce two new vocabulary words: nodes and relationships.
Nodes are entities in relational database terms. Each node is popularly a noun, like a person, an
event, an employee, a product, or a car. A relationship is similar to a relationship in SQL Server in
that it defines that a connection exists between nouns.
A key difference between a relational storage engine and a graph database storage engine is
that as the number of nodes increase, the performance cost stays the same.
Graph databases are popularly traversed through a domain specific language (DSL) called
Gremlin. In Azure SQL Database, graph-like capabilities are implemented throughT-SQL.
DDL Extensions – create node/edge tables
Query Language Extensions – New built-in: MATCH, to support pattern matching and
traversals
80. 80
What is a Graph?
Attendee Session
attends
• A graph is collection of Nodes and Edges
– Nodes: Entities – for example
customer, supplier, product
– Edges: Relationships that various
entities share with each other
– Properties: Node or Edge attributes
81. 81
Why Graph Databases?
Hierarchical or interconnected
data, entities with multiple
parents.
Analyze interconnected data,
materialize new information
from existing facts. Identify non-
obvious connections
Complex many-to-many
relationships. One relation
flexibly connecting multiple
entities.
A
John
Mary
Alice
Shaun
Jacob
Jerry
Natalie
Bob
leads
manages
leadsleads
82. 82
Our approach – Embrace and Extend
Backed by Research
References
J. Fan, A. Gerald, S. Raj and J. M. Patel,
"The case against specialized graph
analytics engines," in CIDR, Asilomar,
CA, 2015.
A. Jindal, S. Madden, M. Castellanos
and M. Hsu, "Graph analytics using
vertica relational database," in IEEE
BigData, Santa Clara, CA, 2015
Matured Product
40+ years of academic and
industry research.
Highly evolved ecosystem,
including tooling and
community support
Build on-prem, cloud,
Hybrid Solutions
Best of both relational
and graph database on a
single platform
Trusted
Used and trusted by
millions of customers for
enterprise and mission
critical workloads.
83. 83
DDL Extensions
CREATE NODE
CREATE TABLE [dbo].[Attendee](
[Attendee_Id] [uniqueidentifier] PRIMARY KEY,
[Attendee_FName] varchar(100),
[Attendee_LName] varchar(100)
) AS NODE
GO
SELECT TOP 5 * FROM Attendee;
84. 84
DDL Extensions
CREATE TABLE attends (Rating integer) AS EDGE;
CREATE TABLE [from] AS EDGE;
CREATE EDGE
SELECT TOP 5 * FROM [from];
85. 85
Query Language Extensions
• Multi-hop navigation and join-free pattern matching using MATCH
predicate
• ASCII-art syntax to facilitate graph traversal
SELECT
Attendee.Attendee_Name AS ‘AttendeeName’,
Session.Session_ID AS ‘SessionName’
FROM
attends a,
Attendee at,
Session s
WHERE
MATCH (Attendee-(attends)->Session)
AND Session.session_name = 'Graph extensions in Microsoft SQL
Server 2017 and Azure SQL Database'
86. 86
Relational vs. Graph
Graph and relational designs can answer the same questions
But if traversal of relationships define the primary application requirements,
Graph can solve this more intuitively and with less code
87. 87
Graph Database Scenarios
Recommendation Systems
Fraud Detection
Content Management
Bill of Materials, product hierarchy
CRM
88. 88
AutomaticTuning
• One-click to enable
• Prevent and mitigate
performance issues
• No app changes needed
• Tuning actions
Create missing indexes
Drop unused/duplicate indexes
Force last good plan
94. 94
Intelligent Insights
• Continuous monitoring
• Disruptive event detection
• Root cause analysis
• Available as diagnostic log
Azure SQL Analytics solution
Stream to Event Hub
Archive to Storage
Root-cause: Hitting resource limits caused by new ad-hoc query 0X9001RTYU. Impacted query 0X9002FGJR started
timing out. Consider stopping the ad-hoc query or increasing your pricing tier.
Disruptive
event
Queries:
0X9003HA4J OK
0X9002FGJR Regressed query
0X901119GI OK
0X900044RJ OK
100. 100
Query Performance Insight
Query Performance Insight allows you to spend less time troubleshooting database
performance by providing the following:
Deeper insight into your databases resource (DTU) consumption.
The top queries by CPU/Duration/Execution count, which can potentially be tuned
for improved performance.
The ability to drill down into the details of a query, view its text and history of
resource utilization.
Performance tuning annotations that show actions performed by SQL Azure
Database Advisor
*Query Performance Insight requires that Query Store is active
on your database. If Query Store is not running, the portal
prompts you to turn it on.
107. 107
Automated discovery and
classification of sensitive data
Labeling (tagging) sensitive data on
column level with persistency
Audit access to sensitive data
Visibility through dashboards and
reports
Hybrid cloud + on-premises
115. 115
Detects suspicious database activities
Just turn it ON
Detects potential
vulnerabilities and SQL
injection attacks
Detects unusual behavior
activities
Actionable alerts which
recommend how to
investigate & remediate
Azure SQL DatabaseApps
Audit
Log
Threat Detection
(1) Turn on Threat Detection
(3) Real-time actionable alerts
*It costs $15/server/month , first 60 days for free.
(2) Possible threat to
access / breach data
121. 121
Service Endpoint
Restrict Access to the DB
from VMs in a given
VNET/Subnet
Separation of duties between network
admin and DB admin
Simplify management of VIPs and
firewall rules;
Server-level configuration
available for SQL Database, SQL Data
Warehouse
126. 126
Orchestration Key ManagementPrivate Connections Monitoring
AZURE EXPRESSROUTE AZURE DATA FACTORY AZURE KEY VAULT OPERATIONS MANAGEMENT SUITE
AZURE SQL DATA WAREHOUSE
DATA FACTORY
DATA FACTORY
AZURE MACHINE LEARNING & MACHINE LEARNING SERVER
AZURE DATA LAKE STORE AZURE DATA LAKE ANALYTICS COSMOS DB WEB & MOBILE APPS
AZURE STREAM ANALYTICS
Power BI
COGNITIVE SERVICESBOT SERVICE Logic App
AZURE ANALYSIS SERVICES
127. 127
SMP vs. MPP Architecture
VS
Scale-up Scale-out
Symmetric Multi-Processing (SMP) vs. Massively Parallel Processing (MPP)
133. 133
Azure SQL DataWarehouse
Azure SQL DataWarehouse offers two different performance tiers:
Optimized for Elasticity On this performance tier, storage and compute are in
separate architectural layers.This tier is ideal for workloads of heavy peaks of
activity, allowing you to scale the compute and storage tiers separately
depending on your needs.
Optimized for Compute Microsoft provides you with the latest hardware for
this performance tier, using NVMe Solid State Disk cache.This way, most
recently accessed data keeps as close as possible to the CPU.This tier provides
the highest level of scalability, by providing you up to 30,000 compute Data
Warehouse Units (cDWU).
139. 139
How to choose your performance tier
Elasticity Compute
Current status Generally available Preview in fall
Regional availability 33 6 (growing over time)
Entry pricing $1.21 / hour $6.05 / hour (preview rate)
Starting scale point 100 DWUs 1000 cDWUs
Max compute scale 6,000 DWUs 30,000 cDWUs
Max storage 240TB (compressed) Unlimited (columnar)
Use of elasticity Dynamic “burst” scaling Incremental scaling
Min memory per query 6GB 15 GB
Language surface area Same Same
140. 140
Hash-distributed tables
A hash distributed table can deliver the highest
query performance for joins and aggregations on
large tables.
141. 141
Round-robin distributed tables
A round-robin table is the simplest table to create and delivers
fast performance when used as a staging table for loads.
144. 144
Data Migration Recommendations
Data FormatConversion
• Date Format, Field delimiters, escaping, field order, encoding
Compression
• Use Gzip, ORC, Parquet
• 7-Zip utility, .NET/JAVA libraries
Export
• BCP for fast export
• Multiple files per large table, one folder per table
Copy
• AZCopy
• Data Movement Library
Tips
• Incorrect format means migration
needs to be entirely repeated
• Exploit bcp options, hints, parallelism
• Multiple compressed files, Split files
• Parallel import, reliable transfer
• Don’t use multiple files in the same
gziped file
• EfficientCopy
• Parallel, Async, Resumable
• Limit concurrent copies if low
bandwidth
• Very Large Data transfer
• Express Route, Import/Export Service
145. 145
Data Loading Recommendations
PolyBase and SSIS (with 2017 Azure feature pack) the fastest method
• Upload to BLOB viaAZCOPY or PowerShell library
• Historical load – use CTAS
• Incremental – use INSERT…SELECT
Use the highest resource class (without sacrificing concurrency)
Increase DWU during load, decrease when done
PolyBase now supports UTF-16 file types.ADLS as a source and target is also supported
Known Issues:
• Does not support extendedASCII
• Does not support custom multi-date format. E.g. 2000-1-6
• No reject files/reason for rejected rows.
147. 147
Azure SQL DataWarehouse
Target workload: Analytics (OLAP)
Store large volumes of data
Consolidate disparate data into a single location
Shape, model, transform and aggregate data
Perform query analysis across large datasets
Ad-hoc reporting across large data volumes
All using simple SQL constructs
148. 148
Azure SQL DataWarehouse
Unsuitable workloads
Operational workloads (OLTP)
High frequency reads & writes
Large numbers of singleton selects
High volume of single row inserts
Data Preparation
Row by row processing needs
Incompatible formats (JSON, XML)
Sourced from General vNext goals slide “2% of Linux on-premises DB market ~$150M”
http://www.bloomberg.com/news/articles/2016-03-07/microsoft-plans-linux-database-in-bid-to-win-sales-from-oracle
Mark R. Murphy
Satya, regarding the announcement that you will release your SQL Server database on the Linux platform, I was wondering if you can walk us through your decision tree just in terms of what you think the potential risks are and what you think the potential rewards are of reaching for that level of openness, if you will. And just how impactful do you think that, that product can be in enhancing Microsoft's share of the database market?
Satya Nadella
Thanks for the question. So the decision logic was driven primarily by what I'd say the increased competitiveness of SQL Server. If you think about where SQL Server now with this new release, SQL Server 2016, it's become a fantastic database for many, many of the workloads, everything from OLTP to data warehousing to BI to advanced analytics. For the Tier 1, this is a capability that's been multiple decades in the work, but here we are with very competitive total cost of ownership, price competitiveness but with a technology that is, in many cases, as Gartner talks about, at the top of the charts when it comes to all of these workloads. So now that we find yourselves with that capability, we're saying, "Look, what's the way to think about market -- all the markets that we can, in fact, take this product to." And the Linux operating system database market is not something that -- which is mostly primarily a Tier 1 segment, is something that we never worked in. And so, therefore, we look at that as an expansion opportunity so we take that. We've already made the call that Azure Linux's FirstClass. We already have 20-plus points of -- or 20-plus percent of VMs in Azure or Linux and we'll all increasingly have Linux via big share of percentage of what is happening in Azure. So for the first time now, we have the ability to go to an enterprise and talk about that entire data estate across Windows and Linux. People don't really move between operating systems. Those choices have been made. But at the same time, now they have a choice around database. And so we think that, that's a very good incremental opportunity for us.
Next steps: create SQL Server vNext slide once messaging finalized
Current status: messaging workstream with Sydney Davis
Planned pillars: new "platform of choice" pillar to supplement existing pillars
Notes: “Any data” my be overselling; won’t have some capabilities at Public Preview but will at GA
Title: SQL Server - The platform of choice
Any data
Access diverse data, including video, streaming, documents, relational, both external data and data internal to your org
Use Polybase to access Hadoop big data and Azure blog storage with the simplicity of t-SQL
You can use Azure DocumentDB, a NoSQL document database service, for native JSON support and JavaScript built directly inside the database engine
Any application
Leverage the t-SQL skills of your talent base to run advanced analytics through R models, and to access structured and unstructured data
Take advantage of Microsoft–created database connectivity drivers and open-source drivers that enable developers to build any application using the platforms and tools of their choice, including Python, Ruby, and Node.js
Anywhere
Flexible on-premises and cloud
Easily backup to the cloud
You can now migrate a SQL Server workload to Azure SQL DB. The parity is there and the notion that SQL Server doesn’t map to Azure SQL DB is no longer the case
Keep more historical data at your fingertips by dynamically stretching tables to the cloud with Stretch Database.
Choice of platform
Aligns to your operating system environment. Today, SQL Server is on Windows/Windows Server, will also be on Ubuntu Linux, and we are targeting additional platforms, including Red Hat Linux
Benefit from continued integration with Windows Server for industry-leading performance, scale and virtualization on Windows.
Note: Tux penguin image created by Larry Ewing
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
Brand new feature – that we’re announcing is in Public Preview *today*!!
Beginnings we saw in VA - expanding to more comprehensive solution
This is a VITAL element of GDPR/ data privacy story - data discovery + classification –
We help you by automatically discovering sensitive data.
You can label it with classifications – and the metatdata is persisted in the DB!
This enables management, visibility. Audit access. Track sensitive data when it leaves DB boundaries. The persistent label will be identified by external apps to handle accordingly, e.g. encrypt.
Manage the policy ACROSS Azure – for all your data! In ASC! Classification framework integrated with MIP for holistic MS data classification story.
It can serve as infrastructure for:
Helping meet data privacy standards and regulatory compliance requirements.
Various security scenarios, such as monitoring (auditing) and alerting on anomalous access to sensitive data.
Controlling access to and hardening the security of databases containing highly sensitive data.
Data Discovery & Classification introduces a set of advanced services and new SQL capabilities, forming a new SQL Information Protection paradigm aimed at protecting the data, not just the database:
Discovery & recommendations – The classification engine scans your database and identifies columns containing potentially sensitive data. It then provides you an easy way to review and apply the appropriate classification recommendations via the Azure portal.
Labeling – Sensitivity classification labels can be persistently tagged on columns using new classification metadata attributes introduced into the SQL Engine. This metadata can then be utilized for advanced sensitivity-based auditing and protection scenarios.
Query result set sensitivity – The sensitivity of query result set is calculated in real time for auditing purposes.
Visibility - The database classification state can be viewed in a detailed dashboard in the portal. Additionally, you can download a report (in Excel format) to be used for compliance & auditing purposes, as well as other needs.
RON
SQL Vulnerability Assessment is our newest security intelligent feature, which was just released to Public Preview
It provides you visibility into the security state of your and allows you to constantly track and improve it over time
It is a built-in security feature in Azure SQL Database and it is also available using the latest SQL Server Management Studio (for SQL OnPrem or SQL on VM)
2) In short, SQL Vulnerability Assessment runs a set of security checks which
Discover sensitive data which is not protected
Identify security misconfigurations that leave your database vulnerable to attack
In addition, it provides a clear report which is very helpful for security audits.
It can help you:
Meet compliance requirements that require database scan reports.
Meet data privacy standards.
Monitor a dynamic database environment where changes are difficult to track.
s
s
s
RON
The second security intelligent feature that I would it to share with you is SQL Threat Detection
It is also a built-in feature in Azure SQL Database, which detects anomalous database activities indicating unusual and potentially harmful attempts to breach the database
1) It is super simple to enable it using Azure portal or standard API and requires no modifications to your application code
2) It provides you a set of world-class algorithms that learn, profile and detect potential SQL injections and unusual behavior patterns
3) It trigger an immediate email & portal alert upon detection ,which includes clear description and actionable investigation and remediation steps
Vulnerability to SQL Injection: This alert is triggered when an application generates a faulty SQL statement in the database. This may indicate a possible vulnerability to SQL injection attacks. There are two possible reasons for the generation of a faulty statement:
A defect in application code that constructs the faulty SQL statement
Application code or stored procedures don't sanitize user input when constructing the faulty SQL statement, which may be exploited for SQL Injection
Potential SQL injection: This alert is triggered when an active exploit happens against an identified application vulnerability to SQL injection. This means the attacker is trying to inject malicious SQL statements using the vulnerable application code or stored procedures.
Access from unusual location: This alert is triggered when there is a change in the access pattern to SQL server, where someone has logged on to the SQL server from an unusual geographical location. In some cases, the alert detects a legitimate action (a new application or developer maintenance). In other cases, the alert detects a malicious action (former employee, external attacker).
Access from unusual Azure data center: This alert is triggered when there is a change in the access pattern to SQL server, where someone has logged on to the SQL server from an unusual Azure data center that was seen on this server during the recent period. In some cases, the alert detects a legitimate action (your new application in Azure, Power BI, Azure SQL Query Editor). In other cases, the alert detects a malicious action from an Azure resource/service (former employee, external attacker).
Access from unfamiliar principal: This alert is triggered when there is a change in the access pattern to SQL server, where someone has logged on to the SQL server using an unusual principal (SQL user). In some cases, the alert detects a legitimate action (new application, developer maintenance). In other cases, the alert detects a malicious action (former employee, external attacker).
Access from a potentially harmful application: This alert is triggered when a potentially harmful application is used to access the database. In some cases, the alert detects penetration testing in action. In other cases, the alert detects an attack using common attack tools.
Brute force SQL credentials: This alert is triggered when there is an abnormal high number of failed logins with different credentials. In some cases, the alert detects penetration testing in action. In other cases, the alert detects brute force attack.
s
s
s
s
Only one geographic region
Server-level, not database-level
s
s
s
s
Add key for the coluors
De-coupled storage from compute & control
Completely elastic
Pay for the data you store and the compute you provision
De-coupled storage from compute & control
Completely elastic
Pay for the data you store and the compute you provision
Data storage and snapshots
Data storage is charged based on Azure Premium Storage rates of €125.39/1 TB/month (€0.18/1 TB/hour). Data storage includes the size of your data warehouse and 7-days of incremental snapshot storage.
Note—Storage transactions are not billed. You only pay for stored data and not storage transactions.
Geo-redundant disaster recovery
Your data warehouse is copied to geo-redundant storage for disaster recovery. Storage for geo-redundant copies is billed at Azure Standard Disk read-access geo-redundant storageof €0.102/GB/month.
Compute is billed at €930.87/100 DWUs/month, unless the data warehouse is paused. Storage is billed at €125.39/1 TB/month.
You cannot opt out of snapshots, as this capability provides your data warehouse with data loss and corruption protection.
DWU: In essence, DWU is a function of memory, CPU and concurrency. Basic DWU, DW100 can have upto 24GB of RAM with lesser concurrency
1 DWU is approximately 7.5 DTU (Database Throughput Unit, used to express the horse power of an OLTP Azure SQL Database) in capacity although they are not exactly comparable.
To calculate your DTU needs, multiply the 7.5 by the total DWU needed, or multiply 9.0 by the total cDWU needed.