Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Data Management Gateway - Deep Dive

•
•
Jean-Pierre Riehl
Jean-Pierre Riehl

--session donnée lors du SQL Saturday Lisbon 2015-- Data Management Gateway (and also AS Connector) is what make modern Microsoft BI stack hybrid. Power BI and Azure Data Factory use that component to interact with On-Prem Data assets. That session is a Deep dive into the DMG and the hybrid architecture involved by Power BI and ADF. How does it work ? Security, Firewall, Certificates, Multiple gateways, Admin delegation, Scale out, Disaster Recovery…. All that topics will be covered during that technical session.

Related slideshows

Unit 1
Unit 1Unit 1
Unit 1
 
Azure Fundamentals Part 2
Azure Fundamentals Part 2Azure Fundamentals Part 2
Azure Fundamentals Part 2
 
What are the Business Benefits of Microsoft Azure
What are the Business Benefits of Microsoft AzureWhat are the Business Benefits of Microsoft Azure
What are the Business Benefits of Microsoft Azure
 
1 of 40
Data Management Gateway Deep Dive
Our Main Sponsors:
Jean-Pierre Riehl Practice Manager Data & BI @djeepy1 - http://blog.djeepy1.net Who am I ? Pure-Player Microsoft  Self-Service BI experts  « Agile BI » projects  Data Strategy  Hybrid-BI MVP SQL Server French PASS Chapter Leader
An hybrid story…
Microsoft Cloud-First strategy…
But your data remains On-Prem…
Here it comes DMG
DMG-aware tools Power BI for O365 PowerBI.com Azure Data Factory
Power BI Excel Power Query Power Pivot Power View Power Map Power BI Tenant SQL Cloud On-Prem Oracle … Power BI Admin Center Data Catalog Q&A Power BI Sites Data Refresh Indexing / Search OData Access
PowerBI.com Live Power BI reports & dashboards Live Query Scheduled Refresh Cloud On-premises
Azure Data Factory
Hybrid features  Data Refresh  Data Movement  Live Query  Indexing Sources  OData Access (Data Virtualization)
One DMG to rule them all DMZ SQL Cloud On-Prem Oracle … Power BI for O365 Data Management Gateway OnPrem Data Sources PowerBI.com Azure Data Factory Excel
DMG DETAILS
Versions of DMG  There are multiple versions of DMG Data Management Gateway Power BI ADF 1.4.5549.1 24/3/2015 Analysis Services Connector Power BI v2 1.4.5576.2 17/4/2015
What’s that  A Windows Service  DIAHostService  A configuration application  Managed by the cloud  Power BI Admin Center  Azure Portal (new version)
Data Management Gateway - Deep Dive
FIRST LOOK AT DMG
Data Sources Supported Source Authentification SQL Server / Oracle Sybase, Teradata, DB2, MySQL Windows, Basic Files (CSV, XML, Text, Excel, Access) Folders Windows SharePoint List (Online) SharePoint List (On-prem) Anonymous Anonymous, Windows OData Feed Anonymous, Windows, Basic Key (Data Market) Azure Marketplace Key (Data Market) Azure HDInsight Azure Blob Storage Azure Table Storage Key (Azure Account) SQL Azure PostgreSQL Basic Teradata, Sybase, DB2, MySQL Windows, Basic SSAS Tabular Windows +SSIS Data Feed
Data Sources supported  Details on data sources supported  https://support.office.com/en-us/article/Supported-Data-Sources- and-Data-Types  Prequisites  Beware of types  Drivers installed  https://support.office.com/en-us/article/Data-source-prerequisites- Power-Query
HOW DOES IT WORK
Cloud On-Prem Data Refresh explained Power BI OnPrem Sources DMG Azure Service Bus 0 Register DMG Register Data Sources 1 Ask for Data Refresh Check permissions Find Gateway 4 Execute Query 5 Get Data 6 Compress, Chunk Queue request2 3 Read request Gateway Cloud Service 7 Update Model
Cloud On-Prem OData access explained Power BI OnPrem Sources DMG 0 Index Sources 2 Resolve Data Source Redirect to local Gateway 4 Execute Query 3 Request Data 1 Search Source Excel Power Query Hybrid Proxy
DMG IN ACTION
Data Source Discovery  Data Sources are declared on « Admin Center »  They are associated to a Gateway  Gateway Cloud Service resolves Data Source and find right Gateway  Resolution = Provider + Server + Catalog  Warning: Case sensitive
DATA SOURCE DISCOVERY
SECURITY
Excel Power Query Power BI DMG Data sources • SQL (1433) • FileShare • Etc. +DNS resolution Azure Service Bus (outbound) • 9350-9354 Power BI Admin Center (outbound) • 80/443 Direct HTTP (inbound) • HTTPS (8050) • HTTP (8051) HTTPS (443) Cloud On-Prem Network Security Credential Manager
Identity Management 2 flavors :  Impersonation by DMG  Impersonation by Data Source How is query authentified on Data Source ?
Impersonation by DMG  Connection identity is impersonated by DMG  Peer-to-peer connection  Credentials are stored by DMG for each source  Credentials can be stored encrypted locally  Credentials can be stored encrypted on Cloud  Credentials are set up locally  Credential Manager is a Click Once application  Credential Manager discuss with DMG
CREDENTIAL MANAGER
Impersonation by Data Source  Used by Live Query  Keep identity of original user  Use the EffectiveUsername method  DMG authenticates to SSAS  DMG passes EffectiveUsername property  SSAS executes the query as if it was original user  DMG must be SSAS Admin
EffectiveUsername
HIGH AVAILABILITY & DISASTER RECOVERY
As easy as scale-out DMGs Cloud On-Prem Power BI … Instance 1 Instance 2 Server A Server B
HA & DR  Gateways and Data sources are managed by Cloud  Credentials are encrypted by MDG  Use of Certificate, self-generated or yours  Need Certificate when registering a new instance  Warning when stored localy  Load Balancing
HA & DR
Going further  Express Route  Mashup Container (aka Power Query)  Monitoring & Logging
Resources  Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help- 5e391ecb-500c-47a3-bd0f-a6173b541044  Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx  Supported Data sources : https://support.office.com/en-us/article/Supported-Data-Sources-and-Data- Types  DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query  SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931  How does DMG work - sequence diagram :  https://support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway- 078D306D-2D28-409C-962B-D80B69A1D0B2  Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653  Store encrypted credentials in the cloud for business continuity : https://support.office.com/en- us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886- da51b8cadc5b  Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint- Conference/2014/SPC397  Mini Nay & Luming Han Session at PASS Summit : http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231
Obrigado! Thank you! Merci ! @djeepy1 - http://blog.djeepy1.net Any questions ?

More Related Content

Data Management Gateway - Deep Dive

  • 3. Jean-Pierre Riehl Practice Manager Data & BI @djeepy1 - http://blog.djeepy1.net Who am I ? Pure-Player Microsoft  Self-Service BI experts  « Agile BI » projects  Data Strategy  Hybrid-BI MVP SQL Server French PASS Chapter Leader
  • 6. But your data remains On-Prem…
  • 8. DMG-aware tools Power BI for O365 PowerBI.com Azure Data Factory
  • 9. Power BI Excel Power Query Power Pivot Power View Power Map Power BI Tenant SQL Cloud On-Prem Oracle … Power BI Admin Center Data Catalog Q&A Power BI Sites Data Refresh Indexing / Search OData Access
  • 10. PowerBI.com Live Power BI reports & dashboards Live Query Scheduled Refresh Cloud On-premises
  • 12. Hybrid features  Data Refresh  Data Movement  Live Query  Indexing Sources  OData Access (Data Virtualization)
  • 13. One DMG to rule them all DMZ SQL Cloud On-Prem Oracle … Power BI for O365 Data Management Gateway OnPrem Data Sources PowerBI.com Azure Data Factory Excel
  • 15. Versions of DMG  There are multiple versions of DMG Data Management Gateway Power BI ADF 1.4.5549.1 24/3/2015 Analysis Services Connector Power BI v2 1.4.5576.2 17/4/2015
  • 16. What’s that  A Windows Service  DIAHostService  A configuration application  Managed by the cloud  Power BI Admin Center  Azure Portal (new version)
  • 19. Data Sources Supported Source Authentification SQL Server / Oracle Sybase, Teradata, DB2, MySQL Windows, Basic Files (CSV, XML, Text, Excel, Access) Folders Windows SharePoint List (Online) SharePoint List (On-prem) Anonymous Anonymous, Windows OData Feed Anonymous, Windows, Basic Key (Data Market) Azure Marketplace Key (Data Market) Azure HDInsight Azure Blob Storage Azure Table Storage Key (Azure Account) SQL Azure PostgreSQL Basic Teradata, Sybase, DB2, MySQL Windows, Basic SSAS Tabular Windows +SSIS Data Feed
  • 20. Data Sources supported  Details on data sources supported  https://support.office.com/en-us/article/Supported-Data-Sources- and-Data-Types  Prequisites  Beware of types  Drivers installed  https://support.office.com/en-us/article/Data-source-prerequisites- Power-Query
  • 21. HOW DOES IT WORK
  • 22. Cloud On-Prem Data Refresh explained Power BI OnPrem Sources DMG Azure Service Bus 0 Register DMG Register Data Sources 1 Ask for Data Refresh Check permissions Find Gateway 4 Execute Query 5 Get Data 6 Compress, Chunk Queue request2 3 Read request Gateway Cloud Service 7 Update Model
  • 23. Cloud On-Prem OData access explained Power BI OnPrem Sources DMG 0 Index Sources 2 Resolve Data Source Redirect to local Gateway 4 Execute Query 3 Request Data 1 Search Source Excel Power Query Hybrid Proxy
  • 25. Data Source Discovery  Data Sources are declared on « Admin Center »  They are associated to a Gateway  Gateway Cloud Service resolves Data Source and find right Gateway  Resolution = Provider + Server + Catalog  Warning: Case sensitive
  • 28. Excel Power Query Power BI DMG Data sources • SQL (1433) • FileShare • Etc. +DNS resolution Azure Service Bus (outbound) • 9350-9354 Power BI Admin Center (outbound) • 80/443 Direct HTTP (inbound) • HTTPS (8050) • HTTP (8051) HTTPS (443) Cloud On-Prem Network Security Credential Manager
  • 29. Identity Management 2 flavors :  Impersonation by DMG  Impersonation by Data Source How is query authentified on Data Source ?
  • 30. Impersonation by DMG  Connection identity is impersonated by DMG  Peer-to-peer connection  Credentials are stored by DMG for each source  Credentials can be stored encrypted locally  Credentials can be stored encrypted on Cloud  Credentials are set up locally  Credential Manager is a Click Once application  Credential Manager discuss with DMG
  • 32. Impersonation by Data Source  Used by Live Query  Keep identity of original user  Use the EffectiveUsername method  DMG authenticates to SSAS  DMG passes EffectiveUsername property  SSAS executes the query as if it was original user  DMG must be SSAS Admin
  • 35. As easy as scale-out DMGs Cloud On-Prem Power BI … Instance 1 Instance 2 Server A Server B
  • 36. HA & DR  Gateways and Data sources are managed by Cloud  Credentials are encrypted by MDG  Use of Certificate, self-generated or yours  Need Certificate when registering a new instance  Warning when stored localy  Load Balancing
  • 38. Going further  Express Route  Mashup Container (aka Power Query)  Monitoring & Logging
  • 39. Resources  Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help- 5e391ecb-500c-47a3-bd0f-a6173b541044  Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx  Supported Data sources : https://support.office.com/en-us/article/Supported-Data-Sources-and-Data- Types  DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query  SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931  How does DMG work - sequence diagram :  https://support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway- 078D306D-2D28-409C-962B-D80B69A1D0B2  Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653  Store encrypted credentials in the cloud for business continuity : https://support.office.com/en- us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886- da51b8cadc5b  Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint- Conference/2014/SPC397  Mini Nay & Luming Han Session at PASS Summit : http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231
  • 40. Obrigado! Thank you! Merci ! @djeepy1 - http://blog.djeepy1.net Any questions ?

Editor's Notes

  1. -Cloud First Strategy, just listen to Satya -Innovation is in the cloud -Examples : PBI, AzureML, HDInsigth, etc. -If you want new technologies you probably will have to use a cloud-based one -But…
  2. -you can put all your data assets in the cloud, even, -that will take a long time -or it won’t never happen (mindset, investment, legal constraints, privacy…) -So what -Microsoft knows about that -They build hybrid cloud, hybrid solutions -They make investments in a technology to remove barriers
  3. -Data Management Gateway -it is the link between Cloud technologies and On-Prem Data
  4. -first, data refresh is trivial. Models are in the cloud, Data is on-prem. -second, indexing data sources into data catalog -third, relating Data Catalog, agnostic access to Data
  5. -
  6. Initiated by Cloud Otherwise, there is no challenge, you can push data from On-Prem (SSIS, Power BI API, etc.)
  7. ADF & Power BI merged in march
  8. Data sources : https://support.office.com/en-us/article/Supported-Data-Sources-and-Data-Types Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931
  9. https://support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway-078D306D-2D28-409C-962B-D80B69A1D0B2
  10. Azure IP adresses (for Azure Service Bus) : http://www.microsoft.com/en-us/download/details.aspx?id=41653