Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?

Editor's Notes

1. Hallo and Welcome to my session about Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
2. My name is Erwin de Kreuk and I’m working as a Lead Data and AI for InSpark a Microsoft Partner in the Netherlands
3. Azure Data Factory is a Hybrid data integration During the session today I will explain how you can use Parameters within Azure DataFactory How you can replace these parameters with MetaData How we can log these dynamic pipelines And a quick walk through of a complete solution with DataBricks and Azure SQL Database as endpoint
4. Hybrid data integration service where you easily extract data from On Prem Sources, Cloud Sources, SaaS application with more then 120 different DataConnectors With visual tools, you can  build, debug, deploy, operationalize and monitor your data pipelines or big data Pipelines Provides an easy way to transform data at scale without any coding required ELT Platform. With Parameters you build a complete dynamically solution and this is what I’m going
5. Passing parameters to ADF or Azure Synapse is quite important as it provides the flexibility required to create dynamic pipelines. To reference a parameter, you must provide the fully qualified name of the parameter. It is worth noting that parameter names are case sensitive. A parameter could be a user input, which means that the parameter is passed from the pipeline layer or could be an input coming from an activity within the pipeline.
6. Global parameters can be used in any pipeline expression. If a pipeline is referencing another resource such as a dataset or data flow, you can pass down the global parameter value via that resource's parameters. Global Parameters are only available in Azure Data Factory and not in Azure Synapse Analytics You can Global Parameters in the Management Hub in ADF You must define the datatype of the Global Parameter Global parameters are referenced as pipeline().globalParameters.<parameterName>.
7. There are two ways to integrate global parameters in your continuous integration and deployment solution: Include global parameters in the ARM template Deploy global parameters via a PowerShell script Or you can enable this box I’m going not that much in detail, but you can find details in the added link
8. Or you can enable this box I will show you later in the Demo how you deploy these parameters to a next environment
9. Create 1 Dataset for all your Activities per Linked Service
10. Create 1 Dataset for all your Activities per Linked Service Explain parameter name Explain add dynamic content.
11. You can define pipeline parameters to pass through values to your dataset example. How this works I will explain in the upcoming demo
12. Use widgets in Databricks notebooks to use these as a Parameter in ADF
13. Make your Linked Service Dynamiccaly, fe if you want to extract data from the same server but from different databases
14. Samenvatting
15. Linked Service DataBase Parameter Create DataBase Parameter Dataset Create Source Parameter SourceDatabase SourceTable SourceSchema Dataset create Datalake Folder FilePath Filename Pipeline Assign Pipeline Parameters
16. Now we have learned to implement a Dynamically set up Pipeline is it time for the next stage How can we fill all the parameters based on metadata
17. Source Name => The name of Source Table without schema name Source Schema The name of Source Schema DataLake Catalog Folder in the DataLake Table Destination Schema Table Destination Table IsActive IsIncremental IsIncremental LastLoadDataTime
18. With the auditing we have created 3 Stored Proc. 1 where we will start the execution 1 where we will end the execution if successful 1 where we will end the execution if not successful
19. With the auditing we have created 3 Stored Proc. 1 where we will start the execution 1 where we will end the execution if successful 1 where we will end the execution if not successful
20. Show Source Parameters table Show [execution].[Pipeline_DataLake_Files] Show Execution pipeline Show Command Pipeline Get Files For Each Pipeline Execution Run Pipeline DEMO 3 Show SP On Begin, On End, On Error Explain Parent Log Id Show Execution Show Table
21. Show Source Parameters table Show [execution].[Pipeline_DataLake_Files] Show Execution pipeline Show Command Pipeline Get Files For Each Pipeline Execution Run Pipeline DEMO 3 Show SP On Begin, On End, On Error Explain Parent Log Id Show Execution Show Table