Essentials On Azure DevOps Services and GitHub Book 6
Essentials On Azure DevOps Services and GitHub Book 6
Essentials On Azure DevOps Services and GitHub Book 6
and GitHub
Conditions and Terms of Use
Microsoft Confidential
http://www.microsoft.com/en-us/legal/intellectualproperty/Permissions/default.aspx
Module 4: Azure Pipelines
Module Overview
Microsoft Confidential
Overview
• Introduction to Azure Pipelines
• Azure Pipelines Agents
• Azure Pipelines - YAML
• Build Pipelines (classic)
• Release Pipelines (classic)
• Library
Microsoft Confidential
Module 4: Azure Pipelines
Microsoft Confidential
Overview
• What is Azure Pipelines?
• Continuous Integration (CI) and Continuous Delivery (CD)
• Using Azure Pipelines
Microsoft Confidential
What is Azure Pipelines?
• Cloud service that you can use to automatically build and test your code project and make it available to
other users
• Works with just about any language or project type
• Fully featured continuous integration (CI) and continuous delivery (CD) service
• Helps to ensure consistent and quality code that's readily available to users
• Start with your code on GitHub, GitLab, Bitbucket, or Azure Repos. Then you can automate the build, test,
and deployment of your code to Microsoft Azure, Google Cloud, Amazon cloud services or on-prem
Microsoft Confidential
Development Struggles
Microsoft Confidential
Continuous Integration (CI) and Continuous Delivery (CD)
Microsoft Confidential
Use Azure Pipelines
• You can use either YAML or the visual designer to define your pipelines.
• When you use YAML, you define your pipeline mostly in code (a YAML file) alongside the rest of the code
for your app.
Microsoft Confidential
Use Azure Pipelines – Visual Designer
• Configure Azure Pipelines to use your Git repo.
• Use the Azure Pipelines visual designer to create and configure your build and release pipelines.
• Push your code to your version control repository. This action triggers your pipeline and runs tasks such as
building or testing code.
• The build creates an artifact that's used by the rest of your pipeline to run tasks such as deploying to
staging or production.
• Your code is now updated, built, tested, and packaged. It can be deployed to any target.
Microsoft Confidential
Use Azure Pipelines – Visual Designer (continued)
• The visual designer is great for users who are new to the world of continuous integration (CI) and
continuous delivery (CD)
• The visual representation of the pipelines makes it easier to get started
Microsoft Confidential
Use Azure Pipelines - YAML
• Configure Azure Pipelines to use your Git repo.
• Edit your azure-pipelines.yml file to define your build and/or release.
• Push your code to your version control repository. This action kicks off the default trigger to build and
deploy and then monitor the results.
• Your code is now updated, built, tested, and packaged. It can be deployed to any target.
• Unified YAML experience, so you can configure each of your pipelines to do CI, CD, or CI and CD together
Microsoft Confidential
Use Azure Pipelines – YAML (continued)
• The pipeline is versioned with your code and follows the same branching structure
• Validate your changes through code reviews in pull requests and branch build policies
• Every branch you use can modify the build policy by modifying the azure-pipelines.yml file.
• A change to the build process might cause a break or result in an unexpected outcome. Because the
change is in version control with the rest of your codebase, you can more easily identify the issue.
• “Export to YAML" feature available for designer build pipelines. Save your pipeline definition, then find
"Export to YAML" on the ... menu.
Microsoft Confidential
Feature Availability
Microsoft Confidential 15
Feature Availability
Microsoft Confidential 16
Lesson Knowledge Check
1. What is Azure Pipelines?
2. What value does Continuous Integration provide?
3. Name the two options available to define Azure Pipelines.
Microsoft Confidential
Lesson Summary
• In this lesson, you learned about:
o Azure Pipelines
o Continuous Integration and Continuous Delivery
o Using Azure Pipelines
Microsoft Confidential
Module 4: Azure Pipelines
Microsoft Confidential
Overview
• Agents
• Capabilities
• Communication
Microsoft Confidential
Agents
• To build your code or deploy your software you need at least one agent. As you add more code and
people, you'll eventually need more.
• When your build or deployment runs, the system begins one or more jobs. An agent is installable software
that runs one build or deployment job at a time.
Microsoft Confidential 22
Agents (continued)
• Microsoft-hosted
o Simplest way to build and deploy
o Maintenance and upgrades are taken care of for you
o Each time you run a pipeline, you get a fresh virtual machine. The virtual machine is discarded after one use.
• Self-hosted
o Set up and manage on your own to run build and deployment jobs
o Gives you more control to install dependent software needed for your builds and deployments
o You can install the agent on Linux, macOS, or Windows machines. You can also install an agent on a Linux Docker
container.
o Can be configured as a service or an interactive process
Microsoft Confidential
Agent Pools
• An agent pool defines the sharing boundary for all agents in that pool
• Organize agents into agent pools
• Scoped to the Azure DevOps organization; so you can share an agent pool across projects.
• A project agent pool provides access to an organization agent pool. When you create a build or release
pipeline, you specify which pool it uses. Pools are scoped to your project in Azure Pipelines, so you can
only use them across build and release pipelines within a project
• To share an agent pool with multiple projects, in each of those projects, you create a project agent pool
pointing to an organization agent pool
• While multiple pools across projects can use the same organization agent pool, multiple pools within a
project cannot use the same organization agent pool
• Each project agent pool can use only one organization agent pool
Microsoft Confidential 24
Capabilities
• Every agent has a set of capabilities that indicate what it can do
• Capabilities are name-value pairs either automatically discovered by the agent software (system
capabilities), or those that you define (user capabilities)
• The agent software automatically determines various system capabilities:
o Name of the machine, type of operating system, and versions of certain software installed on the machine
o Environment variables defined in the machine
• You can specify certain demands of the agent when you author a build or release pipeline or queue a build
or deployment
o The system sends the job only to agents that have capabilities matching the demands specified in the pipeline
o Allows you to direct builds and deployments to specific agents
• View the system capabilities of an agent, and manage its user capabilities by navigating to Agent pools
and selecting the Capabilities tab for the desired agent
• Bump the priority of a queued job. Users with the "Manage" permission on the pool - typically pool
administrators - will see a new "Run next" button on the job details page. Clicking the button will set the
job to be run as soon as possible.
Microsoft Confidential
Communication
• Agents communicate with Azure Pipelines to determine which job it needs to run, and to report the logs
and job status
• Communication is always initiated by the agent (HTTP/HTTPS)
Microsoft Confidential
Communication
• When you use the agent to deploy artifacts to a set of servers, it must have "line of sight" connectivity to
those servers
• The Microsoft-hosted agent pools, by default, have connectivity to Azure websites and servers running in
Azure
• If your on-premises environments do not have connectivity to a Microsoft-hosted agent pool, you'll need
to manually configure a self-hosted agent on on-premises computer(s). The agents must have connectivity
to the target on-premises environments and access to the internet to connect to Azure Pipelines
Microsoft Confidential
Lesson Knowledge Check
1. What are the two types of agents?
2. How does an agent communicate with Azure Pipelines?
Microsoft Confidential
Demo 1: Install an agent
Microsoft Confidential 30
Lesson Summary
• In this lesson, you learned about:
o Agents
o Capabilities
o Communication
Microsoft Confidential
Module 4: Azure Pipelines
Microsoft Confidential
Overview
• Azure Pipelines - YAML
• YAML Schema Reference
• Customize the pipeline
• Multi-stage pipeline with environments
Microsoft Confidential
Azure Pipelines – YAML
• Pipelines are made of one or more stages describing a CI/CD process.
• Stages are the major divisions in a pipeline. Example: "build this app", "run these tests", and "deploy to pre-
production”
• Stages consist of one or more jobs, which are units of work assignable to a particular machine.
• Both stages and jobs may be arranged into dependency graphs. Example: "run this stage before that one"
• Jobs consist of a linear series of steps. Steps can be tasks, scripts, or references to external templates.
• Pipeline
o Stage A
▪ Job 1
• Step 1.1
• Step 1.2
• ...
▪ Job 2
• Step 2.1
• ...
o Stage B
▪ ...
• Create Azure Pipelines from VSCode
Microsoft Confidential 34
YAML Schema Reference
• stage
o A stage is a collection of related jobs. By default, stages run sequentially, starting only after the stage ahead of
them has completed.
• job
o A job is a collection of steps to be run by an agent or on the server. Jobs can be run conditionally, and they may
depend on earlier jobs.
• steps
o Steps are a linear sequence of operations that make up a job. Each step runs in its own process on an agent and
has access to the pipeline workspace on disk. This means environment variables are not preserved between steps
but filesystem changes are.
• task
o Tasks are the building blocks of a pipeline. There is a catalog of tasks available to choose from.
• variables
o Hardcoded values can be added directly, or variable groups can be referenced. Variables may be specified at the
pipeline, stage, or job level.
• trigger
o A trigger specifies what branches will cause a continuous integration build to run.
Microsoft Confidential 35
YAML Schema Reference (continued)
• pool
o Specifies which pool to use for a job of the pipeline. It also holds information about the job's strategy for running.
• server
o Specifies a server job. Only server tasks such as manual intervention or invoking an Azure Function can be run in a
server job.
• script
o Shortcut for the command line task. It will run a script using cmd.exe on Windows and Bash on other platforms.
• powershell
o Shortcut for the PowerShell task. It will run a script in PowerShell on Windows.
• publish
o Shortcut for the Publish Pipeline Artifact task. It will publish (upload) a file or folder as a pipeline artifact that can
be consumed by other jobs and pipelines.
• download
o Shortcut for the Download Pipeline Artifact task. It will download one or more artifacts associated with the current
run to $(Pipeline.Workspace). It can also be used to disable automatic downloading of artifacts in classic release
and deployment jobs.
Microsoft Confidential 36
Customize the pipeline
• Understand the azure-pipelines.yml file
o A pipeline is defined using a YAML file in your repo. Usually, this file is named azure-pipelines.yml and is located at
the root of your repo.
Microsoft Confidential 37
Customize the pipeline (continued)
• Change the platform to build on
o Build your project on Microsoft-hosted agents that already include SDKs and tools for various development
languages. Or, use self-hosted agents with specific tools that you need.
Microsoft Confidential 38
Customize the pipeline (continued)
• Add steps
o Add additional script or task as steps to your pipeline. A task is a pre-packaged script. Use tasks for building,
testing, publishing, or deploying your app.
Microsoft Confidential 39
Customize the pipeline (continued)
• Build across multiple platforms
o Build and test your project on multiple platforms. One way to do it is with strategy and matrix. Use variables to
conveniently put data into various parts of a pipeline.
Microsoft Confidential 40
Customize the pipeline (continued)
• Build using multiple versions
o Build a project using different versions of that language using a matrix of versions and a variable.
Microsoft Confidential 41
Customize the pipeline (continued)
• Customize CI triggers
o Use a trigger: to specify the events when you want to run the pipeline. YAML pipelines are configured by default
with a CI trigger on your default branch (which is usually master).
Microsoft Confidential 42
Multi-stage pipelines with environments
• Multi-stage pipelines with environments
Microsoft Confidential 43
Multi-stage pipelines with environments (continued)
• CD capabilities for your multi-stage YAML pipelines
o Consume artifacts published by your CI pipeline and enable pipeline completion triggers. In multi-stage YAML
pipelines, pipelines are a resource. In your YAML, you can now refer to another pipeline and also enable CD
triggers.
Microsoft Confidential 44
Multi-stage pipelines with environments (continued)
• Additional control of your deployments
• Approvals and checks
o Checks allow you to manage how this resource is used. Changes made to checks are effective immediately,
applicable to all existing and new pipelines.
Microsoft Confidential 45
Lesson Knowledge Check
1. True/False: Pipelines are made of one or more stages describing only CI process.
2. True/False: Stages consist of one or more jobs.
3. Name the shortcut for the command line task.
Microsoft Confidential
Demo 2: Set up an Azure
Pipeline - YAML
Microsoft Confidential 48
Lesson Summary
• In this lesson, you learned about:
o Azure Pipelines - YAML
o YAML Schema Reference
o Customizing the pipeline
o Multi-stage pipeline with environments
Microsoft Confidential
Module 4: Azure Pipelines
Microsoft Confidential
Overview
• Features
• Build Pipelines (classic)
Microsoft Confidential
Build for Your Platform, Speaking Your Language
• One system for multiple platforms
• Build for most platforms:
• Windows
• iOS
• Java (Ant, Maven, or Gradle)
• Linux
Microsoft Confidential
Features
• Real-time Build Status
o A live console view on the web that displays the real-time status of each task
• Variables
o Common parameter values can be abstracted into variables
o Sensitive values can be marked as secret
• CI Support
o Branch filter
• Template
o Reuse pre-defined steps
• Task-based
o Execute any script
o Extensible
Microsoft Confidential
What is a Build Pipeline?
• Defines the configuration for a concrete application build process
• Key settings:
o Name
o Triggers
o Repository mappings
o Build defaults
o Retention policy
• Process settings:
o Tasks
▪ Building block for defining automation in a build pipeline
▪ A packaged script or procedure that has been abstracted with a set of inputs
▪ Every task has its own arguments
▪ Tasks are run in sequence, one after the other, on an agent
▪ To run the same set of tasks in parallel on multiple agents, or to run some tasks without using an agent, use jobs
Microsoft Confidential
Build Templates
• Use a template with pre-defined steps to simplify the effort
• Or create your build pipeline from scratch
• Or create your own template from an existing build pipeline
Microsoft Confidential
Build Triggers
Multiple triggers per build:
• Manual
• Continuous integration
o Batch changes
o Path filters
• Scheduled
• TFVC Gated check-in
• Pull request validation
o Add as many triggers as you wish
• Build completion
o Trigger a new build when a upstream dependency change
Microsoft Confidential
Repository Settings
• Select the type of source repository that contains your source files and specify where your sources are
• Repository type:
o Azure Repos Git or TFVC
o GitHub
o GitHub Enterprise
o Bitbucket Cloud
o External Git
o Subversion
Microsoft Confidential
Variables
• Variables give you a convenient way to get key bits of data into various parts of your build pipeline
• There is a list of predefined build variables
• Predefined build variables give you information about the build setup and environment configuration
during execution.
o Example: working folder, build definition name, source folder being built.
• These variables are automatically set by the system and are read-only.
• Examples of predefined variables:
o Build.DefinitionName
o Build.SourcesDirectory
o Build.StagingDirectory
Microsoft Confidential
Build Configuration Options
• Build number format
• Badge enabled
• Enable/Pause/Disable
• Build job authorization scope
• Demands
Microsoft Confidential
Retention Policy
• Configure how long builds are to be retained by the system
• Conserve storage and reduce clutter
Microsoft Confidential
History
• List of changes that includes who made the change and when the change occurred
• Compare two different versions
• Revert to a specific version
Microsoft Confidential
Export/Import
• Export build pipeline as a JSON file
• Import and reuse in another Project
Microsoft Confidential
Queue a Build
• Web-browser based
• Settings:
o Agent pool
o Branch
o Commit
o Variables/Demands
Microsoft Confidential
View Logs
• Real-time monitoring and logging
• Click on a task to view task specific log
• Logs are saved with the build after completion
Microsoft Confidential
Lesson Knowledge Check
1. What is a Build Pipeline?
2. Name two types of Build triggers.
3. Where would you store the script that you want to execute during Build?
Microsoft Confidential
Demo 3: Build Pipelines
(classic)
Microsoft Confidential
Lesson Summary
• In this lesson, you learned about:
o Build Features
o Build Pipelines
Microsoft Confidential
Module 4: Azure Pipelines
Microsoft Confidential
Overview
• Release Pipelines (classic)
• How Do Release Pipelines Work?
• Artifacts
• Stages
• Triggers
• Variables
• Approvals and Gates
• Retention Policy
• Deployment Groups
• Releases and Deployments
• Creating a Release
• Tracking a Release
Microsoft Confidential
Release Pipelines (classic)
• A release pipeline defines the end-to-end release pipeline for an application to be deployed across
various stages
• Use Azure Pipelines releases by authoring a release pipeline for your application
• Specify the artifacts that make up the application and the release pipeline
• An artifact is a deployable component of your application. It is typically produced through a Continuous
Integration or a build pipeline. Azure Pipelines releases can deploy artifacts that are produced by a wide
range of artifact sources such as Azure Pipelines build, Jenkins, or Team City
• Define the release pipeline using stages, and restrict deployments into or out of an stage using approvals
• Define the automation in each stage using jobs and tasks
• Use variables to generalize your automation and triggers to control when the deployments should be
kicked off automatically
71
Release Pipelines (classic)
72
How Do Release Pipelines (classic) Work?
Microsoft Confidential
Artifacts
• A release is a collection of artifacts.
• An artifact is a deployable component of your application
• Link the appropriate artifact sources to your release pipeline.
• A release pipeline can be linked to multiple artifact sources.
• You specify which version of the artifact to deploy when a release is created.
• Artifact sources:
o Azure Pipelines
o TFVC, Git, and GitHub
o Jenkins
o Azure Container Registry, Docker, and Kubernetes
o Azure Artifacts (NuGet, Maven, and npm)
o External or on-premises TFS
o TeamCity
o Other sources
Microsoft Confidential
Stages
• A stage is a logical and independent entity that represents where you want to deploy a release
• The deployment in a stage may be to a collection of servers, a cloud, or multiple clouds
• Deploy to a stage independently of other stages in the pipeline
• The deployment pipeline of a release to a stage is defined in terms of jobs and tasks
• The physical deployment of a release to a stage is controlled through
approvals and gates, deployment conditions and triggers, and queuing policies
Microsoft Confidential
Stages - Jobs
• Organize your deployment pipeline into jobs. Every deployment pipeline has at least one job
• A job is a series of tasks that run sequentially on the same target
• At design time in your job, you specify a series of tasks that you want to run on a common target.
• At run time (when the release pipeline is triggered), each job is dispatched as one or more jobs to its target
• In a deployment pipeline, the target can be either an agent, a deployment group, or the server
• When the target is an agent, the tasks are run on the computer that hosts the agent
76
Stages – Tasks
• A task is the building block for defining automation in a stage of a release pipeline
• Packaged script or procedure that has been abstracted with a set of inputs
• All the tasks are run in sequence, one after the other, on an agent
Microsoft Confidential
Stages – Queuing Policies
78
Stages - General
79
Triggers
• Configure when releases should be created (release triggers), and when those releases should be
deployed to stages (stage triggers), in your DevOps CI/CD processes
Microsoft Confidential
Triggers – Release Triggers
• Continuous deployment triggers
o Azure Pipelines creates new releases automatically when it detects new artifacts are available
o At present only available for Team Foundation Build artifacts and Git-based sources such as Team Foundation Git,
GitHub, and other Git repositories
• Scheduled release triggers
o Create and start a release at specific times
81
Triggers – Stage Triggers
Trigger deployment to each stage automatically when a release is created by a continuous deployment
trigger, based on:
• The result of deploying to a previous stage in the pipeline
• Filters based on the artifacts
• A predefined schedule
• A pull request that updates the artifacts
• The pull request trigger conditions settings
• Manually by a user
82
Variables
• Custom variables: Define a more generic deployment pipeline once, and then customize it easily for each
stage
o Share values across all of the definitions in a project by using variable groups
o Share values across all of the stages by using release pipeline variables
o Avoid duplication of values, making it easier to update all occurrences as one operation
o Store sensitive values in a way that they cannot be seen or changed by users of the release pipelines
• Default variables: Use information about the context of the particular release, stage, artifacts, or agent in
which the deployment pipeline is being run
Microsoft Confidential
Approvals and Gates
• Approvals and gates give you additional control over the start and completion of the deployment pipeline
• Configure each stage in a release pipeline with pre- and post-deployment conditions that can include
waiting for users to manually approve or reject deployments, and checking with other automated systems
until specific conditions are verified
• Configure a manual intervention to pause the deployment pipeline and prompt users to carry out manual
tasks, then resume or reject the deployment
84
Approvals and Gates (continued)
85
Retention Policy
• Control how long a release is retained for each pipeline
Microsoft Confidential
Deployment Groups
• A deployment group is a logical set of deployment target machines that have agents installed on each one.
• Deployment groups represent the physical environments; for example, "Dev", "Test", "UAT", and
"Production". In effect, a deployment group is just another grouping of agents, much like an agent pool
• When authoring an Azure Pipelines Release pipeline, you can specify the deployment targets for a job
using a deployment group. This makes it easy to define parallel execution of deployment tasks
• Deployment groups:
o Specify the security context and runtime targets for the agents.
o As you create a deployment group, you add users and give them appropriate permissions to administer, manage,
view, and use the group.
o Let you view live logs for each server as a deployment takes place, and download logs for all servers to track your
deployments down to individual machines.
o Enable you to use machine tags to limit deployment to specific sets of target servers.
87
Releases and Deployments
• A release is the package or container that holds a versioned set of artifacts specified in a release pipeline
in your DevOps CI/CD processes..
• A deployment is the action of running the tasks for one stage, which results in the application artifacts
being deployed, tests being run, and whatever other actions are specified for that stage.
Microsoft Confidential
Creating a Release
Releases (and, in some cases, draft releases) can be created:
• By a continuous deployment trigger that creates a release when a new version of the source build artifacts
is available
• By using the Release command in the UI to create a release manually from the Releases or the Builds
summary
• By sending a command over the network to the REST interface
Microsoft Confidential
Tracking a Release
• The Releases page shows a list of releases
• Click on a specific release to see the release log and details.
o See the status for each stage.
Microsoft Confidential
Lesson Knowledge Check
1. What is a Release Pipeline?
2. What is an Artifact?
3. Name two release triggers
4. How would you create a release?
5. True/False: You can create a release from a build
Microsoft Confidential
Demo 4: Creating a Release
Pipeline – (classic)
Microsoft Confidential
Lesson Summary
• In this lesson, you learned about:
o Release Pipelines (classic)
o How Release Pipelines (classic) Work
o Artifacts
o Stages
o Triggers
o Variables
o Approvals and Gates
o Retention Policy
o Deployment Groups
o Releases and Deployments
o Creating a Release
o Tracking a Release
Microsoft Confidential
Module 4: Azure Pipelines
Lesson 6: Library
Microsoft Confidential
Overview
• Library
• Variable Groups
• Task Groups
• Service Connections
• Secure Files
Microsoft Confidential
Library
• Collection of shared build and release assets for a project
• Assets defined in a library can be used in multiple build and release pipelines of the project
• Can be accessed directly in Azure Pipelines
• Contains two types of assets: variable groups and secure files
• Security:
o Control who can define new items in a library, and who can use an existing item
o Roles are defined for library items, and membership of these roles governs the operations you can perform on
those items
97
Variable Groups
• Use a variable group to store values that you want to make available across multiple build and release
pipelines
• Variable groups are defined and managed in the Library tab of the Pipelines hub
98
Task Groups
• Standardize and centrally manage deployment steps for all your applications
• Allow you to encapsulate a sequence of tasks, already defined in a build or a release pipeline, into a single
reusable task that can be added to a build or release pipeline, just like any other task
• Automatically added to the task catalog, ready to be added to other release and build pipelines
• Stored at the project level, and are not accessible outside the project scope
99
Service Connections
• Connect to external and remote services to execute tasks for a build or deployment by defining service
connections in Azure Pipelines
• Define and manage service connections from the Admin settings of your project:
o https://dev.azure.com/{organization}/{project}/_admin/_services
• Created at project scope. A service connection created in one project is not visible in another project.
100
Secure Files
• Use the Secure Files library to store files such as signing certificates, Apple Provisioning Profiles, Android
Keystore files, and SSH keys on the server without having to commit them to your source repository
• Defined and managed in the Library tab in Azure Pipelines
• The contents of the secure files are encrypted and can only be used during the build or release pipeline by
referencing them from a task
• Available across multiple build and release pipelines in the project based on the security settings
• Secure files follow the library security model
• There's a size limit of 10 MB for each secure file
101
Lesson Knowledge Check
1. Why would you use a variable group?
2. True/False: There's a size limit of 10 MB for each secure file
Microsoft Confidential
Demo 5: Library
Microsoft Confidential
Lesson Summary
• In this lesson, you learned about:
o Library
o Variable Groups
o Task Groups
o Service Connections
o Secure Files
Microsoft Confidential
Module Summary
• In this module, you learned about:
o Azure Pipelines Key Concepts
o Azure Artifacts Agents
o Azure Pipelines - YAML
o Build Pipelines (classic)
o Release Pipelines (classic)
o Library
Microsoft Confidential
Lab: Azure Pipelines
Microsoft Confidential
Microsoft Confidential
Microsoft Confidential