Create Deployment Pipelines

Define the pipeline structure

A deployment pipeline moves content between environments in a controlled way. The deployment process lets you clone content from one stage in the deployment pipeline to another, typically from development to test, and from test to production. Each stage holds one workspace, so creating a pipeline means deciding the stages first and filling them afterwards.

You can create a pipeline from the deployment pipelines entry point in Fabric, or from a specific workspace. If you create a pipeline from a workspace, the workspace is automatically assigned to the pipeline.

When you create a pipeline, you define how many stages it should have and what they should be called. By default, the pipeline has three stages named Development, Test, and Production. You can accept these default stages or change the number of stages and their names. A pipeline can have anywhere between 2 and 10 stages.

The structure is a one-time decision. In the creation dialog you can add another stage, delete stages, or rename them. After that, the number of stages and their names are permanent and can't be changed. So agree on the list of stages, and on what each one is called, before you select Create.

A team that later wants one more stage, for example a pre-production stage between Test and Production, cannot add it to the existing pipeline.

Share the pipeline and choose public stages

Access to a pipeline and access to the workspaces inside it are two different things. Pipeline permissions and workspace permissions are granted and managed separately.

Pipelines only have one permission, Admin, which is required for sharing, editing, and deleting a pipeline. When you share a pipeline with others, they receive access to the pipeline and become pipeline admins. Pipeline access doesn't grant permissions to view or take actions on the workspace content. Microsoft 365 groups aren't supported as pipeline admins.

A public stage controls what content consumers see. By default, the final stage of the pipeline is made public. A consumer of a public stage without access to the pipeline sees it as a regular workspace, without the stage name and the deployment pipeline icon next to the workspace name.

You can have as many public stages as you want, or none at all. To change the public status of a stage at any time, go to the pipeline stage settings and check or clear the Make this stage public box. This is the one part of the structure that stays editable after the pipeline is created.

Assign a workspace to a stage

A pipeline is empty until its stages have workspaces. Adding content to the pipeline is done by assigning a workspace to a pipeline stage, and you can assign a workspace to any stage. This is how a team that already has separate workspaces for each environment brings them under one pipeline.

To assign a workspace, the pipeline stage has to be vacant. To assign a workspace to a stage that already has another workspace, unassign the current workspace from that stage and then assign the new one.

Only workspaces that can be assigned to a deployment pipeline appear in the dropdown list. A workspace qualifies when all of these conditions apply:

• You're an admin of the workspace.
• The workspace isn't assigned to any other deployment pipeline.
• The workspace resides on a Fabric capacity.
• You have at least workspace contributor permissions for the workspaces in its adjacent stages.

The permission to remember has two halves. Assigning a workspace to a stage requires pipeline admin and workspace admin of the workspace to be assigned.

Unassigning has a cost. Whenever you unassign a workspace, you lose all of the stage's deployment history and its configured rules. Unassigning a workspace requires one of two roles: pipeline admin, or workspace admin through the Unassign Workspace API.

If you try to delete a workspace from its workspace settings while it is assigned to a deployment pipeline, the deletion fails. To remove such a workspace, unassign it from the pipeline first and then remove it.

Understand item pairing

Pairing decides what a deployment overwrites, so it explains most surprises in a pipeline. Pairing is the process by which an item in one stage of the deployment pipeline is associated with the same item in the adjacent stage. When you deploy content from one stage to another, the items being copied from the source stage overwrite the paired item in the target stage. If items aren't paired, even if they have the same name and type, the item in the target stage isn't overwritten during a deploy.

Pairing happens in one of two ways.

1. At assignment. When a workspace is assigned to a deployment stage, the deployment pipeline attempts to pair items. The pairing criteria are item name, item type, and folder location, and the folder is used as a tie breaker when a stage contains two or more items with the same name and type.
2. At deployment. When an unpaired item is copied from one stage to another using the Deploy button, a copy of the item is created in the next stage and paired with the item being deployed. A deploy of an unpaired item is known as a clean deploy.

Two rules describe how pairing behaves over time.

Pairing survives a rename. Once items are paired, renaming them doesn't unpair the items, so there can be paired items with different names. A later deployment therefore still goes to the renamed item, because it follows the pairing and not the name. On subsequent deployments, each deployed item overrides the metadata of its paired item, including its name if it was changed.

Pairing is not created later by hand. Adding a new item to a workspace that's part of a pipeline doesn't automatically pair it to an identical item in an adjacent stage. There's no way to manually pair items except by following the pairing rules.

Paired items appear on the same line in the pipeline content list, and items that aren't paired appear on a line by themselves. Read that list before a deployment: it tells you what will be overwritten.

Handle duplicate names

Duplicate names are where assignment goes wrong, so it helps to know the exact outcomes.

If a single item in each stage has the same name and type, then pairing occurs. If there's more than one item in a stage that has the same name and type, then items are paired if they're in the same folder.

Take a report named Sales that sits in Folder A in Development, and a Test workspace that holds three reports named Sales.

The folder breaks the tie: only the copy in Folder A is paired, and the other two stay as unpaired items.

The tie breaker cannot help when the folder is the same too. If two or more items in the workspace to be paired have the same name, type and folder, pairing fails. The fix is to move one item to a different folder or change its name, so that there are no longer two items that match an existing item in the other stage.

Unpaired items with the same name also appear after assignment, when someone creates an item directly in a later stage. If items aren't paired, even if they have the same name, type, and folder, they don't overwrite on a deployment: a duplicate copy is created and paired with the item in the previous stage. For Power BI items the general limitations are stricter: when you deploy one for the first time and another item in the target stage has the same name and type, the deployment fails. In both descriptions the existing unpaired item is never overwritten, so create items in the first stage and let deployments carry them forward.

Deploy to an empty stage

You do not need a workspace for every stage in advance. Every time you deploy to a vacant stage, a new workspace is created.

The first time you deploy content, deployment pipelines checks if you have permissions. If you have permissions, the content of the workspace is copied to the stage you're deploying to, and a new workspace for that stage is created on the capacity. If you don't have permissions, the workspace is created but the content isn't copied.

Deploying to an empty stage requires pipeline admin and source workspace contributor.

Ownership. The deploying user automatically becomes the owner of the cloned semantic models, and the only admin of the new workspace. Permissions aren't copied during deployment, for a workspace or for a specific item.

Data. After the deployment is complete, refresh the semantic models so that you can use the newly copied content. The semantic model refresh is required because data isn't copied from one stage to another.

Choose full, selective, or backward deployment

Deployment pipelines offer three options for deploying your content.

• Full deployment deploys all your content to the target stage.
• Selective deployment lets you select which content to deploy to the target stage.
• Backward deployment deploys content from a later stage to an earlier stage in the pipeline.

You can deploy content to any adjacent stage, in either direction. The backward direction has a condition. Currently, backward deployment is only possible when the target stage is empty, which means that it has no workspace assigned to it. Backward deployment is also only possible when deploying all the items, so you can't selectively deploy items backwards.

That condition fits one common start. To create a pipeline from a production workspace, assign the workspace to the production stage and use backward deployment to deploy to the test stage, and then to the development stage.

Deploying into a stage that already has content follows the pairing. Once you have content in a pipeline stage, you can deploy it to the next stage even if the next stage workspace has content, and paired items are overwritten. New content is deployed as an addition to the content already there, and updated content replaces some of it. Items that don't exist in the source stage remain as is.

To deploy content to another stage, at least one item must be selected. After you choose how to deploy your content, you can review your deployment and leave a note. You can review the deployment history to see the last time content was deployed to each stage.

Know what a deployment copies

A deployment copies definitions, not everything you see in the workspace. Knowing the two lists tells you what the target stage looks like afterwards.

During deployment, these item properties are copied and overwrite the item properties at the target stage: data sources, parameters, report visuals, report pages, dashboard tiles, model metadata, and item relationships. The overwrite operation only overwrites the content of the item: the item's ID, URL, and permissions remain unchanged.

These item properties aren't copied during deployment: data, URL, ID, permissions, workspace settings, app content and settings, and personal bookmarks. Data isn't copied, only metadata is copied.

These semantic model properties are also not copied: role assignment, refresh schedule, data source credentials, query caching settings, and endorsement settings. In the target stage, item properties that aren't copied remain as they were before deployment.

A quick way to remember it: the definition of the item travels, while anything that belongs to the environment stays.

Finish the first deployment by hand

Because the environment settings do not travel, a first deployment leaves manual work in the target stage. The refresh of the semantic models is the first item. The gateway is the second.

After the initial deployment, the gateway associated with the target item is not automatically mapped to the corresponding data source. You need to configure this mapping manually through the target item's settings page, and then verify that the data refresh completes successfully. Subsequent deployments do not alter or reset this gateway configuration.

Apps are separate as well. The deployment process doesn't include updating the app content or settings, so you update the app manually in the required pipeline stage.

Later deployments are gentler with data. Data in the target item, such as a semantic model or dataflow, is kept when possible. If there are no changes to an item that holds the data, the data is kept as it was before the deployment.

With a small change such as adding or removing a table, Fabric in many cases keeps the original data. For breaking schema changes, or changes in the data source connection, a full refresh is required.

Check the permissions for each action

Plan access before the first release, because each action asks for a different combination. The lowest deployment pipeline permission is pipeline admin, and it's required for all deployment pipeline operations. A pipeline admin that isn't assigned a workspace role can view the pipeline and share it with others, but can't view the content of the workspace and can't perform deployments.

In words: To deploy items to the next stage, you must be a pipeline admin and a workspace contributor to both the source and target stages. Contributor is the minimum, so the member and admin roles also qualify. To deploy dataflows, you must also be the owner of the deployed item.

Comparing two stages requires pipeline admin and workspace contributor, member, or admin for both stages. Viewing deployment history requires only pipeline admin. Creating a pipeline requires a user with a Pro, Premium Per User, or Premium license.

Setting deployment rules has its own requirements, which belong to the configuration of a pipeline.

Respect the limits

A few hard limits decide whether a deployment can run at all.

• The maximum number of items that can be deployed in a single deployment is 300.
• The deployment fails if any of the items have circular or self dependencies, for example when item A references item B and item B references item A.
• Any item that isn't on the list of supported items isn't supported.
• The workspace must reside on a Fabric capacity.

The supported items include the lakehouse, notebook, environment, pipeline, Dataflow Gen2, copy job, warehouse, eventhouse, eventstream, KQL database, SQL database, and variable library. Some supported items are in preview, including the Power BI report and semantic model.

A workspace with more than 300 items therefore needs more than one deployment, and selective deployment is the tool for splitting it.

Key takeaways

• Structure: a pipeline has between 2 and 10 stages. Three stages named Development, Test, and Production are only the default.
• Assignment: assigning a workspace needs pipeline admin plus admin of that workspace. Unassigning loses the stage's deployment history and its configured rules.
• Pairing: at assignment, items are paired by name and type, with the folder as a tie breaker. A rename does not unpair items.
• Empty stage: the deployment creates the workspace, the deploying user is its only admin, and the semantic models need a refresh because data is not copied.
• Backward deployment: it is possible only when the target stage has no workspace, which is how you build a pipeline from an existing production workspace.
• Not copied: data, permissions, and the refresh schedule of a semantic model. Map the gateway by hand after the first deployment.
• Deploying to a stage with a workspace: it needs at least contributor on both the source and the target workspace.