Configure Deployment Pipelines

Pick the right deployment rule

A deployment copies an item as it is in the source stage, but the stages are rarely identical. When you're working in a deployment pipeline, different stages might have different configurations. For example, the development stage might query sample data from the database, while the test and production stages query the entire database.

You can configure data source rules, parameter rules, and default lakehouse rules. A data source rule determines the data sources of an item, a parameter rule determines the values of its parameters, and a default lakehouse rule determines the default lakehouse for a notebook. Which rule you can create depends on the item type.

In words: Dataflows Gen1 and semantic models support data source rules and parameter rules. Paginated reports and mirrored databases support data source rules only. The default lakehouse rule only applies to notebooks.

The table is the full list of item types that you can configure rules for. Only items of a type that you can create rules for are listed when you open the deployment rules of a stage. A report, a lakehouse, and a Dataflow Gen2 are not on that list.

Data source rules have limits that decide whether they are the right tool.

Data source rules only work when you change data sources from the same type. A rule can replace one SQL Server database with another, but it cannot turn a SQL Server source into a SharePoint source.

For dataflows Gen1 and semantic models, data source rules can be defined for a fixed list of data sources, which includes Azure SQL Server, SQL Server, Oracle, and SharePoint. For other data sources, the recommendation is to use parameters to configure the data source. The value of such a parameter can then be set for each stage with a parameter rule.

Create rules in the target stage

A rule belongs to the stage that receives the content. A rule is defined in the stage whose content it changes, under the appropriate item. For example, you can define a rule for a semantic model in the production stage to point to a production database instead of one in the test stage.

Deployment rules can't be created in the development stage.

Two requirements apply to the person. To create a deployment rule, you must be the owner of the item you're creating a rule for. Data source rules are grayed out if you're not the owner of the item, or if the item doesn't contain any data sources. Viewing or setting a rule requires pipeline admin, a contributor, member, or admin role on the target workspace, and ownership of the item.

Timing is the part that surprises people. Deployment rules only take effect the next time you deploy to that stage. Saving a rule changes nothing in the item that is already in the stage. After you create the rules, deploy the items from the source stage to the target stage where the rules were created.

Once a rule is defined, content deployed to that stage inherits the value defined in the rule, and the rule always applies as long as it's unchanged and valid. You set it once, and every later deployment applies it again.

One detail avoids confusion when you review a release. If you create rules and then compare the stages before you deploy, the comparison is done based on the rules that were created, even though they didn't take effect yet.

Keep rules valid

A rule points at something inside the source item, so changes to that item can break it.

If the data source or parameter defined in a rule is changed or removed from the item it points to in the source stage, the rule isn't valid anymore, and deployment fails. When a release starts to fail right after a developer cleaned up parameters or replaced a source, check the rules of the target stage first.

Rules can also disappear. When an item is removed or deleted, its rules are deleted too, and these rules can't be restored. When you unassign and reassign a workspace to reestablish connections, rules for that workspace are lost and have to be configured again.

Some combinations are not supported at all. Parameter rules aren't supported for paginated reports. Data source rules can't be added for semantic models and dataflows Gen1 on data sources that are parametrized.

Let autobinding keep connections

Most connections between items need no rule, because the pipeline rebinds them for you. If there's a connection between two items, deployment pipelines always tries to maintain this connection. For example, a report always depends on the semantic model connected to it. This behavior is called autobinding, and its result depends on where the linked item is.

The linked item exists in the target stage. Deployment pipelines automatically connect the deployed item to the item it depends on in the deployed stage. For example, a paginated report deployed from development to test automatically connects to the semantic model that was previously deployed to the test stage.

The linked item doesn't exist in the target stage. Deployment pipelines fail a deployment if an item has a dependency on another item, and the item providing the data isn't deployed and doesn't reside in the target stage.

The cure for that failure is in the deployment dialog. To avoid failed deployments due to dependent items not being deployed, use the Select related button. Select related automatically selects all the related items that provide dependencies to the items you're about to deploy.

Autobind across pipelines

Large teams often split content over several pipelines, for example one for semantic models and one for reports. Autobinding also works between them. Deployment pipelines automatically binds items that are connected across pipelines, if they're in the same pipeline stage.

Take a report in the development stage of pipeline B that depends on a semantic model in the development stage of pipeline A. You deploy the report to the test stage of pipeline B. If the test stage of pipeline A has a copy of the semantic model, the deployment succeeds, and the report in the test stage of pipeline B is connected to the semantic model in the test stage of pipeline A. If the test stage of pipeline A has no copy of the semantic model, the deployment fails, because the report can't be connected to the semantic model it depends on.

Two conditions decide whether stages in different pipelines correspond.

Position, not name. The phrase same pipeline stage refers to the numeric index of the stage in the deployment pipeline, not the display name. Stages in the same position are the same pipeline stage even when their names differ, and two stages that share a name aren't the same pipeline stage when they're in different positions.

Same number of stages. Each pipeline must have the same number of stages: if pipeline A has 3 stages and pipeline B has 5 stages, autobinding can't succeed.

Avoid autobinding when a connection must stay fixed

Sometimes rebinding is the wrong outcome. For example, you might have one pipeline for developing organizational semantic models and another for creating reports, and want all the reports to always be connected to semantic models in the production stage.

There are three methods you can use to avoid using autobinding.

1. Don't connect the item to corresponding stages: when the items aren't connected in the same stage, deployment pipelines keeps the original connection. For example, a report in the development stage of pipeline B that's connected to a semantic model in the production stage of pipeline A remains connected to that production model when it's deployed to the test stage of pipeline B.
2. Define a parameter rule, an option that isn't available for reports and can only be used with semantic models and dataflows.
3. Connect your reports, dashboards, and tiles to a proxy semantic model or dataflow that isn't connected to a pipeline.

Parameters interact with autobinding too. When a parameter controls the connection, autobinding after deployment doesn't take place. In those cases, rebind the items after the deployment by changing the parameter value, or by using parameter rules. If you're using parameter rules to rebind items, the parameters must be of type Text.

Deploy a lakehouse

A lakehouse holds data, so the first question is what a deployment does to that data. The answer is nothing. For a lakehouse, only metadata is tracked, and deployment operations never overwrite data in tables or files. Tables, Spark views, and folders in the Files section are not tracked and are not overwritten.

If no dependency mapping is configured, a new empty lakehouse with the same name is created in the target workspace, and notebooks and Spark job definitions are remapped to reference the new lakehouse. If the lakehouse dependency is mapped to a different lakehouse, a new empty lakehouse with the same name is still created, but notebook and Spark job definition references point to the mapped lakehouse instead.

SQL analytics endpoints and semantic models are provisioned as part of the lakehouse deployment. The tables do not arrive with it, so the new lakehouse starts without data in its own stage.

Shortcuts are metadata, so they do travel. Shortcut definitions are synced across deployment pipeline stages. What happens to the target of a shortcut depends on its kind.

Shortcuts with external targets, such as ADLS Gen2 and Amazon S3, keep the same targets across all stages after deployment. Shortcuts with internal targets, which are OneLake shortcuts in the same workspace, are automatically remapped across stages. The target tables, folders, and files of a remapped shortcut aren't created automatically, so you must create them in the target workspace after deployment.

If a shortcut needs to point to different locations in different stages, for example an Amazon S3 folder in development and an ADLS Gen2 folder in production, use variables in the shortcut definition. The alternative is to update the shortcut definition manually after deployment.

A deployment overrides the state of shortcuts in the target workspace: all shortcuts in the target lakehouse are updated or deleted based on the source lakehouse, and new shortcuts are created.

Deploy a notebook

A notebook depends on two items: its default lakehouse and its attached environment. Notebook deployment supports auto-binding for the default lakehouse and attached environment when dependent items are in the same workspace. During deployment, Fabric can rebind these dependencies to corresponding items in the target workspace. A notebook that is deployed with its lakehouse therefore reads the lakehouse of its own stage without any rule.

A rule is for the cases where that is not what you want. If you need a specific target-stage default lakehouse, configure a deployment rule to override auto-binding. Deployment rules take priority over auto-binding: if a deployment rule is configured, it overrides the auto-bound lakehouse.

Create notebook deployment rules on the target stage, for example Test or Production, not on Development.

The default lakehouse rule maps the source-stage default lakehouse to the target-stage default lakehouse. It offers three choices.

Same with source lakehouse keeps the same default lakehouse setting as the source stage. N/A removes the default lakehouse setting in the target stage. Other replaces the source-stage default lakehouse with a different lakehouse in the target stage.

With Other you provide the lakehouse ID, the lakehouse name, and the lakehouse workspace ID, and the lakehouse ID is required. You can get the lakehouse ID from the item URL.

Bind Direct Lake models and Dataflow Gen2

Two item types do not follow the general autobinding behavior, and both appear often in data engineering solutions.

Direct Lake semantic models. When you deploy a Direct Lake semantic model, it doesn't automatically bind to items in the target stage. For example, if a lakehouse is a source for a Direct Lake semantic model and they're both deployed to the next stage, the Direct Lake semantic model in the target stage still binds to the lakehouse in the source stage. Use data source rules to bind it to an item in the target stage. Other types of semantic models automatically bind to the paired item in the target stage.

Dataflow Gen2. Autobinding isn't supported for dataflows Gen2. Deployment of a semantic model that uses a Dataflow Gen2 (CI/CD) item as a data source is not supported. If a dataflow is being refreshed during deployment, the deployment fails.

Use a variable library for stage values

Rules cover a short list of item types. A variable library is the general way to give an item a different value in each stage. A variable library is an item within the workspace that contains a list of variables, along with their respective values for each stage of the release pipeline. Variables can be basic types, like string, integer, and boolean, or complex types such as item references. The variables can be referenced by consumer items within the same workspace.

Each variable can have multiple value sets, which are alternative sets of values for different environments such as dev, test, and prod. There's always one, and only one, active value set in a variable library at a time. You can have a different active value set for each stage of a deployment pipeline.

The items that support the variable library include the pipeline, the shortcut for a lakehouse, the notebook, Dataflow Gen2, the copy job, user data functions, and the eventstream.

That gives the working model. The variable library is one of the items that a deployment pipeline can deploy. After a one-time setting of the active value set for each stage, the correct value is automatically used in that stage.

The first deployment needs one check. When a variable library is deployed to an empty stage, the value set named Default is active in the new target workspace. If the target environment requires another value set, set it as active and save the variable library in the target workspace before running dependent items.

A few limits complete the picture.

• A variable library can have up to 1,000 variables and up to 1,000 value sets, as long as the alternative value sets total fewer than 10,000 cells and the item's size doesn't exceed 1 MB.
• You can't delete a value set while it's active: first configure another value set to be active.

Automate deployments

Everything a person does in the pipeline page can also be scripted, which is how a deployment becomes one step of a larger release. You can deploy content programmatically by using the Power BI or Fabric deployment pipelines REST APIs. The Power BI deployment pipelines APIs only apply to Power BI items.

With the APIs you can integrate Fabric into DevOps tools such as Azure DevOps or GitHub Actions, and schedule pipeline deployments to happen automatically at a specific time. In Azure DevOps you can run the calls from PowerShell or use the Power BI automation tools extension, and both work with a service principal or a user.

Automation does not remove limits, and it adds a few of its own.

• Deployment by using APIs is subject to the same limitations as the deployment pipelines user interface.
• A service principal can't configure OAuth credentials, and after it deploys new items it becomes the owner of the deployed paginated reports and semantic models, so a refresh can't be completed.
• Deploying dataflows by using a service principal isn't supported.
• Creating a customized pipeline of 2 to 10 stages is currently supported only through the UI.

Key takeaways

• Rule types: the default lakehouse rule exists for notebooks only. Reports are not on the list of items that can have deployment rules.
• Where and when: rules are created on the target stage, never in the development stage, and they take effect at the next deployment to that stage.
• Autobinding: a deployment fails when an item depends on something that is not in the target stage. Select related adds the dependencies.
• Across pipelines: stages correspond by position, and both pipelines need the same number of stages. To keep a fixed connection, connect the item to a stage that does not correspond.
• Direct Lake: the model stays bound to the lakehouse of the source stage until a data source rule binds it to the target.
• Notebooks: a default lakehouse rule set to Other, with the lakehouse ID, overrides auto-binding.
• Lakehouse: a deployment creates a new empty lakehouse in the target stage, because only metadata travels.
• Shortcuts: external shortcuts keep the same target in every stage. Use variables in the shortcut definition when the target must differ.
• Variable library: one value set is active in each stage, and after a first deployment to an empty stage it is Default.