What Is Data Pipeline Orchestration?

A practical guide to schedules, dependencies, ETL tools and workflow engines.

Published by Polysync · 16 November 2026

Data pipeline orchestration is the automated coordination of the jobs in a data pipeline: it decides when each job starts, what it waits for and what happens when one fails, and it records the outcome of every run. The jobs themselves keep running in the tools that own them, such as Azure Data Factory, Azure Databricks or Power BI.

Orchestration matters when a result depends on several steps: ingesting data, transforming it, then publishing it to a report or another system. A clock can start each step, but it cannot tell whether the previous step finished successfully.

Orchestration is more than scheduling

Scheduling specifies when work may start. Orchestration adds the relationships and decisions around it: wait for a prerequisite, run independent tasks in parallel, hold a step after failure, or run cleanup after completion.

These ideas overlap: a scheduler can launch a workflow, and a workflow engine can include scheduling. But if each job starts at 2:00 because the prior one usually finishes by 1:55, the sequence depends on a timing assumption, not a confirmed dependency.

A Power BI refresh at 6:00 could succeed while the Databricks job that supplies it is still running, leaving yesterday's data in the report without a refresh error. An orchestration layer makes the relationship explicit: start the next task only when its prerequisite reaches the required outcome.

For the difference between a schedule and a job dependency, see workflow orchestration versus job scheduling.

What does a data pipeline orchestrator do?

Depending on the product, orchestration can include:

  • Scheduling the first jobs in a workflow.
  • Waiting on dependencies and their success, failure or completion conditions.
  • Passing a value, such as a row count or file name, from one job to the next.
  • Limiting concurrent work to respect shared capacity or service limits.
  • Retrying eligible failures and recording each run's outcome.

How dependencies form a workflow

A directed acyclic graph (DAG) describes tasks as nodes and their dependencies as directed edges. “Acyclic” means dependencies cannot loop back to a task; a cycle would leave work waiting on itself, so the orchestrator must reject it.

Dependencies are not only a list of steps. They describe which outcomes allow the next step to proceed. Common trigger conditions include:

  • On Success: proceed only after the upstream task succeeds. This is the usual condition for a transformation that requires newly loaded data.
  • On Failure: run a notification or an explicitly designed recovery step after an upstream failure.
  • On Completion: run cleanup or bookkeeping after the upstream task has finished, whether it succeeded or failed.

A graph can have branches that run independently and later meet at a task that needs both. The orchestrator tracks graph state rather than relying on someone to infer progress from separate consoles. Tools differ on edge cases, such as whether a cancelled upstream run releases downstream work, so check how yours handles them. See the scheduling and reliability guide.

How is data pipeline orchestration different from an ETL tool?

ETL (or ELT, when data is loaded before it is transformed) describes a data-processing pattern: extract data from a source, transform it, and load it to a destination. An ETL tool provides capabilities for one or more of those steps. Data pipeline orchestration coordinates those jobs with other work and decides when they run, what each waits for and what happens when one fails.

The boundary is not always a product boundary. A data integration platform may include a pipeline designer, transformations, schedules and dependency handling in one place. In that case, the same platform is doing both data processing and orchestration. In another architecture, the extraction might run in one service, a notebook in another, and a report refresh in a third. The orchestration question is how those separately owned jobs fit together.

ETL orchestration is therefore one form of data pipeline orchestration: it coordinates ETL jobs and their prerequisites. The broader term also covers pipelines that include steps such as model scoring, file preparation or report refreshes, whether or not each step is conventionally called ETL. For the narrower terminology and examples, see What is ETL orchestration?

How is an orchestrator different from a workflow engine?

“Workflow engine” is a broad term for software that advances work through steps, such as an approval or service-call chain. Azure Logic Apps, AWS Step Functions and Temporal are examples. Data pipeline tools include code-first options such as Apache Airflow, Dagster and Prefect, and native options such as Azure Data Factory pipelines and Databricks Lakeflow Jobs. Categories overlap: judge by the jobs a tool coordinates and its operational model, not its label.

A workflow engine can be a good fit when it meets those needs. A data-focused orchestrator is one option when the main problem is sequencing data jobs across platform boundaries. See the Logic Apps comparison for one example of how the approaches differ.

Example: Azure Data Factory, Databricks and Power BI

Consider a reporting workflow with an Azure Data Factory pipeline that loads source data, an Azure Databricks job that transforms it, and a Power BI semantic model refresh that publishes the result. The example is about coordination, not a prescribed architecture: each service can still perform the work it was chosen for.

  1. Start the Data Factory pipeline from the orchestrator's schedule. If the external orchestrator owns this start, disable a competing native trigger for the same load to avoid running it twice.
  2. Allow the Databricks transformation task to start after the load reports the required successful outcome.
  3. Allow the Power BI refresh task to start after the transformation has completed successfully.
  4. Record each task's status so an operator can see which work ran, which work waited and where a failure occurred.

This sequence replaces “start the notebook a few minutes after the pipeline usually starts” with declared dependencies. If the load runs longer, the transformation waits. If a task fails, the downstream success path does not simply proceed as if the data were ready. The team can then decide whether to correct the input, retry a task, or follow a separate failure path.

A Data Factory pipeline can run a Databricks job with its Azure Databricks Job activity, then call the Power BI refresh API via a Web activity, given the required permissions. Fabric has Databricks and semantic model refresh activities, with capacity and model prerequisites for the latter. Checked 16 November 2026.

If one team owns all three steps, those native pipeline capabilities may be enough. A separate orchestration layer is worth comparing when the chain spans several independently owned workflows, a second report depends on the same job, or operators need one run history across them. The Azure data orchestration guide compares approaches; the wider cross-platform orchestration guide covers workflows that span services or clouds.

When a native scheduler is enough

A separate orchestrator is not automatically an improvement. If one platform owns the workflow, its native scheduler and dependency graph cover the jobs, and its monitoring gives enough context, that may be simpler than adding another component and integration to maintain.

Revisit the decision when the work crosses platform boundaries, the graph is split across separate schedules, a person has to check whether an upstream job finished, or run history no longer explains the complete result. These are signals to compare options, not proof that one specific product is required. When native cloud schedulers are enough offers a more detailed decision test.

Code-driven and configuration-driven orchestration

Code-driven approaches define workflows in a programming language, which suits teams that need custom logic, code review and version-controlled definitions. The workflow engine must be operated, either by the team or through a managed service.

A configuration-driven approach stores task settings and dependencies for an orchestrator to read at runtime. Teams can change schedules or dependencies in a UI without changing platform jobs or deploying orchestration code. It is not “no engineering”: someone connects platforms, configures permissions, chooses parameters and operates workflows. In Azure, “metadata-driven” often means a team's own control-table framework read by a parent Data Factory pipeline; the team owns that engine.

Put transformation logic in the platform or codebase designed to perform it. The orchestration definition describes which existing jobs participate, how they depend on one another and what execution rules apply. That division can reduce duplicated glue while keeping each platform responsible for its work.

Choosing an approach

Map a representative pipeline from input to final consumer: list its jobs, owners, dependencies, failure paths and where an operator checks a run. Ask whether one native platform can cover it; whether dependencies use outcomes or approximate times; who owns definitions, credentials, changes and retries; and whether existing jobs can remain in place.

A small workflow may need only a native scheduler. A cross-platform workflow may benefit from a shared dependency graph, but it also introduces another system to configure and operate. The right choice is the simplest one that makes the required ordering, failure handling and run state explicit.

Where Polysync fits

Polysync is a data pipeline orchestration service for Azure-first teams. It schedules and triggers the jobs they already have in Azure Data Factory, Databricks, Microsoft Fabric and other Azure data and AI services in the right order, from one dependency graph. AWS and Google Cloud jobs can join the same graph.

Jobs are imported, not rewritten. Polysync connects to 20+ services, discovers supported jobs and imports their parameters. Data processing stays in the connected platform. No orchestration code. Teams configure schedules, dependencies, retries with back-off and concurrency limits in the web app. One run history. An optional AI monitoring agent is off until an administrator enables it; it explains failures and can retry only within the authority that administrator sets.

Polysync is not the right choice when one platform already runs and monitors the complete workflow, or when your team prefers workflows defined as code. Compare the Airflow option and read about orchestrating AI workloads.

Frequently asked questions

What is data pipeline orchestration?

Data pipeline orchestration is the automated coordination of the jobs in a data pipeline. It decides when each job starts, what it waits for and what happens when one fails, and it records the outcome of every run. The jobs themselves keep running in the tools that own them.

How is orchestration different from an ETL tool?

An ETL tool extracts, transforms or loads data. An orchestrator coordinates when those jobs run and how they depend on other jobs. Many platforms include both capabilities, so the distinction is about the job each part performs, not a strict product boundary.

Do I need an orchestration platform if I already use Azure Data Factory?

Not necessarily. If the workflow stays in Data Factory and its pipeline graph covers the dependencies, schedules and recovery you need, its native capabilities may be enough. A separate orchestrator becomes relevant when work crosses platform boundaries or you need one view of a larger workflow.

What is a DAG in data pipeline orchestration?

A directed acyclic graph, or DAG, represents tasks as nodes and dependencies as directed edges. Acyclic means the dependency path cannot loop back on itself, so the orchestrator can determine a valid order for eligible tasks.


Related: Data pipeline orchestration for Azure · Orchestrate ADF, Databricks and Fabric · What is ETL orchestration? · When native cloud schedulers are enough · Compare tools · Read orchestration answers · Browse topic guides

Microsoft Learn references, checked 16 November 2026: Data Factory pipelines and activities · Data Factory Databricks Job activity · Data Factory Web activity · Power BI refresh API · Fabric Azure Databricks activity · Fabric semantic model refresh activity.

Microsoft, Azure, Azure Data Factory, Azure Databricks, Microsoft Fabric and Power BI are trademarks of the Microsoft group of companies. All other product names, logos and brands are the property of their respective owners and are used for identification purposes only.

Data pipeline orchestration for Azure-first teams

Keep your ADF, Databricks, Fabric and Power BI jobs. Polysync connects them with shared scheduling, cross-platform dependencies and one run history, including jobs on AWS and Google Cloud. No orchestration code to write.

Learn more about Polysync