How do I stop a scheduled pipeline from running twice or overlapping itself?

Set the job's maximum concurrent runs to 1, so a new run cannot start while the previous one is still active, and decide what should happen to the extra run: wait, be skipped, or be rejected. Then remove the cause: a schedule more frequent than the job's run time, two triggers for the same job, or a retry that starts while the original is still running.

Common causes

Cause Example
Run time longer than the interval A job scheduled every 15 minutes takes 20 minutes on month-end data
Two triggers A schedule in the platform and another in an orchestrator both start the same job
Retry overlap A retry fires while the platform is still finishing the original run
Duplicate dependency release Two parents each start the child instead of the child waiting for both
Daylight-saving changeover A schedule in the repeated hour runs twice (see scheduling in Sydney time)

Controls by tool

Tool Setting
Azure Data Factory Pipeline concurrency; tumbling-window triggers also have max concurrency
Databricks Lakeflow Jobs Maximum concurrent runs on the job
Apache Airflow max_active_runs on the DAG
Polysync Max Concurrent Runs on the task (1 = strictly sequential); a run that would exceed it is rejected rather than queued

In Polysync, a task with several parents waits until every parent's latest run satisfies its condition and then runs once, not once per parent.

Wait, skip or reject?

  • Wait when every run must happen in order, for example incremental loads.
  • Skip or reject when only the latest run matters, for example a refresh that will catch up next time.

Frequently asked questions

Why did my pipeline run twice at the same time?

Most often because two things trigger it, or because one run was still active when the next schedule fired. Check the trigger source of each run in the run history.

Is it safe to let a job overlap itself?

Only if it is idempotent and the target can handle parallel writes. Loads that append or update state usually are not.

Where can I see what started each run?

In Polysync every run records whether it was Scheduled, Manual, Dependency or Retry, and the Monitor page shows the history.

Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.

Data pipeline orchestration for Azure-first teams

Keep your ADF, Databricks, Fabric and Power BI jobs. Polysync connects them with shared scheduling, cross-platform dependencies and one run history, including jobs on AWS and Google Cloud. No orchestration code to write.

Learn more about Polysync