How do I stop a scheduled pipeline from running twice or overlapping itself?
Set the job's maximum concurrent runs to 1, so a new run cannot start while the previous one is still active, and decide what should happen to the extra run: wait, be skipped, or be rejected. Then remove the cause: a schedule more frequent than the job's run time, two triggers for the same job, or a retry that starts while the original is still running.
Common causes
| Cause | Example |
|---|---|
| Run time longer than the interval | A job scheduled every 15 minutes takes 20 minutes on month-end data |
| Two triggers | A schedule in the platform and another in an orchestrator both start the same job |
| Retry overlap | A retry fires while the platform is still finishing the original run |
| Duplicate dependency release | Two parents each start the child instead of the child waiting for both |
| Daylight-saving changeover | A schedule in the repeated hour runs twice (see scheduling in Sydney time) |
Controls by tool
| Tool | Setting |
|---|---|
| Azure Data Factory | Pipeline concurrency; tumbling-window triggers also have max concurrency |
| Databricks Lakeflow Jobs | Maximum concurrent runs on the job |
| Apache Airflow | max_active_runs on the DAG |
| Polysync | Max Concurrent Runs on the task (1 = strictly sequential); a run that would exceed it is rejected rather than queued |
In Polysync, a task with several parents waits until every parent's latest run satisfies its condition and then runs once, not once per parent.
Wait, skip or reject?
- Wait when every run must happen in order, for example incremental loads.
- Skip or reject when only the latest run matters, for example a refresh that will catch up next time.
Frequently asked questions
Why did my pipeline run twice at the same time?
Most often because two things trigger it, or because one run was still active when the next schedule fired. Check the trigger source of each run in the run history.
Is it safe to let a job overlap itself?
Only if it is idempotent and the target can handle parallel writes. Loads that append or update state usually are not.
Where can I see what started each run?
In Polysync every run records whether it was Scheduled, Manual, Dependency or Retry, and the Monitor page shows the history.
Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.