How do I limit concurrent runs against a Databricks workspace from several tools?
Each tool limits only its own runs: a Databricks job has a maximum concurrent runs setting, and an ADF pipeline has its own concurrency setting, but nothing adds them up. To cap the total, route the runs through one place that holds a shared budget, such as a queue you build or an orchestrator with a workspace-wide concurrency cap, and size that cap below the workspace's own limit. Databricks documents a limit of 2,000 concurrent task runs per workspace and returns HTTP 429 for a run that cannot start, but cluster or warehouse capacity is often the tighter limit in practice.
Why per-job limits are not enough
| Setting | Limits | Does not limit |
|---|---|---|
| Databricks job maximum concurrent runs | Copies of that one job | Other jobs on the same cluster or warehouse |
| ADF pipeline concurrency | Runs of that one pipeline | Databricks runs started by other pipelines or tools |
| Cluster autoscaling | Nodes per cluster | How many jobs queue up for it |
When a nightly ADF pipeline, a notebook schedule and an ad-hoc workflow all start Databricks work at 02:00, each is within its own limit and the shared capacity is still overloaded.
The shared-budget approach
Put every scheduled start through one gate that counts active runs for that workspace:
- Build it: a queue table or message queue plus a small service that releases work when a slot is free.
- Use an orchestrator's limit: in Polysync, a contention profile linked to the Databricks platform does this. Choose Concurrency Contention, set Max Concurrent Tasks, and pick what happens when the limit is reached: wait in the queue, reject immediately, or reject only low-priority runs. Queued runs are released in priority order.
The profile applies to every task on that platform, whether a schedule, a dependency, a manual run or a retry started it. Runs started outside Polysync are not counted, so the cap should leave headroom for them.
Rate as well as concurrency
If the problem is how fast runs start rather than how many run at once (for example API rate limits), use a rate limit instead: token bucket for short bursts, leaky bucket for a steady flow, or sliding window for "at most N per period". See handling API rate limits.
Frequently asked questions
What is the Databricks concurrent run limit?
Databricks documents a limit of 2,000 concurrent task runs per workspace, 1,000 tasks per job and 10,000 job creations per hour (Lakeflow Jobs limitations, checked 7 November 2026). Practical limits are usually your cluster and warehouse capacity.
Should one job be allowed to run twice at once?
Usually not for loads and transforms. Set the job's maximum concurrent runs to 1, or in Polysync set the task's Max Concurrent Runs to 1, so a late run is not overlapped by the next schedule.
Can I cap all platforms together?
Yes. A Polysync contention profile can be scoped globally instead of per platform.
Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.