Pipeline scheduling, dependencies and reliability
A reliable pipeline is mostly a set of small, explicit decisions: when each job starts, what it waits for, how many copies may run at once, how often it may hit an API, how long it may run, and whether a failure is worth retrying. Most incidents in scheduled data work come from one of those decisions being implicit: a time gap standing in for a dependency, a retry that repeats a side effect, or two tools driving the same cluster with no shared limit.
The building blocks
| Decision | What to set | Polysync setting |
|---|---|---|
| When does it start? | A schedule in the right time zone | Visual schedule builder, one time zone per schedule, several schedules per task, or a Quartz cron expression |
| What does it wait for? | Dependencies with a condition | On Success, On Failure or On Completion per parent; a task with several parents waits for all of them |
| Can it overlap itself? | A per-job concurrency cap | Max Concurrent Runs on the task (1 = strictly sequential) |
| Can it overwhelm a shared system? | A shared cap or rate limit | Contention profiles: concurrency caps, or token-bucket, leaky-bucket and sliding-window rate limits, per platform or global |
| How long may it run? | A timeout | Execution Timeout on the task |
| What if it fails? | A retry policy, only where a retry is safe | Retry count with fixed, linear or exponential backoff and a maximum delay |
Dependencies beat time gaps
A schedule that starts the transform at 02:30 because the load "usually finishes by 02:15" fails silently the night the load is late. A dependency starts the transform when the load has actually succeeded, and an On Failure dependency can start a clean-up or alert task instead. See How to trigger a Databricks job after an ADF pipeline for a worked example.
Retries are not free
A retry is only safe when running the job again cannot repeat a side effect that already happened, such as an append without a key, an email, or a payment. Transient errors (throttling, a brief network failure, a capacity spike) usually recover; credential, permission and configuration errors never do. The retry guidance page goes through each class.
Frequently asked questions
Should I use cron expressions or a visual scheduler?
Either works if the result is right. Visual builders reduce mistakes with day-of-week numbering, last-business-day rules and time zones; cron expressions are compact and portable. Polysync accepts both and shows the next three run times either way.
What is the difference between a concurrency cap and a rate limit?
A concurrency cap limits how many runs are active at the same moment. A rate limit limits how many runs start within a period. A cluster with fixed capacity needs a concurrency cap; an API with a requests-per-minute quota needs a rate limit.
What happens to a child task when its parent is cancelled?
In Polysync, a cancelled parent run releases no children under any condition. Only Completed, Failed or TimedOut outcomes can satisfy a dependency.
Answers in this topic
- How do I limit concurrent runs against a Databricks workspace from several tools?Per-job caps versus one shared budget for everything that drives a workspace.
- How do I stop orchestrated jobs from hitting cloud API rate limits (HTTP 429)?Rate-limit job starts with token bucket, leaky bucket or sliding window, not just retries.
- How do I schedule data pipelines in Sydney time across daylight-saving changes?Schedule in a named time zone, not a UTC offset, and check what happens on changeover nights.
- How do I stop a scheduled pipeline from running twice or overlapping itself?Cap concurrent runs per job, and decide whether an extra run should wait, be skipped or be rejected.