Pipeline scheduling, dependencies and reliability

A reliable pipeline is mostly a set of small, explicit decisions: when each job starts, what it waits for, how many copies may run at once, how often it may hit an API, how long it may run, and whether a failure is worth retrying. Most incidents in scheduled data work come from one of those decisions being implicit: a time gap standing in for a dependency, a retry that repeats a side effect, or two tools driving the same cluster with no shared limit.

The building blocks

Decision What to set Polysync setting
When does it start? A schedule in the right time zone Visual schedule builder, one time zone per schedule, several schedules per task, or a Quartz cron expression
What does it wait for? Dependencies with a condition On Success, On Failure or On Completion per parent; a task with several parents waits for all of them
Can it overlap itself? A per-job concurrency cap Max Concurrent Runs on the task (1 = strictly sequential)
Can it overwhelm a shared system? A shared cap or rate limit Contention profiles: concurrency caps, or token-bucket, leaky-bucket and sliding-window rate limits, per platform or global
How long may it run? A timeout Execution Timeout on the task
What if it fails? A retry policy, only where a retry is safe Retry count with fixed, linear or exponential backoff and a maximum delay

Dependencies beat time gaps

A schedule that starts the transform at 02:30 because the load "usually finishes by 02:15" fails silently the night the load is late. A dependency starts the transform when the load has actually succeeded, and an On Failure dependency can start a clean-up or alert task instead. See How to trigger a Databricks job after an ADF pipeline for a worked example.

Retries are not free

A retry is only safe when running the job again cannot repeat a side effect that already happened, such as an append without a key, an email, or a payment. Transient errors (throttling, a brief network failure, a capacity spike) usually recover; credential, permission and configuration errors never do. The retry guidance page goes through each class.

Frequently asked questions

Should I use cron expressions or a visual scheduler?

Either works if the result is right. Visual builders reduce mistakes with day-of-week numbering, last-business-day rules and time zones; cron expressions are compact and portable. Polysync accepts both and shows the next three run times either way.

What is the difference between a concurrency cap and a rate limit?

A concurrency cap limits how many runs are active at the same moment. A rate limit limits how many runs start within a period. A cluster with fixed capacity needs a concurrency cap; an API with a requests-per-minute quota needs a rate limit.

What happens to a child task when its parent is cancelled?

In Polysync, a cancelled parent run releases no children under any condition. Only Completed, Failed or TimedOut outcomes can satisfy a dependency.

Answers in this topic

Data pipeline orchestration for Azure-first teams

Keep your ADF, Databricks, Fabric and Power BI jobs. Polysync connects them with shared scheduling, cross-platform dependencies and one run history, including jobs on AWS and Google Cloud. No orchestration code to write.

Learn more about Polysync