Cross-platform and multi-cloud data orchestration
Cross-platform orchestration means one system decides when each job runs, what it waits for and what happens when it fails, even though the jobs themselves run in different services: an Azure Data Factory pipeline, a Databricks job, an AWS Glue job, a Google Cloud Composer DAG. Each of those services has its own scheduler, and each scheduler only sees its own work. Once a workflow crosses from one service to another, something has to sit above them.
Why native schedulers stop being enough
Native schedulers are good at work inside their own boundary. Azure Data Factory sequences activities within a pipeline. Databricks Lakeflow Jobs sequences tasks within a job. AWS Step Functions sequences states within a state machine. The gaps appear at the edges:
- A dependency on work elsewhere. "Run the Databricks job only after the ADF load succeeds" has no native home. Teams bridge it with polling scripts, webhooks, storage-event triggers or a fixed time gap.
- Passing values across the edge. An output from one service (a row count, a watermark, a file name) has to reach the next service's input. Without a shared layer, that usually means a shared table or a queue.
- One view of runs. Each service has its own run history. Answering "what failed overnight, and what did it block?" means opening several portals.
- Shared limits. A cluster, a capacity or an API quota is consumed by several tools at once, but each tool only limits its own runs.
The native cloud schedulers article gives a five-point test for when you have crossed that boundary.
What a cross-platform control plane does
A control plane does not replace the services underneath. It calls them through their own APIs and watches the result. In Polysync that means:
| Capability | How it works |
|---|---|
| Discovery | Connect a platform once; Polysync lists its pipelines, jobs, notebooks or functions and imports them with their parameters. |
| Dependencies | Any task can depend on any other task, on any supported service, with an On Success, On Failure or On Completion condition. |
| Parameter mapping | A named output of a parent run can be copied into a named input of a child run, across platforms and clouds. |
| Scheduling | Visual schedules with a time zone each, and several schedules per task. |
| Shared limits | Contention profiles cap concurrency or release rate per platform, or globally, across every task that uses it. |
| Monitoring | One run history, retry chains and dependency overlays for every platform. |
How to choose an approach
- Everything is in one service. Use that service's scheduler. A control plane adds little.
- Mostly code-defined workflows run by engineers. A code-first orchestrator such as Apache Airflow is a good fit. See Polysync vs Apache Airflow.
- Several managed services, owned by a data platform team. A configuration-driven control plane avoids writing and hosting orchestration code. That is the case Polysync is built for.
Frequently asked questions
What is the difference between orchestration and scheduling?
Scheduling decides when a job starts. Orchestration also decides what each job waits for, what runs after it, what values pass between jobs, and what happens on failure. A cron entry is scheduling; a dependency graph with conditions and retries is orchestration.
Can one orchestrator run jobs in Azure, AWS and Google Cloud?
Yes, if it calls each cloud's job APIs and holds credentials for each. Polysync supports 20+ services across the three clouds, including Azure Data Factory, Databricks, Microsoft Fabric, AWS Glue, AWS Step Functions and Google Cloud Composer.
Does a cross-platform orchestrator move my data?
Polysync does not. It starts jobs, passes small parameter values between them and records the outcome. The data stays in the services that already process it.
Comparisons in this topic
- Polysync vs Apache AirflowCode-first DAGs you run yourself versus a configuration-driven control plane over existing cloud jobs.
- Polysync vs OrchestraTwo control planes over existing tools, compared on pricing model, features, hosting and buying.
Answers in this topic
- Workflow orchestration vs job scheduling: what is the difference?Scheduling answers "when"; orchestration also answers "after what", "with what" and "what if it fails".
- Can Azure Data Factory wait for a job running in AWS or Google Cloud?REST polling, storage signals or an orchestrator dependency to make ADF wait for another cloud.