Guide

Automating Cross-Platform Data Pipelines

A practical walkthrough for teams whose pipelines no longer fit inside one platform.

The Real Problem Is Coordination, Not Compute

Most teams do not have a transformation problem. ADF, Databricks, Functions, Logic Apps, Synapse, Fabric, and Cloud Composer all do their respective jobs well. The pain shows up between them: the ADF pipeline that finishes in the morning, the Databricks job that should follow it but instead runs on a fixed schedule and hopes ADF is done, the Cloud Function that nobody is sure ran at all.

What This Walkthrough Covers

Automating a cross-platform pipeline comes down to one question: what is ready to run now? Seven things have to be in place to answer it reliably, and this walkthrough sets them out in the order you would build them. The pattern is not tied to any one orchestrator. Each step ends with how it looks in Polysync, so the walkthrough doubles as a tour of the product.

One example runs throughout: a nightly customer refresh. An Azure Data Factory pipeline copies new customer records into Azure storage. A Databricks notebook builds a gold table from them. A Google Cloud Function enriches the new rows. A Power BI dataset publishes the result. Four services across three clouds, and before any of this is automated, three of them run on separate clocks and hope the others have finished.

The running example: an Azure Data Factory pipeline copies raw records, a Databricks notebook builds a gold table, a Google Cloud Function enriches the new rows, and a Power BI dataset publishes the result. — The running example: four services across three clouds, joined by one dependency chain.

Step 1. Connect Each Platform

Before anything can run, the orchestrator has to be able to call each platform for you. Two choices are involved, and they are easy to confuse.

How does the orchestrator authenticate? Every connection picks an authentication method. For the Azure services, one of the options is the Polysync Service Principal: a multi-tenant application that Polysync owns, which you grant a role to in your own tenant. There is no long-lived secret for you to hold or rotate. The other choices are your own service principal (with a secret or a certificate), and, on the other clouds, the platform's own methods such as an AWS access key or assume-role, or a Google service account key, application default credentials, workload identity federation or an impersonated service account.

Where does a supplied credential live? If you use your own credential, it is stored in a secret vault and referenced by name, rather than pasted into a configuration screen. Polysync reads from Azure Key Vault, AWS Secrets Manager, Google Cloud Secret Manager and HashiCorp Vault, and can also hold the credential in its own managed vault if you would rather not link one of yours. The value is read when a task is dispatched, not copied into the task's configuration.

Two ways to connect a platform. Top: the Polysync Service Principal, granted a role in your tenant, with no secret to hold. Bottom: your own credential stored in a secret vault and read at dispatch time. — An authentication method decides how Polysync is identified; a vault decides where your credential is kept.

Today Polysync connects to Azure Data Factory, Microsoft Fabric, Synapse Pipelines, Functions, Logic Apps, AKS, Batch, Automation, AI Foundry and OpenAI, Azure SQL stored procedures and Power BI; to Azure Databricks; to AWS Step Functions, Glue, Lambda, SageMaker and Batch; and to Google Cloud Composer, Vertex AI, Cloud Functions and Dataflow.

Step 2. Discover Jobs Instead of Redefining Them

Once a platform is connected, you should not have to retype what already lives there. Polysync reads the platform's own resources and imports them as Jobs: the ADF pipeline, the Databricks notebook, the Cloud Function, the Power BI dataset. The parameters each resource accepts come with it.

Keep two words apart. A Job is the resource on the platform. A Task is a configured run of that Job: parameters bound, dependencies attached, a schedule or trigger. The same Job can back several Tasks, so one notebook can serve both the nightly refresh and a one-off backfill without being copied.

In the example, the four platforms are connected and Polysync imports the copy pipeline, the notebook, the Cloud Function and the dataset refresh. Nothing is rewritten; the resources stay where they are.

Step 3. Wire the Dependencies, Not the Clock

A pipeline is a set of relationships, not a set of times. "Run the notebook at 06:15 because the copy usually finishes at 06:10" encodes a guess that will be wrong on the one night it matters. Tell the orchestrator the relationship and let it work out the time.

Each edge in the graph carries a trigger condition:

On Success. Run only if the upstream task succeeded. This is the chain in the example: copy, then transform, then enrich, then publish.
On Failure. Run only if the upstream failed, which is where an alert or a compensation step belongs.
On Completion. Run whatever the outcome, which is where cleanup belongs.
On Skipped. Run only if the upstream was skipped by an earlier condition, which keeps branch-and-merge graphs correct.

Conditions are resolved across the whole graph, not just the immediate parent, so a skip deep in a branch behaves the way you would expect. A cycle is rejected before a run starts: an accidental loop between two tasks is reported as a configuration error, not discovered as a run that never finishes.

Step 4. Pass Values Between Tasks

Useful pipelines hand values along. The copy pipeline knows how many rows it moved, and the notebook needs that count as a parameter. The Cloud Function writes a file and returns its path, and the Power BI refresh needs to know which partition changed. Without parameter mapping, that value travels through a shared file or a table that somebody maintains by hand.

Polysync maps an output of one task into a parameter of another, across platforms: the row count from the ADF pipeline becomes an input to the Databricks notebook, and the notebook's partition name becomes an input to the Cloud Function. The mapping is part of the task configuration, so it is visible and changeable without touching transformation code.

The output of the ADF copy task, a row count of 41,208, is mapped into the rowCount parameter of the Databricks notebook task. — One task's output becomes another task's parameter, declared in the configuration.

Step 5. Put Governance on the Queue

Every managed service has hard limits: Databricks job concurrency per workspace, Logic Apps action throughput, Cloud Functions concurrent executions. Parallel triggers turn those limits into HTTP 429 responses, dropped runs and unpredictable spend.

A concurrency profile applies two controls, scoped to a single platform connection:

Leaky bucket caps how many tasks are in flight against the platform at once.
Rolling window caps how many are dispatched in any sliding period.

Because the budget is per connection, a busy Databricks workspace cannot hold up work aimed at an unrelated ADF factory. The queue is also where waiting happens: a task whose dependencies are not yet met waits there, rather than polling the platform and piling up activity.

Step 6. Schedule the Roots

Schedules attach to root tasks, the ones with nothing upstream. They use standard cron with a timezone, and there is a visual builder if you would rather not count asterisks. A task can carry more than one schedule.

Everything downstream runs because the graph says it is ready, not because it was given its own clock. In the example, only the ADF copy is scheduled. The transform, the enrichment and the publish have no times of their own.

Step 7. Watch It in One Place

The point of all of the above is to answer "what is happening right now" on one screen. Polysync shows queue depth, the tasks currently dispatched, recent run history, durations and status breakdowns across every connected platform, so a run that spans four services reads as one run instead of four consoles.

What You End Up With

A pipeline that no longer depends on someone remembering which portal to open first, and a schedule that states a relationship rather than a hope. The graph triggers itself, the queue applies the limits, and one view tells the truth about what ran. The orchestrator becomes the system of record for what your data platform actually did last night.

Microsoft, Azure, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Functions, Logic Apps, Microsoft Fabric and Microsoft Marketplace are trademarks of the Microsoft group of companies. All other product names, logos and brands are the property of their respective owners and are used for identification purposes only.