ADF vs Polysync: When ADF Stops Being Enough

A pragmatic comparison for teams already invested in Azure Data Factory.

Start With What ADF Does Well

Azure Data Factory is a strong data integration service. Within a single factory, it gives you visual pipelines, a large library of connectors, parameterised datasets and linked services, tumbling-window and schedule triggers, and an integration runtime that can reach hybrid sources. For a team whose data lives inside Azure, ADF is often the right primary tool and is not something Polysync attempts to replace.

The conversation gets interesting when ADF stops being the only thing in the room.

Where ADF's Orchestration Model Runs Out

ADF's scheduling and dependency primitives are designed around the assumption that the work being coordinated is also in ADF. Once the assumption breaks, three gaps appear quickly.

1. Dependencies stop at the factory boundary. ADF activity dependencies (dependsOn) only resolve between activities inside the same pipeline. Triggering one pipeline from another is possible via the Execute Pipeline activity, but a Databricks job kicked off elsewhere, a Cloud Function on GCP, or an external arrival event cannot participate as a first-class node in an ADF dependency graph.
2. Triggers fire from a clock, not from a graph. Schedule and tumbling-window triggers know when to run, but they do not natively express "run this pipeline only after that unrelated workload finishes successfully on another platform." Teams typically bridge that gap with custom code, a queue, or a third system that polls.
3. Concurrency control is per-pipeline, not per-platform. ADF lets you cap a single pipeline's concurrent runs. It does not give you a global budget that says "no more than 8 things running against this Databricks workspace, across all sources." When the same Databricks cluster is being driven by ADF, a notebook job, and an ad-hoc workflow, that budget has to live somewhere outside ADF.

Changing a Schedule Should Not Be a Deployment

In ADF, a trigger is part of the factory's definition. Changing when a pipeline runs means editing the trigger and publishing the factory, the same path a pipeline change takes. Publishing restarts the trigger with the new definition. For a team with Git integration and CI/CD, that is a pull request, a review and a deploy window. It is a reasonable amount of ceremony for a change to pipeline logic. It is a lot for "move this job fifteen minutes earlier", and it means the person who wants the change usually cannot make it alone.

When schedules and dependencies are data rather than artifacts, the change is an edit. An operator moves the time in the UI; the scheduler picks up the new definition on its next cycle. No branch, no review, no deploy. The audit trail is the record of the change itself, not a commit that has to be released before it counts.

Monitoring Stops at the Factory Door

ADF Monitor shows ADF runs. That is exactly the right scope when ADF is the only thing running. The moment a run depends on a Databricks job, a Logic App, a Fabric pipeline or a Cloud Function, "did the whole run succeed?" is a question no single console answers. Azure Monitor and Log Analytics can collect the pieces, but only once diagnostic settings are wired for every resource, and each service still keeps its own notion of a run. The result is a status question answered by opening four monitors and comparing timestamps.

A control plane keeps one run history. Every task (whatever platform actually executes it) records a status, a duration and its place in the dependency graph, and a retry is traced back to the original attempt. Assembling the cross-platform timeline stops being a side project that someone maintains.

You Pay for Orchestration, Including the Waiting

Azure Data Factory bills per activity run, which is an ordinary way for a consumption service to price coordination. The consequence worth understanding is that an activity which only orchestrates is an activity run like any other. A Lookup, a Wait, a Web call, each pass through an Until loop. They all count, whether or not a byte of data moves.

That makes the polling pattern expensive in a specific way. Using ADF to watch for something that happens elsewhere, such as a file to land or another platform's job to finish, means repeated activities that do no data work. The pipeline is billed for waiting. A control plane that waits on the other platforms' own tasks does so at the queue: a dependency is resolved when the upstream run reports completion, and no platform activity is created in the meantime. The cost follows the work, not the wait.

Metadata-Driven Pipelines and the Glue-Code Bill

ADF can be made metadata-driven, and experienced teams do it: a control table is read by a Lookup, the rows drive a ForEach, an If or Switch chooses the activity, and the outcome is written back to the table. It is a sound pattern. It is also a framework: a parent pipeline, a table schema, error handling and parameter contracts that someone owns and maintains, and that every organisation builds a slightly different version of.

The cost is not the first pipeline. It is the second platform, where the same framework has to be rebuilt because ADF cannot iterate over a Databricks job or a Cloud Function as a first-class node. Then the third platform, and then the person who leaves. With Polysync the metadata is the product rather than something you assemble: jobs are discovered from each connected platform with their parameters, tasks and dependencies are rows, and there is no parent pipeline to maintain or hand over.

What Polysync Adds Around ADF

Polysync is a control plane that treats ADF pipelines as first-class nodes in a larger dependency graph. It does not run the transformations itself; ADF still does that. Polysync decides when ADF runs, what runs alongside it, and what waits for it.

• Cross-platform DAG. An ADF pipeline can depend on a Databricks job, which can depend on a Synapse pipeline, which can depend on a Google Cloud Function. Trigger conditions (on success, on failure, on completion, on skipped) are resolved recursively across the whole graph.
• Auto-discovery of pipelines. Polysync connects to your factory and discovers existing pipelines, exposing them as Jobs you can configure into Tasks. You are not retyping pipeline names or parameters.
• Parameter mapping across platforms. The output of an ADF pipeline (a row count, a file path, a status) can be mapped into the parameters of a downstream task on a different platform without writing a custom passthrough.
• Unified concurrency and rate limits. Concurrency profiles use leaky-bucket and rolling-window controls scoped per platform connection. An ADF run, a Databricks job, and a Logic App invocation can each contribute to the same budget when they target the same downstream resource.
• One monitoring view. Every run across every connected platform shows up in a single dashboard with status, duration, and history; you do not bounce between ADF Monitor, Databricks job runs, and Logic Apps run history to assemble the picture.
• Standard cron with a visual builder. Schedules attach to root tasks and are expressed in standard cron syntax with timezone awareness. Multiple schedules per task are supported; tumbling windows are not the only available shape.
• Changes without a deployment. Schedules, dependencies and parameters are configuration, so changing them is an edit rather than a release. The scheduler picks up the new definition on its next cycle.
• Waiting without activity cost. A task waits for its upstream dependencies at the queue; the platform is only called once they are satisfied, so nothing accrues while a run waits.
• No orchestration code to maintain. There is no parent pipeline or control-table framework to build, own and hand over. The graph, the parameters and the schedules are the configuration.

Side by Side

Will I Still Need ADF?

Yes. Polysync does not move bytes. If you use ADF today to copy from on-prem SQL into ADLS, that activity is still in ADF and still uses your integration runtime. Polysync calls the ADF pipeline, waits for it, captures its result, and decides what runs next.

When ADF Alone Is Still the Right Answer

If every pipeline you operate lives inside one ADF instance, your dependencies are simple, and you have no plans to add Databricks, Functions, Logic Apps, Synapse, Fabric, or Google Cloud services into the same flow, ADF's built-in scheduling is fine. Adding a separate orchestrator before you need one is overhead, not insurance.

The point at which most teams cross the line is the second platform: the first Databricks job, the first Cloud Function, the first AKS workload that needs to coordinate with an ADF pipeline. That is the moment a dedicated orchestrator starts paying for itself.

---

Related: When native cloud schedulers are enough · What is ETL orchestration? · Automating cross-platform data pipelines

Microsoft, Azure, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Functions, Logic Apps, Microsoft Fabric and Microsoft Marketplace are trademarks of the Microsoft group of companies. All other product names, logos and brands are the property of their respective owners and are used for identification purposes only.