How do I trigger a Databricks job after an Azure Data Factory pipeline finishes?
There are four common ways: call the Databricks work from inside the ADF pipeline with ADF's Azure Databricks activity; call the Databricks Jobs API run-now endpoint from a Web activity at the end of the pipeline; have ADF write a file and start the Databricks job with a file-arrival trigger; or let an external orchestrator start the Databricks job when the ADF pipeline's run succeeds. The first is simplest when one team owns both; the last keeps the two independent and gives you one place to see the chain.
Option 1: Azure Databricks activity inside the ADF pipeline
ADF has native activities that run a Databricks notebook, JAR or Python file on a cluster. Add one after the load activities with a success dependency.
- Good for: one pipeline, one team, notebook-shaped work.
- Watch out for: the activity runs a notebook or file, not an existing multi-task Databricks job with its own settings; the Databricks work's schedule and ownership now live in ADF.
Option 2: Web activity calling the Jobs API
Add a Web activity that posts to the Databricks Jobs API run-now endpoint with the job ID and parameters. To wait for the result, poll runs/get in an Until loop.
- Good for: starting an existing Databricks job as-is.
- Watch out for: you own the polling loop, token handling and error mapping. The ADF run shows "Web activity succeeded", not the Databricks outcome, unless you build the loop.
Option 3: file-arrival trigger in Databricks
Have the ADF pipeline write a marker or data file to cloud storage, and configure the Databricks job with a file arrival trigger.
- Good for: loose coupling between teams.
- Watch out for: a partial write or a failed run that still writes the file starts the job anyway. Passing values (a batch ID, a watermark) means encoding them in the file.
Option 4: an external dependency
An orchestrator above both services starts the Databricks job when the ADF run succeeds. In Polysync:
- Connect the Data Factory and the Databricks workspace as platforms; their pipelines and jobs are discovered and imported with their parameters.
- Create a task for the ADF pipeline and one for the Databricks job.
- Add a dependency on the Databricks task: parent = ADF task, condition = On Success.
- Optionally map an ADF output to a Databricks input. ADF outputs come from the pipeline's
pipelineReturnValue(set with a Set Variable activity); Databricks inputs are sent as job parameters onrun-now. - Add an On Failure dependency to an alert or clean-up task if you want one.
- Good for: jobs owned by different teams, factories in different subscriptions, or chains that continue into Power BI, Fabric or another cloud.
- Watch out for: it is another service to subscribe to. The ADF and Databricks jobs themselves do not change.
Comparison
| Runs existing Databricks job | Waits for real success | Passes values | Code to own | |
|---|---|---|---|---|
| Databricks activity in ADF | No (notebook/JAR/Python) | Yes | Yes, activity parameters | Little |
| Web activity + Jobs API | Yes | Only with a polling loop | Yes | Polling, auth, errors |
| File-arrival trigger | Yes | No, waits for a file | Only via file contents | Little |
| Orchestrator dependency | Yes | Yes | Yes, output to input | None |
Frequently asked questions
Can a Databricks job depend directly on an ADF pipeline?
Not natively. Lakeflow Jobs dependencies are between tasks inside a Databricks job. Work in ADF reaches Databricks through an ADF activity, an API call, a storage trigger or an external orchestrator.
How do I pass a value from ADF to the Databricks job?
Return it from the ADF pipeline (pipelineReturnValue) and send it as a Databricks job parameter. With ADF's own activities you pass it as an activity parameter; with Polysync you map the ADF output to the Databricks input on the dependency.
What if the ADF pipeline fails?
With a success condition, the Databricks job does not start. Add a failure path (an ADF failure dependency, or an On Failure dependency in Polysync) if someone should be told.
Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.