Can Azure Data Factory wait for a job running in AWS or Google Cloud?

Not natively. Azure Data Factory has no built-in activity that starts or waits for an AWS Glue job, an AWS Step Functions execution or a Google Cloud job. You can make a pipeline wait by calling the other cloud's API from a Web activity and polling in an Until loop, by waiting for a file or message the other job writes when it finishes, or by putting both jobs under an orchestrator that starts the ADF pipeline when the other cloud's job succeeds.

Option 1: call and poll the other cloud's API

Use a Web activity to start the job (or to read the status of a job started elsewhere), then an Until loop with a Wait activity to poll until it reaches a final state.

  • Watch out for: AWS APIs require Signature Version 4 request signing, which a plain Web activity cannot do on its own, so teams usually put an Azure Function or an AWS API Gateway endpoint in between. Google Cloud APIs need an OAuth token. You own the polling, timeouts and error mapping.

Option 2: wait for a signal

Have the AWS or Google Cloud job write a marker file to storage ADF can see, and start the ADF pipeline with a storage event trigger, or poll with a Get Metadata activity.

  • Watch out for: cross-cloud storage access, and the fact that a file proves something was written, not that the job succeeded.

Option 3: an external orchestrator

Put both jobs under an orchestrator that can call each cloud. In Polysync:

  1. Connect the AWS account or Google Cloud project and the Data Factory as platforms. Polysync supports AWS Glue, Lambda, Step Functions, SageMaker and Batch, and Google Cloud Composer, Dataflow, Cloud Functions and Vertex AI.
  2. Create a task for the cross-cloud job and one for the ADF pipeline.
  3. Make the ADF task depend on the other task with On Success. Polysync starts the ADF pipeline only after the other cloud's run succeeds, and the reverse direction works the same way.

Where the other cloud's job returns outputs, they can be mapped into the ADF pipeline's parameters.

Frequently asked questions

Does Azure Data Factory have an AWS Glue activity?

No. ADF can copy data from Amazon S3 and other AWS sources with connectors, but it cannot run or wait for a Glue job without a custom call.

Can ADF trigger a Google Cloud Composer DAG?

Only through a custom REST call to the Airflow API with a Google OAuth token, typically via an Azure Function. Polysync has a Cloud Composer DAG job type that does this directly.

What is the simplest option if I only have one cross-cloud dependency?

A marker file and a storage trigger is often the least work. As the number of cross-cloud dependencies grows, an orchestrator can become cheaper than maintaining the glue.

Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.

Data pipeline orchestration for Azure-first teams

Keep your ADF, Databricks, Fabric and Power BI jobs. Polysync connects them with shared scheduling, cross-platform dependencies and one run history, including jobs on AWS and Google Cloud. No orchestration code to write.

Learn more about Polysync