Answers to common data orchestration questions
Direct answers to the questions data engineers ask about cross-platform scheduling, dependencies, retries, concurrency, credentials and cost.
Cross-platform and multi-cloud data orchestration
Workflow orchestration vs job scheduling: what is the difference?
Scheduling answers "when"; orchestration also answers "after what", "with what" and "what if it fails".
Can Azure Data Factory wait for a job running in AWS or Google Cloud?
REST polling, storage signals or an orchestrator dependency to make ADF wait for another cloud.
Orchestrating the Azure data platform
How do I trigger a Databricks job after an Azure Data Factory pipeline finishes?
ADF's Databricks activity, a Jobs API call, a file-arrival trigger or an orchestrator dependency, compared.
How do I refresh a Power BI semantic model only after my ETL succeeds?
Replace a clock-time scheduled refresh with a refresh that waits for the load to succeed.
How do I orchestrate Microsoft Fabric pipelines alongside existing Azure Data Factory pipelines?
Keeping one overnight chain working while pipelines move from ADF to Fabric.
Do I need Apache Airflow if I already use Azure Data Factory?
Add Airflow for code-first workflows; not just to sequence pipelines ADF already runs.
Pipeline scheduling, dependencies and reliability
How do I limit concurrent runs against a Databricks workspace from several tools?
Per-job caps versus one shared budget for everything that drives a workspace.
How do I stop orchestrated jobs from hitting cloud API rate limits (HTTP 429)?
Rate-limit job starts with token bucket, leaky bucket or sliding window, not just retries.
How do I schedule data pipelines in Sydney time across daylight-saving changes?
Schedule in a named time zone, not a UTC offset, and check what happens on changeover nights.
How do I stop a scheduled pipeline from running twice or overlapping itself?
Cap concurrent runs per job, and decide whether an extra run should wait, be skipped or be rejected.
Orchestrating AI workloads and AI-assisted operations
How do I schedule Azure OpenAI batch jobs with dependencies on my data loads?
Start an Azure OpenAI batch after the load, wait for it, and hand its output file to the next task.
What is an AI monitoring agent for data pipelines?
AI that reviews failed runs, explains them with evidence, and acts only within limits you set.