How do I schedule Azure OpenAI batch jobs with dependencies on my data loads?

An Azure OpenAI batch job is asynchronous: you submit a request file, the service processes it within a completion window (24 hours by default), and you poll until the batch is completed, failed or expired. To schedule it after a data load, make the batch submission depend on the load succeeding, poll the batch to a final state, and pass its output_file_id to a downstream step that downloads and processes the results.

The moving parts

  1. Prepare the input. An upstream job writes the JSONL request file and uploads it through the Files API, which returns a file ID (file-…).
  2. Submit the batch. POST /openai/batches with the input file ID, the endpoint (for example /chat/completions) and the completion window.
  3. Wait. Poll GET /openai/batches/{id}. validating, in_progress and finalizing mean it is still running.
  4. Collect. On completed, read output_file_id (and error_file_id if any) and hand them to the next step.

Doing it with Polysync

Polysync imports an Azure OpenAI Batch Job for every chat-capable deployment it discovers.

  1. Set the job's OpenAI Input File ID attribute. The file ID can be resolved from a vault reference at run time, so the upload step can store a new ID each night.
  2. Make the batch task depend on the load task with On Success.
  3. Polysync submits the batch, polls it (validating, in progress and finalizing count as Running; completed is Success; failed or expired is Failed), and can cancel it.
  4. On success it records output_file_id, error_file_id and the request counts (total, completed, failed) as outputs. Map output_file_id into the input of the task that processes the results.

Protecting quota

Batch, chat completion and embedding calls on the same deployment share its quota. Link a rate-limit or concurrency contention profile to the Azure OpenAI platform so that scheduled and dependency-driven runs do not all start together.

Frequently asked questions

How long can an Azure OpenAI batch take?

Up to its completion window, 24 hours by default. A batch that does not finish in time ends as expired.

Can I chain two batch jobs?

Yes. Make the second depend on the first, and pass the first batch's output to whatever prepares the second batch's input file.

Why does my batch fail with input_file_not_found?

The file was usually uploaded to a different Azure OpenAI resource from the one running the batch. Files are scoped to a resource.

Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.

Data pipeline orchestration for Azure-first teams

Keep your ADF, Databricks, Fabric and Power BI jobs. Polysync connects them with shared scheduling, cross-platform dependencies and one run history, including jobs on AWS and Google Cloud. No orchestration code to write.

Learn more about Polysync