How do I schedule Azure OpenAI batch jobs with dependencies on my data loads?
An Azure OpenAI batch job is asynchronous: you submit a request file, the service processes it within a completion window (24 hours by default), and you poll until the batch is completed, failed or expired. To schedule it after a data load, make the batch submission depend on the load succeeding, poll the batch to a final state, and pass its output_file_id to a downstream step that downloads and processes the results.
The moving parts
- Prepare the input. An upstream job writes the JSONL request file and uploads it through the Files API, which returns a file ID (
file-…). - Submit the batch.
POST /openai/batcheswith the input file ID, the endpoint (for example/chat/completions) and the completion window. - Wait. Poll
GET /openai/batches/{id}.validating,in_progressandfinalizingmean it is still running. - Collect. On
completed, readoutput_file_id(anderror_file_idif any) and hand them to the next step.
Doing it with Polysync
Polysync imports an Azure OpenAI Batch Job for every chat-capable deployment it discovers.
- Set the job's OpenAI Input File ID attribute. The file ID can be resolved from a vault reference at run time, so the upload step can store a new ID each night.
- Make the batch task depend on the load task with On Success.
- Polysync submits the batch, polls it (validating, in progress and finalizing count as Running; completed is Success; failed or expired is Failed), and can cancel it.
- On success it records
output_file_id,error_file_idand the request counts (total, completed, failed) as outputs. Mapoutput_file_idinto the input of the task that processes the results.
Protecting quota
Batch, chat completion and embedding calls on the same deployment share its quota. Link a rate-limit or concurrency contention profile to the Azure OpenAI platform so that scheduled and dependency-driven runs do not all start together.
Frequently asked questions
How long can an Azure OpenAI batch take?
Up to its completion window, 24 hours by default. A batch that does not finish in time ends as expired.
Can I chain two batch jobs?
Yes. Make the second depend on the first, and pass the first batch's output to whatever prepares the second batch's input file.
Why does my batch fail with input_file_not_found?
The file was usually uploaded to a different Azure OpenAI resource from the one running the batch. Files are scoped to a resource.
Microsoft, Azure, Azure Data Factory, Microsoft Fabric, Power BI, Logic Apps and Microsoft Marketplace are trademarks of the Microsoft group of companies. Apache Airflow and Airflow are trademarks of the Apache Software Foundation. Databricks and Lakeflow are trademarks of Databricks, Inc. All other product names are the property of their respective owners and are used for identification only. No endorsement by their owners is implied.