Databricks Notebook Job

The Databricks Notebook job type runs a single notebook in a Databricks workspace as a one-off run via POST jobs/runs/submit (Jobs API 2.1). The notebook is identified by its workspace path — Polysync stores that path as the Job's External Id.

This job type is supported on the Azure Databricks platform.

Required job fields

External Id — the notebook path (e.g., /Users/alice@contoso.com/etl/sales-load).
Job Type — Databricks Notebook (set automatically on import).

Compute precedence: job-level compute attributes → Platform attributes. Both the Job and the Platform can carry Compute Type and the cluster-shape attributes (Cluster Version, Cluster Node Type, Number of Worker Nodes, Existing Cluster ID, Existing Instance Pool); a Job-level value overrides the Platform's, and an empty Job-level value falls back to the Platform's.

Job discovery

Polysync calls workspace/list?path={DefaultNotebookPath} (default /) and recursively descends, importing every object where object_type == "NOTEBOOK" as a Polysync Job. For each notebook, workspace/export is fetched and dbutils.widgets.{text,dropdown,combobox,multiselect} declarations are parsed to seed the Job's parameter list with the widget names. Declared default values are not imported — set the value you want in the Polysync editor before running.

Parameter handling

Notebook widgets are exposed by Databricks as strings only. The provider sends Input + Input&Output parameters in the notebook_task.base_parameters dictionary, exactly as configured:

{
  "notebook_task": {
    "notebook_path": "<external-id>",
    "base_parameters": {
      "<parameter-name>": "<value-coerced-to-string>"
    }
  }
}

Direction	Sent in `runs/submit`	Updated from response
`Input`	✅	❌
`Output`	❌	✅ (JSON exit object)
`Input&Output`	✅	✅ (JSON exit object)

Outputs are read back from jobs/runs/get-output after the run succeeds: if dbutils.notebook.exit(...) returns a JSON object whose keys match parameters marked Output or Input & Output, those parameters are updated with the returned values (the same contract as ADF's pipelineReturnValue). Plain-text exit values are ignored by Polysync — they remain visible on the Databricks run page only. Persist larger results to a shared store (Delta table, ADLS path).

All values are sent as JSON strings regardless of the Polysync Data Type — pick the Data Type that helps the editor validate the value, but be aware your notebook will always receive a string from dbutils.widgets.get.

Execution flow

Polysync builds the notebook_task body, attaches the compute block resolved from the Platform, then calls POST jobs/runs/submit (Jobs API 2.1).
The response run_id becomes the Polysync RunId.

Status is polled via GET jobs/runs/get?run_id={runId} and decoded from the combination of life_cycle_state and result_state:

Databricks state	Polysync status
`life_cycle_state = PENDING / RUNNING / TERMINATING`	Running
`life_cycle_state = SKIPPED`	Cancelled
`life_cycle_state = INTERNAL_ERROR`	Failed
`result_state = SUCCESS`	Success
`result_state = FAILED` / `TIMEDOUT`	Failed
`result_state = CANCELED`	Cancelled

When the run reaches result_state = SUCCESS, Polysync reads the notebook exit value via GET jobs/runs/get-output?run_id={runId} and fills any Output / Input & Output parameters from a JSON-object exit value.
Cancel is supported via POST jobs/runs/cancel ({ "run_id": <run_id> }).

Monitor URL

{workspace_url}/#job/{run_id}/run/1

If the submit response includes run_page_url, Polysync uses that value verbatim.

Best practices

Always declare the notebook's widgets explicitly with dbutils.widgets.text/dropdown — they're what Polysync's parameter importer reads.
Return a JSON object from dbutils.notebook.exit(...) when values must flow back into Polysync outputs; plain-text exits are not read. Persist large results to Delta / object storage instead.
Use Serverless compute or Existing Instance Pool to keep run startup latency low — runs/submit provisions a new cluster otherwise.

Troubleshooting

Failure message looks generic — Databricks reports Workload failed, see run output for details. as the run state. Polysync folds the notebook's exception text (from the run output's error field) into the failure message, after the dash.
Run fails with RESOURCE_DOES_NOT_EXIST — the External Id is not a valid notebook path in the workspace. Re-import from the Platform editor to pick the correct path.
Widgets not picked up on import — workspace/export may have returned a SOURCE rather than a JUPYTER notebook; ensure dbutils.widgets calls are at the top level of cells (not inside conditional blocks) so the importer's regex can find them.
Run stays in PENDING for a long time — the compute block is spinning up a new cluster; switch the Platform's Compute Type to ExistingInteractiveCluster, ExistingInstancePool, or Serverless for faster starts.

Documentation