Documentation

Fabric Spark Job Definition Job

The Fabric Spark Job Definition job type runs a Microsoft Fabric Spark Job Definition item via POST /workspaces/{wsId}/items/{itemId}/jobs/instances?jobType=sparkjob. The artefact is identified by its item ID — Polysync stores that ID in the Job's External Id.

This job type is supported on the Microsoft Fabric platform.

Required job fields

  • External Id — the Fabric item ID (GUID) of the Spark Job Definition.
  • Job Type — Fabric Spark Job Definition (set automatically on import).

Job discovery

GET /workspaces/{workspaceId}/items?type=SparkJobDefinition (paginated with continuationToken).

Parameter handling

Spark Job Definitions consume parameters via the command line. Polysync flattens Input + Input&Output parameters into the commandLineArguments string in --name value form:

{
  "executionData": {
    "commandLineArguments": "--name1 value1 --name2 value2"
  }
}
Direction Sent in jobs/instances Updated from response
Input ✅ (as CLI arg) ❌
Output ❌ ❌ (not supported)
Input&Output ✅ (as CLI arg) ❌ (not supported)

Output parameters are not supported.

Values are stringified — your Spark entry-point must parse the CLI arguments itself (e.g., argparse, Args(args).getOrElse).

Execution flow

Same as Fabric Notebook:

  1. POST jobs/instances?jobType=sparkjob.
  2. Poll GET .../jobs/instances/{instanceId} for status (same mapping table).
  3. Cancel via POST /jobs/instances/{instanceId}/cancel.

Monitor URL

https://app.fabric.microsoft.com/groups/{workspaceId}/sparkjobdefinitions/{itemId}

Best practices

  • Quote values containing spaces in your Polysync parameter values — Polysync forwards them verbatim and your CLI parser sees them as separate tokens otherwise.
  • Avoid Polysync parameter names containing - or = to keep the generated argument list parseable.

Troubleshooting

  • Application fails parsing args — check the order and quoting of CLI args in the Fabric run log; the Polysync forward order is the parameter order on the Job.