Documentation

ADF Pipeline Job

The ADF Pipeline job type wraps a single pipeline in an Azure Data Factory and runs it via the Data Factory REST API (Pipelines/CreateRun). The pipeline is identified by its name in ADF — Polysync stores that name as the Job's External Id.

This job type is supported on the Azure Data Factory platform.

Required job fields

  • External Id — the pipeline name as it appears in the factory. Polysync calls pipelines/{externalId}?api-version=2018-06-01 to read the pipeline definition, so it must match exactly (case-sensitive in ADF).
  • Job Type — ADF Pipeline. Set automatically when the Job is imported from the platform.

There are no extra job-level attributes — all execution options come from the Platform record and the Job's parameter values.

Job discovery (import from platform)

Click Get Pipelines in the Azure Data Factory platform editor (or use the AI Copilot's list_platform_pipelines → import_platform_jobs tools). Polysync calls GET pipelines?api-version=2018-06-01, then for each pipeline:

  1. Creates a Job with External Id = pipeline.name and Job Type = ADF Pipeline.
  2. Reads properties.parameters from the pipeline definition and creates one Polysync parameter per entry. The ADF parameter type (String, Int, Bool, Object, Array, Float, SecureString) is mapped to a Polysync data type so values are sent with the right JSON token shape at run time.
  3. Copies the pipeline description.

Use Sync Parameters on the Job afterwards to re-read the schema if the pipeline definition changes in ADF — this preserves any Tasks already linked to the Job.

Parameter handling

ADF pipeline parameters are passed as a JSON object in the CreateRun request body. Polysync builds that object from the Job's parameter values, one ADF parameter per Polysync parameter:

{
  "<parameter-name-1>": <value-with-declared-type>,
  "<parameter-name-2>": <value-with-declared-type>
}

Direction (Input / Output / Input&Output)

Each Polysync parameter has a Direction attribute on the Parameters tab of the Job editor. Direction controls whether the parameter is sent to ADF, read back from the response, or both:

Direction Sent in CreateRun request Updated from response
Input ✅ ❌
Output ❌ ✅
Input&Output ✅ ✅
(unset) ✅ ❌

Output-only parameters are deliberately omitted from the request body — this matches ADF's behaviour, where pipeline parameters are inputs only and outputs are returned through pipelineReturnValue.

Data types

The Polysync parameter's Data Type controls how the value is serialized into the JSON request:

Polysync Data Type JSON token sent to ADF
String JSON string "…"
Int JSON number (no quotes)
Float JSON number (no quotes)
Bool JSON true / false
JSON Raw JSON object — embedded verbatim
JsonArray Raw JSON array — embedded verbatim

Always set the right Data Type — sending a number or boolean as a string will cause ADF to fail the run with a type mismatch when the pipeline parameter is declared as Int or Bool.

Output parameters (pipelineReturnValue)

When the pipeline completes with Status = Succeeded, Polysync reads the run's pipelineReturnValue object and writes matching values back into parameters whose Direction is Output or Input&Output:

  • The pipeline must populate pipelineReturnValue via a Set Variable activity at root level, with variableType = Pipeline return value (this is ADF's standard mechanism for returning data from a pipeline).
  • Matching is case-insensitive on the top-level property name. Extra properties in pipelineReturnValue are ignored.
  • If the pipeline does not set a pipelineReturnValue, or the run did not succeed, output parameters are left untouched.

Example pipeline definition:

{
  "activities": [
    {
      "name": "ReturnRowCount",
      "type": "SetVariable",
      "typeProperties": {
        "variableName": "pipelineReturnValue",
        "value": [
          { "key": "rowCount", "value": { "type": "Expression", "content": "@variables('count')" } }
        ]
      }
    }
  ]
}

Polysync setup on the Job:

Parameter Data Type Direction
rowCount Int Output

Execution flow

  1. Polysync resolves the ADF pipeline through the ARM client (SubscriptionId / ResourceGroupName / FactoryName from the Platform), then filters the Job's parameters to Input + Input&Output and posts them to the CreateRun API.

  2. The response runId is captured and the Polysync run is returned immediately with Status = Running and a deep-link Monitor URL.

  3. Status is polled via GET pipelineruns/{runId}?api-version=2018-06-01 and mapped to Polysync's status:

    ADF status Polysync status
    Succeeded / Completed Success
    Failed Failed
    Cancelled Cancelled
    Queued / Pending Starting
    InProgress / Running Running
    (other) Unknown

  4. On Succeeded, output parameters are extracted as described above.

  5. Cancel is supported — Polysync calls POST pipelineruns/{runId}/cancel?api-version=2018-06-01. Runs that are already in a terminal state return 409 and are reported as non-cancellable.

Monitor URL

https://adf.azure.com/en-us/monitoring/pipelineruns/{runId}
  ?factory=/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}
   /providers/Microsoft.DataFactory/factories/{factoryName}

Opens the ADF Studio monitoring view for the specific pipeline run.

Best practices

  • Match each Polysync parameter's Data Type to the ADF pipeline parameter's type exactly — mismatches cause silent string coercion in String parameters and hard failures in typed parameters.
  • Use Object / Array parameters in ADF together with Polysync JSON / JsonArray to pass structured data without escaping headaches.
  • Avoid SecureString pipeline parameters for credentials — store secrets in the linked services' Key Vault references instead, and use Polysync parameters only for non-sensitive run-time values.
  • When you change a pipeline's parameter list in ADF, click Sync Parameters on the Job before the next run rather than recreating the Job — this preserves the Tasks already linked to it.

Troubleshooting

  • PipelineNotFound / 404 on GetPipeline — the External Id on the Job does not match the pipeline name in ADF (names are case-sensitive in the REST API). Re-import the Job from the platform editor.
  • Run fails immediately with a parameter type error — a Polysync Data Type is wider than the ADF parameter type (e.g., Polysync sending a string where ADF declares Int). Edit the parameter on the Parameters tab and set the matching Data Type.
  • Output parameter stays empty after a successful run — confirm the pipeline writes to pipelineReturnValue (Set Variable with variableName = pipelineReturnValue at root), the property name in the return value matches the Polysync parameter name (case-insensitive), and the Polysync parameter's Direction is Output or Input&Output.
  • Authorization failure (401/403) on CreateRun — the identity used by the Platform is missing the Data Factory Contributor role (or equivalent Microsoft.DataFactory/factories/pipelineruns/*/action permissions) on the factory.
  • Cancel returned false — the run was already Succeeded / Failed / Cancelled. Cancellation is only possible while the run is Queued, Pending, InProgress, or Running.