Dataflow Flex Template Job

The Dataflow Flex Template job type launches an Apache Beam Flex Template (a container image specified by a GCS-hosted spec file) on Google Dataflow via POST projects/{project}/locations/{location}/flexTemplates:launch. The discovery artefact is an existing Dataflow Job that was previously launched from a Flex Template — Polysync stores its Job ID in the External Id and the container spec GCS path in the Dataflow Flex Template Container Spec Path job attribute.

This job type is supported on the Google Dataflow platform.

Required job fields

External Id — a representative Dataflow Job ID launched from the template (used to seed the parameter list).
Job Type — Dataflow Flex Template (set automatically on import).
Dataflow Flex Template Container Spec Path (job attribute) — required; the gs://…/specs/… path of the Flex Template container spec.

Job discovery

Same as Dataflow Classic Template — the provider classifies jobs as Flex when their metadata shows a container userAgent and a Flex template type label.

Parameter handling

Polysync sends Input + Input&Output parameters nested under launchParameter.parameters:

{
  "launchParameter": {
    "jobName": "<polysync-name>-<timestamp>",
    "containerSpecGcsPath": "<Dataflow Flex Template Container Spec Path>",
    "parameters": {
      "<param-1>": "<value-as-string>",
      "<param-2>": "<value-as-string>"
    },
    "environment": {}
  }
}

Direction	Sent in `launchParameter.parameters`	Updated from response
`Input`	✅	❌
`Output`	❌	⚠ Via `sdkPipelineOptions`
`Input&Output`	✅	⚠ Via `sdkPipelineOptions`

Values are sent as strings (no Dataflow-side type coercion at submit time — Flex Template metadata controls coercion).

Output parameters

Same as Dataflow Classic Template — values come from environment.sdkPipelineOptions on each status poll.

Execution flow

Same as Dataflow Classic Template once the job is launched — same status polling endpoint, same JOB_STATE_* mapping, same cancel mechanism.

Monitor URL

https://console.cloud.google.com/dataflow/jobs/{location}/{jobId}?project={project}

Best practices

Maintain a single container spec per Flex Template and version it alongside your image tag.
Declare your template's parameters in the spec's metadata.parameters block so the Polysync importer picks up the correct names and defaults.

Troubleshooting

HTTP 400 with BAD_TEMPLATE — the container spec path is invalid or the spec JSON is malformed. Validate with gcloud dataflow flex-template build.
Job stays Starting — the worker is pulling the container image for the first time; subsequent runs should be faster.

Documentation