Documentation

AWS Batch

AWS Batch is the AWS managed batch-computing service that runs containerized jobs on EC2, Fargate, or EKS compute environments fronted by job queues. Polysync integrates with AWS Batch using the official AWS SDK for .NET v4 (AWSSDK.Batch and AWSSDK.SecurityToken). Polysync can list job definitions, submit jobs to a queue with parameter substitution, poll job status, cancel queued or running jobs, and surface a direct link to the AWS Batch console for monitoring.

Required attributes

  • Region — the AWS region in which the Batch compute environments, queues, and job definitions are deployed (e.g., us-east-1, eu-west-1). All Batch API calls are region-scoped.

Optional platform-level defaults

  • Default Job Queue — the name or ARN of the Batch job queue used when a job does not specify its own queue. Every SubmitJob call requires a queue, so this default avoids per-job configuration when all jobs target the same queue.

Authentication methods

  • Web Identity Federation ⭐ (recommended for Polysync SaaS) — Polysync exchanges its Microsoft Entra ID workload identity token for short-lived AWS credentials via sts:AssumeRoleWithWebIdentity. No long-lived secrets stored. Required attributes: Role ARN.
    • In AWS, create an IAM Identity Provider (OIDC) trusting Polysync's Entra ID issuer (https://login.microsoftonline.com/<polysync-tenant-id>/v2.0) with audience sts.amazonaws.com.
    • Create an IAM role whose trust policy allows sts:AssumeRoleWithWebIdentity from that provider with a condition on the Polysync workload identity's sub / oid claim.
  • Access Key — Provide Access Key Id, Secret Access Key, and optionally Session Token. Simplest, but the secret must be rotated and stored in a Secret Vault.
  • Assume Role — Provide a bootstrap Access Key Id and Secret Access Key, plus the Role ARN to assume. The bootstrap user only needs sts:AssumeRole on the target role; the assumed role holds the Batch permissions.
  • Instance Profile — Uses the host EC2/ECS instance profile. Only viable when Polysync is deployed inside AWS.

IAM permissions checklist

The role / user used by Polysync needs (at minimum):

  • batch:DescribeJobQueues — connectivity test and queue lookup.
  • batch:DescribeJobDefinitions — discover job definitions (Polysync filters by Status=ACTIVE).
  • batch:SubmitJob — submit jobs (scope to specific queue / definition ARNs in production).
  • batch:DescribeJobs — poll job status.
  • batch:TerminateJob — cancel queued or running jobs.
  • iam:PassRole (on the job definition's job role ARN, if one is declared) — required for Batch to launch containers under that role.

The job definition's own job role / execution role needs whatever permissions its container requires (S3, ECR pulls, CloudWatch Logs, etc.) — these are defined on the role attached to the job definition, not on the Polysync caller.

Job discovery

Polysync calls DescribeJobDefinitions with Status="ACTIVE" (paginated via NextToken). The API returns every active revision of every definition; Polysync deduplicates by name and keeps only the highest revision per name so each definition shows up once in the editor.

Each Job's ExternalId is the Job Definition Name (not the ARN). At submit time, AWS Batch resolves a bare definition name to the latest ACTIVE revision automatically — this matches the natural Batch developer workflow where publishing a new revision automatically rolls forward consumers. To pin a specific revision, set the Job's ExternalId to <name>:<revision> or the full ARN.

Parameter conventions

  • Input parameters are passed as Dictionary<string, string> to SubmitJob. AWS Batch substitutes them into the Job Definition's Ref::<name> placeholders in the container command or environment variables. All values are sent as strings.
  • Output parameters are not supported. DescribeJobs returns status, status reason, and the container exit code — there is no structured output payload. Job artifacts live in the locations the container writes them (S3, EFS, FSx, CloudWatch Logs).

Execution flow

  1. ExecutePipelineAsync resolves the effective job queue (job override → platform default), generates a unique JobName (Polysync-<guid>, max 128 chars), builds the parameter dictionary, applies optional fair-share controls (ShareIdentifier, SchedulingPriorityOverride), and calls SubmitJob.
  2. The provider returns a PipelineRun with the Batch JobId as the RunId — Batch accepts the JobId directly on DescribeJobs and TerminateJob.
  3. GetPipelineRunStatusAsync calls DescribeJobs(Jobs=[jobId]) and maps the status:
    • SUBMITTED / PENDING / RUNNABLE / STARTING / RUNNING → Running
    • SUCCEEDED → Success
    • FAILED with a StatusReason containing "Cancelled by Polysync" or "Terminated" → Cancelled
    • FAILED (any other reason) → Failed (StatusReason and container exit code surfaced on the run message)
  4. CancelPipelineRunAsync calls TerminateJobAsync(JobId, Reason="Cancelled by Polysync"). TerminateJob works on both queued (SUBMITTED/PENDING/RUNNABLE) and running (STARTING/RUNNING) jobs — Polysync does not need to differentiate by current state.

Monitor URL

https://{region}.console.aws.amazon.com/batch/home?region={region}#jobs/detail/{jobId}

This deep-links into the AWS Batch console for the specific job, where you can see the assigned compute environment, container details, CloudWatch Logs link, and the job's state-transition history.

Fair-share scheduling

When the target job queue uses a fair-share scheduling policy, jobs may be required to declare a ShareIdentifier and optionally override the policy's scheduling priority. Polysync exposes both as per-job attributes:

  • Share Identifier — string tag bucketing this job into a fair-share group.
  • Scheduling Priority Override — integer priority within the share (higher = sooner).

If the queue does not use a fair-share scheduler, leave both blank.

Troubleshooting

  • ClientException: Queue does not exist — the job queue name or ARN supplied on the platform default or the job override does not exist in this region.
  • ClientException: Job definition X does not exist — confirm the definition name is correct and has at least one ACTIVE revision (DescribeJobDefinitions --status ACTIVE --job-definition-name X).
  • AccessDeniedException on SubmitJob — the caller is missing batch:SubmitJob on the queue/definition ARNs, or is missing iam:PassRole on the definition's job role.
  • Jobs stuck in RUNNABLE — the compute environment cannot satisfy the job's resource requirements (vCPU/memory/GPU). Check the compute environment's max vCPU, the instance types, and any service quotas (EC2, Fargate vCPU limits).
  • Job lands in FAILED immediately with Essential container in task exited — the container's command failed at startup. Inspect the CloudWatch Logs stream linked from the job detail page; the most common causes are missing parameters (Ref::X not supplied), missing IAM permissions inside the container, or an incorrect container image.
  • Web Identity Federation InvalidIdentityToken — check that the IAM Identity Provider's thumbprint matches login.microsoftonline.com, the audience is sts.amazonaws.com, and the role trust policy allows the Polysync workload identity's sub / oid.
  • No structured output parameters — this is by design. Use the container's output sink (S3, CloudWatch Logs, EFS) to retrieve job results.

Scope notes

This provider integrates AWS Batch single jobs — submitting one Batch job definition per Polysync job. Multi-node parallel jobs and array jobs are supported transparently by AWS Batch itself (declared inside the job definition), but Polysync does not yet expose array-job size overrides or per-child status as first-class concepts; an array job appears as a single Polysync run whose status reflects the parent job's aggregate state.