Documentation

AWS Glue

AWS Glue is the AWS managed ETL service. Polysync uses the official AWS SDK for .NET v4 (AWSSDK.Glue and AWSSDK.SecurityToken) to discover Glue jobs, run them with parameter values, poll their status, and surface a direct link to the AWS Glue Studio console for monitoring.

Required attributes

  • Region — the AWS region containing the Glue catalog (e.g., us-east-1, ap-southeast-2). All Glue API calls are region-scoped.

Optional platform-level defaults

These are applied when a run starts, with this precedence: job-level override → platform default → the Glue job definition's own value.

  • Worker Type — Glue worker type for the run (e.g. Standard, G.1X, G.2X). Leave blank to use the job definition's setting.
  • Number Of Workers — number of workers for the run (positive integer). Leave blank to use the job definition's setting.
  • Version — the Glue version (e.g. 4.0). Sent to the run as the --glue-version argument; the authoritative version normally lives on the job definition.
  • Timeout (minutes) — run timeout in minutes (positive integer). Leave blank to use the job definition's setting.
  • Max Retries — shown for editor intent only: Glue reads the retry count from the job definition and ignores this value on the run-start API.

Authentication methods

  • Web Identity Federation ⭐ (recommended for Polysync SaaS) — Polysync exchanges its Microsoft Entra ID workload identity token for short-lived AWS credentials via sts:AssumeRoleWithWebIdentity. No long-lived secrets stored. Required attributes: Role ARN.
    • In AWS, create an IAM Identity Provider (OIDC) trusting Polysync's Entra ID issuer (https://login.microsoftonline.com/<polysync-tenant-id>/v2.0) with audience sts.amazonaws.com.
    • Create an IAM role whose trust policy allows sts:AssumeRoleWithWebIdentity from that provider with a condition on the Polysync workload identity's sub/oid claim.
  • Access Key — Provide Access Key Id, Secret Access Key, and optionally Session Token. Simplest, but the secret must be rotated and stored in a Secret Vault.
  • Assume Role — Provide a bootstrap Access Key Id and Secret Access Key, plus the Role ARN to assume. The bootstrap user only needs sts:AssumeRole on the target role; the assumed role holds the Glue permissions.
  • Instance Profile — Uses the host EC2/ECS instance profile. Only viable when Polysync is deployed inside AWS; not applicable to the standard Azure SaaS deployment.

IAM permissions checklist

The role / user used to call Glue must hold (at minimum):

  • glue:ListJobs — discover available jobs.
  • glue:GetJobs, glue:GetJob — read job definitions, including parameter defaults.
  • glue:StartJobRun — submit a run with arguments.
  • glue:GetJobRun — poll status.

Cancellation of an in-flight Glue run is not supported by this provider — no stop permission is required.

Plus any IAM permissions Glue itself needs to access the script's S3 location and the job's data sources (typically defined on the Glue job's IAM role, not the Polysync caller).

Supported jobs

  • AWS Glue Job — submits a Glue job run with parameter arguments.

Troubleshooting

  • Web Identity Federation InvalidIdentityToken — check that the IAM Identity Provider's thumbprint matches login.microsoftonline.com, the audience is sts.amazonaws.com, and the role trust policy allows the Polysync workload identity's sub / oid.
  • AccessDenied on listing jobs — the caller's IAM principal is missing glue:ListJobs / glue:GetJobs in the target region.