Connect Amazon S3 to Permute

What you need

• An S3 bucket containing CSV, TSV, pipe-separated, semicolon-separated, JSON, NDJSON, or Parquet objects.
• Permission to create an IAM role and policy in the bucket's AWS account.
• A Permute workspace where you can create connectors.

1. Create a read-only IAM role

Create a role in the AWS account that owns the bucket. Its trust policy must allow Permute's production S3 assumer role and require your external ID (the external ID prevents a confused-deputy authorization failure).

Permute issues a different external ID for each connector. Replace <YOUR_UNIQUE_EXTERNAL_ID> below with that connector's value. Don't reuse a value from another connector. This value is a matching condition for role assumption (it isn't an S3 key, AWS resource ID, or password). The production Permute principal is arn:aws:iam::474668386399:role/PermuteS3ConnectorAssumer.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::474668386399:role/PermuteS3ConnectorAssumer"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<YOUR_UNIQUE_EXTERNAL_ID>"
        }
      }
    }
  ]
}

2. Grant access to selected prefixes

Attach a policy to the new role. Replace the bucket name and prefixes with the paths Permute should read (the example keeps unrelated bucket contents outside the role).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListSelectedPrefixes",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<BUCKET_NAME>",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "exports/*",
            "snapshots/*",
            "tables/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadSelectedObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET_NAME>/exports/*",
        "arn:aws:s3:::<BUCKET_NAME>/snapshots/*",
        "arn:aws:s3:::<BUCKET_NAME>/tables/*"
      ]
    }
  ]
}

Keep s3:GetObjectVersion when bucket versioning is enabled. For objects encrypted with a customer-managed KMS key, also grant kms:Decrypt on that key and allow the role in the KMS key policy, then enter the same key ARN in Permute. AWS-managed S3 encryption needs no extra KMS permission.

3. Define table mappings

Add one mapping for each Permute table. Enter the table name, S3 file path, file format, and file layout. Patterns are bucket-relative (don't include the bucket name): * matches one path segment, ** matches nested paths, and ? matches one character.

Storage layouts

Mapping fields

• Table name starts with a letter and contains only letters, numbers, and underscores. _source_files is reserved.
• S3 file path selects objects relative to the configured bucket and may contain wildcards.
• File format is CSV, TSV, pipe-separated, semicolon-separated, JSON, NDJSON, or Parquet.
• File layout specifies one record per file, multiple records per file, or one full snapshot file. Snapshot layouts also specify whether the newest file is determined by a date in its path or its S3 last-modified time.

4. Connect and sync

1. Open your Permute workspace and select Connectors.
2. Select Amazon S3.
3. Copy the Permute role ARN and server-issued external ID into the role trust policy.
4. Enter the bucket name, AWS region, customer role ARN, and any customer-managed KMS key ARNs.
5. Add each table and configure its file path, format, and layout.
6. Create the connector. Permute discovers the schemas and starts the initial sync.

How sync works

Each successful sync materializes the current state selected by each mapping and replaces the connector's prior BlockDocument tables. One-record and multiple-record layouts combine every current match. A removed object disappears on the next successful sync; a new snapshot becomes current only when its configured ordering wins.

Versioned objects are pinned to their S3 version while Permute reads them. For unversioned objects, Permute checks the ETag, size, and modification time again after reading and fails the sync if an object changed. A failed refresh leaves the prior successful materialization available.

Lineage

Permute adds three reserved columns to each materialized row:

• _source_file: the complete s3:// path.
• _source_version: the S3 version ID, or the ETag for an unversioned object.
• _source_row: the row number within that source object.

The connector also publishes _source_files, a manifest containing each selected object, its mapping, version, modification time, and size.

Limits and boundaries

• V1 supports the commercial AWS partition. AWS GovCloud and AWS China roles aren't supported.
• One mapping can scan at most 10,000 current objects under the literal prefix before its first wildcard. Use a narrower leading prefix for larger buckets.
• CSV, TSV, pipe-separated, and semicolon-separated files must contain a header row.
• A one-record-per-file mapping fails if any matched object produces zero rows or more than one row. JSON arrays and other multi-row inputs belong in a multiple-records or snapshot mapping.
• JSON can contain one object, an array of objects, or another shape DuckDB can read as rows.
• Parquet files combine columns by name. Hive path partitions aren't inferred as columns.
• Iceberg and Delta tables should connect through their catalog so transaction metadata remains authoritative.
• Input columns named _source_file, _source_version, or _source_row are rejected.

Troubleshooting

AWS references