Batch export files to Amazon S3 with curl and AWS CLI

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

Use a small AWS SDK signer to generate pre-signed PUT URLs, then let curl upload each file to Amazon S3. The Bash script below exports the regular files at the top of a directory, including hidden and empty files, and reports failure if any selected file fails. AWS CLI supplies credential checks and an independent download check; aws s3 presign itself signs downloads, not uploads.

Prerequisites

This example targets Linux with Bash, GNU stat, file, curl, AWS CLI v2, and Corepack installed. Use Node.js 24.15 or later in the 24.x line, or Node.js 26.x. These are example prerequisites rather than the AWS SDK’s runtime minimum; Node runs the TypeScript signer without a build step. If AWS CLI is missing, follow its installation guide⁠.

The signer was checked with Node.js 24.15.0, 26.0.0, 26.5.0, and 26.8.1. The complete upload uses AWS SDK 3.1148.0, Yarn 4.12.0, AWS CLI 2.37.10, and curl 8.22.0. The dependency pins below make that setup reproducible; they do not require downgrading an existing CLI installation.

You also need an existing general purpose S3 bucket, its region, and an authenticated AWS profile with access to your destination prefix. Use your organization’s existing login process. The CLI and SDK both honor AWS_PROFILE; session credentials supplied through environment variables must include AWS_SESSION_TOKEN. Do not put credentials in these scripts.

Create a separate project with the two signer dependencies. This block leaves your shell in its original directory and refuses an existing project directory:

(
  set -euo pipefail
  for tool in node corepack aws curl file stat; do
    command -v "$tool" >/dev/null || { echo "Missing prerequisite: $tool" >&2; exit 1; }
  done
  mkdir s3-batch-export
  printf '%s\n' '{"private":true,"type":"module","packageManager":"yarn@4.12.0"}' \
    > s3-batch-export/package.json
  printf 'nodeLinker: node-modules\n' > s3-batch-export/.s3-batch-export.yarnrc.yml
  touch s3-batch-export/yarn.lock
  cd -- s3-batch-export
  YARN_RC_FILENAME=.s3-batch-export.yarnrc.yml corepack yarn add --exact \
    @aws-sdk/client-s3@3.1148.0 @aws-sdk/s3-request-presigner@3.1148.0
)

The empty yarn.lock makes this a separate Yarn project. The named configuration file keeps the installation from inheriting enclosing .yarnrc.yml files. If installation fails after creation, inspect the new directory and retry the same YARN_RC_FILENAME=… corepack yarn add command inside it; do not rerun mkdir or delete an existing project. After successful installation, enter it for the remaining examples:

cd -- s3-batch-export

Understand the components

Amazon S3 is a scalable object store. A pre-signed URL grants time-limited permission to perform one specific kind of S3 request without exposing your AWS credentials to whoever performs it. The signature covers the HTTP method, the bucket and key, the expiry, and any headers you choose to sign, so a URL signed for a GET cannot be replayed as a PUT.

It is not a one-shot token. You can reuse the URL until it expires, and a successful PUT replaces the object at that key. Keep it out of logs, shell tracing, and shared terminal output. AWS describes these permissions and replacement behavior in its presigned upload guide⁠.

The AWS CLI presign command⁠ authorizes GET requests. Its output cannot authorize curl -T. To sign a PUT you need a signer that lets you name the operation, which is what @aws-sdk/s3-request-presigner⁠ does. AWS CLI checks your identity and retrieves uploaded bytes; curl performs the transfer.

Provide the required IAM permissions

The signature itself is computed locally, so no permission is checked while the URL is being minted. The permissions that matter are the signer's own, because S3 evaluates the signing identity's policy when the pre-signed URL is redeemed. Ask the bucket administrator for permission scoped to your destination, for example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::your-bucket-name/exports/*"
    }
  ]
}

Replace your-bucket-name with your bucket and exports/ with your chosen prefix.

Generate pre-signed PUT URLs with the AWS SDK

Save this as presign-put.ts. It prints one URL and exits, which makes it easy to call from a shell loop later:

// presign-put.ts - print a pre-signed URL that authorizes PUT, and only PUT.
import { parseArgs } from 'node:util'

import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

const { values } = parseArgs({
  options: {
    bucket: { type: 'string' },
    key: { type: 'string' },
    region: { type: 'string' },
    'content-type': { type: 'string', default: 'application/octet-stream' },
    'expires-in': { type: 'string', default: '3600' },
  },
})

const { bucket, key, region } = values
if (!bucket || !key || !region) {
  throw new Error('Usage: node presign-put.ts --bucket B --key K --region R [--content-type T]')
}

const expiresIn = Number(values['expires-in'])
// SigV4 caps a pre-signed URL at 7 days. Reject longer values here rather than
// relying on the SDK's generic expiry error; fractional values are rejected too.
if (!Number.isInteger(expiresIn) || expiresIn < 1 || expiresIn > 604800) {
  throw new Error(`--expires-in must be between 1 and 604800 seconds, got: ${values['expires-in']}`)
}

const client = new S3Client({
  region,
  maxAttempts: 1,
  // curl supplies the body later; omit optional checksum metadata here.
  requestChecksumCalculation: 'WHEN_REQUIRED',
})

const url = await getSignedUrl(
  client,
  new PutObjectCommand({ Bucket: bucket, Key: key, ContentType: values['content-type'] }),
  {
    expiresIn,
    // Require the uploader to send the Content-Type used when signing.
    signableHeaders: new Set(['content-type']),
  },
)

console.log(url)

Run it:

node presign-put.ts \
  --bucket your-bucket-name \
  --key exports/localfile.txt \
  --region your-region \
  --content-type text/plain \
  --expires-in 3600

Replace the bucket, key, and region with your values. The URL carries the signature and its expiry in the query string; it contains no secret key, but anyone holding it can write that one object, repeatedly, until it expires, so keep it out of logs.

--expires-in is a ceiling, not a guarantee. If you sign with temporary credentials, from an IAM role, an SSO session or sts:AssumeRole, the URL stops working the moment those credentials expire, which is often well before the expiry you asked for.

Upload files with curl

Use the curl -T flag to specify the local file to upload. Adding --retry makes the upload more resilient to transient network issues. The Content-Type header is not optional here: it is one of the signed headers, so it must match the --content-type you passed to the signer. The subshell keeps strict error handling local to this example, even when pasted into an existing Bash session.

(
  set -euo pipefail

  CONTENT_TYPE=$(file -b --mime-type -- ./localfile.txt)

  # Sign for that exact type, then send that exact type
  URL=$(node presign-put.ts \
    --bucket your-bucket-name \
    --key exports/localfile.txt \
    --region your-region \
    --content-type "$CONTENT_TYPE")

  if status=$(curl --disable -fsS --globoff --retry 3 --retry-delay 2 \
    --connect-timeout 10 --max-time 300 --retry-max-time 300 \
    -T ./localfile.txt \
    -H "Content-Type: $CONTENT_TYPE" \
    -o /dev/null -w '%{http_code}' \
    "$URL") && [[ "$status" =~ ^2[0-9][0-9]$ ]]; then
    echo 'Upload completed.'
  else
    echo 'Upload failed: expected an HTTP 2xx response.' >&2
    exit 1
  fi
)

Replace ./localfile.txt with your file path and the bucket and region with your own values. --disable ignores a caller’s .curlrc; --globoff keeps braces and brackets in filenames literal. -f makes curl exit non-zero on an HTTP error instead of printing S3's XML error body as if it were a success, and -sS keeps the progress meter quiet while still showing real errors. The explicit 2xx check also rejects redirects, which -f alone does not treat as errors. Do not add -L: following a redirect changes the signed request. Each attempt has a 300-second deadline, including data transfer, so a slow valid file can fail. Retry scheduling also stops after 300 seconds; a retry already started can run for another 300 seconds. Inspect the destination before retrying after an uncertain response. The curl manual⁠ explains these transfer, retry, and filename options.

Automate batch file exports

Save this as batch-export.sh beside presign-put.ts. It processes one file at a time, keeps going after an individual failure, and exits nonzero if any selected file fails. It skips directories and symlinks, accepts UTF-8 filenames, and treats an empty directory as a successful batch of zero files. Keep the input directory unchanged during the run.

#!/usr/bin/env bash

set -euo pipefail # Exit on error, undefined variable, or pipe failure
export LC_ALL=C AWS_MAX_ATTEMPTS=1 AWS_PAGER=""
trap 'echo "Batch interrupted; inspect uploaded objects before retrying." >&2; exit 130' INT
trap 'echo "Batch interrupted; inspect uploaded objects before retrying." >&2; exit 143' TERM

BUCKET="your-bucket-name"
EXPIRE=3600                     # URL validity in seconds (1 hour)
FILES_DIR="/path/to/your/files" # Directory containing files to upload
REGION="your-region"            # Your S3 bucket region
PRESIGN="./presign-put.ts"      # The signer from the previous section
PREFIX="${S3_PREFIX:-exports/}" # Use a separate prefix for this batch
TRANSFER_SECONDS="${S3_TRANSFER_SECONDS:-300}"

# --- pre-flight checks ---

if [[ ! -d "$FILES_DIR" || ! -r "$FILES_DIR" || ! -x "$FILES_DIR" ]]; then
  echo "Error: Directory '$FILES_DIR' must exist and be readable/searchable." >&2
  exit 1
fi

if [[ ! -f "$PRESIGN" ]]; then
  echo "Error: Signer '$PRESIGN' not found. See the previous section." >&2
  exit 1
fi

for tool in aws node curl file stat; do
  command -v "$tool" >/dev/null || { echo "Missing prerequisite: $tool" >&2; exit 1; }
done

if [[ ! "$TRANSFER_SECONDS" =~ ^[1-9][0-9]*$ ]]; then
  echo "Error: S3_TRANSFER_SECONDS must be a positive integer." >&2
  exit 1
fi

# Verify the credentials the signer will pick up are present and valid.
if ! caller_arn=$(aws sts get-caller-identity --query Arn --output text 2>/dev/null); then
  echo "Error: AWS credentials are not configured properly or are invalid." >&2
  echo "Check your selected AWS profile or session credentials." >&2
  exit 1
fi
echo "AWS credentials verified for: $caller_arn"

echo "Starting batch export from '$FILES_DIR' to bucket '$BUCKET' in region '$REGION'..."

# --- processing loop ---

failed=0
uploaded=0

# dotglob: a hidden file is an ordinary object to S3, and silently leaving
# .env.example behind is data loss. nullglob: an empty directory must not run
# the loop once with a literal '*'.
shopt -s dotglob nullglob

for file in "$FILES_DIR"/*; do
  filename=${file##*/}

  # Test for a symlink before -f, which follows them: a link inside FILES_DIR
  # would otherwise upload bytes from outside it, under a name from inside it.
  if [[ -L "$file" ]]; then
    echo "  Skipping '$filename': symbolic link."
    continue
  fi

  # Skip directories and anything else that is not a regular file.
  [[ -f "$file" ]] || continue

  echo "Processing '$filename'..."

  # Node's argv decoding is lossy for malformed UTF-8; reject before signing a key.
  if ! printf '%s' "$filename" | node --input-type=module -e \
    'import { readFileSync } from "node:fs"; new TextDecoder("utf-8", {fatal:true}).decode(readFileSync(0))' \
    2>/dev/null; then
    echo "  Error: Filename is not valid UTF-8." >&2
    failed=$((failed + 1))
    continue
  fi

  if [[ ! -r "$file" ]] || ! file_size=$(stat -c%s -- "$file"); then
    echo "  Error: Cannot read or inspect '$filename'." >&2
    failed=$((failed + 1))
    continue
  fi
  # This example limits each single PUT to 5 GiB; use multipart for larger files.
  if ((file_size > 5368709120)); then
    echo "  Skipping '$filename': $((file_size / 1024 / 1024)) MiB exceeds this script’s 5 GiB limit." >&2
    echo "  Use 'aws s3 cp' instead, which switches to multipart automatically." >&2
    failed=$((failed + 1))
    continue
  fi

  if ! content_type=$(file -b --mime-type -- "$file"); then
    echo "  Error: Cannot determine Content-Type for '$filename'." >&2
    failed=$((failed + 1))
    continue
  fi

  echo "  Signing a PUT for '$filename' (Content-Type: $content_type)..."
  if ! url=$(node "$PRESIGN" \
    --bucket "$BUCKET" \
    --key "$PREFIX$filename" \
    --region "$REGION" \
    --content-type "$content_type" \
    --expires-in "$EXPIRE"); then
    echo "  Error: Failed to sign a URL for '$filename'." >&2
    failed=$((failed + 1))
    continue
  fi

  echo "  Uploading '$filename'..."
  # The Content-Type must match the signature. curl -f does not reject redirects;
  # require a completed 2xx upload without replaying a signed body at another URL.
  if status=$(curl --disable -fsS --globoff --retry 3 --retry-delay 2 \
    --connect-timeout 10 --max-time "$TRANSFER_SECONDS" --retry-max-time "$TRANSFER_SECONDS" \
    -T "$file" \
    -H "Content-Type: $content_type" \
    -o /dev/null -w '%{http_code}' \
    "$url") && [[ "$status" =~ ^2[0-9][0-9]$ ]]; then
    echo "  Successfully uploaded '$filename'."
    uploaded=$((uploaded + 1))
  else
    echo "  Error: Failed to upload '$filename'." >&2
    failed=$((failed + 1))
  fi
done

echo "Batch export completed: $uploaded uploaded, $failed failed."

# Surface partial failure to the caller: 'set -e' cannot do this for us, because
# every failure above was deliberately caught so the loop could continue.
if ((failed > 0)); then
  exit 1
fi

Replace the bucket, input directory, and region in the script, then run it from this project:

bash batch-export.sh

For two successful files, the summary is Batch export completed: 2 uploaded, 0 failed. A failed transfer contributes to the failure count even if later files succeed. A missing or unreadable input directory fails before any transfer. S3_PREFIX overrides exports/; include the trailing slash. Names retain spaces, brackets, percent signs, emoji, and newlines. Malformed UTF-8 names are counted as failures because they cannot be mapped reliably through Node’s string arguments.

This script caps files at 5 GiB and uses the same 300-second transfer and retry limits as the single upload. Set S3_TRANSFER_SECONDS higher for slower transfers. Each file has at most four attempts, including the initial request. A retry sends the whole file again; it does not resume an interrupted PUT. If interrupted, the script returns nonzero without a completion summary. Completed objects remain, and a lost response can leave a committed object, so inspect the prefix before restarting.

Uploads replace existing objects with the same key. Use a fresh prefix to separate batches; nested directories are skipped rather than flattened. Repeating a batch replaces its successful keys and does not delete unrelated keys.

Verify a known file independently by downloading it and comparing bytes. This creates a fresh local directory and leaves existing files alone:

(
  set -euo pipefail
  command -v cmp >/dev/null || { echo 'Missing prerequisite: cmp' >&2; exit 1; }
  mkdir s3-download-check || exit $?
  aws s3api get-object --bucket your-bucket-name --key exports/localfile.txt \
    --region your-region s3-download-check/localfile.txt >/dev/null || exit $?
  cmp -- ./localfile.txt s3-download-check/localfile.txt || exit $?
  echo 'Downloaded bytes match.'
)

Compare every expected file when checking a complete batch. A success count alone cannot detect an omitted file or a wrong object. If the download fails, inspect and remove only this check’s partial download directory before retrying the block.

Apply security best practices

Scope the signing identity’s access to the batch prefix and choose a URL lifetime long enough for the transfer. Temporary credentials can expire earlier than --expires-in, as AWS explains in its presigned URL guide⁠. The scripts do not change bucket policies or encryption settings. For a bucket using a customer managed KMS key, confirm the administrator has granted the required key access before uploading.

Signing Content-Type binds that header, not the file’s contents or safety. Anyone holding the URL can submit different bytes with that same header. The independent download check protects your verification from accidentally comparing the wrong object; it does not turn the URL into a content validation policy.

Troubleshoot common issues

• Credential check fails: Renew your existing login and check AWS_PROFILE or session environment variables. A successful aws sts get-caller-identity identifies the CLI’s caller; it does not prove that S3 permits a write. The signer also resolves its own credentials.
• Upload returns HTTP 403: Check the bucket region, key, URL expiry, signing permissions, and the exact signed Content-Type. The upload block deliberately discards S3’s response body, so its generic failure cannot identify which of those causes applies. Do not log the signed URL while diagnosing it. AWS’s upload troubleshooting guidance⁠ covers these checks.
• Transfer times out: The deadline also applies to valid files below the size limit. Inspect the object, adjust the transfer deadline if appropriate, and rerun with a fresh URL. Authentication errors are not fixed by adding retries.
• File is skipped as too large: This script enforces a 5 GiB single-PUT ceiling. Use a multipart client for a larger file.

For larger files, aws s3 cp⁠ handles multipart uploading. This is a separate credentialed transfer, not a pre-signed curl request:

aws s3 cp ./large_file.zip s3://your-bucket-name/exports/large_file.zip --region your-region

Conclusion

Use a fresh prefix for your first file export, download known objects to check their bytes, and check the script’s exit status before treating the batch as complete. This open source workflow keeps signing and transfer separate while making partial failures visible.

If your files already pass through Transloadit processing, the 🤖 /s3/store Robot can export Step results to S3 within the same Assembly.