Skip to main content

Multipart upload for large files

For large files, multipart upload is dramatically more reliable than a single PUT:

  • Parts upload in parallel — much higher throughput.
  • A failed part retries individually — no need to restart the whole transfer.
  • Required for objects larger than 5 GB.

Every modern AWS SDK and the AWS CLI switch to multipart automatically. This recipe shows the easy path for the common SDKs and the full low-level protocol when you need fine-grained control.

Easy mode​

AWS CLI​

The AWS CLI auto-detects file size and switches to multipart at 8 MB by default:

aws --endpoint https://s3.filebase.io s3 cp ./video.mp4 s3://my-bucket/

Tune the threshold and chunk size globally:

aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 64MB
aws configure set default.s3.max_concurrent_requests 16

AWS SDK for JavaScript v3​

import { Upload } from '@aws-sdk/lib-storage';
import { createReadStream } from 'node:fs';

const upload = new Upload({
  client: s3,
  params: {
    Bucket: 'my-bucket',
    Key: 'video.mp4',
    Body: createReadStream('./video.mp4'),
  },
  partSize: 8 * 1024 * 1024, // 8 MB parts
  queueSize: 4,              // 4 in-flight uploads
});

upload.on('httpUploadProgress', (p) => {
  console.log(`${p.loaded} / ${p.total}`);
});

await upload.done();

AWS SDK for Python (boto3)​

from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,
    multipart_chunksize=8 * 1024 * 1024,
    max_concurrency=4,
)

s3.upload_file('./video.mp4', 'my-bucket', 'video.mp4', Config=config)

AWS SDK for Go v2​

import "github.com/aws/aws-sdk-go-v2/feature/s3/manager"

uploader := manager.NewUploader(client, func(u *manager.Uploader) {
    u.PartSize = 8 * 1024 * 1024
    u.Concurrency = 4
})

f, _ := os.Open("./video.mp4")
defer f.Close()

uploader.Upload(ctx, &s3.PutObjectInput{
    Bucket: aws.String("my-bucket"),
    Key:    aws.String("video.mp4"),
    Body:   f,
})

Manual multipart, by hand​

Write the three calls yourself when you need:

  • To upload parts from many sources (e.g. multipart browser upload, distributed workers).
  • Custom retry logic.
  • Resumable uploads with checkpoints.
AWS SDK for JavaScript v3
import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand,
} from '@aws-sdk/client-s3';

async function multipartUpload(
  bucket: string,
  key: string,
  parts: Buffer[],
) {
  // 1. Start the upload
  const { UploadId } = await s3.send(
    new CreateMultipartUploadCommand({
      Bucket: bucket,
      Key: key,
      ContentType: 'video/mp4',
    }),
  );

  if (!UploadId) throw new Error('No UploadId');

  try {
    // 2. Upload each part (5 MB minimum, except the last)
    const uploaded: { ETag: string; PartNumber: number }[] = [];

    for (let i = 0; i < parts.length; i++) {
      const partNumber = i + 1;
      const { ETag } = await s3.send(
        new UploadPartCommand({
          Bucket: bucket,
          Key: key,
          UploadId,
          PartNumber: partNumber,
          Body: parts[i],
        }),
      );
      if (!ETag) throw new Error(`Part ${partNumber} returned no ETag`);
      uploaded.push({ ETag, PartNumber: partNumber });
    }

    // 3. Complete
    await s3.send(
      new CompleteMultipartUploadCommand({
        Bucket: bucket,
        Key: key,
        UploadId,
        MultipartUpload: { Parts: uploaded },
      }),
    );
  } catch (err) {
    // 4. On error, abort to free part storage
    await s3.send(
      new AbortMultipartUploadCommand({
        Bucket: bucket,
        Key: key,
        UploadId,
      }),
    );
    throw err;
  }
}

Resumable uploads​

The full multipart protocol naturally supports resume:

  1. On the first attempt, CreateMultipartUpload and persist the UploadId somewhere (database, local file).
  2. As parts succeed, persist the (PartNumber, ETag) pair.
  3. On retry after a crash:
    • Read the persisted UploadId and completed parts.
    • Call ListParts(UploadId) to confirm what's actually uploaded.
    • Skip already-uploaded parts; resume from the next.
    • Call CompleteMultipartUpload with the full list.

Don't forget cleanup: if the user cancels permanently, call AbortMultipartUpload to free part storage. Otherwise orphaned parts count toward your storage quota.

Choosing part size​

File sizeRecommended part size
< 100 MBSingle PUT, no multipart
100 MB – 1 GB8 MB
1 GB – 10 GB16 MB
10 GB – 100 GB64 MB
100 GB – 1 TB128 MB
> 1 TB256 MB

Constraints:

  • 5 MiB minimum per part (except the last).
  • 5 GiB maximum per part.
  • 10,000 parts maximum per upload.

For the largest objects, larger parts reduce overhead. For smaller objects, smaller parts give finer-grained retry.

Cleaning up failed uploads​

Orphan multipart uploads count toward your storage. List them periodically:

aws --endpoint https://s3.filebase.io s3api list-multipart-uploads --bucket my-bucket

Abort each by ID and key:

aws --endpoint https://s3.filebase.io s3api abort-multipart-upload \
  --bucket my-bucket \
  --key video.mp4 \
  --upload-id <id>

What's next​