Skip to main content

Multipart upload

Multipart upload lets you split a large object into parts, upload them in parallel (or out of order, with retries), and then assemble the final object server-side. Filebase's multipart implementation is wire-compatible with AWS S3 — every SDK and tool that knows how to do multipart upload against AWS works against Filebase.

When to use it​

Object sizeRecommendation
Under 100 MBSingle PutObject is fine
100 MB – 5 GBUse multipart for parallelism and resumability
Over 5 GBRequired — single PutObject is capped at 5 GB
Over 1 TBAllowed up to 5 TB, but consider whether splitting at the application layer makes sense

The protocol​

Three calls:

  1. CreateMultipartUpload — gives you an UploadId that ties the parts together.
  2. UploadPart — called once per part. Each part is 5 MiB to 5 GiB. The last part can be smaller. Up to 10,000 parts.
  3. CompleteMultipartUpload — assembles the parts in order and persists the final object.

If anything goes wrong, call AbortMultipartUpload to discard the upload and free the part storage.

High-level uploaders​

You almost never write the three calls yourself — every modern AWS SDK has a high-level uploader that switches to multipart automatically.

AWS CLI​

aws --endpoint https://s3.filebase.io s3 cp ./large-file.bin s3://my-bucket/

The CLI uses multipart automatically once the file exceeds 8 MB.

To tune the threshold or part size:

aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 64MB

AWS SDK for JavaScript v3​

import { Upload } from '@aws-sdk/lib-storage';
import { S3Client } from '@aws-sdk/client-s3';
import { createReadStream } from 'node:fs';

const s3 = new S3Client({
  endpoint: 'https://s3.filebase.io',
  region: 'auto',
  credentials: { accessKeyId: KEY, secretAccessKey: SECRET },
});

const upload = new Upload({
  client: s3,
  params: {
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    Body: createReadStream('./large-file.bin'),
  },
  partSize: 1024 * 1024 * 8, // 8 MB parts
  queueSize: 4, // 4 concurrent parts
});

upload.on('httpUploadProgress', (p) => {
  console.log(`${p.loaded} / ${p.total} bytes`);
});

await upload.done();

AWS SDK for Python (boto3)​

s3.upload_file(
    Filename='./large-file.bin',
    Bucket='my-bucket',
    Key='large-file.bin',
    Config=boto3.s3.transfer.TransferConfig(
        multipart_threshold=8 * 1024 * 1024,
        multipart_chunksize=8 * 1024 * 1024,
        max_concurrency=4,
    ),
)

AWS SDK for Go v2​

import "github.com/aws/aws-sdk-go-v2/feature/s3/manager"

uploader := manager.NewUploader(client, func(u *manager.Uploader) {
    u.PartSize = 8 * 1024 * 1024 // 8 MB
    u.Concurrency = 4
})

f, _ := os.Open("./large-file.bin")
defer f.Close()

_, err := uploader.Upload(ctx, &s3.PutObjectInput{
    Bucket: aws.String("my-bucket"),
    Key:    aws.String("large-file.bin"),
    Body:   f,
})

Low-level multipart, by hand​

If you need fine-grained control (for example, uploading parts from disparate sources, or retrying in unusual ways), use the three-call protocol directly:

AWS SDK for JavaScript v3
import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand,
} from '@aws-sdk/client-s3';

// 1. Start the upload
const { UploadId } = await s3.send(
  new CreateMultipartUploadCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    ContentType: 'application/octet-stream',
  }),
);

// 2. Upload each part (5 MB minimum, except the last)
const parts = [];
for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
  const body = readPart(partNumber);
  const { ETag } = await s3.send(
    new UploadPartCommand({
      Bucket: 'my-bucket',
      Key: 'large-file.bin',
      UploadId,
      PartNumber: partNumber,
      Body: body,
    }),
  );
  parts.push({ ETag, PartNumber: partNumber });
}

// 3. Finish
await s3.send(
  new CompleteMultipartUploadCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    UploadId,
    MultipartUpload: { Parts: parts },
  }),
);

// On error, abort to free part storage
// await s3.send(new AbortMultipartUploadCommand({ Bucket, Key, UploadId }));

Handling failures​

If UploadPart fails, retry the same part with the same PartNumber and UploadId. Parts can be uploaded in any order — CompleteMultipartUpload reassembles them by part number.

If your process crashes mid-upload, Filebase keeps the parts around indefinitely. List in-progress uploads with ListMultipartUploads and either resume them or abort them with AbortMultipartUpload. Aborting frees the storage; not aborting and not completing leaves orphaned parts that count toward your storage quota.

Browser uploads with pre-signed URLs​

For direct browser-to-Filebase multipart uploads, generate a pre-signed URL for each part. See browser uploads.

ETag for multipart objects​

Multipart object ETags follow AWS's composite-hash format: <md5-of-md5s>-<part-count>. Tools that integrity-check by ETag (e.g., the AWS CLI when downloading) handle multipart ETags automatically.

SHA256 checksum for multipart objects​

Filebase recomputes the SHA256 over the reassembled object's bytes during CompleteMultipartUpload, so the x-amz-checksum-sha256 returned on subsequent GET/HEAD responses is a full-object hash — the same value as sha256sum on the locally-reconstructed file. The accompanying x-amz-checksum-type: FULL_OBJECT header tells SDKs not to try to verify it as a composite-of-part-hashes (which would otherwise be the default given the composite-shaped ETag).

See object checksums for examples of reading and verifying the value.

What's next​