Multipart upload

​

Multipart upload

​

Benefits of Multipart Upload

• Efficient: You can upload large files more quickly and efficiently by splitting them into smaller parts.
• Resumable: If an upload of a part fails, you can retransmit that part without affecting other parts.
• Parallel Uploads: Parts can be uploaded independently, allowing you to upload multiple parts in parallel to reduce the overall time taken.

​

Multipart Upload Workflow

1. Initiate Multipart Upload: You can initiate a multipart upload from any API Route, Workflow or Function using the File Storage Action. This will initiate the multipart upload and return the url where the separate parts have to be sent to.
2. Upload Parts: Upload each part using the upload url returned in the first step and the respective part number. Each part must be between 5 MB and 5 GB in size, except the last part, which can be smaller.
3. Complete Multipart Upload: After uploading all parts, complete the multipart upload to assemble the object from the parts.
4. Abort Multipart Upload: If you don’t want to finish your multipart upload, you should abort it. Otherwise the parts that have already been uploaded will occupy space in your bucket.

​

Initiate Multipart Upload

​

Upload Parts

​

Step-by-Step Instructions

1. Get the File:
   The example below uses a simple HTML file input element.
   Example: const file = document.getElementById('fileInput').files[0]
2. Define Chunk Size:
   The chunk size defines the size of each file part to be uploaded.
   It needs to be at least 5MB. We recommend a chunk size of 10 MB.
   Example: const chunkSize = 10 * 1024 * 1024; // 10 MB
3. Define Total Parts:
   Determine the total number of parts by dividing the file size by the chunk size and rounding up.
   Example: const totalParts = Math.ceil(file.size / chunkSize);
4. Initialize an Array to Store Parts Metadata:
   Create an array to store the part number and ETag for each uploaded part. These will later be needed for completing the multipart upload
5. Loop Over and Slice the File:
   Loop through the file, slicing it into chunks based on the chunk size.
   Use file.slice(start, end) to create each chunk.
6. Create FormData and Append Chunk and Part ID:
   Create a FormData object for each chunk.
   Append the chunk and the part ID to the FormData object.
7. Send a POST request with the FormData to the presigned URL:
   The example below uses the fetch API, but you can use any other method to send the request, such as Axios or XMLHttpRequest.
   The url that the request is sent to, is the url that you receive in the Initiate Multipart Upload step.
8. Retrieve the ETag from the response headers & update parts array:
   Push an object containing the PartNumber and ETag into the parts array.
9. Error handling:
   The example below uses a simple check if the response was ok and prints the error message to the console.
   You can adjust this to your needs and add a custom retry logic if needed.

const file = document.getElementById(`fileInput`).files[0]

const url = 'https://upload.fastgen-storage.com/multipart-upload/9+hSeDIyK8d5g2mjP6U5oFglvGMcaGhz6G2P3qH+SWkL3CHq9kWftmNM7aQsxcCyvI85nujFEii3MwNnZp6Nunq2wegjPTvqKc/CWKT9WHGvp2jw1//4jWPwhvxREt5Auqtm1SjP4SvhT0c7V+pjBM8J0WVyH72FQ2/cuC/kv0Xh4rLgS9epR6H+DJ8mrTuf1OboeGmOxqJSaAQ3+V7//TQdKtNvfi+S8BEGCxg0eVQ8U026oR4ejBWFTpSWsPHBaU+umjiY0dgHVdyUuQej/1TO4q+U5vhchGQxbxZ5jD+0iRegPUzmPGXyH6JzSBfNJtlcdRNtSW+9Es7Cdo61XfuPVwpa2xsdAdElVGpN3kXdfvR833Di+u1MWehEzeGmzuSy/llaIv0Ztbdq0tz9PrlsPT1Xot6c9jVA4cLK3fMnglRCBTnnPWe3ZWKDK0L7mQrBQ/fc11pq/faldFnup/6A22v5Iwe7G5sSsgHdJPgnst+pjX2m01BpXULnp0GCmys5NEuxMw/Rxf49e1htlGPrESt+x3aMLL8FqRGIOwpKWpLZvnmwq2GkqOuYmpNOn8DaZF2h1xQ7QOnwF5bVfBE7RSdP3VypBwJf/AFT1faWu9ydLHU0jGw51pdS3zBKS7GQ5wzN+VPi2xj82V9qZw=='

const chunkSize = 10 * 1024 * 1024 // 10 MB
const totalParts = Math.ceil(file.size / chunkSize)

const parts = []

for (let partId = 1; partId <= totalParts; partId++) {
  const start = (partId - 1) * chunkSize
  const end = Math.min(file.size, partId * chunkSize)
  const chunk = file.slice(start, end)

  try {
    const formData = new FormData()
    formData.append('file', chunk)
    formData.append('partId', partId.toString())

    const response = await fetch(url, {
      method: 'POST',
      body: formData
    })

    const etag = await response.headers.get('etag')
    parts.push({ PartNumber: partId, ETag: etag })

    if (!response.ok) {
      console.error(await response.text())
      return
    }
  } catch (err) {
    console.error('catch error: ', err)
  }
}

​

Complete Multipart Upload

const completeUploadUrl = url + '/complete'
await fetch(completeUploadUrl, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    parts,
    size: file.size
  })
})

​

Abort Multipart Upload

async function abortUpload(url: string) {
  return await fetch(url, {
    method: 'DELETE'
  })
}