Multipart upload

​

Multipart upload

Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object’s data. You can upload these object parts independently and in any order. After all parts of your object are uploaded, the multipart upload assembly process puts all the object parts together into a single object.

​

Benefits of Multipart Upload

• Efficient: You can upload large files more quickly and efficiently by splitting them into smaller parts.
• Resumable: If an upload of a part fails, you can retransmit that part without affecting other parts.
• Parallel Uploads: Parts can be uploaded independently, allowing you to upload multiple parts in parallel to reduce the overall time taken.

​

Multipart Upload Workflow

1. Initiate Multipart Upload: You can initiate a multipart upload from any API Route, Workflow or Function using the File Storage Action. This will initiate the multipart upload and return the url where the separate parts have to be sent to.

2. Upload Parts: Upload each part using the upload url returned in the first step and the respective part number. Each part must be between 5 MB and 5 GB in size, except the last part, which can be smaller.

3. Complete Multipart Upload: After uploading all parts, complete the multipart upload to assemble the object from the parts.

4. Abort Multipart Upload: If you don’t want to finish your multipart upload, you should abort it. Otherwise the parts that have already been uploaded will occupy space in your bucket.

​

Initiate Multipart Upload

As mentioned above, you can initiate a multipart upload using the File Storage Action via the Upload Url.

This will return the url that is then later used for uploading each part & completing or aborting your multipart upload.

​

Upload Parts

In order to upload a large file, you first have to divide it into smaller chunks. These chunks are then uploaded separately to the presigned URL that is described in the previous step. This method is efficient for uploading large files, ensuring that if a part fails to upload, only that part needs to be retried. The process involves splitting the file, calculating the number of parts, uploading each part, and gathering necessary metadata for completing the multipart upload.

​

Step-by-Step Instructions

1. Get the File:
   The example below uses a simple HTML file input element.
   Example: const file = document.getElementById('fileInput').files[0]

2. Define Chunk Size:
   The chunk size defines the size of each file part to be uploaded.
   It needs to be at least 5MB. We recommend a chunk size of 10 MB.
   Example: const chunkSize = 10 * 1024 * 1024; // 10 MB

3. Define Total Parts:
   Determine the total number of parts by dividing the file size by the chunk size and rounding up.
   Example: const totalParts = Math.ceil(file.size / chunkSize);

4. Initialize an Array to Store Parts Metadata:
   Create an array to store the part number and ETag for each uploaded part. These will later be needed for completing the multipart upload

5. Loop Over and Slice the File:
   Loop through the file, slicing it into chunks based on the chunk size.
   Use file.slice(start, end) to create each chunk.

6. Create FormData and Append Chunk and Part ID:
   Create a FormData object for each chunk.
   Append the chunk and the part ID to the FormData object.

7. Send a POST request with the FormData to the presigned URL:
   The example below uses the fetch API, but you can use any other method to send the request, such as Axios or XMLHttpRequest.
   The url that the request is sent to, is the url that you receive in the Initiate Multipart Upload step.

8. Retrieve the ETag from the response headers & update parts array:
   Push an object containing the PartNumber and ETag into the parts array.

9. Error handling:
   The example below uses a simple check if the response was ok and prints the error message to the console.
   You can adjust this to your needs and add a custom retry logic if needed.

By following these steps, you can effectively manage the multipart upload of a large file. Each chunk is uploaded independently, and the parts array collects the necessary metadata for completing the multipart upload.

const file = document.getElementById(`fileInput`).files[0]

const url = 'https://upload.fastgen-storage.com/multipart-upload/9+hSeDIyK8d5g2mjP6U5oFglvGMcaGhz6G2P3qH+SWkL3CHq9kWftmNM7aQsxcCyvI85nujFEii3MwNnZp6Nunq2wegjPTvqKc/CWKT9WHGvp2jw1//4jWPwhvxREt5Auqtm1SjP4SvhT0c7V+pjBM8J0WVyH72FQ2/cuC/kv0Xh4rLgS9epR6H+DJ8mrTuf1OboeGmOxqJSaAQ3+V7//TQdKtNvfi+S8BEGCxg0eVQ8U026oR4ejBWFTpSWsPHBaU+umjiY0dgHVdyUuQej/1TO4q+U5vhchGQxbxZ5jD+0iRegPUzmPGXyH6JzSBfNJtlcdRNtSW+9Es7Cdo61XfuPVwpa2xsdAdElVGpN3kXdfvR833Di+u1MWehEzeGmzuSy/llaIv0Ztbdq0tz9PrlsPT1Xot6c9jVA4cLK3fMnglRCBTnnPWe3ZWKDK0L7mQrBQ/fc11pq/faldFnup/6A22v5Iwe7G5sSsgHdJPgnst+pjX2m01BpXULnp0GCmys5NEuxMw/Rxf49e1htlGPrESt+x3aMLL8FqRGIOwpKWpLZvnmwq2GkqOuYmpNOn8DaZF2h1xQ7QOnwF5bVfBE7RSdP3VypBwJf/AFT1faWu9ydLHU0jGw51pdS3zBKS7GQ5wzN+VPi2xj82V9qZw=='

const chunkSize = 10 * 1024 * 1024 // 10 MB
const totalParts = Math.ceil(file.size / chunkSize)

const parts = []

for (let partId = 1; partId <= totalParts; partId++) {
  const start = (partId - 1) * chunkSize
  const end = Math.min(file.size, partId * chunkSize)
  const chunk = file.slice(start, end)

  try {
    const formData = new FormData()
    formData.append('file', chunk)
    formData.append('partId', partId.toString())

    const response = await fetch(url, {
      method: 'POST',
      body: formData
    })

    const etag = await response.headers.get('etag')
    parts.push({ PartNumber: partId, ETag: etag })

    if (!response.ok) {
      console.error(await response.text())
      return
    }
  } catch (err) {
    console.error('catch error: ', err)
  }
}

​

Complete Multipart Upload

Once all parts are uploaded, the multipart upload process is completed by sending a request with the collected parts metadata to the completion endpoint.

The url that the request is sent to, is the url that you receive in the Initiate Multipart Upload step.

const completeUploadUrl = url + '/complete'
await fetch(completeUploadUrl, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    parts,
    size: file.size
  })
})

​

Abort Multipart Upload

If you need to abort the multipart upload, you can send a DELETE request to the URL.

The url that the request is sent to, is the url that you receive in the Initiate Multipart Upload step.

async function abortUpload(url: string) {
  return await fetch(url, {
    method: 'DELETE'
  })
}