Skip to content

Refactor the put_object method to avoid temporary file write #26

@brunocalza

Description

@brunocalza

While uploading an object, we first write all contents in a temporary file before sending it to Hoku.

This is clearly bad for performance, especially for big files. Ideally, we would stream the bytes from the request directly to Hoku. We didn't do this in the first version because of:

  • The Hoku SDK method add_reader required a AsyncSeek (this is not true anymore)
  • We need a way to calculate the md5 of the file and pass it as metadata

So, writing to a temp file was the easiest way out of this.

One solution is to wrap the body request in an AsyncRead implementation that calculates the MD5 and passes it to add_reader. But we wouldn't be able to get the final md5 hash value before calling add_reader. So in addition to that, we would need a way to add metadata to an existing Hoku object (we don't have that Hoku operation right now).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions