Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parse.Files cannot be created if over 512MB #9283

Open
dalyaidan1 opened this issue Aug 15, 2024 · 2 comments · May be fixed by #9286
Open

Parse.Files cannot be created if over 512MB #9283

dalyaidan1 opened this issue Aug 15, 2024 · 2 comments · May be fixed by #9286
Labels
type:feature New feature or improvement of existing feature

Comments

@dalyaidan1
Copy link

New Issue Checklist

Issue Description

Files larger than 512MB are not able to be uploaded to Parse Server as Parse.Files.

The files are converted to stings, which is more than Node can handle.

Steps to reproduce

Upload a file bigger than 512MB.

Actual Outcome

Error: Cannot create a string longer than 0x1fffffe8 characters

    at Object.slice (node:buffer:654:37)

    at Buffer.toString (node:buffer:824:14)

    at createHandler (/project/server/node_modules/parse-server/lib/Routers/FilesRouter.js:153:29)

    at Layer.handle [as handle_request] (/project/server/node_modules/express/lib/router/layer.js:95:5)

    at next (/project/server/node_modules/express/lib/router/route.js:144:13)

    at handleParseSession (/project/server/node_modules/parse-server/lib/middlewares.js:281:7)

    at Layer.handle [as handle_request] (/project/server/node_modules/express/lib/router/layer.js:95:5)

    at next (/project/server/node_modules/express/lib/router/route.js:144:13)

    at handleRateLimit (/project/server/node_modules/parse-server/lib/middlewares.js:275:3)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {

  code: 'ERR_STRING_TOO_LONG'

This appears to come from this line:
https://github.com/parse-community/parse-server/blob/6.3.1/src/Routers/FilesRouter.js#L175

Expected Outcome

The Parse.File to be created.

Environment

Server

  • Parse Server version: 6.3.1
  • Operating system: Linux
  • Local or remote host (AWS, Azure, Google Cloud, Heroku, Digital Ocean, etc): Aptible

Database

  • System (MongoDB or Postgres): MongoDB
  • Database version: 4.0
  • Local or remote host (MongoDB Atlas, mLab, AWS, Azure, Google Cloud, etc): Aptible

Client

  • SDK (iOS, Android, JavaScript, PHP, Unity, etc): Android
  • SDK version: 4.2.0

Logs
@parse-github-assistant Parse GitHub Assistant
Copy link

Thanks for opening this issue!

  • 🚀 You can help us to fix this issue faster by opening a pull request with a failing test. See our Contribution Guide for how to make a pull request, or read our New Contributor's Guide if this is your first time contributing.

@mtrezza mtrezza added the type:feature New feature or improvement of existing feature label Aug 15, 2024
@mtrezza
Copy link
Member

mtrezza commented Aug 15, 2024 •
edited
Loading

This is an expected limitation of the V8 engine. Large files should be uploaded in a different way, like streaming or multi-part. Not sure what the file adapter already supports. A "workaround" could be to break the base64 conversion up into parts instead of converting the whole string at once. But even then a var would hold the entire file in memory, which may again run into limitations. Maybe the conversion can be made unnecessary if req.body is already a buffer and we could simply const file = new Parse.File(filename, { buffer: fileBuffer }, contentType);.

@dalyaidan1 dalyaidan1 linked a pull request Aug 18, 2024 that will close this issue
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:feature New feature or improvement of existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Allow large Parse File uploads dalyaidan1/parse-server
2 participants
@mtrezza @dalyaidan1