BUG - Unable to upload a large file

[leoraba]

Description

The application is unable to handle large file uploads, such as a 270MB submission file.

Steps to Reproduce

1. Use the POST /data endpoint to upload a submission file of approximately 270MB.
2. The application responds with a 200 OK status, returning "PROCESSING" and a new submission ID, indicating that the upload has started successfully.

Actual behaviour:

• Shortly after the initial response, the application stops working and becomes unresponsive.
• Upon reviewing the logs, it's evident that the application crashes while attempting to process the large 270MB file. with following error:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Expected Behavior:

• The application should successfully handle large file uploads without crashing, providing progress feedback, and processing the submission data accordingly.

Possible Fix

Optimize file upload logic to prevent timeouts and crashes, possibly by streaming the file in chunks or setting appropriate memory limits