Problem: MemoryError when storing large AIPs over SWIFT

[s-tittel]

Expected behaviour
Storing large AIPs (in this case a ~280 GB .7z file) in a SWIFT location should work.

Current behaviour
The storage service throws the following exception:

ERROR     2024-08-06 06:24:04  django.request.tastypie:resources:_handle_500:309:  Internal Server Error: /api/v2/file/
Traceback (most recent call last):
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/tastypie/resources.py", line 221, in wrapper
    response = callback(request, *args, **kwargs)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/tastypie/resources.py", line 456, in dispatch_list
    return self.dispatch('list', request, **kwargs)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/tastypie/resources.py", line 488, in dispatch
    response = method(request, **kwargs)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/tastypie/resources.py", line 1399, in post_list
    updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
  File "/usr/lib/archivematica/storage-service/locations/api/resources.py", line 1135, in obj_create
    self._store_bundle(bundle)
  File "/usr/lib/archivematica/storage-service/locations/api/resources.py", line 1066, in _store_bundle
    bundle.obj.store_aip(
  File "/usr/lib/archivematica/storage-service/locations/models/package.py", line 872, in store_aip
    self.create_replicas()
  File "/usr/lib/archivematica/storage-service/locations/models/package.py", line 1542, in create_replicas
    self.replicate(replicator_loc)
  File "/usr/lib/archivematica/storage-service/locations/models/package.py", line 676, in replicate
    src_space.move_to_storage_service(
  File "/usr/lib/archivematica/storage-service/locations/models/space.py", line 337, in move_to_storage_service
    self.get_child_space().move_to_storage_service(
  File "/usr/lib/archivematica/storage-service/locations/models/swift.py", line 180, in move_to_storage_service
    self._download_file(src_path, dest_path)
  File "/usr/lib/archivematica/storage-service/locations/models/swift.py", line 159, in _download_file
    headers, content = self.connection.get_object(self.container, remote_path)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/swiftclient/client.py", line 1918, in get_object
    rheaders, body = self._retry(None, get_object, container, obj,
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/swiftclient/client.py", line 1797, in _retry
    rv = func(self.url, self.token, *args,
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/swiftclient/client.py", line 1269, in get_object
    object_body = resp.read()
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/swiftclient/client.py", line 453, in releasing_read
    chunk = self.resp.raw.read(*args, **kwargs)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/urllib3/response.py", line 567, in read
    data = self._fp_read(amt) if not fp_closed else b""
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.10/site-packages/urllib3/response.py", line 533, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
  File "/usr/lib/python3.10/http/client.py", line 482, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.10/http/client.py", line 631, in _safe_read
    data = self.fp.read(amt)
MemoryError
ERROR     2024-08-06 06:24:05  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/

Steps to reproduce
Ingest a large SIP.

Environment
OS: Ubuntu 22.04.4 LTS
Archivematica: V1.15

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged (if applicable)
• Details about this issue have been added to the release notes (if applicable)