Disk space is not freed up after objects are deleted

[skliarie]

It seems that java somehow keeps open handle on binaryData, even if the object was deleted, thus preventing OS to delete the file and reclaim the freed up disk space.

Restarting container frees up the disk space correctly.

[afranken]

@skliarie correct, all uploaded files are deleted when the application is shut down.
S3Mock is meant to be used for testing your application locally, shut down after the tests are done.

[skliarie]

We use S3Mock (with persistence) for a quick and dirty backups.. Daily overwrites cause cause disk usage to grow.
It would be nice for the disk space to be freed up in such case.

[afranken]

@skliarie
looked at the code again, we do indeed delete everything when S3Mock shuts down (via https://docs.oracle.com/javase/8/docs/api/java/io/File.html#deleteOnExit-- ), but all files relating to a key are deleted immediately when https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html is called.
Not sure why disk space is not freed up, maybe File#deleteOnExit makes the JVM keep a file handle open even if the file is deleted regularly?

How do you reproduce this?

I refactored the "deleteOnExit" code, but I would like to reproduce the issue before / after:
#2173

[skliarie]

To reproduce:

1. Upload a big file to S3Mock, delete it, repeat several times
2. Observe used disk space not shrinking after each delete. On upload more disk space is used (e.g. old one is not recycled).

[afranken]

which OS are you using?

I just tested S3Mock directly on Ubuntu and disk space is freed up immediately when I delete an object.
This is different on MacOS, where disk space is freed up only after I shut down S3Mock.

[skliarie]

I use adobe/s3mock docker image from DockerHub with retainFilesOnExit: "true"
The /data volume is mapped from local directory

[afranken]

and which OS are you using to host the container?

[skliarie]

The docker runs on Ubuntu 22.04.4 LTS
Thank you for fixing the bug!

awaiting for the image with the fix to show up on dockerhub.

[afranken]

I fixed handling of InputStreams and OutputStreams that weren't closed in all cases.

Calling DELETE on the API works, the file is deleted immediately (at least on my freshly installed Ubuntu server 24.04.1 VM).
The used space you see on your system is lost somewhere in between, and I can't exactly pinpoint where.

BEFORE

This is how I it looks before the stream fixes with a ~640MB file.
Before upload: 6.1GB used
After upload: 8.2GB used
After delete: 7.5GB used

Even after shutting down S3Mock, the disk space is not released.

franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  6.1G  6.5G  49% /

--CALL PUT ON API
--S3MOCK STORES INCOMING STREAM IN TMP FILE
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  8.8G  3.8G  70% /

--S3MOCK DELETES TMP FILE
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  8.2G  4.4G  65% /

-- CALL DELETE ON API
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  7.5G  5.0G  61% /

AFTER

This is how I it looks after the stream fixes with the same ~640MB file.
This is from the current main which includes several dependency updates (Spring 3.3 -> 3.4) and the Jetty to Tomcat refactoring as well.

Before upload: 6.1GB used
After upload: 6.7GB used
After delete: 6.1GB used

--BEFORE UPLOAD
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  6.1G  6.5G  49% /

--CALL PUT ON API
--S3MOCK STORES INCOMING STREAM IN TMP FILE
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  6.7G  5.9G  54% /

--S3MOCK TRANSFERS TMP FILE TO BUCKET STORAGE
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  7.3G  5.3G  59% /

--S3MOCK DELETES TMP FILE
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  6.7G  5.9G  54% /

-- CALL DELETE ON API
franken@franken-vmware-ubuntu-2404:~$ df -kh
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  6.1G  6.5G  49% /

[afranken]

The fix will be released with 4.0.0, but there are a few other fixes and features I'm looking into, thanks for your patience.

Not 100% sure if I can keep file storage compatibility with 4.0.0 as I'm currently working on supporting versions, which may need incompatible changes in directories or files.