Skip to content

Use sparse files (FALLOC_FL_PUNCH_HOLE) to deallocate unused pages on disk #1063

@CAFxX

Description

@CAFxX

The README currently says:

bbolt/README.md

Lines 886 to 891 in dbe1e6d

* Because of the way pages are laid out on disk, Bolt cannot truncate data files
and return free pages back to the disk. Instead, Bolt maintains a free list
of unused pages within its data file. These free pages can be reused by later
transactions. This works well for many use cases as databases generally tend
to grow. However, it's important to note that deleting large chunks of data
will not allow you to reclaim that space on disk.

The missing ability to truncate to return space to the filesystem could be mitigated, at least on ext4/xfs/btrfs on linux, by having bbolt call

unix.Fallocate(fd, unix.FALLOC_FL_PUNCH_HOLE|unix.FALLOC_FL_KEEP_SIZE, offset, length)

either eagerly whenever a page is added on the freelist or lazily later during a background operation, with offset the offset of the page in bytes from the start of the file, and length the size of the page in bytes.

This would instruct the kernel/fs to free the space on disk corresponding to that page, thereby making it available again to the rest of the system.tio

Given that the actual effectiveness of this approach depends on the combination of (at least) OS, OS version, filesystem, filesystem page size, database page size, and physical disk page size, whether to perform the Fallocate call should be controlled by a database option, defaulting to false.

Note: this is the same as #125, that was automatically closed as stale due to no activity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions