Truncate AO relation segment files before unlink #668
Conversation
Problem: In cases where an AO table is created, some data is inserted, and then the table is dropped, file descriptor of segfile will be opened still. It stays open until session ends. Thus, disc space is not freed until the end of the session. Cause: There are 2 issues: 1) mdcreate_ao is used to create a segment file during data insertion. For creation, it calls PathNameOpenFile without closing. File descriptor returned by PathNameOpenFile in mdcreate_ao is not stored anywhere except the virtual file level (fd.c), where it will be cleaned some time later only when the number of opened descriptors reaches threshold. 2) second issue happens when we run out of space. This problem is related to file descriptors left open because of an abnormal abort of the insert command. In function BufferedAppendWrite, once free space has ended, ereport is thrown. Before that, the file is not closed, so after the long jump it remained open. Fix: Add FileClose to mdcreate_ao and BufferedAppendWrite (before throwing ereport).
|
Allure report https://allure-ee.adsw.io/launch/61185 |
|
Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932873 |
|
Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932874 |
|
Failed job Regression tests with Postgres on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932866 |
|
Failed job Regression tests with Postgres on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932865 |
|
Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932867 |
|
Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932868 |
|
Failed job Behave tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/932871 |
Problem: Segment files of AO tables are not truncated before unlink. As result, after relation drop if there are file descriptors of segment files, not closed by some backend process, disk space is not returned to the OS. Cause: In mdunlinkfork, for an AO relation mdunlink_ao is called instead of truncating all segment files. And mdunlink_ao doesn't perform truncation of segment files. Fix: Add truncation of segment files for AO tables into mdunlink_ao_perFile, called from mdunlink_ao for each segment file. For that purpose static function do_truncate was renamed and moved to accessible api in fd.h. + Reverted previous commit 20aa6e1
whitehawk
changed the title
|
Allure report https://allure-ee.adsw.io/launch/61953 |
|
Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/973082 |
|
Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/973083 |
| @@ -145,4 +145,6 @@ extern const char *FileGetFilename(File file); | |||
| extern void FileSetIsWorkfile(File file); | |||
| extern void FileSetIsTempFile(File file, bool isTempFile); | |||
|
|
|||
| extern int TruncateFileByName(FileName fileName); | |||
There was a problem hiding this comment.
Why not simply put public do_truncate to md.h like mdunlink?
Problem:
Segment files of AO tables are not truncated before unlink.
As result, after relation drop if there are file descriptors of segment files,
not closed by some backend process, disk space is not returned to the OS.
Cause:
In mdunlinkfork, for an AO relation mdunlink_ao is called instead of truncating
all segment files. And mdunlink_ao doesn't perform truncation of segment files.
Fix:
Add truncation of segment files for AO tables into mdunlink_ao_perFile, called
from mdunlink_ao for each segment file.
For that purpose static function do_truncate was renamed and moved to
accessible api in fd.h.