Enhance Build Pipeline with Debug and Core Analysis Support

[edespino]

Fixes: #765

Adds comprehensive debug build support and automated core dump analysis to the Cloudberry build pipeline. Key features:

• Debug build capability with preserved symbols and debug-specific RPMs
• Automated core dump detection and analysis during test execution
• Core file correlation with test failures
• Enhanced test result reporting with core dump status
• Improved artifact management for debug builds

The changes enable better debugging of test failures and provide more detailed information about process crashes during testing.

I expect the ic-good-opt-off group to fail as the mirror_replay test suite is dumping a core. I have filed the following for that issue: Core Dump in mirror_replay Test Suite During Execution #782

After the checks have run and ic-good-opt-off has failed as expected with core dump, I plan to enable the skip ci optionis PR to allow it to be committed after receiving approval.

Benign edit to re-trigger the workflow.

[avamingli]

After the checks have run and ic-good-opt-off has failed as expected with core dump, I plan to enable the skip ci optionis PR to allow it to be committed after receiving approval.

If we do this with a skip CI tag, any later pull requests will fail and cannot be merged. I’m not optimistic that we can fix the core dump issue #782 in a short time.

How about we configure the system to allow the mirror_replay case for now? This way, we can separate the core dumps and report them independently. When someone eventually fixes the mirror_replay issue, we can update the configuration to set the expected core dump case to NULL, ensuring that nothing is blocked.

[edespino]

If we do this with a skip CI tag, any later pull requests will fail and cannot be merged. I’m not optimistic that we can fix the core dump issue #782 in a short time.

How about we configure the system to allow the mirror_replay case for now? This way, we can separate the core dumps and report them independently. When someone eventually fixes the mirror_replay issue, we can update the configuration to set the expected core dump case to NULL, ensuring that nothing is blocked.

The best I can do is to disable (ignore) the mirror_replay test for now. Is this what you meant? I do not have a way to exclude the core checking for a specific test suite within a test schedule. Will disabling mirror_replay suffice for now?

[avamingli]

The best I can do is to disable (ignore) the mirror_replay test for now. Is this what you meant?

This is indeed a way to address the issue at the moment. However, it means we will miss out on a test case.
On the positive side, it will allow us to promptly detect core dumps when they occur.

cc: @my-ship-it

[edespino]

There are a couple of options:

1. I could place mirror_replay into it's own schedule, add a makefile target for it, enhance the core file check to allow a non-failure state at the matrix test group level. We could then unwind when it is fixed.
2. Instead of signaling a failure, I could simply flag core files (if found) as a warning and not a fatal scenario. I can add this as an annotation in the UI and a yellow warning symbol in the summary.

Personally, I prefer option 1 over 2. I do not know if developers tend to review the details of the workflow action only when there are CI check status failures. A core file warning condition could go unnoticed if the test case passes as is the case with mirror_replay.

[edespino]

There are a couple of options:

1. I could place mirror_replay into it's own schedule, add a makefile target for it, enhance the core file check to allow a non-failure state at the matrix test group level. We could then unwind when it is fixed.
2. Instead of signaling a failure, I could simply flag core files (if found) as a warning and not a fatal scenario. I can add this as an annotation in the UI and a yellow warning symbol in the summary.

Personally, I prefer option 1 over 2. I do not know if developers tend to review the details of the workflow action only when there are CI check status failures. A core file warning condition could go unnoticed if the test case passes as is the case with mirror_replay.

@avamingli & @my-ship-it Any thoughts on which approach to take? I have other test suites I would like to enable that is blocked on getting this work completed.

[my-ship-it]

There are a couple of options:

1. I could place mirror_replay into it's own schedule, add a makefile target for it, enhance the core file check to allow a non-failure state at the matrix test group level. We could then unwind when it is fixed.
2. Instead of signaling a failure, I could simply flag core files (if found) as a warning and not a fatal scenario. I can add this as an annotation in the UI and a yellow warning symbol in the summary.

Personally, I prefer option 1 over 2. I do not know if developers tend to review the details of the workflow action only when there are CI check status failures. A core file warning condition could go unnoticed if the test case passes as is the case with mirror_replay.

Hey Ed, thanks for making the enhancement.
I prefer option 1, in addition, it's better we can add a FIXME to remind us to fix this problem later.

[edespino]

@avamingli & @my-ship-it Please check out the workflow. You will see a new test group ic-fixme which contains mirror_replay. It succeeds but the core does not cause the test to fail. You will see that warnings are available with the test summary. Let me know what you think.