fix(securitycenter): issues related to flaky tests in SHA & ETD

[vijaykanthm]

Description

Fixes #4866 #4871 Flakiness in tests due to Concurrent Modification error.

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

• I have followed Contributing Guidelines from CONTRIBUTING.MD
• Tests pass: go test -v ./.. (see Testing)
• Code formatted: gofmt (see Formatting)
• Vetting pass: go vet (see Formatting)
• These samples need a new API enabled in testing projects to pass (let us know which ones)
• These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
• This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
• This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
• Please merge this PR for me once it is approved

[code-review-assist]

Hi team,

I'm currently reviewing this pull request and will post my detailed review in a few minutes. In the meantime, here's a quick summary to help you and other reviewers get up to speed:

This pull request addresses flaky tests in the golang-samples repository, specifically those related to Security Health Analytics (SHA) and Event Threat Detection (ETD). The core issue was a concurrent modification error causing test instability.

Here's a breakdown of the changes:

• Added retry mechanism: A retryOperation function was implemented to handle transient errors during cleanup of existing custom modules. This improves the robustness of the test suite.
• Added Mutex: A mutex (mu sync.Mutex) was added to protect shared resources in the test functions, preventing race conditions and concurrent modification errors. This addresses the root cause of the flakiness.
• Refactored cleanup functions: The cleanupExistingCustomModules and cleanupCustomModule functions were extracted and improved for better readability and maintainability. These functions now efficiently clean up test resources before and after test execution.
• Renamed test functions: Several test functions were renamed for clarity and consistency (e.g., TestCreateCustomModule to TestCreateSHACustomModule).

The changes are primarily focused on securitycenter/management_api/event_threat_detection/event_threat_detection_custom_module_test.go and securitycenter/management_api/security_health_analytics_custom_module_test.go. The line numbers in the diff clearly show the additions of the mutex, retry logic, and refactoring of the cleanup functions. Specific line changes include the addition of the sync package import (lines 26 in both files), the declaration of the mutex (lines 38 in both files), and the implementation of the retry logic (lines 48-52 in both files).

I'll provide a more detailed review shortly, but this summary should give you a good overview of the changes and their intent.

And now, a little haiku to lighten the mood:

Tests once flaky,
Now run smooth, a mutex's grace,
Code review's delight.

[code-review-assist]

Gemini encountered an error creating the review. You can try again by commenting @code-review-assist review.