Fix flaky backup test: Handle 409 'version not available' error

[nammn]

Summary

Fixes flaky test TestBackupForMongodb::test_deploy_same_mdb_again_with_orphaned_backup by clearing monitoring/backup agent credentials when authentication is disabled.

Why It's Flaky

The test creates mdb-four-zero (auth enabled), deletes it with autoTerminateOnDeletion=False (orphaned backup), then creates mdb-four-two (auth disabled).

Race condition:

• Passes: OM cleans up mdb-four-zero hosts before backup request → only healthy mdb-four-two hosts checked
• Fails: mdb-four-zero hosts still registered with stale credentials → monitoring auth fails → no version info → 409

The 409 occurs because OM checks version info across all hosts in the project, not just the target deployment.

Root Cause Analysis

The Bug

When re-deploying MongoDB with auth disabled after a previous deployment with auth enabled:

1. Stale credentials remain in the monitoring agent template
2. Ops Manager copies these credentials to per-host monitoring config
3. Monitoring agent attempts SCRAM-SHA-1 authentication against MongoDB (which has auth disabled)
4. Authentication fails → no metrics collected → no version info → 409 error

Evidence from Failure

Task ID:
https://spruce.mongodb.com/task/mongodb_kubernetes_e2e_om80_kind_ubi_e2e_om_ops_manager_backup_294d005d333d64faff0d753ad57bb25b5941917f_26_01_23_12_36_41/tests?execution=1&sorts=STATUS%3AASC

1. Monitoring agent log shows authentication failures despite globalAuthUsername = <unset>:

[2026-01-23T14:25:55.748+0000] [header.info] [::0] globalAuthUsername = <unset>

[2026-01-23T14:26:05.181+0000] [metrics.dbstats.collector-mdb-four-zero-0...error] 
Failed to get connectionStatus. Err: `auth error: sasl conversation error: 
unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.`

2. Automation config shows disabled: true but credentials still present:

{
  "auth": {
    "disabled": true,
    "autoUser": "mms-automation-agent",
    "autoPwd": "vDMC07Aj7t0QzU+xOEJrSiuz1/C/2Kmwq2tw7FLoHVuNre4C..."
  }
}

-> in om we don't care whether its disabled or not, we only set user and pw
-> in automation we use those as we receive them, automation doesn't check for disabled or not

3. Test failure after 600s timeout:

Status: 409 (Conflict), Detail: Backup failed to start: MongoDB version information 
is not yet available for one of your hosts.

Causal Chain

Stale credentials in config (auth.disabled=true but autoUser/autoPwd set)
    ↓
Ops Manager copies credentials to per-host monitoring config
    ↓
Monitoring agent attempts SCRAM-SHA-1 authentication
    ↓
MongoDB rejects (auth is disabled)

Validation

Unit Test Added

TestDisableAuthenticationClearsAgentCredentials in controllers/operator/authentication/authentication_test.go:

• ✅ Passes with fix - credentials cleared via MERGO_DELETE

Evergreen Patches (6/6 passed)

Patch	Status	URL
1	✅	https://evergreen.mongodb.com/version/697727c8a39dc400072e5a81
2	✅	https://evergreen.mongodb.com/version/697727d9a39dc400072e5a9f
3	✅	https://evergreen.mongodb.com/version/697727e201c5ae0007a50dbb
4	✅	https://evergreen.mongodb.com/version/6977408884379900078dd545
5	✅	https://evergreen.mongodb.com/version/6977409184379900078dd54e
6	✅	https://evergreen.mongodb.com/version/6977409adfe34400070ea867

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.7.0 Release Notes

New Features

• Allows users to override any Ops Manager emptyDir mount with their own PVCs via overrides statefulSet.spec.volumeClaimTemplates.
• Added support for auto embeddings in MongoDB Community to automatically generate vector embeddings for the vector search data. This document can be followed for detailed documentation
• MongoDBSearch: Updated the default mongodb/mongodb-search image version to 0.60.1. This is the version MCK uses if .spec.version is not specified.
• Added support for configurable ValidatingWebhookConfiguration name via operator.webhook.name helm value.

Bug Fixes

• Fix an issue to ensure that hosts are consistently removed from Ops Manager monitoring during AppDB scale-down events.
• Fixed an issue where monitoring agents would fail after disabling TLS on a MongoDB deployment.
• Persistent Volume Claim resize fix: Fixed an issue where the Operator ignored namespaces when listing PVCs, causing conflicts with resizing PVCs of the same name. Now, PVCs are filtered by both name and namespace for accurate resizing.
• Fixed a panic that occurred when the domain names for a horizon was empty. Now, if the domain names are not valid (RFC 1123), the validation will fail before reconciling.
• MongoDBMultiCluster, MongoDB: Fix an issue where the operator skipped host removal when an external domain was used, leaving monitoring hosts in Ops Manager even after workloads were correctly removed from the cluster.
• Fixed an issue where the Operator could crash when TLS certificates are configured using the certificatesSecretsPrefix field without additional TLS settings.
• MongoDBOpsManager, AppDB: Block removing a member cluster while it still has non-zero members. This prevents scaling down without the preserved configuration and avoids unexpected issues.
• Fixed an issue where the monitoring agent failed to report version information to Ops Manager when a MongoDB deployment with authentication disabled was created after a previous deployment with authentication enabled had been deleted.
• The operator now clears stale agent credentials from monitoring and backup agent configs when authentication is disabled, preventing authentication failures against MongoDB instances that have auth disabled.

[nammn]

without the fix the agentTemplateUserName and Password are still

		config.SetAgentUserName("mms-automation")
		config.SetAgentPassword("stale-password-from-previous-deployment")