ACK RDS controller false diff when adopting a previously retained DBInstance

[eTimS]

Hello,
Describe the bug
Create and then delete a DBInstance with the annotation services.k8s.aws/deletion-policy: retain. Then adopting the DBInstance with the following annotations leads to permanent diff :

metadata:
  annotations:
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain

Steps to reproduce
Use rds-chart:1.6.0 with the following values (all settings have been copied in case they are relevant):

aws:
  region: eu-west-3
deployment:
  replicas: 3
leaderElection:
  enabled: true
metrics:
  service:
    create: true
reconcile:
  defaultMaxConcurrentSyncs: 10
  defaultResyncPeriod: 180 # Debug value to show the diff message in the controller logs every 3 minutes instead of 10 hours
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: <insert your IRSA ARN here>

Create the following manifest:

postgres-retain-adopt-permanent-diff.yaml :

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  annotations:
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
  name: postgres-retain-adopt-permanent-diff
  namespace: default
spec:
  allocatedStorage: 40
  autoMinorVersionUpgrade: true
  backupRetentionPeriod: 30
  caCertificateIdentifier: rds-ca-rsa2048-g1
  copyTagsToSnapshot: true
  dbInstanceClass: db.t4g.medium
  dbInstanceIdentifier: postgres-retain-adopt-permanent-diff
  dbName: maindb
  dbParameterGroupName: postgres-retain-adopt-permanent-diff
  dbSubnetGroupName: <INSERT-YOUR-DB-SUBNET-GROUP-NAME-HERE>
  engine: postgres
  engineVersion: '17'
  masterUserPassword:
    key: password
    name: postgres-retain-adopt-permanent-diff-password
    namespace: default
  masterUsername: dbadmin
  multiAZ: false
  performanceInsightsEnabled: true
  performanceInsightsRetentionPeriod: 7
  preferredBackupWindow: 02:00-03:00
  preferredMaintenanceWindow: sun:00:15-sun:01:15
  publiclyAccessible: false
  storageEncrypted: true
  storageType: gp2
  tags:
  - key: env
    value: dev
---
apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBParameterGroup
metadata:
  annotations:
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
  name: postgres-retain-adopt-permanent-diff
  namespace: default
spec:
  description: Parameter group for postgres-retain-adopt-permanent-diff RDS database
  family: postgres17
  name: postgres-retain-adopt-permanent-diff
  parameterOverrides:
    hot_standby_feedback: '1'
    # log_checkpoints: '1'
    log_connections: '1'
    log_lock_waits: '1'
    log_min_duration_statement: '10000'
    log_rotation_age: '1440'
    shared_preload_libraries: pg_stat_statements
  tags:
  - key: env
    value: dev
---
apiVersion: v1
data:
  password: eW91ci1wYXNzd29yZA==
kind: Secret
metadata:
  name: postgres-retain-adopt-permanent-diff-password
  namespace: default
type: Opaque

After applying the manifest, wait for the DBInstance to be synched and wait 3 or 4 minutes more (reconcile delay) and observe that no drift event gets logged in the ack-rds controller logs:

2025-09-03 13:10:08.467	
{"level":"error","ts":"2025-09-03T11:10:08.467Z","msg":"Reconciler error","controller":"dbinstance","controllerGroup":"rds.services.k8s.aws","controllerKind":"DBInstance","DBInstance":{"name":"postgres-retain-adopt-permanent-diff","namespace":"default"},"namespace":"default","name":"postgres-retain-adopt-permanent-diff","reconcileID":"4509164c-2ad5-4453-8c78-b8e336adde63","error":"operation error RDS: CreateDBInstance, https response error StatusCode: 404, RequestID: d341c9da-46da-4e34-87d7-e28c8dc9b7e2, DBParameterGroupNotFound: DBParameterGroup not found: postgres-retain-adopt-permanent-diff","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
	
	
2025-09-03 13:10:08.822	
{"level":"info","ts":"2025-09-03T11:10:08.822Z","logger":"ackrt","msg":"created new resource","kind":"DBParameterGroup","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":1}
	
	
2025-09-03 13:10:09.860	
{"level":"info","ts":"2025-09-03T11:10:09.860Z","logger":"ackrt","msg":"created new resource","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":1}

Now, delete the manifest (using kubectl delete -f <file_name> in my tests), then wait a few seconds for the resources to be deleted in Kubernetes and then create the manifest again.

After one minute, notice in the controller logs that resources are not deleted as expected, but then DBInstance is detected as drifted with null fields:

Logs (part2) :

2025-09-03 13:22:24.135	
{"level":"info","ts":"2025-09-03T11:22:24.134Z","logger":"ackrt","msg":"AWS resource will not be deleted - deletion policy set to retain","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","generation":3}
	
	
2025-09-03 13:22:24.345	
{"level":"info","ts":"2025-09-03T11:22:24.345Z","logger":"ackrt","msg":"AWS resource will not be deleted - deletion policy set to retain","kind":"DBParameterGroup","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","generation":2}
	
	
2025-09-03 13:22:52.839	
{"level":"info","ts":"2025-09-03T11:22:52.839Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":1,"diff":[{"Path":{"Parts":["Spec","MasterUserPassword"]},"A":"","B":"default/postgres-retain-adopt-permanent-diff-password.password"},{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","PerformanceInsightsKMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}
	
	
2025-09-03 13:22:52.980	
{"level":"info","ts":"2025-09-03T11:22:52.980Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":1,"diff":[{"Path":{"Parts":["Spec","MasterUserPassword"]},"A":"","B":"default/postgres-retain-adopt-permanent-diff-password.password"},{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","PerformanceInsightsKMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}
	
	
2025-09-03 13:22:53.986	
{"level":"info","ts":"2025-09-03T11:22:53.986Z","logger":"ackrt","msg":"updated resource","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":1}
	
	
2025-09-03 13:22:54.320	
{"level":"info","ts":"2025-09-03T11:22:54.320Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}
	
	
2025-09-03 13:22:55.070	
{"level":"info","ts":"2025-09-03T11:22:55.070Z","logger":"ackrt","msg":"updated resource","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":2}
	
	
2025-09-03 13:23:24.368	
{"level":"info","ts":"2025-09-03T11:23:24.368Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}
	
	
2025-09-03 13:23:54.614	
{"level":"info","ts":"2025-09-03T11:23:54.614Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}
	
	
2025-09-03 13:24:24.767	
{"level":"info","ts":"2025-09-03T11:24:24.767Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"default","name":"postgres-retain-adopt-permanent-diff","account":"383425823718","role":"","region":"eu-west-3","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","DatabaseInsightsMode"]},"A":null,"B":"standard"},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","KMSKeyID"]},"A":null,"B":"arn:aws:kms:eu-west-3:383425823718:key/2c1e7e73-bc59-41c5-bd37-2f0055b2157b"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"postgresql-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0}]}

This issue seems to be related in some way to #2616

Expected outcome
We expect no diff to be detected since the "postgres-retain-adopt-permanent-diff" is appropriately configured and synced.

Environment

• Kubernetes version
  1.31
• Using EKS (yes/no), if so version?
  eks.34
• AWS service targeted (S3, RDS, etc.)
  RDS

[github-actions]

Hello @eTimS 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[eTimS]

PS : also notice that the frequency of the drift message is much highier than one message per 3minutes (180s reconcile delay). This may or may not be normal. I haven't looked at the code. Maybe the behaviour is a bit different after a successful adoption of a resource.