Auto-Restoring deleted ServiceImports and EndPointSlices

[davidohana]

Not sure whether to call this a bug or an enhancement.

What happened:

• Deleting a service import or an endpoint slice locally - it stays deleted locally and deletion is synced to the broker.
• Deleting a service import or an endpoint slice from the broker, nothing happen

What you expected to happen:

The agent shall recreate the service import / endpoint slice after some time as the "source of truth" is the export and it still exists. Currently this only happens after lighthouse agent restart.

Additional notes:

To compare, if I delete a k8s endpoint slice, it is recreated immediately.
I realize that those edge cases are only possible if someone with sufficient permissions delete those objects manually.

Slack discussion link: https://kubernetes.slack.com/archives/C010RJV694M/p1645966829687039

[vthapar]

@davidohana Could you clarify what you mean by deleting locally? I expect you mean deleting locally on the source cluster where export exists, but just wanted to be sure it is so.

[davidohana]

Yes, that what I mean - deleting on the origin cluster.

[dfarrell07]

There are likely other objects that would fail in similar ways if a user deletes them, because they are internal objects that we don't expect users to mess with. To handle this properly we would need to make a more systematic review of the code base, which would likely be benefited by an EP and a broader discussion about what should change and how.

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[dfarrell07]

@vthapar pointed out that finalizers may be a good, low-overhead way of doing this.

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[dfarrell07]

Discussing this again, it's important to note that monitoring resources cross-many-clusters for deletion comes with overhead that doesn't exist for the parallel of an endpoint slices in one K8s cluster. If there's a usecase for this, we'd be happy to discuss that and prioritize this work. Again, this likely applies to other resources and finalizers might enable this with reasonable overhead.

This might be kinda-related to submariner-io/enhancements#161, or at least a good time to think about finalizers for various resources.