Update Scaled Node with extended resources from MachineClass NodeTemplate

[elankath]

How to categorize this issue?

/area auto-scaling
/kind enhancement
/priority 3

What would you like to be added:
We have enhanced our CA fork to support extended resources for scale-from-zero. PR: gardener/autoscaler#334. However, scheduler cannot assign Pods to the scaled up Node.

Currently, stakeholders are manually using tools such as Kyverno policies to update the Node.Status.Capacity object with extended resources, but this is inconvenient and somewhat error prone due to possible divergence of the values in the CA Node Group and Kyverno policy. Since the MCM is the managing controller and already reads the MachineClass it is best suited to update the Node object with the extended resources specified in the MachineClass

Why is this needed:
Support for assigning Pods to Nodes scaled by CA belonging to NodeGroups declared with extended resources. Background covered at: gardener/autoscaler#132

UPDATE:

We additionally need to introduce support for virtual extended resources that should not trigger rollout. Node.Status.Capacity should be changed dynamically without a node rollout for these virtual extended resources.

Related

gardener/gardener#11691
gardener/gardener-extension-provider-aws#1238