Metal3 future architecture with multi-tenancy

[lentzi90]

I want to take the opportunity to gather three proposals in one discussion since the discussions anyway seem to get tangled.
The proposals are:

• DHCP-less IPAM
• CAPI multi-tenancy for CAPM3
• BareMetalHost API decomposition

In my head they fit together beautifully. There is just one open question about where/how the metaData and networkData should be handled. But I'm probably missing a lot of details.

To get started, I have attempted to draw a simplistic diagram of how things could fit together. I don't know if this is useful/understandable but I hope so.

(I know there are many resources missing in this picture but I didn't want to clutter it more. If I missed something essential please let me know!)

CAPI multi-tenancy is mostly about the vertical dotted line. The point is to strap CAPM3 of its privileges to manage any/all BMHs. Instead the user must explicitly give it credentials to the resources it should use. This is the equivalent of how other CAPI providers accept cloud credentials to get access to a specific tenant in GCP/AWS/OpenStack where they manage the cloud resources.

BMH API decomposition is more about the horizontal dotted line as I see it. Here we want to separate the consumer of a BMH from the admin who makes these baremetal resources available. This is also where the DHCP-less proposal comes in I think, because the admin will need to handle preprovisioning, whereas the consumer may need to have control over the network after provisioning.

That last part is an open question for me, as mentioned. I'm not sure if the consumer should always control the networkData for example. Or perhaps this must be configurable? In some situations it should be the admin and in other the consumer?

I would be very happy to hear opinions on this. Do you agree with what I wrote? Does the diagram make sense? What have I missed? 😄

[Rozzii]

For additional context: https://groups.google.com/g/metal3-dev/c/Wl2DIdr_ZQ0

[hardys]

I'm not sure if the consumer should always control the networkData for example. Or perhaps this must be configurable? In some situations it should be the admin and in other the consumer?

I think it depends on the environment, and we likely need to enable some amount of flexibility e.g:

• Infrastructure team owns all the networking - defines preprovisioningNetworkData, and optionally networkData (both applied directly to the BMH, so we'd likely need some way to prevent consumer configuration via the Machine)
• Consumer owns Machine networking - infrastructure team defines preprovisioningNetworkData and enables some consumer-owned IPPool for Machine networking

I think in the first case both networkData secrets would exist in the BMH namespace, but in the second the Machine-derived networkData would exist in the consumer/machine namespace?

Would be great to get some additional perspectives on this /cc @dtantsur @pierrecregut @Rozzii @zaneb

[zaneb]

Does the networkData (not preprovisioningNetworkData) actually do anything other than get provided to the user's image? If so then it doesn't matter if they can control it, since they can choose what they want to do with it anyway.
I think ironic does validate the format but that's it?

In practice I think the preprovisioningNetworkData field is now the home for Infra team--owned data (since it flows through to being the default networkData as well), and the networkData` field is the de facto home for consumer-owned data (even though there is no such thing in the OpenStack model).

There's no concept of user-defined metaData in OpenStack either. It seems less likely that anyone is relying on that (but I could be wrong), so maybe we can just leave that on the Infra team side where it logically belongs.

[dtantsur]

Could you please add resource description for completeness? I'm not sure I remember the difference between BareMetalImage, BareMetalInventory and BareMetalAllocation (this may be a sign that the names are not obvious).

[lentzi90]

Updated. I think perhaps the BareMetalImage should rather be named something like BareMetalHostClaim?

[dtantsur]

One issue: I don't think HardwareDetails can possibly exist in the consumer namespace: it's created when there is no consumer.

[lentzi90]

Good point! I had not thought of that. The consumer will need access to it though. One option would be to allow read only access to HardwareDetails in the BMH namespace. That sounds like a security issue waiting to happen though. I think we need to go through the BareMetalInventory somehow. For example HardwareDetails related to the BareMetalInventory could be copied over...

[zaneb]

At the time I wrote the doc I think I imagined that people would use the hardware-classification-controller to label BMHs based on their HardwareDetails and then users would match on those labels they cared about in the BareMetalInventory resource.
In practice nobody is doing that so we might need a new plan. (Or maybe not? Maybe nobody is doing it because it is only useful once you have multitenancy?)

Going in the other direction, after provisioning the user needs to know at least the IP address. We could easily add that to the BareMetalImage Status. But picking and choosing what we think the user needs to know is a recipe for disappointment, copying the whole HardwareDetails into the status is a recipe for the kind of size issues that drove us to create a new CRD in the first place, and copying the whole CR is a recipe for chaos.

[zaneb]

BTW in CAPI BareMetalShutdownRequest is managed by a Metal3Remediation. (There's also a Metal3RemediationTemplate.)