L3 domain for Prefixes

[PieterL75]

We have a mix of environments, all 'owning' a whole 10.0.0.0/8, we call them 'Layer3 domains'.
Within a layer3 domain, all IP addresses have to be unique.

On top of that, each Layer3 domain can contain different VRF's, each separating the purpose of the prefix (customer shared, customer dedicated, management) etc..

So we can have

• L3Domain 1
  • VRF1 - 10.10.0.0/16
  • VRF2 - 10.20.0.0/16
• L3Domain 2
  • VRF3 - 10.10.0.0/16
  • VRF4 - 10.40.0.0/16

But we cannot have

• L3Domain 1
  • VRF5 - 10.50.0.0/16
  • VRF6 - 10.50.0.0/16

I've tried to model the above in Netbox, but it looks like it does not have that concept of a Layer3 domain.
Netbox does have a VRF, but that VRF has no relation to other VRF's to check the used prefixes.

Is there an option, to create like the VLANGroups (which we refer to as Layer2Domains), something like 'VRFGroups'.
Within a VRFGroup all IPs and Prefixes need to be unique

Pieter

[jeremystretch]

What you're describing is the function of the VRF model.

Netbox does have a VRF, but that VRF has no relation to other VRF's to check the used prefixes.

Prefixes can be imported/exported among VRFs using route targets, which mimics their real-world function.

[PieterL75]

@DanSheps
I've been looking into the documentation concerning Custom Validators https://demo.netbox.dev/static/docs/customization/custom-validation/
It's not that clear, on where you have to define them.
I can see that the coniguration.py needs to be updates with the validator, but where do I define the Class ?
Do I have to do that in a plugin ? or is there a specific file/folder to put then in, so that it is not overwritten with a git pull request

Pieter

[DanSheps]

You could do it as a plugin, I don't think we have a specific folder though for it.

[PieterL75]

Looks like I have to put the validation code inside the configuration.py itself...
well,, that works at least :-)

I dont think a plugin its classes is loaded before the configuration.py is processed, so I would not be able to add custom validators via a plugin.
If there are better ways, let me know !

[bluikko]

@PieterL75 I raised this issue (of where to put the validators) some time ago - there is no "official" location, I ended up putting it in netbox/ for example netbox/customvalidators.py and then validator is called from configuration.py say customvalidators.doodooaddressvalidator

[PieterL75]

Also found that I cannot use netbox models in the custom validators.. as the validators are loaded before netbox models are fully loaded (I think).
btw: there is now a good location for custom files : #8210
that local folder is excluded from git

[PieterL75]

What about this approach :
Currently, I can only create one 10.0.0.0/8 aggregate and that one will contain ALL prefixes, independent of the VRF they are part of.

What if the aggregate is extended with a 'VRF members' multivalue field.
Then I can add all VRF that share an 'l3domain' to that aggregate. It also gives a VRF independent view of how much the ip space in in use and what prefixes are available.
One VRF needs to be able to be member of multiple aggregates. (ex 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12)
Aggregates without a VRF are 'global' and the prefixes out of those can be used anywhere (with or without a VRF)

This way, I could create multiple 10.0.0.0/8 aggregates, that each contain an unique ip space, and is shared across multiple vrf's

One of the goals that I an accomplish with this, is a proper documentation of overlapping IP spaces, that is shared in different VRF's.

[candlerb]

Aggregates aren't really intended for that purpose though. An Aggregate is "where did I get this block of address space from?", not "how is it deployed in my network?". You can't have multiple instances of the 10.0.0.0/8 aggregate, for example - it all comes from RFC1918.

I also don't think you want to mark an aggregate as belonging to a particular VRF or set of VRFs, because you could slice it up in different ways - i.e. assign different chunks within that aggregate to different sites, different VRFs etc. That sort of assignment belongs on a "container prefix", not an "aggregate".

[PieterL75]

In my lingo an 'aggregate' is a route summarization in BGP, before handing over to another AS.
as I have multiple rfc1918 routing domains (yeah I know.. but that's how the company grew by taking over other companies) it make sense to have different aggregates of the 10.0.0.0/8 range, each linked to one specific VRF or l3domain

My public ip space is a whole different story, for that netbox fits just right

[PieterL75]

I'm still struggling with this... I'm not able to document my ISP network with the current rules of Netbox...
What if, we make the VRF field multivalue ?
Then I could

• create one VRF that is the 'L3 Domain',
• create other VRFs that indicate the route isolation of prefixes out of that L3 domain
• link a prefix to both a L3domainvrf and a isolationvrf
• each vrf still has to verify that the prefix assigned is unique in the vrf

If I create a new prefix, the l3domain vrf will make sure it is unique in my rfc1918 IP space of that environment, and the isolation vrf can tell me how the routing is done for that prefix.

Coming to think of it.. it would solve a lot of my problems.
We're building a VRF that routes between the different L3 domains in an effort to phase out the many vrf's and L3 domains we have. This means that we have one VRF that could contain prefixes of different L3domain...
Translated in the idea above :

• 10.10.10.0/24 - VRF: L3D-ONE, VRF-HYBRID
• 10.10.20.0/24 - VRF: L3D-ONE, VRF-CUST1
• 10.30.10.0/24 - VRF: L3D-TWO, VRF-HYBRID
• 10.30.20.0/24 - VRF: L3D-TWO, VRF-CUST2
• 100.74.10.0/24 - VRF: L3D-MAIN, VRF-HYBRID
  With the rule the prefix has to be unique in each VRF...

For sure a solution that does not deviate a lot from the current implementation of the VRF, but creates super flexibility

Pieter

[PieterL75]

hm. I can see where netbox can get into problems with multiple vrfs on one prefix...

    def get_child_prefixes(self):
        """
        Return all Prefixes within this Prefix and VRF. If this Prefix is a container in the global table, return child
        Prefixes belonging to any VRF.
        """

My idea is to change that (and other def's) to "Return all Prefixes within this Prefix and either assigned VRFs'

• 10.10.0.0/16 - VRF: L3D-ONE (container)
  • 10.10.10.0/24 - VRF: L3D-ONE, VRF-HYBRID
  • 10.10.16.0/21 - VRF: L3D-ONE (container: Customers)
    • 10.10.16.0/22 - VRF: L3D-ONE, VRF-CUST1 (container, CUST1)
      • 10.10.16.0/24 - VRF: L3D-ONE, VRF-CUST1
      • 10.10.17.0/24 - VRF: L3D-ONE, VRF-CUST2
      • 10.10.18.0/24 - VRF: L3D-ONE, VRF-CUST1
      • 10.10.19.0/24 - VRF: L3D-ONE, VRF-CUST2
• 10.10.0.0/16 - VRF: L3D-TWO (container)
  • 10.10.16.0/21 - VRF: L3D-TWO (container: Customers)
    • 10.10.20.0/24 - VRF: L3D-TWO, VRF-CUST2
    • 10.10.21.0/24 - VRF: L3D-TWO, VRF-CUST1

getting the childs of 10.10.16.0/21 VRF: L3D-ONE should return

• 10.10.16.0/22 - VRF: L3D-ONE, VRF-CUST1 (container, CUST1)
  • 10.10.16.0/22 - VRF: L3D-ONE, VRF-CUST1
  • 10.10.17.0/24 - VRF: L3D-ONE, VRF-CUST2
  • 10.10.18.0/24 - VRF: L3D-ONE, VRF-CUST1
  • 10.10.19.0/24 - VRF: L3D-ONE, VRF-CUST2

getting the childs of 10.10.16.0/22 - VRF: L3D-ONE, VRF-CUST1 should return

• 10.10.16.0/24 - VRF: L3D-ONE, VRF-CUST1 (because it is in L3D-ONE, VRF-CUST1)
• 10.10.17.0/24 - VRF: L3D-ONE, VRF-CUST2 (because it is in L3D-ONE)
• 10.10.18.0/24 - VRF: L3D-ONE, VRF-CUST1 (because it is in VRF-L3D-ONE, VRF-CUST1)
• 10.10.19.0/24 - VRF: L3D-ONE, VRF-CUST2 (because it is in L3D-ONE)
• 10.10.21.0/24 - VRF: L3D-TWO, VRF-CUST1 (because it is in VRF-CUST1)

thoughts ?
pieter

[DanSheps]

I still don't understand what you are trying to do.

What equipment are you using in the real world, and how is all of this defined in that equipment?

[iunderwood]

The use of a Layer 3 domain via nesting VRFs is an improper way to use VRFs. Prevention of network repetition within a VRF represents the real-world operation of VRFs and is the default setting.

The problem here is an application of unique business logic. At a minimum, this would need a custom field to identify the layer 3 domian, L3D-ONE for instance. This would need to be applied to the prefixes. Then, that logic would need to verify prefix uniqueness across all entries where the layer 3 domain was selected or entered. It would be necessary to disable "enforce unique space" on each of the customer VRFs in order for this to function.

[DetunizedGravity]

I disagree with the business logic argument. IP unicity, hence prefix unicity, is a necessary constraint across one's whole L3 network, no matter the number of interconnected routers and VRF that constitute it. Unicity within a VRF is "the default setting" for obvious reasons. But unicity cannot be limited to one VRF or routing across the network cannot work. Routing in itself can hardly be considered "business logic."

Which is why enhancing Netbox to allow us to enforce a fundamental of routing makes perfect sense to me .

[DetunizedGravity]

Since I can't seem to reply on @jeremystretch 's comment on how one should use only one VRF if one wishes to enforce prefix unicity (technical difficulty: nothing happens when I click on the "reply" field), and since I respectfully disagree (sorry about that :) ) I will make a new comment. Here goes.

In a multi-tenant network one may implement one L3VPN by tenant over several interconnected routers by using one VRF per tenant per router, each VRF exchanging all its routes with the other VRF of the same tenant on the other routers through a shared route target. In that context, IP unicity is desirable across all VRF of the same tenant in most if not all cases.

From Netbox standpoint there is one VRF per router, and there is no technical way to enforce unicity across them, as explained in introduction of this thread.

Another similar case, in the same context, is if we want to register both the tenants' IP plan and keep control of the supernet from which are alloted /30 to locally connect our VRF on our CE and the tenants' physical devices. These prefixes are logically in their respective VRF, but we might want to ensure unicity across all VRF and devices, to keep things exploitable and easy to troubleshoot.

One way to support this second use case is to use a pseudo-VRF to contain the interconnection IP plan, but it is twisting the model, and we lose information in the process.

To support both use cases, I think the proposal made here works, if we add the the tenant to the L3 domain model : a L3 domain would be defined by a list of at least one VRF and a list of at least one tenant.

I know that both proposals could lead to L3 domains overlapping (a same VRF, a same tenant and thus a same prefix could belong to more that one), but I fail to find an issue with that.

[PieterL75]

I don't think mapping a layer3 to a tenant is a good idea.
In my use cases, the prefix is where we put the tenant.

I like the drawing of Brian above. A littke extended:

L3 domain1 ----< VRF1 ----< Prefix
          |           `---< IPAddress
           ----< VRF2 ----< Prefix
                      `---< IPAddress

L3 domain2 ----< VRF3 ----< Prefix
                      `---< IPAddress

IF this gets implemented, then I would no make the l3 domain mandatory, but rather optional.
A VRF without l3 domain, just stands on its own, and preserves uniqueness of prefixes in the vrf.

But if a L3Domain is created and assigned to a VRF, then the uniqueness of the prefix needs to be across all vrf's of that layer3 domain..

Maybe I should give it a try in my fork to see if I can make this work

[DetunizedGravity]

L3 domains overlapping (a same VRF, a same tenant and thus a same prefix could belong to more that one)

That's kind-of the whole point of VRFs: they allow overlapping address space.

Yes. By giving control over the exported routes between VRF. Same export being the very reason one cannot map a VRF to only one L3 domain. I happen to work on a network where each VRF "partially" belongs to between 2 and 3 L3 domains, because of design choices and hardware limitations.

Which is why I proposed to use tenancy as an additional discriminant, On second thought it does not sound like a great idea, I agree.

I thought of making possible to add prefixes directly to L3 domains, independently of VRF, in addition to the existing proposal, but it would not have been practical.

What about defining a L3 domain as the intersection of the set of prefixes and addresses belonging to a list of VRF (@PieterL75 's proposal) and the set of sub-prefixes and addresses included in a list of supernets (NOT prefix model instances, since they must not be linked to any VRF for their intended purpose)?

This way one would be able to say, e.g. "I want one L3 domain per tenant (one list of VRF per tenant) and another L3 domain for the CGN IP plan across all VRFs".

Edit: typo

[candlerb]

I happen to work on a network where each VRF belongs to between 2 and 3 L3 domains, because of design choices and hardware limitations.

Maybe we're using the term "L3 domain" differently. I am using it to mean a namespace: a scope for IP address uniqueness only.

I think you are saying that you want to associate the same prefix with multiple VRFs? In that case, I'm not sure how to deal with uniqueness within sets of VRFs and between sets of VRFs. You could perhaps have:

VRF >---*---< L3 Domain -----< Prefix/IPAddress
                  |
                  ^
              Namespace

(* = many to many join). This lets you inject a prefix or IP address into a set of VRFs. However, each distinct grouping of VRFs would need to be created as an L3 Domain.

[PieterL75]

I would not put the L3Domain between the vrf and the prefix, but really like you drew it the first time.

L3 domain1 ----< VRF1 ----< Prefix
          |           `---< IPAddress
           ----< VRF2 ----< Prefix
                      `---< IPAddress

(unless that is design wise for the models better of course)

For the definition : I see a Layer3 domain as a place where all IP's and prefixes are unique, independent of what VRF they are assigned to.

In my view: One VRF can belong to none, one or more L3Domains
None : Uniqueness of the prefix is only in this VRF
One: Uniqueness of the prefix has to be checked on all VRFs of this one L3Domain
More: Uniqueness of the prefix has to be checked on all VRFs of all the L3Domain

the last one might be a little confusing, but in my 'world' it makes sense.
We are migrating multiple environments (old companies) into one network. Each company will get it's Layer3 domain for the RFC1918 address space. Within a company, we have multiple VRF's each for its purpose, but we can use one prefix only once in the entire company's routing domain (combination of vrfs).
Here we have multiple VRF's with ONE Layer3 domain.

We have multiple environments like that, and are building a routing between them.
This requires a new VRF that connects all Layer3domains of all companies together.
Here we have the one VRF to more layer3 domains.

The logic then is:
If I provision a prefix in a company'vrf, it has to be unique on all vrfs of that company, including the interconnect-vrf, but not the other company's vrfs
if I provision a prefix in the interconnect-vrf it has to be unique in all company vrf's

[candlerb]

In my view: One VRF can belong to none, one or more L3Domains

In that case you need a many-to-many join:

L3 Domain >---*---< VRF -----< Prefix

Note that A ----< B is one-to-many: "[one] A has-many B", "each B belongs-to [one] A".

[DetunizedGravity]

I've given this topic a lot more thoughts and I think we must go back to fundamentals. This is not going in the right direction.

We ask for L3 domains to ensure a unicity of addresses constraint. Why do we need unicity? Two reasons: (i) to allow routing within the domain and for (ii) business reasons (technical debt, exploitability, etc. You name it.)

Now that we know what we want, what do we have in terms of containers for IP addresses that can be used to build L3 domains ?

• Supernets (I intentionally do not use the prefix term not to avoid confusion with the netbox model)
  2 IPs that do not share an ancestor cannot be identical, hence a supernet is by design of IP addressing one boundary of a L3 domain.

• VRF / routing tables (e.g. routers unable to implement VRF, which may be represented in Netbox by a pseudo VRF limited to that device)
  A VRF is at heart a routing table, and a routing table is not supposed to enforce unicity of prefixes or addresses. There is no reason for that, technically. There can be business reasons, which is why Netbox offers the option in its model, but that what it is: an option.
  Moreover VRF can leak/share routes for subnets from one shared supernet to allow inter-VRF routing, while wanting to keep subnets from another supernet private from other VRFs. To model that we will need L3 domains to be able to encompass parts of a set of VRF without embarking the whole content of all VRF of the set.
  In short, a VRF or a routing table can only be one boundary of a L3 domain for business purpose/practical reasons and it must remain optional.

My conclusion: any design of a L3 domain model based on a collection of VRF and all their content can only model a small subset of reality and is bound to fail.

So, to synthetise, a L3 domain is defined by :

• one supernet (the prefix model in netbox might not be the ideal candidate for that purpose, though)
• the topological boundaries of the network where its subnets are routed.

I have no better proposal for the topological part than "a set of VRF". Asking Netbox to deduce it from the network topology would be way too complex.

Finally, for practical reason, it make sense to merge L3 domains that share the same topological boundaries.

So the final model would be:
VRFs <---one to many--- L3 domain ---one to many---> supernets
And the model rule is "any subnet of the linked supernets must be unique across all the linked VRF".
To simply enforce unicity of all prefixes across all listed VRFs one only needs to use 0/0 as the supernet.

[PieterL75]

I like the idea of combining supernets into a l3 domain.
You can indeed say that it is the prefix/supernet that defines the l3domain, and that the vrf is an optional carrier/transport layer of it.
That way, one l3 domain can live across vrf's..

Could we expand aggregates to the function of the supernets ?
I know that currently, we can only create one 10.0.0.0/8 aggregate. but if we can add the layer3 domain to the aggregates, then we could create more 10.0.0.0/8 each linked to its layer domain.

I'm sure there are many other options, so your thoughts ?

[DetunizedGravity]

After thinking about a few way to do it and always finding counter arguments against each I finally came to my sense. I spent to much time trying to find a model. I was probably overthinking that stuff, and falling into the overengineering trap. So once again I went back to fundamentals. The L3 domain is the model we are trying to design and request from core maintainer. This model is the one providing the feature we want, and thus the one that may need to gather all other attributes we may need such as tenancy, description, tags, role, scope (if we want to add physical scope as another type of boundary to the domain), etc. Moreover, we have absolutely no need to apply any model logic between supernets as used in l3 domains. There is no mathematical or technical constraint between supernets used in L3 domains beyond being well formed. I don't care if they overlap with other doamin (I want to be able to make them overlap!). I don't even care if they overlap **inside** a given domain. It doesn't change a damn thing. It all boils down to: we don't need a model to represent supernets in a L3 domain. A collection of values of the "IP address" data type is enough. By the way, how comes I can't find the content of the emails we exchange on the subject in the thread on github? Are they not automatically included? Le jeu. 31 mars 2022 à 22:18, PieterL75 ***@***.***> a écrit :

…

I like the idea of combining supernets into a l3 domain. You can indeed say that it is the prefix/supernet that defines the l3domain, and that the vrf is an optional carrier/transport layer of it. That way, one l3 domain can live across vrf's.. Could we expand aggregates to the function of the supernets ? I know that currently, we can only create one 10.0.0.0/8 aggregate. but if we can add the layer3 domain to the aggregates, then we could create more 10.0.0.0/8 each linked to its layer domain. I'm sure there are many other options, so your thoughts ? — Reply to this email directly, view it on GitHub <#7608 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF45H7UMQNC4OLLZDOW6JDVCYCBJANCNFSM5GOKOYMA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[DetunizedGravity]

OK. So just as I write "how come what we say is not brought back to github", I discover that it actually is. I was just impatient. Let me die.

[DetunizedGravity]

I just thought that IP ranges may also define boundaries of L3 domains. I don't know the data-type and/or models that are behind those and if they may be used that way.

[ajackson79]

This would definitely be a great feature. Our solution today for this is to reserve the prefix in all shared vrfs. This works, but is not pretty to look at and can be a bit cumbersome to use.