Render configuration templates natively #7547

@renatoalmeidaoliveira commented on GitHub (Jan 30, 2023):

@jeremystretch I think it would be nice to have a "composable" environment for configuration templates, for example:

when deploying a device you can have:

A base template for AAA, SNMP, license and other stuff
A particular configuration for some device type, or scenario

That way having several base templates, and a configuration template that is a composable of the bases may be useful

Another point is how to add variables inside that template, in such way it could be used to render some configuration to deploy a new device or inside a script.

@renatoalmeidaoliveira commented on GitHub (Jan 30, 2023): @jeremystretch I think it would be nice to have a "composable" environment for configuration templates, for example: when deploying a device you can have: - A base template for AAA, SNMP, license and other stuff - A particular configuration for some device type, or scenario That way having several base templates, and a configuration template that is a composable of the bases may be useful Another point is how to add variables inside that template, in such way it could be used to render some configuration to deploy a new device or inside a script.

adam commented

@ryanmerolle commented on GitHub (Feb 2, 2023):

@carlbuchmann, @titom73, @ClausHolbechArista

There might be some good synergies here with ansible avd if you have thoughts.

@ryanmerolle commented on GitHub (Feb 2, 2023): @carlbuchmann, @titom73, @ClausHolbechArista There might be some good synergies here with ansible avd if you have thoughts.

adam commented

@jeremystretch commented on GitHub (Feb 8, 2023):

@renatoalmeidaoliveira my intent here is to determine per device the appropriate base template, which can pull in other templates via {% include %} tags. These inclusion templates can be reused by many templates. This approach has worked very well for me in the past, and mimics NetBox's own template hierarchy.

@jeremystretch commented on GitHub (Feb 8, 2023): @renatoalmeidaoliveira my intent here is to determine per device the appropriate base template, which can pull in other templates via `{% include %}` tags. These inclusion templates can be reused by many templates. This approach has worked very well for me in the past, and mimics NetBox's own template hierarchy.

adam commented

@renatoalmeidaoliveira commented on GitHub (Feb 8, 2023):

Intresting, very similar to the templates environment, that way the user gonna have several "base" templates and for a full configuration he build a template import the base ones.
It seens interesting, but how you gonna import "as a template" some model attribute?

@renatoalmeidaoliveira commented on GitHub (Feb 8, 2023): Intresting, very similar to the templates environment, that way the user gonna have several "base" templates and for a full configuration he build a template import the base ones. It seens interesting, but how you gonna import "as a template" some model attribute?

adam commented

@jeremystretch commented on GitHub (Feb 10, 2023):

So far, I've got the template rendering working as planned, however I'm getting a little hung up on deciding how best to relate devices to specific templates. To review, each device should reference exactly one (if any) configuration template, which can be rendered using context data to produce the desired configuration of that device.

It's entirely possible to designate a single global base template for all devices, and employ logic to evaluate a device's role, platform, etc. to determine which other relevant template to pull in (via {% include %} tags). However, this is probably not ideal, and I expect that people will want some mechanism native to NetBox for selecting which of several base templates to use for a device. I see a few options, detailed below.

Option A: Add a `template` ForeignKey to Device

This is pretty straightforward: Just add a ForeignKey to the Device model so that each device can point to its appropriate ConfigTemplate. This allows for very efficient queries at the cost of requiring explicit assignment for each device. It's also the safest option, as a misconfiguration will affect only the single device on which it's made.

Option B: Add a `template` ForeignKey to DeviceRole and/or Platform

Rather than specifying templates per device, we could specify them per device role and/or platform. This would require much less explicit configuration than option A, however we run into an issue if a device's role and platform each indicate a different template. It's also likely users may want to classify devices by some other attribute, such as assigned tag, for the purpose of determining which template to render.

(This option could be combined with option A, where a device's individually-defined template would take precedence over that associated with its role or platform.)

Option C: Add Many-to-Many Assignments on ConfigTemplate

This approaches the problem from a different direction, associating each template with a set of device roles and/or platforms. To determine the appropriate template for a device, a query is made for a template with matching role & platform. (This is similar in concept to how config contexts are retrieved.) This grants a great deal of flexibility, however it would also require some form of tie-breaking mechanism in case multiple matching templates are found.

For example, suppose you associate template A with device role X and template B with platform Y. A device having role X and platform Y would match on both templates. We would need to include a factor such as numeric weighting to prefer one over the other; or return an error.

As with option B, there's a danger here in that a minor change to a template (e.g. removing an assigned role) can break template rendering for a large number of devices; this may be an acceptable drawback considering the flexibility afforded. However, it also complicates retrieving associated device templates in bulk.

This approach could also be combined with option A, which would provide an alternative, explicit assignment for more conservative deployments.

@jeremystretch commented on GitHub (Feb 10, 2023): So far, I've got the template rendering working as planned, however I'm getting a little hung up on deciding how best to relate devices to specific templates. To review, each device should reference exactly one (if any) configuration template, which can be rendered using context data to produce the desired configuration of that device. It's entirely possible to designate a single global base template for all devices, and employ logic to evaluate a device's role, platform, etc. to determine which other relevant template to pull in (via `{% include %}` tags). However, this is probably not ideal, and I expect that people will want some mechanism native to NetBox for selecting which of several base templates to use for a device. I see a few options, detailed below. ### Option A: Add a `template` ForeignKey to Device This is pretty straightforward: Just add a ForeignKey to the Device model so that each device can point to its appropriate ConfigTemplate. This allows for very efficient queries at the cost of requiring explicit assignment for each device. It's also the safest option, as a misconfiguration will affect only the single device on which it's made. ### Option B: Add a `template` ForeignKey to DeviceRole and/or Platform Rather than specifying templates per device, we could specify them per device role and/or platform. This would require much less explicit configuration than option A, however we run into an issue if a device's role and platform each indicate a different template. It's also likely users may want to classify devices by some other attribute, such as assigned tag, for the purpose of determining which template to render. (This option could be combined with option A, where a device's individually-defined template would take precedence over that associated with its role or platform.) ### Option C: Add Many-to-Many Assignments on ConfigTemplate This approaches the problem from a different direction, associating each template with a set of device roles and/or platforms. To determine the appropriate template for a device, a query is made for a template with matching role & platform. (This is similar in concept to how config contexts are retrieved.) This grants a great deal of flexibility, however it would also require some form of tie-breaking mechanism in case multiple matching templates are found. For example, suppose you associate template A with device role X and template B with platform Y. A device having role X and platform Y would match on both templates. We would need to include a factor such as numeric weighting to prefer one over the other; or return an error. As with option B, there's a danger here in that a minor change to a template (e.g. removing an assigned role) can break template rendering for a large number of devices; this may be an acceptable drawback considering the flexibility afforded. However, it also complicates retrieving associated device templates in bulk. This approach could also be combined with option A, which would provide an alternative, explicit assignment for more conservative deployments.

adam commented

@falz commented on GitHub (Feb 10, 2023):

Fond of Option B but allowing option A to override the role/platform per device seems like the sweet spot to me.

Option C sounds good on paper but seems like excess complexity.

Note that we wrote our own config generator using Netbox, and may have further feedback on this based on our experiences once we see what data is available to the template. Will chime in with more details on that when the time is appropriate.

@falz commented on GitHub (Feb 10, 2023): Fond of Option **B** but allowing option **A** to override the role/platform per device seems like the sweet spot to me. Option C sounds good on paper but seems like excess complexity. Note that we wrote our own config generator using Netbox, and may have further feedback on this based on our experiences once we see what data is available to the template. Will chime in with more details on that when the time is appropriate.

adam commented

@zn0k commented on GitHub (Feb 10, 2023):

I concur that option C, while powerful, could get very confusing. Given that the penalty is overwriting the config of a production device and taking it down, simple seems better.

A mixture of options A and B with a slight tweak seems best to me as well. Being able to set defaults via a combination of platform (so we can differentiate between IOS and Junos, for example) and role while retaining the ability to override on a per-device basis by directly assigning a template would be reasonably simple but also allow enough escape hatches to solve slightly trickier situations. I do think it's necessary to allow the combination of platform and role to determine the template - many networks are multi-platform. Or, if you're on Cisco, you're not very unlikely to run multiple dialects between legacy and newer devices.

@zn0k commented on GitHub (Feb 10, 2023): I concur that option C, while powerful, could get very confusing. Given that the penalty is overwriting the config of a production device and taking it down, simple seems better. A mixture of options A and B with a slight tweak seems best to me as well. Being able to set defaults via a combination of platform (so we can differentiate between IOS and Junos, for example) and role while retaining the ability to override on a per-device basis by directly assigning a template would be reasonably simple but also allow enough escape hatches to solve slightly trickier situations. I do think it's necessary to allow the combination of platform and role to determine the template - many networks are multi-platform. Or, if you're on Cisco, you're not very unlikely to run multiple dialects between legacy and newer devices.

adam commented

@PieterL75 commented on GitHub (Feb 10, 2023):

I would like to be able to run multiple templates to a device.

Initial template to setup the device (hostname, tacacs, snmp, mgmt ip, mgmt route, ....)
MLAG template to setup an MLAG
Layer2 template to setup STP, VLAN
Layer3 template to add routing
other templates that do small things

An ofcourse, each template has to be valid for that vendor/syntax.

imo, this should only create a config out of a template, but never deploy it direct.
That does not seems like the task of NetBox

@PieterL75 commented on GitHub (Feb 10, 2023): I would like to be able to run multiple templates to a device. - Initial template to setup the device (hostname, tacacs, snmp, mgmt ip, mgmt route, ....) - MLAG template to setup an MLAG - Layer2 template to setup STP, VLAN - Layer3 template to add routing - other templates that do small things An ofcourse, each template has to be valid for that vendor/syntax. imo, this should only create a config out of a template, but never deploy it direct. That does not seems like the task of NetBox

adam commented

@jeremystretch commented on GitHub (Feb 10, 2023):

I would like to be able to run multiple templates to a device.

You can do this by creating a base template which includes other (reusable) templates, in the desired order and subject to any logic asserting specific device attributes. For example:

{% include 'common/system.jinja2' %}
{% include 'common/interfaces.jinja2' %}
{% include 'common/access-lists.jinja2' %}
{% if device.role.slug == 'some-special-role' %}
  {% include 'special/other-stuff.jinja2' %}
{% endif %}

You'll likely have different base templates for each role/platform.

I also want to point out that you do not need to create a corresponding ConfigTemplate instance in NetBox to accomplish this: Included templates can be populated directly from other remotely-sourced data files. You'll only need to create a ConfigTemplate for each template that you want to render directly.

imo, this should only create a config out of a template, but never deploy it direct.

100% 👍 - There are plenty of other tools for that.

@jeremystretch commented on GitHub (Feb 10, 2023): > I would like to be able to run multiple templates to a device. You can do this by creating a base template which includes other (reusable) templates, in the desired order and subject to any logic asserting specific device attributes. For example: ``` {% include 'common/system.jinja2' %} {% include 'common/interfaces.jinja2' %} {% include 'common/access-lists.jinja2' %} {% if device.role.slug == 'some-special-role' %} {% include 'special/other-stuff.jinja2' %} {% endif %} ``` You'll likely have different base templates for each role/platform. I also want to point out that you do *not* need to create a corresponding ConfigTemplate instance in NetBox to accomplish this: Included templates can be populated directly from other remotely-sourced data files. You'll only need to create a ConfigTemplate for each template that you want to render directly. > imo, this should only create a config out of a template, but never deploy it direct. 100% :+1: - There are plenty of other tools for that.

adam commented

@ryanmerolle commented on GitHub (Feb 10, 2023):

Not trying to pile on, but just share a current workflow to broaden an understanding of how users currently approach.

I currently do this off box, and my templates get applied to a "function" and platform. Functions are assigned to device roles (the netbox model). Here are a few examples:

NetBox Device Roles: border_leaf, edge_leaf, storage_leaf, spine, internet_firewall, office_firewall, b2b_firewall, console_server, mgmt_core, mgmt_access, console_server, voice_gateway, user_l2_leaf, user_l3_leaf, pdu, ups
NetBox Platforms: eos, ios, nx-os, opengear, fortios, panos, aos
Templates Assigned to a network function: b2b_peering, leaf_peering, compute_access, storage_leaf, underlay_spine_leaf, evpn_overlay, vxlan, base, wan

I think map my made up functions to NetBox Device Roles like:

device_role_definitions:
  border_leaf:
    - base (applies the device platform's base)
    - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config)
    - wan (bgp, prefix-lists, community-lists, route-maps, circuit terminations w/descriptions, bfd, etc.)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - vxlan
    - border_leaf (specific border_leaf only config)
    - storage_leaf (specific config if a location is so small that storage_leaf role is consolidated on this device)
  storage_leaf:
    - base (applies the device platform's base)
    - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - vxlan
    - storage_leaf (storage network specific items)
  spine:
    - base (applies the device platform's base)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - spine (specific spine only config like summary networks)

All of this is defined by using include templates like @jeremystretch mentioned. "Functions" are not really modeled in netbox, but in the templates that define each device role. All of the above templates are created as needed, and if I bring up a JUNOS border_leaf in the future, the template would fail (as it should) until platform specific templates are created for all the boarder_leaf functions.

I probably would continue to do this off box because I produce additional artifacts and upload all to git:

Single config file per device in cli platform format
Single config file per device in json/yaml form
Single markdown document per device
Topology Docs (markdown, yaml, etc) per site
Snapshot of netbox data used for the config build (per device, per site, etc)

@ryanmerolle commented on GitHub (Feb 10, 2023): Not trying to pile on, but just share a current workflow to broaden an understanding of how users currently approach. I currently do this off box, and my templates get applied to a "function" and platform. Functions are assigned to device roles (the netbox model). Here are a few examples: - **NetBox Device Roles:** border_leaf, edge_leaf, storage_leaf, spine, internet_firewall, office_firewall, b2b_firewall, console_server, mgmt_core, mgmt_access, console_server, voice_gateway, user_l2_leaf, user_l3_leaf, pdu, ups - **NetBox Platforms:** eos, ios, nx-os, opengear, fortios, panos, aos - **Templates Assigned to a network function:** b2b_peering, leaf_peering, compute_access, storage_leaf, underlay_spine_leaf, evpn_overlay, vxlan, base, wan I think map my made up functions to NetBox Device Roles like: ```yaml device_role_definitions: border_leaf: - base (applies the device platform's base) - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config) - wan (bgp, prefix-lists, community-lists, route-maps, circuit terminations w/descriptions, bfd, etc.) - underlay_spine_leaf (fiber uplinks, bgp peering, etc.) - evpn_overlay (evpn, bgp, etc) - vxlan - border_leaf (specific border_leaf only config) - storage_leaf (specific config if a location is so small that storage_leaf role is consolidated on this device) storage_leaf: - base (applies the device platform's base) - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config) - underlay_spine_leaf (fiber uplinks, bgp peering, etc.) - evpn_overlay (evpn, bgp, etc) - vxlan - storage_leaf (storage network specific items) spine: - base (applies the device platform's base) - underlay_spine_leaf (fiber uplinks, bgp peering, etc.) - evpn_overlay (evpn, bgp, etc) - spine (specific spine only config like summary networks) ``` All of this is defined by using include templates like @jeremystretch mentioned. "Functions" are not really modeled in netbox, but in the templates that define each device role. All of the above templates are created as needed, and if I bring up a JUNOS border_leaf in the future, the template would fail (as it should) until platform specific templates are created for all the boarder_leaf functions. I probably would continue to do this off box because I produce additional artifacts and upload all to git: - Single config file per device in cli platform format - Single config file per device in json/yaml form - Single markdown document per device - Topology Docs (markdown, yaml, etc) per site - Snapshot of netbox data used for the config build (per device, per site, etc)

adam commented

@ryanmerolle commented on GitHub (Feb 10, 2023):

Oh I forgot to add, I feel like the Option C, similar to how config_context can be applied ultimate gives use the most flexibility, but its likely a trade off with complexity/confusion.

@ryanmerolle commented on GitHub (Feb 10, 2023): Oh I forgot to add, I feel like the Option C, similar to how config_context can be applied ultimate gives use the most flexibility, but its likely a trade off with complexity/confusion.

adam commented

@jeremystretch commented on GitHub (Feb 10, 2023):

For now, I've just added the foreign key from Device to ConfigTemplate, since it seems like we'll almost certainly want the ability to specify an explicit assignment regardless. Making really great progress with this already!

@jeremystretch commented on GitHub (Feb 10, 2023): For now, I've just added the foreign key from Device to ConfigTemplate, since it seems like we'll almost certainly want the ability to specify an explicit assignment regardless. Making really great progress with this already! ![Screenshot 2023-02-10 at 16-34-47 dmi01-akron-sw01 - Config NetBox](https://user-images.githubusercontent.com/13487278/218203500-95dab75b-f66d-4519-b440-0b2a8e88949c.png)

adam commented

@davidgwatkins commented on GitHub (Feb 12, 2023):

This is really awesome progress! Any chance that could be modified to be Device to Many? We have J2 templates that are CLI for Day 0 / out-of-band / sneakernet, then NETCONF for Day 2 use cases; which gets rendered is based on device status. Perhaps this is covered under the use case for multiple component templates, but we are already using includes.

@davidgwatkins commented on GitHub (Feb 12, 2023): This is really awesome progress! Any chance that could be modified to be Device to Many? We have J2 templates that are CLI for Day 0 / out-of-band / sneakernet, then NETCONF for Day 2 use cases; which gets rendered is based on device status. Perhaps this is covered under the use case for multiple component templates, but we are already using includes.

adam commented

@jeremystretch commented on GitHub (Feb 17, 2023):

Pretty happy with where this ended up. No doubt we'll continue to refine & improve the implementation from here, but I'm calling it done for the purposes of this particular FR.

Any chance that could be modified to be Device to Many?

You'll be able to implement conditional logic within the template itself, which could (for example) check for the presence of a flag indicating which type of config to render, and pull in the relevant subordinate template files.

@jeremystretch commented on GitHub (Feb 17, 2023): Pretty happy with where this ended up. No doubt we'll continue to refine & improve the implementation from here, but I'm calling it done for the purposes of this particular FR. > Any chance that could be modified to be Device to Many? You'll be able to implement conditional logic within the template itself, which could (for example) check for the presence of a flag indicating which type of config to render, and pull in the relevant subordinate template files.

adam commented

@dmulyalin commented on GitHub (Mar 25, 2023):

This is great.

Late to the party, but would like to share an idea.

It would be good to associate config templates with services as well. Say service L3VPN for customer assigned to the device, service has template associated matching device platform to produce requied configuration.

@dmulyalin commented on GitHub (Mar 25, 2023): This is great. Late to the party, but would like to share an idea. It would be good to associate config templates with services as well. Say service L3VPN for customer assigned to the device, service has template associated matching device platform to produce requied configuration.

adam commented