Allow reading config context data from a file on disk #6319

New Issue

adam · 2025-12-29T19:39:19+01:00

adam commented

2025-12-29 19:39:19 +01:00

Originally created by @jeremystretch on GitHub (Apr 7, 2022).

Originally assigned to: @jeremystretch on GitHub.

NetBox version

v3.2.0

Feature type

New functionality

Proposed functionality

Currently, when creating a config context, the user must JSON populate data on the object, which gets stored in the database. This issues proposes providing the option of instead specifying the path to a file on disk (within one of the configured directories) from which data can be read. This will take the place of locally-stored data; a config context instance can define local data or a source file, but not both.

Data from a file will be read at initialization of the config context instance and retained until its deletion. Some degree of caching may also be supported, but further research is needed to determine its impact and feasibility.

Implementation will likely involve the addition of a new configuration parameter to define the permitted file paths.

Use case

This approach allows users to reference external data where necessary e.g. to better accommodate a change control process, while still empowering users to store data locally where sufficient.

Database changes

An optional data_file FilePath field will be added to the ConfigContext model, and the existing data field will become optional. Model validation will ensure that one of these two fields has been populated on save. Additionally, model validation will handle validating the source data upon save.

External dependencies

No response

Originally created by @jeremystretch on GitHub (Apr 7, 2022). Originally assigned to: @jeremystretch on GitHub. ### NetBox version v3.2.0 ### Feature type New functionality ### Proposed functionality Currently, when creating a config context, the user must JSON populate data on the object, which gets stored in the database. This issues proposes providing the option of instead specifying the path to a file on disk (within one of the configured directories) from which data can be read. This will take the place of locally-stored data; a config context instance can define local data _or_ a source file, but not both. Data from a file will be read at initialization of the config context instance and retained until its deletion. Some degree of caching may also be supported, but further research is needed to determine its impact and feasibility. Implementation will likely involve the addition of a new configuration parameter to define the permitted file paths. ### Use case This approach allows users to reference external data where necessary e.g. to better accommodate a change control process, while still empowering users to store data locally where sufficient. ### Database changes An optional `data_file` FilePath field will be added to the ConfigContext model, and the existing `data` field will become optional. Model validation will ensure that one of these two fields has been populated on save. Additionally, model validation will handle validating the source data upon save. ### External dependencies _No response_

adam added the status: accepted type: feature labels 2025-12-29 19:39:19 +01:00

adam closed this issue

2025-12-29 19:39:20 +01:00

adam commented

2025-12-29 19:39:21 +01:00

@jeremystretch commented on GitHub (Apr 7, 2022):

Related: #8505

@jeremystretch commented on GitHub (Apr 7, 2022): Related: #8505

adam commented

2025-12-29 19:39:21 +01:00

@jasonyates commented on GitHub (Apr 10, 2022):

For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file?

One of my hesitations to using config contexts in automation against my devices is there's no peer review process. An integration with Github would allow us to store the config contexts there and require a PR etc. If Netbox then had a supportable API, you could configure a webhook on the GH repository to trigger Netbox to pull the latest versions on a successful merge.

@jasonyates commented on GitHub (Apr 10, 2022): For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file? One of my hesitations to using config contexts in automation against my devices is there's no peer review process. An integration with Github would allow us to store the config contexts there and require a PR etc. If Netbox then had a supportable API, you could configure a webhook on the GH repository to trigger Netbox to pull the latest versions on a successful merge.

adam commented

2025-12-29 19:39:22 +01:00

@jeremystretch commented on GitHub (Apr 10, 2022):

For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file?

You can certainly do that, although the revision tracking function (e.g. git) can operate outside of NetBox. (Given the support for remote storage, it can even happen on some other system entirely.) You would just define in NetBox the path to a file that happens to live in a git repo.

@jeremystretch commented on GitHub (Apr 10, 2022): > For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file? You can certainly do that, although the revision tracking function (e.g. git) can operate outside of NetBox. (Given the support for remote storage, it can even happen on some other system entirely.) You would just define in NetBox the path to a file that happens to live in a git repo.

adam commented

2025-12-29 19:39:22 +01:00

@jcralbino commented on GitHub (Apr 17, 2022):

I would like to see this type of git remote file. The issue I see here is related to how the change logging inside netbox would appear when changes in the config context data are made outside netbox

I would like to have a better view of what changed in the configuration context information also within the change log of netbox

@jcralbino commented on GitHub (Apr 17, 2022): I would like to see this type of git remote file. The issue I see here is related to how the change logging inside netbox would appear when changes in the config context data are made outside netbox I would like to have a better view of what changed in the configuration context information also within the change log of netbox

adam commented

2025-12-29 19:39:22 +01:00

@jeremystretch commented on GitHub (Apr 18, 2022):

I would like to have a better view of what changed in the configuration context information also within the change log of netbox

This would be tracked by e.g. git, not NetBox. NetBox would track only changes to the ConfigContext object itself (such as if the file path is modified). This ensures a clear delineation of revision control.

@jeremystretch commented on GitHub (Apr 18, 2022): > I would like to have a better view of what changed in the configuration context information also within the change log of netbox This would be tracked by e.g. git, not NetBox. NetBox would track only changes to the ConfigContext object itself (such as if the file path is modified). This ensures a clear delineation of revision control.

adam commented

2025-12-29 19:39:22 +01:00

@ryanmerolle commented on GitHub (Apr 28, 2022):

Yea I think this makes sense. Have got track the file and have netbox be able to reference git.

@ryanmerolle commented on GitHub (Apr 28, 2022): Yea I think this makes sense. Have got track the file and have netbox be able to reference git.

adam commented

2025-12-29 19:39:22 +01:00

@jcralbino commented on GitHub (May 4, 2022):

For me that makes sense to have this separation for revision control well defined and I not against it

I would like to have only a easy way to show the information that git has for that object in the user interface.
Maybe we could store that information cached locally and display it in the user interface when we are using git remote files ?

@jcralbino commented on GitHub (May 4, 2022): For me that makes sense to have this separation for revision control well defined and I not against it I would like to have only a easy way to show the information that git has for that object in the user interface. Maybe we could store that information cached locally and display it in the user interface when we are using git remote files ?

adam commented

2025-12-29 19:39:22 +01:00

@jeremystretch commented on GitHub (May 13, 2022):

Currently we are aggregating config context data as part of the database query, which is not possible when dealing with data from files. We have two options for implementing this:

Gather the JSON data from applicable config contexts as part of the query, but render it (along with the data from files) in-memory within the view.
Dictate that all database contexts take precedence over all file-based contexts, or vice versa.
Synchronize the file data to the database via some yet to be determined means.

Personally I strongly prefer the first option, however I'm concerned about performance. The second option, a hybrid approach, would work but assumes that an organization uses predominantly one type or the other. Attempting to keep the database data in sync with the source files is likely to be extremely unreliable (as well as inefficient).

@jeremystretch commented on GitHub (May 13, 2022): Currently we are aggregating config context data as part of the database query, which is not possible when dealing with data from files. We have two options for implementing this: 1. Gather the JSON data from applicable config contexts as part of the query, but render it (along with the data from files) in-memory within the view. 2. Dictate that all database contexts take precedence over all file-based contexts, or vice versa. 3. Synchronize the file data to the database via some yet to be determined means. Personally I strongly prefer the first option, however I'm concerned about performance. The second option, a hybrid approach, would work but assumes that an organization uses predominantly one type or the other. Attempting to keep the database data in sync with the source files is likely to be extremely unreliable (as well as inefficient).

adam commented

2025-12-29 19:39:23 +01:00

@ryanmerolle commented on GitHub (May 18, 2022):

After talking about this a little, it feels like the feature needs to be fleshed out a little.

A couple of items I am thinking about off the bat:

How do users get the file to NetBox? Are we expecting them to have access to the server or container that netbox is running on to then copy the file or git clone a remote repo? That seems shortsighted because most users, besides the deploying admin, will not have access to said file system. This is especially true for the SAAS version of NetBox. - I think a feature may have to come before this to allow git repos or http hosted files via the ui. This could be then used for config_context, export_templates, and other extensibility features
When/how will files be checked for updates and loaded as config_context?

As more items come to mind about gaps OR how to approach this, I will comment. I just rather get the discussion going before going to far down a thought on my own.

Since nautobot constantly lifts ideas from NetBox, I am listing some references form their implementation:

https://github.com/nautobot/nautobot/tree/develop/nautobot/extras/datasources

@ryanmerolle commented on GitHub (May 18, 2022): After talking about this a little, it feels like the feature needs to be fleshed out a little. A couple of items I am thinking about off the bat: 1. How do users get the file to NetBox? Are we expecting them to have access to the server or container that netbox is running on to then copy the file or git clone a remote repo? That seems shortsighted because most users, besides the deploying admin, will not have access to said file system. This is especially true for the SAAS version of NetBox. - *I think a feature may have to come before this to allow git repos or http hosted files via the ui. This could be then used for config_context, export_templates, and other extensibility features* 2. When/how will files be checked for updates and loaded as config_context? As more items come to mind about gaps OR how to approach this, I will comment. I just rather get the discussion going before going to far down a thought on my own. Since nautobot constantly lifts ideas from NetBox, I am listing some references form their implementation: - https://github.com/nautobot/nautobot/tree/develop/nautobot/extras/datasources

adam commented

2025-12-29 19:39:23 +01:00

@jeremystretch commented on GitHub (May 18, 2022):

I think I like the idea of allowing a user to specify a generic "source" URL for a config context. This would point to either a local file (e.g. file://foo/bar/baz.json) or a remote resource. When and how the content of that file gets updated is beyond the scope of NetBox's control. I imagine the most common scenario would see the source file managed by some revision control system (e.g. git).

Regarding the aggregation of data (see my comment above), the most efficient approach is likely to be copying the source data into the database, where it can be aggregated just like "native" config contexts. There are various mechanisms by which we can effect this replication: a UI button, a REST API endpoint, a scheduled task, etc. Ultimately it will be up to the user to determine when this data is replicated, but we should ensures options are available for both automatic and manual synchronizations.

@jeremystretch commented on GitHub (May 18, 2022): I think I like the idea of allowing a user to specify a generic "source" URL for a config context. This would point to either a local file (e.g. `file://foo/bar/baz.json`) or a remote resource. When and how the content of that file gets updated is beyond the scope of NetBox's control. I imagine the most common scenario would see the source file managed by some revision control system (e.g. git). Regarding the aggregation of data (see my comment above), the most efficient approach is likely to be copying the source data into the database, where it can be aggregated just like "native" config contexts. There are various mechanisms by which we can effect this replication: a UI button, a REST API endpoint, a scheduled task, etc. Ultimately it will be up to the user to determine when this data is replicated, but we should ensures options are available for both automatic and manual synchronizations.

adam commented

2025-12-29 19:39:23 +01:00

@jeremystretch commented on GitHub (Jun 28, 2022):

Given that there hasn't been a ton of interest in this proposal, I'm going to shelve it for now. We may want to spend more time considering how we might use this same pattern elsewhere in NetBox and plan accordingly.

@jeremystretch commented on GitHub (Jun 28, 2022): Given that there hasn't been a ton of interest in this proposal, I'm going to shelve it for now. We may want to spend more time considering how we might use this same pattern elsewhere in NetBox and plan accordingly.

adam commented

2025-12-29 19:39:23 +01:00

@ryanmerolle commented on GitHub (Oct 13, 2022):

This would be useful for more than just config context.

I can envision some scripts, reports, plugins that I would build to either read from the latest git commit or push commits to git.

One example would be for configuration compliance. I would connect to the following git repos:

Daily config backups
YAML based config_context
Roles based jinja templates defining intended config

@ryanmerolle commented on GitHub (Oct 13, 2022): This would be useful for more than just config context. I can envision some scripts, reports, plugins that I would build to either read from the latest git commit or push commits to git. One example would be for configuration compliance. I would connect to the following git repos: - Daily config backups - YAML based config_context - Roles based jinja templates defining intended config

adam commented

2025-12-29 19:39:24 +01:00

@jeremystretch commented on GitHub (Jan 21, 2023):

Marking this as blocked by #11558, which seeks to implement support for remote data replication in a more abstract sense.

@jeremystretch commented on GitHub (Jan 21, 2023): Marking this as blocked by #11558, which seeks to implement support for remote data replication in a more abstract sense.

adam commented

2025-12-29 19:39:24 +01:00

@jeremystretch commented on GitHub (Feb 2, 2023):

Happy to report that #11558 has been completed and work on this feature can now move forward.

@jeremystretch commented on GitHub (Feb 2, 2023): Happy to report that #11558 has been completed and work on this feature can now move forward.

Sign in to join this conversation.

Branches Tags

main

update-changelog-comments-docs

feature-removal-issue-type

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#6319