starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

pin *all* dependencies in requirements.txt #3348

New Issue
Closed
opened 2025-12-29 18:28:05 +01:00 by adam · 8 comments
adam commented 2025-12-29 18:28:05 +01:00
Owner
Copy Link

Originally created by @candlerb on GitHub (Feb 19, 2020).

Environment

  • Python version: 3.5.2
  • NetBox version: 2.7.6

Proposed Functionality

In my opinion, netbox should pin the version of all packages it uses - not just direct dependencies but all descendant dependencies as well.

Use Case

People are repeatedly running into issues where Netbox breaks because a child or grandchild dependency package is at an unexpected version.

I personally hit it in #3460. I hit it again with PyYAML installing the wrong version under Ubuntu 16.04. However there are numerous recent cases in google groups where people have broken systems and the fixes involve upgrading or downgrading random packages (e.g. markdown, django-restframework).

I think Netbox has become unsupportable unless one of two things happen:

  1. Release an official docker container, and this is the only supported artefact - i.e. if you choose to install from pip then you are a developer and get to fix your own problems; or
  2. Netbox pins all dependencies to fixed versions in requirements.txt

There may also be some improvement by getting all users to install in a fresh virtualenv (not polluted by any packages they've already installed, or any OS-supplied python packages), but that doesn't solve the problem completely.

Database Changes

None

External Dependencies

This is to some extent caused by a long-standing bug in pip which does not handle recursive dependencies properly. If A installs B and C, and B and C both depend on D (but with different version requirements), a version of D may be installed which meets one requirements but not both, even though a suitable version which meets both is available.

However even if that bug did not exist, pip3 could still end up installing a version of D which meets the declared requirements of B and C, but for some reason does not work in the context of Netbox. This could be because of some missing dependency (Netbox actually uses D directly but doesn't declare it); or it could be because of some other unexpected interaction.

To take one specific example see this post, where the output of "pip3 list" from an end user is very different to what my own system shows - and by extension, is likely to be different to what the Netbox developers have.

If this happens, it means that (a) developers aren't going to be aware of problems; (b) end-users are going to experience these problems first; and (c) we end up trying to support people's systems with problems we haven't seen before and whose systems are in indeterminate states.

Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range.

Originally created by @candlerb on GitHub (Feb 19, 2020). ### Environment * Python version: 3.5.2 * NetBox version: 2.7.6 ### Proposed Functionality In my opinion, netbox should pin the version of *all* packages it uses - not just direct dependencies but all descendant dependencies as well. ### Use Case People are repeatedly running into issues where Netbox breaks because a child or grandchild dependency package is at an unexpected version. I personally hit it in #3460. I hit it again with PyYAML installing the wrong version under Ubuntu 16.04. However there are numerous recent cases in google groups where people have broken systems and the fixes involve upgrading or downgrading random packages (e.g. markdown, django-restframework). I think Netbox has become unsupportable unless one of two things happen: 1. Release an official docker container, and this is the *only* supported artefact - i.e. if you choose to install from pip then you are a developer and get to fix your own problems; or 2. Netbox pins *all* dependencies to fixed versions in requirements.txt There may also be some improvement by getting all users to install in a fresh virtualenv (not polluted by any packages they've already installed, or any OS-supplied python packages), but that doesn't solve the problem completely. ### Database Changes None ### External Dependencies This is to some extent caused by a [long-standing bug in pip](https://github.com/pypa/pip/issues/988) which does not handle recursive dependencies properly. If A installs B and C, and B and C both depend on D (but with different version requirements), a version of D may be installed which meets one requirements but not both, even though a suitable version which meets both is available. However even if that bug did not exist, pip3 could still end up installing a version of D which meets the declared requirements of B and C, but for some reason does not work in the context of Netbox. This could be because of some missing dependency (Netbox actually uses D directly but doesn't declare it); or it could be because of some other unexpected interaction. To take one specific example see [this post](https://groups.google.com/d/msg/netbox-discuss/6e_DkTbBDr8/sbUdMimgBQAJ), where the output of "pip3 list" from an end user is very different to what my own system shows - and by extension, is likely to be different to what the Netbox developers have. If this happens, it means that (a) developers aren't going to be aware of problems; (b) end-users are going to experience these problems first; and (c) we end up trying to support people's systems with problems we haven't seen before and whose systems are in indeterminate states. Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range.
adam closed this issue 2025-12-29 18:28:05 +01:00
adam commented 2025-12-29 18:28:06 +01:00
Author
Owner
Copy Link

@chicks-net commented on GitHub (Feb 19, 2020):

Have you considered pipenv ? It seems to solve the virtualenv creation and python package pinning that you're looking for.

@chicks-net commented on GitHub (Feb 19, 2020): Have you considered [pipenv](https://realpython.com/pipenv-guide/) ? It seems to solve the virtualenv creation and python package pinning that you're looking for.
adam commented 2025-12-29 18:28:06 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Feb 19, 2020):

In my opinion, netbox should pin the version of all packages it uses - not just direct dependencies but all descendant dependencies as well.

Are you asserting that there's something unique to NetBox that requires this, or that this is true for all projects which employ pip for dependency management?

@jeremystretch commented on GitHub (Feb 19, 2020): > In my opinion, netbox should pin the version of all packages it uses - not just direct dependencies but all descendant dependencies as well. Are you asserting that there's something unique to NetBox that requires this, or that this is true for _all_ projects which employ pip for dependency management?
adam commented 2025-12-29 18:28:06 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Feb 19, 2020):

The latter. This is what "pip freeze" exists for.

@candlerb commented on GitHub (Feb 19, 2020): The latter. This is what "pip freeze" exists for.
adam commented 2025-12-29 18:28:07 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Feb 19, 2020):

Release an official docker container, and this is the only supported artefact

Just want to go on record to point out that this is a non-starter: We won't require users to implement an entire layer of abstraction and deploy external services simply to run the application.

I'm not necessarily opposed to adopting this as policy, but we should consider the implications. First, this would effectively require users to run NetBox in a virtual environment (if not using a dedicated server already). This isn't a bad thing IMO, but it does amount to a new installation requirement that needs to be communicated and documented. Incidentally, this initiative has already been captured as #3949.

Second, it means the maintainers must now vet a specific release version for every dependent package: There are currently almost 60, as opposed to the 25 direct dependencies we currently maintain. I don't think this is too big a deal given that we typically update pinned versions only during minor releases (e.g. from 2.6 to 2.7), but it is worth noting that we'd now be managing an entire tree of dependencies.

We'd also be asserting that all dependent packages can be updated only during a minor release. The only exception to this is when a bug or vulnerability is identified in a dependent package (which requires a peer bug report to be submitted against NetBox to track the change).

And finally, I'll point out that this largely prevents bundling NetBox as a distribution package, as pointed out in #2977. I don't think this is a substantial argument though, especially given that it's already largely frustrated by the current pinnings.

So to summarize:

  • Officially require installation within a virtual environment (or dedicated instance)
  • Extend requirements.txt to pin a specific version for all ~60 dependent packages
  • Extend base_requirements.txt to document all ~60 packages and why they are needed
  • Update all pinned versions on every 2.x release
  • Update the pinned version of a specific package as needed per bug reports

Does that sound right?

Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range.

IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release.

@jeremystretch commented on GitHub (Feb 19, 2020): > Release an official docker container, and this is the only supported artefact Just want to go on record to point out that this is a non-starter: We won't require users to implement an entire layer of abstraction and deploy external services simply to run the application. I'm not necessarily opposed to adopting this as policy, but we should consider the implications. First, this would effectively require users to run NetBox in a virtual environment (if not using a dedicated server already). This isn't a bad thing IMO, but it _does_ amount to a new installation requirement that needs to be communicated and documented. Incidentally, this initiative has already been captured as #3949. Second, it means the maintainers must now vet a specific release version for _every_ dependent package: There are currently almost 60, as opposed to the 25 direct dependencies we currently maintain. I don't think this is _too_ big a deal given that we typically update pinned versions only during minor releases (e.g. from 2.6 to 2.7), but it is worth noting that we'd now be managing an entire tree of dependencies. We'd also be asserting that all dependent packages can be updated _only_ during a minor release. The only exception to this is when a bug or vulnerability is identified in a dependent package (which requires a peer bug report to be submitted against NetBox to track the change). And finally, I'll point out that this largely prevents bundling NetBox as a distribution package, as pointed out in #2977. I don't think this is a substantial argument though, especially given that it's already largely frustrated by the current pinnings. So to summarize: * Officially require installation within a virtual environment (or dedicated instance) * Extend `requirements.txt` to pin a specific version for _all_ ~60 dependent packages * Extend `base_requirements.txt` to document all ~60 packages and why they are needed * Update all pinned versions on every 2.x release * Update the pinned version of a specific package as needed per bug reports Does that sound right? > Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range. IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release.
adam commented 2025-12-29 18:28:07 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Feb 19, 2020):

Does that sound right?

Yes, I believe so, thank you.

  • Update the pinned version of a specific package as needed per bug reports

Speaking as a user of Netbox, that would be a big plus point. I am unlikely to be monitoring for security alerts on all of Netbox's recursive package dependencies; but if Netbox comes out with a new release because a dependency had to be updated, I will notice that and will upgrade.

Regarding updating all pinned versions on minor (i.e. 2.x) release: I'd say that's a good time to do it.

IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release.

I think it's right to pin it. If a new Django release doesn't do anything which fixes a problem observable by Netbox users, then there's no need to upgrade. Having a known working version is more important. But as above: if Django makes a point release because of a security issue, then having a corresponding bump in Netbox gives a useful notification of the need to upgrade. Plus, hopefully the Netbox developers have done at least a quick regression test first.

@candlerb commented on GitHub (Feb 19, 2020): > Does that sound right? Yes, I believe so, thank you. > * Update the pinned version of a specific package as needed per bug reports Speaking as a user of Netbox, that would be a big plus point. I am unlikely to be monitoring for security alerts on all of Netbox's recursive package dependencies; but if Netbox comes out with a new release because a dependency had to be updated, I *will* notice that and will upgrade. Regarding updating all pinned versions on minor (i.e. 2.x) release: I'd say that's a good time to do it. > IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release. I think it's right to pin it. If a new Django release doesn't do anything which fixes a problem observable by Netbox users, then there's no need to upgrade. Having a known *working* version is more important. But as above: if Django makes a point release because of a security issue, then having a corresponding bump in Netbox gives a useful notification of the need to upgrade. Plus, hopefully the Netbox developers have done at least a quick regression test first.
adam commented 2025-12-29 18:28:08 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Feb 19, 2020):

There are some common tools out there that can help maintain these workflows and track requirements more intelligently than what requirements.txt provides

  • pyup for automated dependency updates - which could hook in to an automated CI/CD like travis
  • poetry which helps maintain a better list of requirements (using lock files and pyproject.toml) - that is also exportable to requirements.txt for users.
@tyler-8 commented on GitHub (Feb 19, 2020): There are some common tools out there that can help maintain these workflows and track requirements more intelligently than what `requirements.txt` provides - [pyup](https://github.com/pyupio/pyup) for automated dependency updates - which could hook in to an automated CI/CD like travis - [poetry](https://github.com/python-poetry/poetry) which helps maintain a better list of requirements (using lock files and `pyproject.toml`) - that is also exportable to `requirements.txt` for users.
adam commented 2025-12-29 18:28:09 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Feb 25, 2020):

It occurs to me to point out that our CI tests rebuild a virtual environment per requirements.txt on every run, and never seem to encounter a problem with dependency management. Since we're looking at moving to recommend a virtualenv-based installation anyway (see #3949), this may be easily resolved by simply rebuilding the virtual environment from scratch on each upgrade.

@jeremystretch commented on GitHub (Feb 25, 2020): It occurs to me to point out that our CI tests rebuild a virtual environment per `requirements.txt` on every run, and never seem to encounter a problem with dependency management. Since we're looking at moving to recommend a virtualenv-based installation anyway (see #3949), this may be easily resolved by simply rebuilding the virtual environment from scratch on each upgrade.
adam commented 2025-12-29 18:28:09 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Feb 26, 2020):

I've just implemented #3949, which moves the official NetBox implementation into a Python virtual environment. During an upgrade, the virtualenv is destroyed and completely rebuilt, effectively replicating the same process we use for our CI tests. With this approach in place, it's difficult to imagine a dependency issue that would persist for a user but escape our integration tests.

I'm going to close this issue as I believe the us of virtual environments negates the concerns it has raised. We can revisit if any further dependency issues arise.

@jeremystretch commented on GitHub (Feb 26, 2020): I've just implemented #3949, which moves the official NetBox implementation into a Python virtual environment. During an upgrade, the virtualenv is destroyed and completely rebuilt, effectively replicating the same process we use for our CI tests. With this approach in place, it's difficult to imagine a dependency issue that would persist for a user but escape our integration tests. I'm going to close this issue as I believe the us of virtual environments negates the concerns it has raised. We can revisit if any further dependency issues arise.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3348
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?