Introduce support for NetBox plugins #2744

@lampwins commented on GitHub (Jul 18, 2019):

@hellerve yes, the intent is really to set up plugins to look and feel like every other core app in NetBox by allowing them access to the same base classes, templates, etc, while maintaining a clear delineation that they are in fact 3rd party plugins. As stated above:

Plugins will be able to register arbitrary links in NetBox's navigation menu.

In my PoC this was done by optionally registering view names which were dynamically added to the nav menu dropdown under the heading "Plugins" but also grouped by plugin.

@lampwins commented on GitHub (Jul 18, 2019): @hellerve yes, the intent is really to set up plugins to look and feel like every other core app in NetBox by allowing them access to the same base classes, templates, etc, while maintaining a clear delineation that they are in fact 3rd party plugins. As stated above: > Plugins will be able to register arbitrary links in NetBox's navigation menu. In my PoC this was done by optionally registering view names which were dynamically added to the nav menu dropdown under the heading "Plugins" but also grouped by plugin.

adam commented

@hellerve commented on GitHub (Jul 18, 2019):

That sounds great. In my implementation of plugins I also had a bunch of other places where I could place hooks, but I guess that’s hard to make customizable/generalizable. If that’s at all interesting, though, I’d be happy to help draft and eventually implement something like that. I understand if that is out of scope, though!

@hellerve commented on GitHub (Jul 18, 2019): That sounds great. In my implementation of plugins I also had a bunch of other places where I could place hooks, but I guess that’s hard to make customizable/generalizable. If that’s at all interesting, though, I’d be happy to help draft and eventually implement something like that. I understand if that is out of scope, though!

adam commented

@bbock commented on GitHub (Jul 19, 2019):

One use case which I would envision, but I'm not sure is covered by your proposal, would be data validation.

I would love to have a plugin that enforces my naming schema. It would need to be able to reject additions and mondifications of core elements if they do not match. Currently, I'm using a report to find invalid entries, but this is fixing after-the-fact, while I'd love to block inconsistent data in the fist place.

@bbock commented on GitHub (Jul 19, 2019): One use case which I would envision, but I'm not sure is covered by your proposal, would be data validation. I would love to have a plugin that enforces my naming schema. It would need to be able to reject additions and mondifications of core elements if they do not match. Currently, I'm using a report to find invalid entries, but this is fixing after-the-fact, while I'd love to block inconsistent data in the fist place.

adam commented

@hellerve commented on GitHub (Jul 22, 2019):

Another question: will plugins be able to introduce middlewares? This might be an interesting feature for a variety of use cases (we’re currently using a custom middleware for change management, for instance). My version of the plugin system (https://github.com/hellerve/netbox/tree/feat/plugin) was able to support this (simply by exposing a list that gets appended to MIDDLEWARES in settings.py, so not a huge liability on code).

It is also a security concern, however, but I think part of the threat model of a plugin is "it can execute arbitrary code anyway", so it should be trusted; feel free to disagree with me on this!

@hellerve commented on GitHub (Jul 22, 2019): Another question: will plugins be able to introduce middlewares? This might be an interesting feature for a variety of use cases (we’re currently using a custom middleware for change management, for instance). My version of the plugin system (https://github.com/hellerve/netbox/tree/feat/plugin) was able to support this (simply by exposing a list that gets appended to `MIDDLEWARES` in `settings.py`, so not a huge liability on code). It is also a security concern, however, but I think part of the threat model of a plugin is "it can execute arbitrary code anyway", so it should be trusted; feel free to disagree with me on this!

adam commented

@LukeDRussell commented on GitHub (Jul 28, 2019):

Glad to see this being discussed openly, Netbox has grown so much that it would be difficult to properly support all use cases.

I'm hoping these plugins would be able to modify data schemes, or at least add tables. Reference our lengthy debate about the definition and implementation of tenancy https://github.com/netbox-community/netbox/issues/2273.

We've implemented this internally as a Django "overlay" that replaces the "tenant" object with another object that allows many to many. It would be outstanding to turn this into a plugin that we could share with others, and I imagine it would reduce our integration testing with ongoing Netbox releases.

@LukeDRussell commented on GitHub (Jul 28, 2019): Glad to see this being discussed openly, Netbox has grown so much that it would be difficult to properly support all use cases. I'm hoping these plugins would be able to modify data schemes, or at least add tables. Reference our lengthy debate about the definition and implementation of tenancy https://github.com/netbox-community/netbox/issues/2273. We've implemented this internally as a Django "overlay" that replaces the "tenant" object with another object that allows many to many. It would be outstanding to turn this into a plugin that we could share with others, and I imagine it would reduce our integration testing with ongoing Netbox releases.

adam commented

@The-Sec commented on GitHub (Aug 2, 2019):

Let me open with that i love Netbox. I think its a great tool! But i disagree with the feature request as this will no doubly split the Netbox user base into smaller groups. If people have custom use cases and they develop a plugin for it, they will drop support as soon as it launches.

People will create how to's to create fancy dashboards with and plugin they can find and go beyond the goal of what Netbox is. Its a "Source of Truth" not to draw fancy things or make them cool. Its a database we should tried it like one. I think it would make development even slower as people will not read how to develop a plugin but will just email and ask how can i do X to create Y for this special use case i have.

If someone really want to add i see no problem in them sending an PR and if and only if it up to par then approve.

@The-Sec commented on GitHub (Aug 2, 2019): Let me open with that i love Netbox. I think its a great tool! But i disagree with the feature request as this will no doubly split the Netbox user base into smaller groups. If people have custom use cases and they develop a plugin for it, they will drop support as soon as it launches. People will create how to's to create fancy dashboards with and plugin they can find and go beyond the goal of what Netbox is. Its a "Source of Truth" not to draw fancy things or make them cool. Its a database we should tried it like one. I think it would make development even slower as people will not read how to develop a plugin but will just email and ask how can i do X to create Y for this special use case i have. If someone really want to add i see no problem in them sending an PR and if and only if it up to par then approve.

adam commented

@LukeDRussell commented on GitHub (Aug 3, 2019):

The only problem is that Jeremy runs a pretty tight project.

There’s always going to be things that are important to my organisation, but not important to yours or Jeremy’s. There are three options I can see - a public or private fork, custom additions or plugins.

Plugins are the most likely to keep the community on a shared base, so we don’t have to fork when we must have certain features that are inappropriate or not welcome in the core of netbox.

Another use case is to move current core features into supported plugins (tenants and secrets come to mind) to keep the core lean and supportable.

@LukeDRussell commented on GitHub (Aug 3, 2019): The only problem is that Jeremy runs a pretty tight project. There’s always going to be things that are important to my organisation, but not important to yours or Jeremy’s. There are three options I can see - a public or private fork, custom additions or plugins. Plugins are the most likely to keep the community on a shared base, so we don’t have to fork when we must have certain features that are inappropriate or not welcome in the core of netbox. Another use case is to move current core features into supported plugins (tenants and secrets come to mind) to keep the core lean and supportable.

adam commented

@hellerve commented on GitHub (Oct 11, 2019):

Hey, I see that this issue isn’t in accepted state yet? What is missing? Can the community do anything to move this forward?

A roadmap would be extremely helpful as well; I understand that this might be a little much to ask, but if you have anything at hand, that’d be great!

@hellerve commented on GitHub (Oct 11, 2019): Hey, I see that this issue isn’t in `accepted` state yet? What is missing? Can the community do anything to move this forward? A roadmap would be extremely helpful as well; I understand that this might be a little much to ask, but if you have anything at hand, that’d be great!

adam commented

@TheRealBecks commented on GitHub (Nov 15, 2019):

@jeremystretch @lampwins Referencing to @hellerve, what is needed here? Do we need to implement #3408 first?

@TheRealBecks commented on GitHub (Nov 15, 2019): @jeremystretch @lampwins Referencing to @hellerve, what is needed here? Do we need to implement #3408 first?

adam commented

@jeremystretch commented on GitHub (Nov 15, 2019):

First we need to release v2.7, and then probably v2.8, and then we can start looking into plugins. There's an enormous amount of existing backlog that needs to be addressed. Before we introduce a ton more.

@jeremystretch commented on GitHub (Nov 15, 2019): First we need to release v2.7, and then probably v2.8, and _then_ we can start looking into plugins. There's an enormous amount of existing backlog that needs to be addressed. Before we introduce a ton more.

adam commented

@hellerve commented on GitHub (Nov 15, 2019):

That’s more than understandable. I know that people probably know this, but you can see which issues need to be fixed/help for these versions here!

@hellerve commented on GitHub (Nov 15, 2019): That’s more than understandable. I know that people probably know this, but [you can see which issues need to be fixed/help for these versions here](https://github.com/netbox-community/netbox/milestones)!

adam commented

@steffann commented on GitHub (Dec 20, 2019):

While I am okay with supporting plugin source being dumped in a known directory, in my implementation, I found it much easier to support plugins which are simply packaged as standard Django apps, and thus available in the python environment (i.e. pip install <plugin>. From a deployment standpoint, it is much easier to deal with this way. As an aside, I would like to explore this same concept for reports at some point.

I have used the standard setuptools pkg_resources entry-points for plugins in the past, and that can work really well. Installing plugins using setuptools also helps with dependency management (in case plugins have their own dependencies) etc. One limitation of the proposed implementation is that only the plugin is added to INSTALLED_APPS. Plugins that depend on other Django apps would be hard to support.

There are of course many ways to support plugins, but I personally prefer to use existing solutions like this that are already part of very common libraries.

@steffann commented on GitHub (Dec 20, 2019): > While I am okay with supporting plugin source being dumped in a known directory, in my implementation, I found it much easier to support plugins which are simply packaged as standard Django apps, and thus available in the python environment (i.e. `pip install <plugin>`. From a deployment standpoint, it is much easier to deal with this way. As an aside, I would like to explore this same concept for reports at some point. I have used the standard setuptools [pkg_resources entry-points](https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points) for plugins in the past, and that can work really well. Installing plugins using setuptools also helps with dependency management (in case plugins have their own dependencies) etc. One limitation of the proposed implementation is that only the plugin is added to `INSTALLED_APPS`. Plugins that depend on other Django apps would be hard to support. There are of course many ways to support plugins, but I personally prefer to use existing solutions like this that are already part of very common libraries.

adam commented

@steffann commented on GitHub (Dec 30, 2019):

After experimenting a bit I found that the extra complexity of pkg_resources wasn't worth it.

My current experiment is at https://github.com/steffann/netbox/tree/3351-plugins, with an example plugin at https://github.com/steffann/netbox-example-plugin. It currently supports adding extra INSTALLED_APPS, adding URLs under /plugins/<plugin-name>/, adding menu items to existing menus and adding new menus, and adding extra panels to the homepage, ip-address page, device page, and all other pages where adding extra panels might make sense.

This is meant as a start for further discussion, hopefully leading to the status of this issue being changed from gathering feedback to accepted :)

@steffann commented on GitHub (Dec 30, 2019): After experimenting a bit I found that the extra complexity of `pkg_resources` wasn't worth it. My current experiment is at https://github.com/steffann/netbox/tree/3351-plugins, with an example plugin at https://github.com/steffann/netbox-example-plugin. It currently supports adding extra INSTALLED_APPS, adding URLs under `/plugins/<plugin-name>/`, adding menu items to existing menus and adding new menus, and adding extra panels to the homepage, ip-address page, device page, and all other pages where adding extra panels might make sense. This is meant as a start for further discussion, hopefully leading to the status of this issue being changed from gathering feedback to accepted :)

adam commented

@steffann commented on GitHub (Dec 30, 2019):

That was quick :)

@steffann commented on GitHub (Dec 30, 2019): That was quick :)

adam commented

@lampwins commented on GitHub (Dec 30, 2019):

Yeah, my PoC was also basically just what you implemented.

It is extremely important we get the API for plugins correct the first time. This sort of thing has the potential to generate a ton of development noise for us once in the wild so we need to do it justice. To that end, simplicity and elegance in the API are what I am striving for. Django already has an entrenched methodology and ecosystem for this type of architecture and we would be wise to stick to it closely.

Things that are top of mind for me are avoiding the potential for allowing namespace collisions both within the core and within two plugins, data model augmentation vs manipulation, and cleaning up many of our pseudo-private APIs to ready them for more public use (things like signals based events, custom fields, RQ tasks, caching, etc).

Lots to think about and design, but we are thinking about it, don't worry ;)

@lampwins commented on GitHub (Dec 30, 2019): Yeah, my PoC was also basically just what you implemented. It is extremely important we get the API for plugins correct the first time. This sort of thing has the potential to generate a ton of development noise for us once in the wild so we need to do it justice. To that end, simplicity and elegance in the API are what I am striving for. Django already has an entrenched methodology and ecosystem for this type of architecture and we would be wise to stick to it closely. Things that are top of mind for me are avoiding the potential for allowing namespace collisions both within the core and within two plugins, data model augmentation vs manipulation, and cleaning up many of our pseudo-private APIs to ready them for more public use (things like signals based events, custom fields, RQ tasks, caching, etc). Lots to think about and design, but we are thinking about it, don't worry ;)

adam commented

@steffann commented on GitHub (Dec 30, 2019):

I did look at your PoC, and used many ideas from that. Some deviations:

I did combine the PLUGINS and PLUGIN_CONFIG
Allow plugins to require other Django apps
Not depend on default_app_config, https://docs.djangoproject.com/en/2.2/ref/applications/#configuring-applications says "New applications should avoid default_app_config".
I used __init__.py from the package instead for dependencies, version checks etc.
I also went with packages for pip-style compatibility checks. Checking max_version was a bit awkward using LooseVersion because you can't so something like "compatible with 2.6.*". You'd have to set max_version to something like 2.6.999. With packages the version spec can be ~= 2.6.9 to allow >= 2.6.9, < 2.7.
Integrate the API views in the APIRootView.
Allow plugins to add items to any menu
Allow plugins to add panels where possibly appropriate
Implemented the last two with template tags

I have alternated multiple times between __init__.py, a separate Plugin base class and AppConfig properties:

AppConfig would be nice (ignoring the default_app_config deprecation), but only class properties are usable as otherwise we'd be creating multiple instances of the class in multiple places.
A Plugin class is possible, but since it's properties already needed in settings.py it would require instantiating it there, and that's not really the right place to do that. It also invites adding complexity. And where to store the instantiated object? I tried storing it in the plugin settings, but that also feels weird. In the end I decided instantiating an object in settings.py was wrong.
So I ended up with simple variables in __init__.py which can be easily imported and used.

I should re-add the required and default settings though. Those are a good idea!

@steffann commented on GitHub (Dec 30, 2019): I did look at your PoC, and used many ideas from that. Some deviations: - I did combine the `PLUGINS` and `PLUGIN_CONFIG` - Allow plugins to require other Django apps - Not depend on `default_app_config`, https://docs.djangoproject.com/en/2.2/ref/applications/#configuring-applications says "New applications should avoid default_app_config". - I used `__init__.py` from the package instead for dependencies, version checks etc. - I also went with `packages` for pip-style compatibility checks. Checking `max_version` was a bit awkward using `LooseVersion` because you can't so something like "compatible with `2.6.*`". You'd have to set `max_version` to something like `2.6.999`. With `packages` the version spec can be `~= 2.6.9` to allow `>= 2.6.9, < 2.7`. - Integrate the API views in the `APIRootView`. - Allow plugins to add items to any menu - Allow plugins to add panels where possibly appropriate - Implemented the last two with template tags I have alternated multiple times between `__init__.py`, a separate `Plugin` base class and `AppConfig` properties: - `AppConfig` would be nice (ignoring the `default_app_config` deprecation), but only class properties are usable as otherwise we'd be creating multiple instances of the class in multiple places. - A `Plugin` class is possible, but since it's properties already needed in `settings.py` it would require instantiating it there, and that's not really the right place to do that. It also invites adding complexity. And where to store the instantiated object? I tried storing it in the plugin settings, but that also feels weird. In the end I decided instantiating an object in `settings.py` was wrong. - So I ended up with simple variables in `__init__.py` which can be easily imported and used. I should re-add the required and default settings though. Those are a good idea!

adam commented

@steffann commented on GitHub (Dec 30, 2019):

I added a whole bunch of documentation at https://github.com/steffann/netbox/tree/3351-plugins/docs/plugins, as reading and writing documentation from the user's perspective usually gives a good view of where the missing/ugly/etc bits are :)

@steffann commented on GitHub (Dec 30, 2019): I added a whole bunch of documentation at https://github.com/steffann/netbox/tree/3351-plugins/docs/plugins, as reading and writing documentation from the user's perspective usually gives a good view of where the missing/ugly/etc bits are :)

adam commented

@steffann commented on GitHub (Mar 26, 2020):

Development on this is looking great! One thing I realised when writing some code to export interface configurations to Ansible YAML was that it would be great for plugins to also be able to add extra buttons on de Device page under the interface list:

And I'm sure someone could find a good use for similar functionality for pages like /ipam/ip-addresses/. Is this something that you'd be interested in? Willing to propose a patch.

@steffann commented on GitHub (Mar 26, 2020): Development on this is looking great! One thing I realised when writing some code to export interface configurations to Ansible YAML was that it would be great for plugins to also be able to add extra buttons on de Device page under the interface list: <img width="479" alt="image" src="https://user-images.githubusercontent.com/509689/77694854-0f007380-6fab-11ea-9cc9-4eae0ad37425.png"> And I'm sure someone could find a good use for similar functionality for pages like `/ipam/ip-addresses/`. Is this something that you'd be interested in? Willing to propose a patch.

adam commented