starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Implement standard lookup expressions for all filter fields #3295

New Issue
Closed
opened 2025-12-29 18:27:31 +01:00 by adam · 10 comments
adam commented 2025-12-29 18:27:31 +01:00
Owner
Copy Link

Originally created by @lampwins on GitHub (Feb 7, 2020).

Originally assigned to: @lampwins on GitHub.

Environment

  • Python version: 3.7
  • NetBox version: 2.7.4

Proposed Functionality

#3482 proposes implementing negation filters in the API. To realistically implement this, we need to do so in a dynamic way in a base filter class. In this issue, I propose a way to easily do this, but in doing so, to also support a number of other standard lookup expressions, beyond just negation.

I propose the support of these lookup expressions on field types for which is makes logical sense:

  • n - negated exact
  • isw - case insensitive starts with
  • nisw - negated case insensitive starts with
  • iew - case insensitive ends with
  • niew - negated insensitive ends with
  • ic - insensitive contains
  • nic - negated insensitive contains
  • lt - less than
  • nlt - negated less than
  • lte - less than or equal
  • nlte - negated less than or equal
  • gt - greater than
  • ngt - negated greater than
  • gte - greater than or equal
  • ngte - negated greater than or equal

It may seem like a lot, but they are cheap to add and maintain. Only certain fields will support certain lookups. For example, only numeric based fields will get less than/greater than/etc.

These lookup expressions are exposed as <field_name>__<lookup_expr>.

This is implemented by using a custom base filterset class which overrides the get_filters() method to extend fields with more lookup expressions.

Use Case

Today most all filters offer only exact or iexact lookup expressions and only on querset filter() clauses, which also does not allow for negation. Many users often want to lookup based on partial matches of individual fields (instead of the aggregated nature of the q filter). Also pointed out in #3482 is a strong desire to support negation filters (queryset exclude() clauses).

As an example, in networking often a lot of logic is put into a device's hostname. It would be useful to do lookups based on the hostname string starting or ending with a certain string, without having to deal with the potential for unwanted results from the use of the q filter.

Database Changes

None

External Dependencies

This will require the use of a common base filterset class used across the board. This should not present any actual issues, but should still be noted.

Originally created by @lampwins on GitHub (Feb 7, 2020). Originally assigned to: @lampwins on GitHub. <!-- NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED. This form is only for proposing specific new features or enhancements. If you have a general idea or question, please post to our mailing list instead of opening an issue: https://groups.google.com/forum/#!forum/netbox-discuss NOTE: Due to an excessive backlog of feature requests, we are not currently accepting any proposals which significantly extend NetBox's feature scope. Please describe the environment in which you are running NetBox. Be sure that you are running an unmodified instance of the latest stable release before submitting a bug report. --> ### Environment * Python version: 3.7 * NetBox version: 2.7.4 <!-- Describe in detail the new functionality you are proposing. Include any specific changes to work flows, data models, or the user interface. --> ### Proposed Functionality #3482 proposes implementing negation filters in the API. To realistically implement this, we need to do so in a dynamic way in a base filter class. In this issue, I propose a way to easily do this, but in doing so, to also support a number of other standard lookup expressions, beyond just negation. I propose the support of these lookup expressions on field types for which is makes logical sense: - `n` - negated exact - `isw` - case insensitive starts with - `nisw` - negated case insensitive starts with - `iew` - case insensitive ends with - `niew` - negated insensitive ends with - `ic` - insensitive contains - `nic` - negated insensitive contains - `lt` - less than - ~`nlt` - negated less than~ - `lte` - less than or equal - ~`nlte` - negated less than or equal~ - `gt` - greater than - ~`ngt` - negated greater than~ - `gte` - greater than or equal - ~`ngte` - negated greater than or equal~ It may seem like a lot, but they are cheap to add and maintain. Only certain fields will support certain lookups. For example, only numeric based fields will get less than/greater than/etc. These lookup expressions are exposed as `<field_name>__<lookup_expr>`. This is implemented by using a custom base filterset class which overrides the `get_filters()` method to extend fields with more lookup expressions. <!-- Convey an example use case for your proposed feature. Write from the perspective of a NetBox user who would benefit from the proposed functionality and describe how. ---> ### Use Case Today most all filters offer only exact or iexact lookup expressions and only on querset `filter()` clauses, which also does not allow for negation. Many users often want to lookup based on partial matches of individual fields (instead of the aggregated nature of the `q` filter). Also pointed out in #3482 is a strong desire to support negation filters (queryset `exclude()` clauses). As an example, in networking often a lot of logic is put into a device's hostname. It would be useful to do lookups based on the hostname string starting or ending with a certain string, without having to deal with the potential for unwanted results from the use of the `q` filter. <!-- Note any changes to the database schema necessary to support the new feature. For example, does the proposal require adding a new model or field? (Not all new features require database changes.) ---> ### Database Changes None <!-- List any new dependencies on external libraries or services that this new feature would introduce. For example, does the proposal require the installation of a new Python package? (Not all new features introduce new dependencies.) --> ### External Dependencies This will require the use of a common base filterset class used across the board. This should not present any actual issues, but should still be noted.
adam added the status: acceptedtype: feature labels 2025-12-29 18:27:31 +01:00
adam closed this issue 2025-12-29 18:27:31 +01:00
adam commented 2025-12-29 18:27:31 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Feb 8, 2020):

This most likely would resolve the closed #4092

@DanSheps commented on GitHub (Feb 8, 2020): This most likely would resolve the closed #4092
adam commented 2025-12-29 18:27:31 +01:00
Author
Owner
Copy Link

@hSaria commented on GitHub (Feb 9, 2020):

Can we use the same names for those already in Django's QuerySet API? We already expose access to querysets to the user, like custom scripts and reports, so people already used to working with those would feel right at home with this change. Similarly, people that get familiar with the lookups via the API will benefit from knowing them when writing a custom script or report.

@hSaria commented on GitHub (Feb 9, 2020): Can we use the same names for those already in Django's QuerySet API? We already expose access to querysets to the user, like custom scripts and reports, so people already used to working with those would feel right at home with this change. Similarly, people that get familiar with the lookups via the API will benefit from knowing them when writing a custom script or report.
adam commented 2025-12-29 18:27:31 +01:00
Author
Owner
Copy Link

@spolack commented on GitHub (Feb 10, 2020):

Is it possible to test against ´null´ value? At least for exact and negated exact it would be pretty helpful.

@spolack commented on GitHub (Feb 10, 2020): Is it possible to test against ´null´ value? At least for exact and negated exact it would be pretty helpful.
adam commented 2025-12-29 18:27:31 +01:00
Author
Owner
Copy Link

@spolack commented on GitHub (Feb 10, 2020):

Correct me if i'm wrong, it looks like in your proposed filterset the negated lower/greater then filters are redundant.
gt equals nlte
lt equals ngte
lte equals ngt
gte equals nlt

@spolack commented on GitHub (Feb 10, 2020): Correct me if i'm wrong, it looks like in your proposed filterset the negated lower/greater then filters are redundant. gt equals nlte lt equals ngte lte equals ngt gte equals nlt
adam commented 2025-12-29 18:27:31 +01:00
Author
Owner
Copy Link

@lampwins commented on GitHub (Feb 10, 2020):

@hSaria these do map to the underlying django ORM filter expressions by the same names, but my reasoning for doing it this way is two-fold, one it keeps the filter names as short as possible which helps in cutting down on query string length because some clients still impose eventual URL length limits and two, doing it this way means we always use acronyms which instills some consistency because the numeric operators are already implemented as acronyms under the hood.

@lampwins commented on GitHub (Feb 10, 2020): @hSaria these do map to the underlying django ORM filter expressions by the same names, but my reasoning for doing it this way is two-fold, one it keeps the filter names as short as possible which helps in cutting down on query string length because some clients still impose eventual URL length limits and two, doing it this way means we always use acronyms which instills some consistency because the numeric operators are already implemented as acronyms under the hood.
adam commented 2025-12-29 18:27:32 +01:00
Author
Owner
Copy Link

@lampwins commented on GitHub (Feb 10, 2020):

@spolack I will look into supporting the isnull lookup but I have some concerns about being able to generically support it in all cases.

And yes the redundancy of the negated numeric operators was previously pointed out. It's what I get for hastily listing things by hand :)

@lampwins commented on GitHub (Feb 10, 2020): @spolack I will look into supporting the `isnull` lookup but I have some concerns about being able to generically support it in all cases. And yes the redundancy of the negated numeric operators was previously pointed out. It's what I get for hastily listing things by hand :)
adam commented 2025-12-29 18:27:32 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Feb 10, 2020):

I am not sure how I feel about not including filters that are negates that have an equal filter. My view that it makes your intentions very clear when using a negate filter, whereas your intention might not be as clear with a different filter.

If these negate ones map to underlying filters in the ORM, I would honestly leave them in-tact.

@DanSheps commented on GitHub (Feb 10, 2020): I am not sure how I feel about not including filters that are negates that have an equal filter. My view that it makes your intentions very clear when using a negate filter, whereas your intention might not be as clear with a different filter. If these negate ones map to underlying filters in the ORM, I would honestly leave them in-tact.
adam commented 2025-12-29 18:27:32 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Feb 10, 2020):

Another query, do you think we could dump the __in query and replace it with this as well?

@DanSheps commented on GitHub (Feb 10, 2020): Another query, do you think we could dump the __in query and replace it with this as well?
adam commented 2025-12-29 18:27:32 +01:00
Author
Owner
Copy Link

@lampwins commented on GitHub (Feb 10, 2020):

@DanSheps the negation of binary numeric comparisons are unnecessary because the inverse operations are already available. For example, the truth table for:

a < b

Is exactly the same as

not (a >= b)

As for the in filter lookup, it should not be needed because we already support the logical OR operation on almost all fields in the API. Supporting the in lookup would require values to be passed as list.

@lampwins commented on GitHub (Feb 10, 2020): @DanSheps the negation of binary numeric comparisons are unnecessary because the inverse operations are already available. For example, the truth table for: ``` a < b ``` Is exactly the same as ``` not (a >= b) ``` As for the `in` filter lookup, it should not be needed because we already support the logical OR operation on almost all fields in the API. Supporting the `in` lookup would require values to be passed as list.
adam commented 2025-12-29 18:27:32 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Feb 12, 2020):

Yeah, I was just thinking, we do have a id__in filter for at least a few fields, this would allow consolidation I think.

@DanSheps commented on GitHub (Feb 12, 2020): Yeah, I was just thinking, we do have a id__in filter for at least a few fields, this would allow consolidation I think.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3295
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?