starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Provide a mode for API pagination backed by CursorPagination #3019

New Issue
Closed
opened 2025-12-29 18:24:48 +01:00 by adam · 8 comments
adam commented 2025-12-29 18:24:48 +01:00
Owner
Copy Link

Originally created by @ajknv on GitHub (Nov 19, 2019).

Environment

  • Python version: 3.6.8
  • NetBox version: 2.6.7

Proposed Functionality

When making a GET request that will provide paginated results, have an additional query parameter option to request that the pagination be backed by DRFs CursorPagination instead of the current/default behavior of using LimitOffsetPagination.

Use Case

The most specific use case is that of trying to delete all the records that match a particular query, wherein one issues the query and then iterates through all the results and makes delete requests on each corresponding ID. The current pagination behavior creates a subtle bug in this seemingly-straightforward approach, namely that as records are deleted from a given page, the offset for the next page request will now result in the skipping of other valid records; i.e. at the end of the full iteration not all of the records will be deleted. The overall algorithm thus has to be written to "converge" through multiple loops through the query-and-delete iteration until the query itself no longer returns any records. This is more cumbersome, and arguably can be viewed as a violation of the principle of least surprise for a fairly basic operation.

Attempts to work around the problem, such as by gathering all of the IDs on a first pass, also cannot be guaranteed to be reliable in the face of RW access by multiple clients/users. If another client issues a delete during the ID-gathering iteration, the current pagination mechanism will still cause some IDs to potentially be missed.

The problem also generalizes to any attempt to modify the state of (not just delete) all of a set of particular records matching a query, in the face of potential concurrent modification by multiple clients. This is actually even harder to deal with, as it may require implementing semantic-specific diff logic for each kind of update scenario and then verifying the effect on all target records using the same overall convergence approach.

As a separate but potentially significant point, the Django documention also notes that CursorPagination is the only approach with (roughly) constant-time scaling to larger datasets. Thus at least having the option of using it may become an important perf consideration for users with larger deployments managed in Netbox, which I would view as an independent positive of implementing this RFE.

Database Changes

Nothing that I would anticipate.

External Dependencies

None new that I would anticipate.

Originally created by @ajknv on GitHub (Nov 19, 2019). ### Environment * Python version: 3.6.8 * NetBox version: 2.6.7 ### Proposed Functionality When making a GET request that will provide paginated results, have an additional query parameter option to request that the pagination be backed by DRFs CursorPagination instead of the current/default behavior of using LimitOffsetPagination. ### Use Case The most specific use case is that of trying to delete all the records that match a particular query, wherein one issues the query and then iterates through all the results and makes delete requests on each corresponding ID. The current pagination behavior creates a subtle bug in this seemingly-straightforward approach, namely that as records are deleted from a given page, the offset for the next page request will now result in the skipping of other valid records; i.e. at the end of the full iteration not all of the records will be deleted. The overall algorithm thus has to be written to "converge" through multiple loops through the query-and-delete iteration until the query itself no longer returns any records. This is more cumbersome, and arguably can be viewed as a violation of the principle of least surprise for a fairly basic operation. Attempts to work around the problem, such as by gathering all of the IDs on a first pass, also cannot be guaranteed to be reliable in the face of RW access by multiple clients/users. If another client issues a delete during the ID-gathering iteration, the current pagination mechanism will still cause some IDs to potentially be missed. The problem also generalizes to any attempt to modify the state of (not just delete) all of a set of particular records matching a query, in the face of potential concurrent modification by multiple clients. This is actually even harder to deal with, as it may require implementing semantic-specific diff logic for each kind of update scenario and then verifying the effect on all target records using the same overall convergence approach. As a separate but potentially significant point, the Django documention also notes that CursorPagination is the only approach with (roughly) constant-time scaling to larger datasets. Thus at least having the option of using it may become an important perf consideration for users with larger deployments managed in Netbox, which I would view as an independent positive of implementing this RFE. ### Database Changes Nothing that I would anticipate. ### External Dependencies None new that I would anticipate.
adam added the pending closure label 2025-12-29 18:24:48 +01:00
adam closed this issue 2025-12-29 18:24:48 +01:00
adam commented 2025-12-29 18:24:48 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Nov 20, 2019):

I see a few problems with moving to CursorPagination:

The cursor-based pagination presents an opaque "cursor" indicator that the client may use to page through the result set. This pagination style only presents forward and reverse controls, and does not allow the client to navigate to arbitrary positions.

I would say that, this right here is a non-starter for moving to CursorPagination.

Should be an unchanging value, such as a timestamp, slug, or other field that is only set once, on creation.

Not every query will return a completely un-changing field. If depending on what we use as the key, it would be possible to have a query be altered in such a way that you will still have data insertion/loss (if you have an insertion before your cursor, you will potentially be missing any relevant query on that data, as an example)

@DanSheps commented on GitHub (Nov 20, 2019): I see a few problems with moving to CursorPagination: > The cursor-based pagination presents an opaque "cursor" indicator that the client may use to page through the result set. **This pagination style only presents forward and reverse controls, and does not allow the client to navigate to arbitrary positions.** I would say that, this right here is a non-starter for moving to CursorPagination. > Should be an unchanging value, such as a timestamp, slug, or other field that is only set once, on creation. Not every query will return a completely un-changing field. If depending on what we use as the key, it would be possible to have a query be altered in such a way that you will still have data insertion/loss (if you have an insertion before your cursor, you will potentially be missing any relevant query on that data, as an example)
adam commented 2025-12-29 18:24:48 +01:00
Author
Owner
Copy Link

@ajknv commented on GitHub (Nov 20, 2019):

Please note that this RFE is not requesting that the API move to CursorPagination in lieu of the current behavior, but merely to add support for it as an alternate option. Limitations that arise from choosing the alternate mode could of course simply be documented, allowing clients to assess the tradeoffs to any particular use case.

As to the unchanging value concern, seems like the queries could utilize the record (row) ID -- even if only internally -- for the key.

@ajknv commented on GitHub (Nov 20, 2019): Please note that this RFE is not requesting that the API move to CursorPagination in lieu of the current behavior, but merely to add support for it as an alternate option. Limitations that arise from choosing the alternate mode could of course simply be documented, allowing clients to assess the tradeoffs to any particular use case. As to the unchanging value concern, seems like the queries could utilize the record (row) ID -- even if only internally -- for the key.
adam commented 2025-12-29 18:24:49 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Dec 6, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@stale[bot] commented on GitHub (Dec 6, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 18:24:49 +01:00
Author
Owner
Copy Link

@ajknv commented on GitHub (Dec 6, 2019):

RFE activity to preserve interest.

@ajknv commented on GitHub (Dec 6, 2019): RFE activity to preserve interest.
adam commented 2025-12-29 18:24:49 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Dec 6, 2019):

@ajknv Would you be willing to implement these changes as well as own the changes for a period of time (support, maintenance, etc)?

@DanSheps commented on GitHub (Dec 6, 2019): @ajknv Would you be willing to implement these changes as well as own the changes for a period of time (support, maintenance, etc)?
adam commented 2025-12-29 18:24:49 +01:00
Author
Owner
Copy Link

@ajknv commented on GitHub (Dec 6, 2019):

Perhaps

@ajknv commented on GitHub (Dec 6, 2019): Perhaps
adam commented 2025-12-29 18:24:50 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Dec 20, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@stale[bot] commented on GitHub (Dec 20, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 18:24:50 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Dec 27, 2019):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@stale[bot] commented on GitHub (Dec 27, 2019): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pending closure
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3019
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?