starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Provide CursorPagination as an alternative to LimitOffsetPagination #6593

New Issue
Closed
opened 2025-12-29 19:42:45 +01:00 by adam · 4 comments
adam commented 2025-12-29 19:42:45 +01:00
Owner
Copy Link

Originally created by @lyuyangh on GitHub (Jun 23, 2022).

NetBox version

v3.2.5

Feature type

Change to existing functionality

Proposed functionality

The proposed functionality and use cases are well documented in https://github.com/netbox-community/netbox/issues/3699 which was closed by the stale bot. This creates a new issue for it. I am willing to implement and own the changes once it gets accepted. (I've actually already implemented a working version of it. I'll be happy to submit a PR and refine it. )

To reiterate the things that were discussed in the original issue:

Provide an option to use CursorPagination instead of the default LimitOffsetPagination for all GET endpoints that retrieve a list of objects. This will allow clients to retrieve a consistent list of objects even if elements are changed/removed during iteration.

Proposed changes:

  • Interface: provide a new query parameter pagination_mode so clients can choose to use CursorPagination. When in CursorPagination mode, limit can be used to control page size (it has the same functionality as limit in LimitOffsetPagination mode). LimitOffsetPagination is still the default so the change will not effect any existing clients.
  • Implementation: implement a custom Pagination class that uses either CursorPagination or LimitOffsetPagination depending on pagination_mode. Internally, it still uses the pagination implementation provided by Django REST. When in CursorPagination mode, records will be returned in "-create" order.

Use case

1. Get/Delete/Update all records

With LimitOffsetPagination, it is difficult to do this consistently. Take getting all records of a data model as an example: if I have a loop that fetches records page by page, it may end up missing valid records because a record is deleted while the script is running and the offsets are no longer valid.

To work around problems like this, the script needs to iterate through the list multiple times until it gets a "stable" list.

This also applies to attempts to delete or update all records. It is even more difficult to deal with if the update is not idempotent as it is not an option to apply the update multiple times.

2. Pagination on a large dataset

As mentioned in Django REST framework documentation, LimitOffsetPagination may become inefficient on extremely large datasets and CursorPagination has fixed-time properties.

Database changes

None

External dependencies

None

Originally created by @lyuyangh on GitHub (Jun 23, 2022). ### NetBox version v3.2.5 ### Feature type Change to existing functionality ### Proposed functionality The proposed functionality and use cases are well documented in https://github.com/netbox-community/netbox/issues/3699 which was closed by the stale bot. This creates a new issue for it. I am willing to implement and own the changes once it gets accepted. (I've actually already implemented a working version of it. I'll be happy to submit a PR and refine it. ) To reiterate the things that were discussed in the original issue: Provide an option to use **CursorPagination** instead of the default **LimitOffsetPagination** for all GET endpoints that retrieve a list of objects. This will allow clients to retrieve a consistent list of objects even if elements are changed/removed during iteration. ### Proposed changes: - Interface: provide a new query parameter `pagination_mode` so clients can choose to use **CursorPagination**. When in **CursorPagination** mode, `limit` can be used to control page size (it has the same functionality as `limit` in **LimitOffsetPagination** mode). **LimitOffsetPagination** is still the default so the change will not effect any existing clients. - Implementation: implement a custom Pagination class that uses either **CursorPagination** or **LimitOffsetPagination** depending on `pagination_mode`. Internally, it still uses the pagination implementation provided by Django REST. When in **CursorPagination** mode, records will be returned in "-create" order. ### Use case #### 1. Get/Delete/Update all records With **LimitOffsetPagination**, it is difficult to do this consistently. Take getting all records of a data model as an example: if I have a loop that fetches records page by page, it may end up missing valid records because a record is deleted while the script is running and the offsets are no longer valid. To work around problems like this, the script needs to iterate through the list multiple times until it gets a "stable" list. This also applies to attempts to delete or update all records. It is even more difficult to deal with if the update is not idempotent as it is not an option to apply the update multiple times. #### 2. Pagination on a large dataset As mentioned in [Django REST framework documentation](https://www.django-rest-framework.org/api-guide/pagination/#cursorpagination:~:text=Supports%20usage%20with%20very%20large%20datasets.%20With%20extremely%20large%20datasets%20pagination%20using%20offset%2Dbased%20pagination%20styles%20may%20become%20inefficient%20or%20unusable.%20Cursor%20based%20pagination%20schemes%20instead%20have%20fixed%2Dtime%20properties%2C%20and%20do%20not%20slow%20down%20as%20the%20dataset%20size%20increases.), **LimitOffsetPagination** may become inefficient on extremely large datasets and **CursorPagination** has fixed-time properties. ### Database changes None ### External dependencies None
adam added the type: featurestatus: needs ownerpending closure labels 2025-12-29 19:42:45 +01:00
adam closed this issue 2025-12-29 19:42:45 +01:00
adam commented 2025-12-29 19:42:46 +01:00
Author
Owner
Copy Link

@lyuyangh commented on GitHub (Jul 13, 2022):

Hi @jeremystretch! Thanks for reviewing this issue! To provide you with more information, I have submitted a PR that adds support for cursor pagination. I tested it locally and it fixes the issue I mentioned above. This would be a very useful feature for us and I think it would also benefit many other users so I would love to follow through and implement it. Please let me know what you think!

@lyuyangh commented on GitHub (Jul 13, 2022): Hi @jeremystretch! Thanks for reviewing this issue! To provide you with more information, I have submitted a PR that adds support for cursor pagination. I tested it locally and it fixes the issue I mentioned above. This would be a very useful feature for us and I think it would also benefit many other users so I would love to follow through and implement it. Please let me know what you think!
adam commented 2025-12-29 19:42:46 +01:00
Author
Owner
Copy Link

@jsenecal commented on GitHub (Aug 18, 2022):

FWIW, this issue/PR and the test cases it includes seems to offer good functionality. Not sure how many users would benefit from it though. Everything looks good to me, however.

@jsenecal commented on GitHub (Aug 18, 2022): FWIW, this issue/PR and the test cases it includes seems to offer good functionality. Not sure how many users would benefit from it though. Everything looks good to me, however.
adam commented 2025-12-29 19:42:47 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jan 2, 2023):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Jan 2, 2023): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 19:42:47 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Feb 2, 2023):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@github-actions[bot] commented on GitHub (Feb 2, 2023): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pending closure status: needs owner type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#6593
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?