starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

MAX_PAGE_SIZE may not be working as expected #10516

New Issue
Closed
opened 2025-12-29 21:32:30 +01:00 by adam · 5 comments
adam commented 2025-12-29 21:32:30 +01:00
Owner
Copy Link

Originally created by @luisg722 on GitHub (Nov 26, 2024).

Deployment Type

Self-hosted

Triage priority

N/A

NetBox Version

v4.1.7

Python Version

3.11

Steps to Reproduce

When running an API call at higher limits fails. "/api/dcim/interfaces/?format=json&limit=x" page errors out.

Currently at 2875 interfaces.

Confirmed no issue with limit=1000, 1001, 2000, 2300

Confirmed issue at limit=2400 and up

This was tested with MAX_PAGE_SIZE set to 10000 as well as 0 to no avail.

Expected Behavior

Before upgrading to 4.1.7 (running 3.6.1), scripts and APIs were working as expected. Doing API calls would show all expected results.

Observed Behavior

Error page:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server

Originally created by @luisg722 on GitHub (Nov 26, 2024). ### Deployment Type Self-hosted ### Triage priority N/A ### NetBox Version v4.1.7 ### Python Version 3.11 ### Steps to Reproduce When running an API call at higher limits fails. "/api/dcim/interfaces/?format=json&limit=x" page errors out. Currently at 2875 interfaces. Confirmed no issue with limit=1000, 1001, 2000, 2300 Confirmed issue at limit=2400 and up This was tested with MAX_PAGE_SIZE set to 10000 as well as 0 to no avail. ### Expected Behavior Before upgrading to 4.1.7 (running 3.6.1), scripts and APIs were working as expected. Doing API calls would show all expected results. ### Observed Behavior Error page: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request Reason: Error reading from remote server
adam added the type: bug label 2025-12-29 21:32:30 +01:00
adam closed this issue 2025-12-29 21:32:30 +01:00
adam commented 2025-12-29 21:32:31 +01:00
Author
Owner
Copy Link

@bctiemann commented on GitHub (Dec 3, 2024):

The issue I see seems to be that if limit is not specified in the query params, MAX_PAGE_SIZE is not respected here:

d122c334fd/netbox/netbox/api/pagination.py (L39-L53)

(request.query_params has no limit key, so it drops straight to the except and returns self.default_limit, which is PAGINATE_COUNT, which is 50 by default)

But if limit is specified, it seems to be respected in my testing (setting MAX_PAGE_SIZE to 30 in my configuration.py). Thus if you access http://127.0.0.1:8000/api/dcim/interfaces/, you get a Page 1 of 50 items, but if you follow the next link (http://127.0.0.1:8000/api/dcim/interfaces/?limit=50&offset=50) then it clamps the results to 30 for the current page and the params for the next and previous links.

I'm not sure how this behavior squares with your observations, but it does seem surprising and seems like it should be fixed (i.e. perhaps simply by changing line 42 to a get with a default of 0) so that MAX_PAGE_SIZE is enforced regardless of whether limit is specified. But that doesn't sound consistent with what you're seeing.

And I'm not sure what would have changed in the behavior since v3.6.1, since I compared the code block above and it hasn't materially changed.

@luisg722 could you clarify whether you have set custom values for PAGINATE_COUNT and where/how you are specifying that value and MAX_PAGE_SIZE? And could you confirm what your pagination next and previous links look like if you hit http://127.0.0.1:8000/api/dcim/interfaces/?

@bctiemann commented on GitHub (Dec 3, 2024): The issue I see seems to be that if `limit` is not specified in the query params, `MAX_PAGE_SIZE` is not respected here: https://github.com/netbox-community/netbox/blob/d122c334fdd08f277e8ce70953ef89801decd0fc/netbox/netbox/api/pagination.py#L39-L53 (`request.query_params` has no `limit` key, so it drops straight to the `except` and returns `self.default_limit`, which is `PAGINATE_COUNT`, which is 50 by default) But if `limit` is specified, it seems to be respected in my testing (setting `MAX_PAGE_SIZE` to 30 in my `configuration.py`). Thus if you access `http://127.0.0.1:8000/api/dcim/interfaces/`, you get a Page 1 of 50 items, but if you follow the `next` link (`http://127.0.0.1:8000/api/dcim/interfaces/?limit=50&offset=50`) then it clamps the results to 30 for the current page and the params for the `next` and `previous` links. I'm not sure how this behavior squares with your observations, but it does seem surprising and seems like it should be fixed (i.e. perhaps simply by changing line 42 to a `get` with a default of 0) so that `MAX_PAGE_SIZE` is enforced regardless of whether `limit` is specified. But that doesn't sound consistent with what you're seeing. And I'm not sure what would have changed in the behavior since v3.6.1, since I compared the code block above and it hasn't materially changed. @luisg722 could you clarify whether you have set custom values for `PAGINATE_COUNT` and where/how you are specifying that value and `MAX_PAGE_SIZE`? And could you confirm what your pagination `next` and `previous` links look like if you hit `http://127.0.0.1:8000/api/dcim/interfaces/`?
adam commented 2025-12-29 21:32:32 +01:00
Author
Owner
Copy Link

@luisg722 commented on GitHub (Dec 3, 2024):

Thank you for the reply. We are actually doing API calls to JSON using:

https://127.0.0.1:8000/api/dcim/interfaces/?format=json&limit=0

which worked just fine on 3.6.1. We use this for scripting to pull all interface data to find if any devices do not have any "connections" on interfaces they have configured. Currently Paginate count is set to 50 and Max page size is set to 10,000 in the Netbox configuration via the web UI, nothing set on the backend.

After testing a few times, I found that I was able to do successful pulls with limits up to 2800 without issue which would fail earlier so not sure if performance itself can be a factor but again, it worked just fine in 3.6.1.

@luisg722 commented on GitHub (Dec 3, 2024): Thank you for the reply. We are actually doing API calls to JSON using: https://127.0.0.1:8000/api/dcim/interfaces/?format=json&limit=0 which worked just fine on 3.6.1. We use this for scripting to pull all interface data to find if any devices do not have any "connections" on interfaces they have configured. Currently Paginate count is set to 50 and Max page size is set to 10,000 in the Netbox configuration via the web UI, nothing set on the backend. After testing a few times, I found that I was able to do successful pulls with limits up to 2800 without issue which would fail earlier so not sure if performance itself can be a factor but again, it worked just fine in 3.6.1.
adam commented 2025-12-29 21:32:32 +01:00
Author
Owner
Copy Link

@bctiemann commented on GitHub (Dec 3, 2024):

I see, so the issue isn't that MAX_PAGE_SIZE isn't being enforced, correct? If you set it to a small value like 30 or 100, it paginates correctly using that value?

It's very possible that the payload of the responses for Interface is heavier now than it was in 3.6.1 (we have added a lot more data and nested serializers to the object).

If your use case is to make one single unpaginated request with a very high MAX_PAGE_SIZE to avoid pagination, it might be that your script will have to instead consume the pagination at (say) 1000 per page and follow the next links accordingly. I don't know what the policy is for supporting a particular level of reliability for very large API responses as the payload size grows, but I suspect the answer will be that that's what pagination is meant to address.

@bctiemann commented on GitHub (Dec 3, 2024): I see, so the issue isn't that `MAX_PAGE_SIZE` isn't being enforced, correct? If you set it to a small value like 30 or 100, it paginates correctly using that value? It's very possible that the payload of the responses for `Interface` is heavier now than it was in 3.6.1 (we have added a lot more data and nested serializers to the object). If your use case is to make one single unpaginated request with a very high `MAX_PAGE_SIZE` to avoid pagination, it might be that your script will have to instead consume the pagination at (say) 1000 per page and follow the `next` links accordingly. I don't know what the policy is for supporting a particular level of reliability for very large API responses as the payload size grows, but I suspect the answer will be that that's what pagination is meant to address.
adam commented 2025-12-29 21:32:33 +01:00
Author
Owner
Copy Link

@bctiemann commented on GitHub (Dec 5, 2024):

I don't think there is a fix necessary here.

There isn't an SLA for how many records can be expected to be returned from an "unlimited" (limit=0) call, since the payload size can vary dramatically based on your data, and the server performance and network speed also factor in.

We'd recommend using the pagination links to iterate through all records.

@bctiemann commented on GitHub (Dec 5, 2024): I don't think there is a fix necessary here. There isn't an SLA for how many records can be expected to be returned from an "unlimited" (`limit=0`) call, since the payload size can vary dramatically based on your data, and the server performance and network speed also factor in. We'd recommend using the pagination links to iterate through all records.
adam commented 2025-12-29 21:32:33 +01:00
Author
Owner
Copy Link

@luisg722 commented on GitHub (Dec 6, 2024):

Hello.

I did have to end up setting a limit of 1000 and using the page links. Thank you again.

@luisg722 commented on GitHub (Dec 6, 2024): Hello. I did have to end up setting a limit of 1000 and using the page links. Thank you again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#10516
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?