starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

NetBox mixes authors of modifications in the changelog when there are a lot of concurrent modifications #5589

New Issue
Closed
opened 2025-12-29 19:29:48 +01:00 by adam · 13 comments
adam commented 2025-12-29 19:29:48 +01:00
Owner
Copy Link

Originally created by @drygdryg on GitHub (Oct 30, 2021).

NetBox version

v3.0.8

Python version

3.9

Steps to Reproduce

  1. Set up a test local instance of NetBox.
  2. Download the necessary files: https://gist.github.com/drygdryg/f32038ef94514aa5503098dbbb42a46c
  3. Restore backup of the database of my NetBox test instance (netbox.sql) to your NetBox instance according the manual.
  4. Edit test_case.py: configure NETBOX_URL according to your NetBox instance.
  5. Run test_case.py. You need to install the httpx module to run if it's not installed: pip install httpx.
  6. Log in to NetBox using the following credentials:
    username: admin
    password: admin
  7. View Changelog (http://netbox.local/extras/changelog/).

Expected Behavior

According to the changelog, each user should change his device based on this logic:
test_user1 updates Test router 1
test_user2 updates Test router 2
test_user3 updates Test router 3
etc.

Observed Behavior

Users are mixed in the changelog. Some examples:
image
image

Originally created by @drygdryg on GitHub (Oct 30, 2021). ### NetBox version v3.0.8 ### Python version 3.9 ### Steps to Reproduce 1. Set up a test local instance of NetBox. 2. Download the necessary files: https://gist.github.com/drygdryg/f32038ef94514aa5503098dbbb42a46c 3. Restore backup of the database of my NetBox test instance (`netbox.sql`) to your NetBox instance [according the manual](https://netbox.readthedocs.io/en/stable/administration/replicating-netbox/). 4. Edit `test_case.py`: configure `NETBOX_URL` according to your NetBox instance. 5. Run `test_case.py`. You need to install the httpx module to run if it's not installed: `pip install httpx`. 6. Log in to NetBox using the following credentials: username: admin password: admin 7. View Changelog (`http://netbox.local/extras/changelog/`). ### Expected Behavior According to the changelog, each user should change his device based on this logic: `test_user1` updates `Test router 1` `test_user2` updates `Test router 2` `test_user3` updates `Test router 3` etc. ### Observed Behavior Users are mixed in the changelog. Some examples: ![image](https://user-images.githubusercontent.com/43933400/139524544-13b47060-98f6-41fc-8ae5-573ab74058e1.png) ![image](https://user-images.githubusercontent.com/43933400/139524563-a83c1d0d-a30f-47b1-a7a8-20be8a5f0676.png)
adam added the type: bug label 2025-12-29 19:29:48 +01:00
adam closed this issue 2025-12-29 19:29:49 +01:00
adam commented 2025-12-29 19:29:49 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Oct 30, 2021):

Probably same issue as #7657, #5142, #5609 - There's a race condition in ObjectChange handling when using an application server running multiple workers in threads.

I don't really know a lot about django, but isn't this iffy? I'm not sure that is thread safe.

0b0ab9277c/netbox/extras/context_managers.py (L25-L38)

If you just want a workaround, try running gunicorn with threads = 1.

@kkthxbye-code commented on GitHub (Oct 30, 2021): Probably same issue as #7657, #5142, #5609 - There's a race condition in ObjectChange handling when using an application server running multiple workers in threads. I don't really know a lot about django, but isn't this iffy? I'm not sure that is thread safe. https://github.com/netbox-community/netbox/blob/0b0ab9277cdb8f388f2d6aee92d708d5126b9111/netbox/extras/context_managers.py#L25-L38 If you just want a workaround, try running gunicorn with threads = 1.
adam commented 2025-12-29 19:29:49 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Oct 31, 2021):

@drygdryg Can you try this branch and see if it resolves the issue for you? It seems to fix it on my dev instance. https://github.com/tyler-8/netbox/tree/objectchange_threadsafe

[Edit] Hmm... maybe not, looked like it was working for a few test runs. Back to the drawing board I suppose.

@tyler-8 commented on GitHub (Oct 31, 2021): @drygdryg Can you try this branch and see if it resolves the issue for you? It seems to fix it on my dev instance. https://github.com/tyler-8/netbox/tree/objectchange_threadsafe [Edit] Hmm... maybe not, looked like it was working for a few test runs. Back to the drawing board I suppose.
adam commented 2025-12-29 19:29:50 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

I made a new update, try the https://github.com/tyler-8/netbox/tree/objectchange_threadsafe branch now.

@tyler-8 commented on GitHub (Nov 1, 2021): I made a new update, try the https://github.com/tyler-8/netbox/tree/objectchange_threadsafe branch now.
adam commented 2025-12-29 19:29:50 +01:00
Author
Owner
Copy Link

@drygdryg commented on GitHub (Nov 1, 2021):

@drygdryg Can you try this branch and see if it resolves the issue for you? It seems to fix it on my dev instance. https://github.com/tyler-8/netbox/tree/objectchange_threadsafe

@tyler-8 I tested this with 9 workers and 3 threads, and it solved the issue for me, thanks.

@drygdryg commented on GitHub (Nov 1, 2021): > @drygdryg Can you try this branch and see if it resolves the issue for you? It seems to fix it on my dev instance. https://github.com/tyler-8/netbox/tree/objectchange_threadsafe @tyler-8 I tested this with 9 workers and 3 threads, and it solved the issue for me, thanks.
adam commented 2025-12-29 19:29:50 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

@tyler-8 I tested this with 9 workers and 3 threads, and it solved the issue for me, thanks.

Try fewer workers and more threads; such as 4 and 4.

For testing this, we want the threads to be used more heavily and with 9 workers (processes) against your test script there is barely enough load for threads.

@tyler-8 commented on GitHub (Nov 1, 2021): > @tyler-8 I tested this with 9 workers and 3 threads, and it solved the issue for me, thanks. Try fewer workers and more threads; such as 4 and 4. For testing this, we want the threads to be used more heavily and with 9 workers (processes) against your test script there is barely enough load for threads.
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Nov 1, 2021):

Closing this in favour of #7657

@DanSheps commented on GitHub (Nov 1, 2021): Closing this in favour of #7657
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

I'm experimenting a slightly different track with this one now @DanSheps - this particular issue might be more due to using non-unique values for dispatch_uid in the signal .connect() methods when using gunicorn threading. Related to the other issue, but possibly different enough to warrant another solution.

b5e8157700/netbox/extras/context_managers.py (L26)

would become

post_save.connect(handle_changed_object, dispatch_uid=f'handle_changed_object_{request.id}')
@tyler-8 commented on GitHub (Nov 1, 2021): I'm experimenting a slightly different track with this one now @DanSheps - this particular issue might be more due to using non-unique values for `dispatch_uid` in the signal `.connect()` methods when using `gunicorn` threading. Related to the other issue, but possibly different enough to warrant another solution. https://github.com/netbox-community/netbox/blob/b5e81577001f04f489e9fb000d4229e7b7cc9733/netbox/extras/context_managers.py#L26 would become ```python post_save.connect(handle_changed_object, dispatch_uid=f'handle_changed_object_{request.id}') ```
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

@drygdryg - can you try again using this branch instead? https://github.com/tyler-8/netbox/tree/objectchange_dispatch_uids

I'm seeing good results with your script without the use of threading.Lock() like in the previous branch

@tyler-8 commented on GitHub (Nov 1, 2021): @drygdryg - can you try again using this branch instead? https://github.com/tyler-8/netbox/tree/objectchange_dispatch_uids I'm seeing good results with your script without the use of `threading.Lock()` like in the previous branch
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Nov 1, 2021):

@tyler-8 since this one might be semi-unrelated, I will re-open this for you

@DanSheps commented on GitHub (Nov 1, 2021): @tyler-8 since this one might be semi-unrelated, I will re-open this for you
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

Boy this is frustrating to troubleshoot and test, lol.

Initially, using truly unique dispatch_uid seemed to work through a couple test runs of the test_case.py script, but now I'm seeing the user-mixups. On the other hand the original branch https://github.com/tyler-8/netbox/tree/objectchange_threadsafe is still solving the issue consistently.

@tyler-8 commented on GitHub (Nov 1, 2021): Boy this is frustrating to troubleshoot and test, lol. Initially, using truly unique `dispatch_uid` seemed to work through a couple test runs of the `test_case.py` script, but now I'm seeing the user-mixups. On the other hand the original branch https://github.com/tyler-8/netbox/tree/objectchange_threadsafe is still solving the issue consistently.
adam commented 2025-12-29 19:29:51 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

Benchmarks using a modified test_case.py (ThreadPoolExecutor instead of ProcessPoolExecutor)

Client 7 threads, 1 user per thread
gunicorn 8 processes (essentially, no thread-locks)
1:09.33
1:03.45
1:00.36
1:00.26
0:55.067
1:00.82

Client 7 threads, 1 user per thread
gunicorn 2 processes, 5 threads
1:40.41
1:40.45
1:43.61
1:41.18
1:39.32
1:40.59

Client 7 threads, 1 user per thread
gunicorn 5 processes, 3 threads
1:19.74
1:04.45
1:05.68
1:10.65
1:08.53
1:04.65

Client 7 threads, 1 user per thread
gunicorn 4 processes, 4 threads
1:02.83
1:02.15
1:44.20
0:59.19
1:22.28
1:00.38
@tyler-8 commented on GitHub (Nov 1, 2021): Benchmarks using a modified `test_case.py` (ThreadPoolExecutor instead of ProcessPoolExecutor) ``` Client 7 threads, 1 user per thread gunicorn 8 processes (essentially, no thread-locks) 1:09.33 1:03.45 1:00.36 1:00.26 0:55.067 1:00.82 Client 7 threads, 1 user per thread gunicorn 2 processes, 5 threads 1:40.41 1:40.45 1:43.61 1:41.18 1:39.32 1:40.59 Client 7 threads, 1 user per thread gunicorn 5 processes, 3 threads 1:19.74 1:04.45 1:05.68 1:10.65 1:08.53 1:04.65 Client 7 threads, 1 user per thread gunicorn 4 processes, 4 threads 1:02.83 1:02.15 1:44.20 0:59.19 1:22.28 1:00.38 ```
adam commented 2025-12-29 19:29:52 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Nov 1, 2021):

@DanSheps After tinkering with this issue more today (and digging up the old issues that had similar errors/behavior), I think it is in fact the same issue. You can close this one again (sorry :P ) and I'll post my updates in #7657

@tyler-8 commented on GitHub (Nov 1, 2021): @DanSheps After tinkering with this issue more today (and digging up the old issues that had similar errors/behavior), I think it is in fact the same issue. You can close this one again (sorry :P ) and I'll post my updates in #7657
adam commented 2025-12-29 19:29:52 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Nov 3, 2021):

No worries! We'll close this one for now, and revisit if the eventual fix for #7657 is determined not to resolve this particular issue as well.

@jeremystretch commented on GitHub (Nov 3, 2021): No worries! We'll close this one for now, and revisit if the eventual fix for #7657 is determined not to resolve this particular issue as well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#5589
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?