starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Avoid conflicting PATCH requests in parallel(#12338, #12531) #8023

New Issue
Closed
opened 2025-12-29 20:31:25 +01:00 by adam · 5 comments
adam commented 2025-12-29 20:31:25 +01:00
Owner
Copy Link

Originally created by @Tomoyuki-GH on GitHub (May 10, 2023).

NetBox version

v3.5.1

Feature type

Change to existing functionality

Proposed functionality

As in wrote on #12531(#12338), run updates (PATCH request)in parallel could cause field data degradation.

kkthxbye-code said in #12531 comment

As for an immediate workaround for you, you can run your application server with one process/thread.

I would agree with this solution, but I think that potential risk of data collaption may remains.

I do not view thorough code of netbox yet, I think updates should be done in transaction. e.g.

--- /etc/netbox/config/configuration.py.orig	2023-05-10 13:59:22.626761403 +0900
+++ /etc/netbox/config/configuration.py	2023-05-10 13:33:05.363616263 +0900
@@ -77,6 +77,7 @@
                                                     # Max database connection age
     'DISABLE_SERVER_SIDE_CURSORS': _environ_get_and_map('DB_DISABLE_SERVER_SIDE_CURSORS', 'False', _AS_BOOL),
                                                     # Disable the use of server-side cursors transaction pooling
+    'ATOMIC_REQUESTS': True,
 }

 # Redis database settings. Redis is used for caching and for queuing background tasks such as webhook events. A separate
--- /opt/netbox/netbox/netbox/api/viewsets/mixins.py.orig	2023-05-07 14:48:30.000000000 +0900
+++ /opt/netbox/netbox/netbox/api/viewsets/mixins.py	2023-05-10 13:10:09.798559651 +0900
@@ -116,6 +116,13 @@
         partial = kwargs.pop('partial', False)
         serializer = BulkOperationSerializer(data=request.data, many=True)
         serializer.is_valid(raise_exception=True)
+        ##################
+        target = self.queryset.select_for_update().filter(
+            pk__in=[o['id'] for o in serializer.data]
+        )
+        for t in target:
+            pass
+        ###################
         qs = self.get_queryset().filter(
             pk__in=[o['id'] for o in serializer.data]
         )

Of course these additional code impacts performance. Quick test shows 50% performance degrade.

update count No tran With tran
field1 37 21
field2 38 16
field3 39 20
Total 114 57
(in 10 sec.)

Use case

Use case of 20-100 workers in netbox operation per day, or so.

Database changes

No response

External dependencies

No response

Originally created by @Tomoyuki-GH on GitHub (May 10, 2023). ### NetBox version v3.5.1 ### Feature type Change to existing functionality ### Proposed functionality As in wrote on #12531(#12338), run updates (PATCH request)in parallel could cause field data degradation. [kkthxbye-code](https://github.com/kkthxbye-code) said in #12531 [comment](https://github.com/netbox-community/netbox/issues/12531#issuecomment-1540024228) > As for an immediate workaround for you, you can run your application server with one process/thread. I would agree with this solution, but I think that potential risk of data collaption may remains. I do not view thorough code of netbox yet, I think updates should be done in transaction. e.g. ```patch --- /etc/netbox/config/configuration.py.orig 2023-05-10 13:59:22.626761403 +0900 +++ /etc/netbox/config/configuration.py 2023-05-10 13:33:05.363616263 +0900 @@ -77,6 +77,7 @@ # Max database connection age 'DISABLE_SERVER_SIDE_CURSORS': _environ_get_and_map('DB_DISABLE_SERVER_SIDE_CURSORS', 'False', _AS_BOOL), # Disable the use of server-side cursors transaction pooling + 'ATOMIC_REQUESTS': True, } # Redis database settings. Redis is used for caching and for queuing background tasks such as webhook events. A separate --- /opt/netbox/netbox/netbox/api/viewsets/mixins.py.orig 2023-05-07 14:48:30.000000000 +0900 +++ /opt/netbox/netbox/netbox/api/viewsets/mixins.py 2023-05-10 13:10:09.798559651 +0900 @@ -116,6 +116,13 @@ partial = kwargs.pop('partial', False) serializer = BulkOperationSerializer(data=request.data, many=True) serializer.is_valid(raise_exception=True) + ################## + target = self.queryset.select_for_update().filter( + pk__in=[o['id'] for o in serializer.data] + ) + for t in target: + pass + ################### qs = self.get_queryset().filter( pk__in=[o['id'] for o in serializer.data] ) ``` Of course these additional code impacts performance. Quick test shows 50% performance degrade. |update count| No tran| With tran| |----|----|----| |field1| 37 | 21 | |field2| 38| 16| |field3| 39| 20| |Total| 114| 57| (in 10 sec.) ### Use case Use case of 20-100 workers in netbox operation per day, or so. ### Database changes _No response_ ### External dependencies _No response_
adam added the type: featurepending closure labels 2025-12-29 20:31:25 +01:00
adam closed this issue 2025-12-29 20:31:25 +01:00
adam commented 2025-12-29 20:31:25 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (May 10, 2023):

I would agree with this solution, but I think that potential risk of data collaption may remains.

Did you try. I don't see any way that issues could happen in a case where only one request is handled at a time.

I do not view thorough code of netbox yet, I think updates should be done in transaction. e.g.

You are misunderstanding the issue. Updates are already done in a transaction, that's not the issue. The issue is that django-request-framework saves the entire row when handling PATCH request. This causes scenarios like this where two requests patch the same object:

Imagine an object TestObject with fields name and value, initial state TestObject(name="testname", values="testvalue")

Request 1: Initiate PATCH name = "newname"
Request 2: Initiate PATCH value = "newvalue"
Request 1: Get TestObject(name="testname", values="testvalue")
Request 2: Get TestObject(name="testname", values="testvalue")
Request 1: Apply update of all fields, UPDATE testobject SET name="newname", value="testvalue"
Request 2: Apply update of all fields, UPDATE testobject SET name="testname", value="newvalue"

New state will be TestObject(name="testname", values="newvalue") overwriting the changes from Request 1.

Here there is no database transaction issue, the issue is in python code as the object is retrieved before changes by another thread and thus overwrites the later value. Using select_for_updates locks the row, so the second request can't select the row before the first request returns the lock. This is one solution, which as you saw causes some performance issues.

The other solution, as I also linked, is to make django-request-framework only update provided fields in PATCH requests. This would solve the issue without a performance penalty.

I don't really have any strong opinions on whether either solution is worth it though. The second one seems like a decent improvement, but I'm not sure if any unintended side effects could arise.

@kkthxbye-code commented on GitHub (May 10, 2023): > I would agree with this solution, but I think that potential risk of data collaption may remains. Did you try. I don't see any way that issues could happen in a case where only one request is handled at a time. > I do not view thorough code of netbox yet, I think updates should be done in transaction. e.g. You are misunderstanding the issue. Updates are already done in a transaction, that's not the issue. The issue is that django-request-framework saves the entire row when handling PATCH request. This causes scenarios like this where two requests patch the same object: Imagine an object TestObject with fields name and value, initial state TestObject(name="testname", values="testvalue") Request 1: Initiate PATCH name = "newname" Request 2: Initiate PATCH value = "newvalue" Request 1: Get TestObject(name="testname", values="testvalue") Request 2: Get TestObject(name="testname", values="testvalue") Request 1: Apply update of all fields, UPDATE testobject SET name="newname", value="testvalue" Request 2: Apply update of all fields, UPDATE testobject SET name="testname", value="newvalue" New state will be TestObject(name="testname", values="newvalue") overwriting the changes from Request 1. Here there is no database transaction issue, the issue is in python code as the object is retrieved before changes by another thread and thus overwrites the later value. Using select_for_updates locks the row, so the second request can't select the row before the first request returns the lock. This is one solution, which as you saw causes some performance issues. The other solution, as I also linked, is to make django-request-framework only update provided fields in PATCH requests. This would solve the issue without a performance penalty. I don't really have any strong opinions on whether either solution is worth it though. The second one seems like a decent improvement, but I'm not sure if any unintended side effects could arise.
adam commented 2025-12-29 20:31:26 +01:00
Author
Owner
Copy Link

@Tomoyuki-GH commented on GitHub (May 10, 2023):

ok, I've got to know that is Django's behavior and none the Netbox issue. If I(we) would improve these, we have to dive into changing Django's framework and its extensions usages.
Thanks, anyway.

@Tomoyuki-GH commented on GitHub (May 10, 2023): ok, I've got to know that is Django's behavior and none the Netbox issue. If I(we) would improve these, we have to dive into changing Django's framework and its extensions usages. Thanks, anyway.
adam commented 2025-12-29 20:31:26 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (May 10, 2023):

It is correct that the core issue is in django-rest-framework, but as they do not intend to change the functionality, the correct way to mitigate the concurrency issues would be in netbox. The question is if it is necessary and if it's worth the effort to implement and I'm not sure I have any strong opinions about it.

As I linked in the other issue, django-rest-framework seems to suggest using the following extension:

http://chibisov.github.io/drf-extensions/docs/#partialupdateserializermixin

It is probably possible to take inspiration from the implementation there:

31f81e7b48/rest_framework_extensions/serializers.py (L32)

@kkthxbye-code commented on GitHub (May 10, 2023): It is correct that the core issue is in django-rest-framework, but as they do not intend to change the functionality, the correct way to mitigate the concurrency issues would be in netbox. The question is if it is necessary and if it's worth the effort to implement and I'm not sure I have any strong opinions about it. As I linked in the other issue, django-rest-framework seems to suggest using the following extension: http://chibisov.github.io/drf-extensions/docs/#partialupdateserializermixin It is probably possible to take inspiration from the implementation there: https://github.com/chibisov/drf-extensions/blob/31f81e7b486871b0bd3633fad58dc2c4567bde7f/rest_framework_extensions/serializers.py#L32
adam commented 2025-12-29 20:31:26 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Aug 9, 2023):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Aug 9, 2023): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 20:31:26 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Sep 9, 2023):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@github-actions[bot] commented on GitHub (Sep 9, 2023): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pending closure type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8023
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?