Optimize bulk updates of custom field values when custom fields are added/removed #10933

New Issue

adam · 2025-12-29T21:37:56+01:00

adam commented

2025-12-29 21:37:56 +01:00

Originally created by @jeremystretch on GitHub (Mar 21, 2025).

Originally assigned to: @jeremystretch on GitHub.

NetBox version

v4.2.5

Feature type

Change to existing functionality

Proposed functionality

This FR intends to capture an opportunity to improve performance when a custom field is added to or removed from a model which has many thousands of records. Whenever a custom field is added, its populate_initial_data() method is called to automatically populate the field's default value on all objects:

7db0765ed2/netbox/extras/models/customfields.py (L279-L289)

The current approach is suboptimal, as it requires loading each individual object into memory to effect the change. I believe we can optimize this by mutating the JSON data directly within the query:

from django.db.models import F, Func, Value
from django.db.models import JSONField

def bulk_update_cf_value(model, field_name, new_value):
    model.objects.update(
        custom_field_data=Func(
            F('custom_field_data'),
            Value([field_name]),
            Value(new_value, JSONField()),
            function='jsonb_set'
        )
    )

bulk_update_cf_value(Site, "test1", "abc123")

This allows us to update the value of a specific key for all objects without first loading each into memory.

Similar approaches will be needed for the remove_stale_data() and rename_object_data() methods on CustomField as well.

Use case

This should result in a dramatic reduction in the time required to bulk update custom field data, particularly when many thousands of records are affected.

Database changes

N/A

External dependencies

N/A

Originally created by @jeremystretch on GitHub (Mar 21, 2025). Originally assigned to: @jeremystretch on GitHub. ### NetBox version v4.2.5 ### Feature type Change to existing functionality ### Proposed functionality This FR intends to capture an opportunity to improve performance when a custom field is added to or removed from a model which has many thousands of records. Whenever a custom field is added, its `populate_initial_data()` method is called to automatically populate the field's default value on all objects: https://github.com/netbox-community/netbox/blob/7db0765ed2bf7098bab17cb71e389ffbfde34ceb/netbox/extras/models/customfields.py#L279-L289 The current approach is suboptimal, as it requires loading each individual object into memory to effect the change. I believe we can optimize this by mutating the JSON data directly within the query: ```python from django.db.models import F, Func, Value from django.db.models import JSONField def bulk_update_cf_value(model, field_name, new_value): model.objects.update( custom_field_data=Func( F('custom_field_data'), Value([field_name]), Value(new_value, JSONField()), function='jsonb_set' ) ) bulk_update_cf_value(Site, "test1", "abc123") ``` This allows us to update the value of a specific key for all objects without first loading each into memory. Similar approaches will be needed for the `remove_stale_data()` and `rename_object_data()` methods on CustomField as well. ### Use case This should result in a dramatic reduction in the time required to bulk update custom field data, particularly when many thousands of records are affected. ### Database changes N/A ### External dependencies N/A

adam added the status: accepted type: feature complexity: medium labels 2025-12-29 21:37:56 +01:00

adam closed this issue

2025-12-29 21:37:56 +01:00

Sign in to join this conversation.

Branches Tags

main

update-changelog-comments-docs

feature-removal-issue-type

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#10933