API error using a custom field with an object value #9324

Closed
opened 2025-12-29 20:48:26 +01:00 by adam · 11 comments
Owner

Originally created by @peteeckel on GitHub (Mar 5, 2024).

Deployment Type

Self-hosted

NetBox Version

v4.0.0-dev

Python Version

3.11

Steps to Reproduce

Unfortunately I didn't have the time to minimise the problem yet, but I can reliably reproduce it with NetBox DNS.

To reproduce it, create a custom field named ipaddress_dns_zone_id of type netbox_dns.Zone for ipam.IPAddress.

Until ca56c8b9ef creating or updating objects containing that custom field worked perfectly. Starting from 78e284c14f I get an error with a custom field on ipam.IPAddress objects using the API. The following request (with existing IP Address and Zone object IDs) causes an exception:

curl -X "POST" "https://192.168.106.105/api/ipam/ip-addresses/" \
     -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' \
     -d $'{
  "custom_fields": {
    "ipaddress_dns_zone_id": "1"
  },
  "address": "10.1.1.4/24"
}'

Expected Behavior

The API call succeeds and an IP address is created which is linked to a DNS record.

Observed Behavior

The stack trace is:

Traceback (most recent call last):
  File "/opt/netbox/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/netbox/api/viewsets/__init__.py", line 135, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
    ^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/ipam/api/views.py", line 109, in create
    return super().create(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/mixins.py", line 18, in create
    serializer.is_valid(raise_exception=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 227, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 426, in run_validation
    value = self.to_internal_value(data)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/netbox/api/serializers/base.py", line 45, in to_internal_value
    return super().to_internal_value(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 483, in to_internal_value
    validated_value = field.run_validation(primitive_value)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/fields.py", line 547, in run_validation
    value = self.to_internal_value(data)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/extras/api/customfields.py", line 85, in to_internal_value
    if serializer.is_valid():
       ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 227, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 428, in run_validation
    self.run_validators(value)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 461, in run_validators
    super().run_validators(to_validate)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/fields.py", line 560, in run_validators
    validator(value, self)
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 148, in __call__
    self.enforce_required_fields(attrs, serializer)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 106, in enforce_required_fields
    missing_items = {
                    
  File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 109, in <dictcomp>
    if serializer.fields[field_name].source not in attrs
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: TypeError at /api/ipam/ip-addresses/
Exception Value: argument of type 'Zone' is not iterable

The exact same API request on the same database instance succeeds with the previous commit. Currently I don't really have an idea where to start with this. So far I tried the following:

  • Used the latest commit of the feature branch: The error persists.
  • Tried a different object type from the NetBox core model: The error does not occur. Update: Only object types that are used in a validator are affected
  • Tried a different object type from NetBox DNS model: The error does not occur. Update: Only object types that are used in a validator are affected
  • Refactored all nested serializers to the new approach with nested=True (except two of them that are required to avoid circular references): The error persists. Update: The issue occurs when the non-nested serializer with nested=True is used
  • Tentatively removed all fields referring to other (nested or non-nested) serializers from ZoneSerializer: The error persists. Update: It is sufficient to have one field addressing a serializer and referred to in a validator in the model

I'm particularly curious why any piece of code should try to iterate over a model instance, which makes me suspect that there might be an error somewhere causing the wrong object to be passed.

Originally created by @peteeckel on GitHub (Mar 5, 2024). ### Deployment Type Self-hosted ### NetBox Version v4.0.0-dev ### Python Version 3.11 ### Steps to Reproduce Unfortunately I didn't have the time to minimise the problem yet, but I can reliably reproduce it with NetBox DNS. To reproduce it, create a custom field named `ipaddress_dns_zone_id ` of type `netbox_dns.Zone` for `ipam.IPAddress`. Until ca56c8b9ef8a9c2238090b0a500e85361f9655b9 creating or updating objects containing that custom field worked perfectly. Starting from 78e284c14f05eeb18d67ba90ed80a6b75e9d8cc6 I get an error with a custom field on `ipam.IPAddress` objects using the API. The following request (with existing IP Address and Zone object IDs) causes an exception: ``` curl -X "POST" "https://192.168.106.105/api/ipam/ip-addresses/" \ -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \ -H 'Content-Type: application/json; charset=utf-8' \ -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' \ -d $'{ "custom_fields": { "ipaddress_dns_zone_id": "1" }, "address": "10.1.1.4/24" }' ``` ### Expected Behavior The API call succeeds and an IP address is created which is linked to a DNS record. ### Observed Behavior The stack trace is: ``` Traceback (most recent call last): File "/opt/netbox/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper return view_func(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view return self.dispatch(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/netbox/netbox/api/viewsets/__init__.py", line 135, in dispatch return super().dispatch(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch response = self.handle_exception(exc) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception self.raise_uncaught_exception(exc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception raise exc ^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch response = handler(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/netbox/ipam/api/views.py", line 109, in create return super().create(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/mixins.py", line 18, in create serializer.is_valid(raise_exception=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 227, in is_valid self._validated_data = self.run_validation(self.initial_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 426, in run_validation value = self.to_internal_value(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/netbox/netbox/api/serializers/base.py", line 45, in to_internal_value return super().to_internal_value(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 483, in to_internal_value validated_value = field.run_validation(primitive_value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/fields.py", line 547, in run_validation value = self.to_internal_value(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/netbox/extras/api/customfields.py", line 85, in to_internal_value if serializer.is_valid(): ^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 227, in is_valid self._validated_data = self.run_validation(self.initial_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 428, in run_validation self.run_validators(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/serializers.py", line 461, in run_validators super().run_validators(to_validate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/fields.py", line 560, in run_validators validator(value, self) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 148, in __call__ self.enforce_required_fields(attrs, serializer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 106, in enforce_required_fields missing_items = { File "/opt/netbox/lib64/python3.11/site-packages/rest_framework/validators.py", line 109, in <dictcomp> if serializer.fields[field_name].source not in attrs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Exception Type: TypeError at /api/ipam/ip-addresses/ Exception Value: argument of type 'Zone' is not iterable ``` The exact same API request on the same database instance succeeds with the previous commit. Currently I don't really have an idea where to start with this. So far I tried the following: * Used the latest commit of the `feature` branch: The error persists. * Tried a different object type from the NetBox core model: The error does not occur. _Update: Only object types that are used in a validator are affected_ * Tried a different object type from NetBox DNS model: The error does not occur. _Update: Only object types that are used in a validator are affected_ * Refactored all nested serializers to the new approach with `nested=True` (except two of them that are required to avoid circular references): The error persists. _Update: The issue occurs when the non-nested serializer with `nested=True` is used_ * Tentatively removed all fields referring to other (nested or non-nested) serializers from `ZoneSerializer`: The error persists. _Update: It is sufficient to have one field addressing a serializer and referred to in a validator in the model_ I'm particularly curious why any piece of code should try to iterate over a model instance, which makes me suspect that there might be an error somewhere causing the wrong object to be passed.
adam added the type: bug label 2025-12-29 20:48:26 +01:00
adam closed this issue 2025-12-29 20:48:27 +01:00
Author
Owner

@peteeckel commented on GitHub (Mar 5, 2024):

Sorry, closing this until I can figure out what exactly happens.

@peteeckel commented on GitHub (Mar 5, 2024): Sorry, closing this until I can figure out what exactly happens.
Author
Owner

@peteeckel commented on GitHub (Mar 5, 2024):

The second closing of this ticket was unintentional, it's late ...

I can narrow it down to netbox/extras/api/customfields.py. If I checkout that file from ca56c8b9ef, the error goes away.

The culprit is in this diff:

diff --git a/netbox/extras/api/customfields.py b/netbox/extras/api/customfields.py
index 6cd3a245e..5a814f9bd 100644
--- a/netbox/extras/api/customfields.py
+++ b/netbox/extras/api/customfields.py
@@ -58,11 +58,11 @@ class CustomFieldsDataField(Field):
         for cf in self._get_custom_fields():
             value = cf.deserialize(obj.get(cf.name))
             if value is not None and cf.type == CustomFieldTypeChoices.TYPE_OBJECT:
-                serializer = get_serializer_for_model(cf.object_type.model_class(), prefix=NESTED_SERIALIZER_PREFIX)
-                value = serializer(value, context=self.parent.context).data
+                serializer = get_serializer_for_model(cf.object_type.model_class())
+                value = serializer(value, nested=True, context=self.parent.context).data
             elif value is not None and cf.type == CustomFieldTypeChoices.TYPE_MULTIOBJECT:
-                serializer = get_serializer_for_model(cf.object_type.model_class(), prefix=NESTED_SERIALIZER_PREFIX)
-                value = serializer(value, many=True, context=self.parent.context).data
+                serializer = get_serializer_for_model(cf.object_type.model_class())
+                value = serializer(value, nested=True, many=True, context=self.parent.context).data
             data[cf.name] = value
 
         return data

So it seems the custom field serialization is working with the old nested serializer, but not with the standard serializer with nested=True.

Sorry again for the close/reopen spam.

@peteeckel commented on GitHub (Mar 5, 2024): The second closing of this ticket was unintentional, it's late ... I can narrow it down to `netbox/extras/api/customfields.py`. If I checkout that file from ca56c8b9ef8a9c2238090b0a500e85361f9655b9, the error goes away. The culprit is in this diff: ``` diff --git a/netbox/extras/api/customfields.py b/netbox/extras/api/customfields.py index 6cd3a245e..5a814f9bd 100644 --- a/netbox/extras/api/customfields.py +++ b/netbox/extras/api/customfields.py @@ -58,11 +58,11 @@ class CustomFieldsDataField(Field): for cf in self._get_custom_fields(): value = cf.deserialize(obj.get(cf.name)) if value is not None and cf.type == CustomFieldTypeChoices.TYPE_OBJECT: - serializer = get_serializer_for_model(cf.object_type.model_class(), prefix=NESTED_SERIALIZER_PREFIX) - value = serializer(value, context=self.parent.context).data + serializer = get_serializer_for_model(cf.object_type.model_class()) + value = serializer(value, nested=True, context=self.parent.context).data elif value is not None and cf.type == CustomFieldTypeChoices.TYPE_MULTIOBJECT: - serializer = get_serializer_for_model(cf.object_type.model_class(), prefix=NESTED_SERIALIZER_PREFIX) - value = serializer(value, many=True, context=self.parent.context).data + serializer = get_serializer_for_model(cf.object_type.model_class()) + value = serializer(value, nested=True, many=True, context=self.parent.context).data data[cf.name] = value return data ``` So it seems the custom field serialization is working with the old nested serializer, but not with the standard serializer with `nested=True`. Sorry again for the close/reopen spam.
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

The same action executed in the GUI finishes without any problems, so seems to be purely a REST API issue.

@peteeckel commented on GitHub (Mar 6, 2024): The same action executed in the GUI finishes without any problems, so seems to be purely a REST API issue.
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

Another piece of information: The serializer works, both with the old and the new paradigm:

    view = NestedViewSerializer(
        many=False,
        read_only=False,
        required=False,
        default=None,
        help_text="View the zone belongs to",
    )
    view = ViewSerializer(
        nested=True,
        many=False,
        read_only=False,
        required=False,
        default=None,
        help_text="View the zone belongs to",
    )

The results are identical in both cases:

curl "https://192.168.106.105/api/plugins/netbox-dns/zones/1/?brief=1" \
     -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \
     -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP'
{
  "id": 1,
  "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/1/",
  "name": "zone1.example.com",
  "view": {
    "id": 2,
    "url": "http://192.168.106.105/api/plugins/netbox-dns/views/2/",
    "display": "external",
    "name": "external",
    "description": ""
  },
  "display": "[external] zone1.example.com",
  "status": "active",
  "description": "",
  "rfc2317_prefix": null,
  "active": true
}
curl "https://192.168.106.105/api/plugins/netbox-dns/zones/1/ \
     -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \
     -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP'
{
  "id": 1,
  "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/1/",
  "name": "zone1.example.com",
  "view": {
    "id": 2,
    "url": "http://192.168.106.105/api/plugins/netbox-dns/views/2/",
    "display": "external",
    "name": "external",
    "description": ""
  },
  "display": "[external] zone1.example.com",
  "nameservers": [
    {
      "id": 2,
      "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/2/",
      "display": "ns2.example.com",
      "name": "ns2.example.com"
    },
    {
      "id": 4,
      "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/4/",
      "display": "ns4.example.com",
      "name": "ns4.example.com"
    }
  ],
  "status": "active",
  "description": "",
  "tags": [
    {
      "id": 1,
      "url": "http://192.168.106.105/api/extras/tags/1/",
      "display": "tag1",
      "name": "tag1",
      "slug": "tag1",
      "color": "ff0000"
    }
  ],
  "created": "2023-12-21T15:22:09.231204Z",
  "last_updated": "2024-03-06T09:38:23.072286Z",
  "default_ttl": 86400,
  "soa_ttl": 86400,
  "soa_mname": {
    "id": 1,
    "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/1/",
    "display": "ns1.example.com",
    "name": "ns1.example.com"
  },
  "soa_rname": "hostmaster.example.com",
  "soa_serial": 1709717904,
  "soa_serial_auto": true,
  "soa_refresh": 172800,
  "soa_retry": 7200,
  "soa_expire": 2592000,
  "soa_minimum": 3600,
  "rfc2317_prefix": null,
  "rfc2317_parent_managed": false,
  "rfc2317_parent_zone": null,
  "rfc2317_child_zones": [],
  "registrar": null,
  "registry_domain_id": null,
  "registrant": null,
  "tech_c": null,
  "admin_c": null,
  "billing_c": null,
  "active": true,
  "custom_fields": {},
  "tenant": null
}
@peteeckel commented on GitHub (Mar 6, 2024): Another piece of information: The serializer works, both with the old and the new paradigm: ```python view = NestedViewSerializer( many=False, read_only=False, required=False, default=None, help_text="View the zone belongs to", ) ``` ```python view = ViewSerializer( nested=True, many=False, read_only=False, required=False, default=None, help_text="View the zone belongs to", ) ``` The results are identical in both cases: ```bash curl "https://192.168.106.105/api/plugins/netbox-dns/zones/1/?brief=1" \ -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \ -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' ``` ```json { "id": 1, "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/1/", "name": "zone1.example.com", "view": { "id": 2, "url": "http://192.168.106.105/api/plugins/netbox-dns/views/2/", "display": "external", "name": "external", "description": "" }, "display": "[external] zone1.example.com", "status": "active", "description": "", "rfc2317_prefix": null, "active": true } ``` ```bash curl "https://192.168.106.105/api/plugins/netbox-dns/zones/1/ \ -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \ -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' ``` ```json { "id": 1, "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/1/", "name": "zone1.example.com", "view": { "id": 2, "url": "http://192.168.106.105/api/plugins/netbox-dns/views/2/", "display": "external", "name": "external", "description": "" }, "display": "[external] zone1.example.com", "nameservers": [ { "id": 2, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/2/", "display": "ns2.example.com", "name": "ns2.example.com" }, { "id": 4, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/4/", "display": "ns4.example.com", "name": "ns4.example.com" } ], "status": "active", "description": "", "tags": [ { "id": 1, "url": "http://192.168.106.105/api/extras/tags/1/", "display": "tag1", "name": "tag1", "slug": "tag1", "color": "ff0000" } ], "created": "2023-12-21T15:22:09.231204Z", "last_updated": "2024-03-06T09:38:23.072286Z", "default_ttl": 86400, "soa_ttl": 86400, "soa_mname": { "id": 1, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/1/", "display": "ns1.example.com", "name": "ns1.example.com" }, "soa_rname": "hostmaster.example.com", "soa_serial": 1709717904, "soa_serial_auto": true, "soa_refresh": 172800, "soa_retry": 7200, "soa_expire": 2592000, "soa_minimum": 3600, "rfc2317_prefix": null, "rfc2317_parent_managed": false, "rfc2317_parent_zone": null, "rfc2317_child_zones": [], "registrar": null, "registry_domain_id": null, "registrant": null, "tech_c": null, "admin_c": null, "billing_c": null, "active": true, "custom_fields": {}, "tenant": null } ```
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

Digging deeper, the difference between the NestedZoneSerializer and the ZoneSerializer is that the latter has a validator attached (inherited from the Zone model), while the former has none. It is the validator that fails with the exception.

NestedZoneSerializer(context={'request': <rest_framework.request.Request: POST '/api/ipam/ip-addresses/'>, 'format': None, 'view': <ipam.api.views.IPAddressViewSet object>, 'custom_fields': <RestrictedQuerySet [<CustomField: Test Device>, <CustomField: Disable PTR>, <CustomField: Name>, <CustomField: TTL>, <CustomField: Zone>]>}, data='1'):
    id = IntegerField(label='ID', read_only=True)
    url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:zone-detail')
    display = SerializerMethodField(read_only=True)
    name = CharField(max_length=255, required=True)
    view = NestedViewSerializer(help_text='View the zone belongs to', read_only=True, required=False):
        id = IntegerField(label='ID', read_only=True)
        url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:view-detail')
        display = SerializerMethodField(read_only=True)
        name = CharField(max_length=255, validators=[<UniqueValidator(queryset=View.objects.all())>])
    status = ChoiceField(allow_blank=True, choices=[('active', 'Active'), ('reserved', 'Reserved'), ('deprecated', 'Deprecated'), ('parked', 'Parked')], required=False)
    active = BooleanField(allow_null=True, read_only=True, required=False)
    rfc2317_prefix = ModelField(allow_null=True, help_text='RFC2317 IPv4 prefix prefix with a length of at least 25 bits', label='RCF2317 Prefix', model_field=<netbox_dns.fields.rfc2317.RFC2317NetworkField: rfc2317_prefix>, required=False, validators=[<function validate_ipv4>, <function validate_prefix>, <function validate_rfc2317>])
ZoneSerializer(context={'request': <rest_framework.request.Request: POST '/api/ipam/ip-addresses/'>, 'format': None, 'view': <ipam.api.views.IPAddressViewSet object>, 'custom_fields': <RestrictedQuerySet [<CustomField: Test Device>, <CustomField: Disable PTR>, <CustomField: Name>, <CustomField: TTL>, <CustomField: Zone>]>}, data='1', nested=True):
    id = IntegerField(label='ID', read_only=True)
    url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:zone-detail')
    name = CharField(max_length=255, required=True)
    view = NestedViewSerializer(default=None, help_text='View the zone belongs to', read_only=False, required=False):
        id = IntegerField(label='ID', read_only=True)
        url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:view-detail')
        display = SerializerMethodField(read_only=True)
        name = CharField(max_length=255, validators=[<UniqueValidator(queryset=View.objects.all())>])
    display = SerializerMethodField(read_only=True)
    status = ChoiceField(allow_blank=True, choices=[('active', 'Active'), ('reserved', 'Reserved'), ('deprecated', 'Deprecated'), ('parked', 'Parked')], required=False)
    description = CharField(allow_blank=True, max_length=200, required=False)
    rfc2317_prefix = ModelField(allow_null=True, help_text='RFC2317 IPv4 prefix prefix with a length of at least 25 bits', label='RCF2317 Prefix', model_field=<netbox_dns.fields.rfc2317.RFC2317NetworkField: rfc2317_prefix>, required=False, validators=[<function validate_ipv4>, <function validate_prefix>, <function validate_rfc2317>])
    active = BooleanField(allow_null=True, read_only=True, required=False)
    class Meta:
        validators = [<UniqueTogetherValidator(queryset=Zone.objects.all(), fields=('view', 'name'))>]

I tentatively removed the unique_together validator from the model, and now the request no longer fails. This is of course not the solution, but it confirms that the difference between the two serializers is in fact the cause of the problem.

    class Meta:
[...]
        unique_together = (
            "view",
            "name",
        )
@peteeckel commented on GitHub (Mar 6, 2024): Digging deeper, the difference between the `NestedZoneSerializer` and the `ZoneSerializer` is that the latter has a validator attached (inherited from the `Zone` model), while the former has none. It is the validator that fails with the exception. ```python NestedZoneSerializer(context={'request': <rest_framework.request.Request: POST '/api/ipam/ip-addresses/'>, 'format': None, 'view': <ipam.api.views.IPAddressViewSet object>, 'custom_fields': <RestrictedQuerySet [<CustomField: Test Device>, <CustomField: Disable PTR>, <CustomField: Name>, <CustomField: TTL>, <CustomField: Zone>]>}, data='1'): id = IntegerField(label='ID', read_only=True) url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:zone-detail') display = SerializerMethodField(read_only=True) name = CharField(max_length=255, required=True) view = NestedViewSerializer(help_text='View the zone belongs to', read_only=True, required=False): id = IntegerField(label='ID', read_only=True) url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:view-detail') display = SerializerMethodField(read_only=True) name = CharField(max_length=255, validators=[<UniqueValidator(queryset=View.objects.all())>]) status = ChoiceField(allow_blank=True, choices=[('active', 'Active'), ('reserved', 'Reserved'), ('deprecated', 'Deprecated'), ('parked', 'Parked')], required=False) active = BooleanField(allow_null=True, read_only=True, required=False) rfc2317_prefix = ModelField(allow_null=True, help_text='RFC2317 IPv4 prefix prefix with a length of at least 25 bits', label='RCF2317 Prefix', model_field=<netbox_dns.fields.rfc2317.RFC2317NetworkField: rfc2317_prefix>, required=False, validators=[<function validate_ipv4>, <function validate_prefix>, <function validate_rfc2317>]) ``` ```python ZoneSerializer(context={'request': <rest_framework.request.Request: POST '/api/ipam/ip-addresses/'>, 'format': None, 'view': <ipam.api.views.IPAddressViewSet object>, 'custom_fields': <RestrictedQuerySet [<CustomField: Test Device>, <CustomField: Disable PTR>, <CustomField: Name>, <CustomField: TTL>, <CustomField: Zone>]>}, data='1', nested=True): id = IntegerField(label='ID', read_only=True) url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:zone-detail') name = CharField(max_length=255, required=True) view = NestedViewSerializer(default=None, help_text='View the zone belongs to', read_only=False, required=False): id = IntegerField(label='ID', read_only=True) url = HyperlinkedIdentityField(view_name='plugins-api:netbox_dns-api:view-detail') display = SerializerMethodField(read_only=True) name = CharField(max_length=255, validators=[<UniqueValidator(queryset=View.objects.all())>]) display = SerializerMethodField(read_only=True) status = ChoiceField(allow_blank=True, choices=[('active', 'Active'), ('reserved', 'Reserved'), ('deprecated', 'Deprecated'), ('parked', 'Parked')], required=False) description = CharField(allow_blank=True, max_length=200, required=False) rfc2317_prefix = ModelField(allow_null=True, help_text='RFC2317 IPv4 prefix prefix with a length of at least 25 bits', label='RCF2317 Prefix', model_field=<netbox_dns.fields.rfc2317.RFC2317NetworkField: rfc2317_prefix>, required=False, validators=[<function validate_ipv4>, <function validate_prefix>, <function validate_rfc2317>]) active = BooleanField(allow_null=True, read_only=True, required=False) class Meta: validators = [<UniqueTogetherValidator(queryset=Zone.objects.all(), fields=('view', 'name'))>] ``` I tentatively removed the `unique_together` validator from the model, and now the request no longer fails. This is of course not the solution, but it confirms that the difference between the two serializers is in fact the cause of the problem. ```python class Meta: [...] unique_together = ( "view", "name", ) ```
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

The validator itself is not the problem - creating objects works perfectly and the validator returns without raising an exception:

curl -X "POST" "https://192.168.106.105/api/plugins/netbox-dns/zones/" \
     -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' \
     -d $'{
  "soa_expire": "1800",
  "soa_rname": "hostmaster.example.com",
  "soa_minimum": "7200",
  "name": "zonex6.example.com",
  "soa_mname": {
    "name": "ns1.example.com"
  },
  "soa_retry": "3600",
  "soa_serial": "1000",
  "default_ttl": "86400",
  "nameservers": [
    {
      "name": "ns2.example.com"
    },
    {
      "name": "ns4.example.com"
    }
  ],
  "soa_ttl": "3600",
  "soa_refresh": "86400"
}'
{
  "id": 23,
  "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/23/",
  "name": "zonex6.example.com",
  "view": null,
  "display": "zonex6.example.com",
  "nameservers": [
    {
      "id": 2,
      "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/2/",
      "display": "ns2.example.com",
      "name": "ns2.example.com"
    },
    {
      "id": 4,
      "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/4/",
      "display": "ns4.example.com",
      "name": "ns4.example.com"
    }
  ],
  "status": "active",
  "description": "",
  "tags": [],
  "created": "2024-03-06T12:16:21.564333Z",
  "last_updated": "2024-03-06T12:16:21.688141Z",
  "default_ttl": 86400,
  "soa_ttl": 3600,
  "soa_mname": {
    "id": 1,
    "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/1/",
    "display": "ns1.example.com",
    "name": "ns1.example.com"
  },
  "soa_rname": "hostmaster.example.com",
  "soa_serial": 1709727382,
  "soa_serial_auto": true,
  "soa_refresh": 86400,
  "soa_retry": 3600,
  "soa_expire": 1800,
  "soa_minimum": 7200,
  "rfc2317_prefix": null,
  "rfc2317_parent_managed": false,
  "rfc2317_parent_zone": null,
  "rfc2317_child_zones": [],
  "registrar": null,
  "registry_domain_id": null,
  "registrant": null,
  "tech_c": null,
  "admin_c": null,
  "billing_c": null,
  "active": null,
  "custom_fields": {},
  "tenant": null
}
@peteeckel commented on GitHub (Mar 6, 2024): The validator itself is not the problem - creating objects works perfectly and the validator returns without raising an exception: ```bash curl -X "POST" "https://192.168.106.105/api/plugins/netbox-dns/zones/" \ -H 'Authorization: Token de003e9acebcdbe142eb04959b963b77fb2f1bb6' \ -H 'Content-Type: application/json; charset=utf-8' \ -H 'Cookie: csrftoken=y42qrauR8Z5S9o1xAiN0gbnVVPqDgmNP' \ -d $'{ "soa_expire": "1800", "soa_rname": "hostmaster.example.com", "soa_minimum": "7200", "name": "zonex6.example.com", "soa_mname": { "name": "ns1.example.com" }, "soa_retry": "3600", "soa_serial": "1000", "default_ttl": "86400", "nameservers": [ { "name": "ns2.example.com" }, { "name": "ns4.example.com" } ], "soa_ttl": "3600", "soa_refresh": "86400" }' ``` ``` { "id": 23, "url": "http://192.168.106.105/api/plugins/netbox-dns/zones/23/", "name": "zonex6.example.com", "view": null, "display": "zonex6.example.com", "nameservers": [ { "id": 2, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/2/", "display": "ns2.example.com", "name": "ns2.example.com" }, { "id": 4, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/4/", "display": "ns4.example.com", "name": "ns4.example.com" } ], "status": "active", "description": "", "tags": [], "created": "2024-03-06T12:16:21.564333Z", "last_updated": "2024-03-06T12:16:21.688141Z", "default_ttl": 86400, "soa_ttl": 3600, "soa_mname": { "id": 1, "url": "http://192.168.106.105/api/plugins/netbox-dns/nameservers/1/", "display": "ns1.example.com", "name": "ns1.example.com" }, "soa_rname": "hostmaster.example.com", "soa_serial": 1709727382, "soa_serial_auto": true, "soa_refresh": 86400, "soa_retry": 3600, "soa_expire": 1800, "soa_minimum": 7200, "rfc2317_prefix": null, "rfc2317_parent_managed": false, "rfc2317_parent_zone": null, "rfc2317_child_zones": [], "registrar": null, "registry_domain_id": null, "registrant": null, "tech_c": null, "admin_c": null, "billing_c": null, "active": null, "custom_fields": {}, "tenant": null } ```
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

The exception is raised by the validator because the arguments passed to __call__ are wrong. Normally, the Validator instance is called with two arguments: attrs and serializer. attrs is an OrderedDict containing the data for the object to be validated:

OrderedDict([('name', 'zonex6.example.com'), ('view', None), ('nameservers', [<NameServer: ns2.example.com>, <NameServer: ns4.example.com>]), ('default_ttl', 86400), ('soa_ttl', 3600), ('soa_mname', <NameServer: ns1.example.com>), ('soa_rname', 'hostmaster.example.com'), ('soa_serial', 1000), ('soa_refresh', 86400), ('soa_retry', 3600), ('soa_expire', 1800), ('soa_minimum', 7200), ('custom_field_data', {})])

But when the validator is called by NetBox in the error situation, an object, in this case a Zone object, is passed for attrs. That causes the exception to be raised as it is not iterable.

@peteeckel commented on GitHub (Mar 6, 2024): The exception is raised by the validator because the arguments passed to `__call__` are wrong. Normally, the Validator instance is called with two arguments: `attrs` and `serializer`. `attrs` is an OrderedDict containing the data for the object to be validated: ```python OrderedDict([('name', 'zonex6.example.com'), ('view', None), ('nameservers', [<NameServer: ns2.example.com>, <NameServer: ns4.example.com>]), ('default_ttl', 86400), ('soa_ttl', 3600), ('soa_mname', <NameServer: ns1.example.com>), ('soa_rname', 'hostmaster.example.com'), ('soa_serial', 1000), ('soa_refresh', 86400), ('soa_retry', 3600), ('soa_expire', 1800), ('soa_minimum', 7200), ('custom_field_data', {})]) ``` But when the validator is called by NetBox in the error situation, an object, in this case a `Zone` object, is passed for `attrs`. That causes the exception to be raised as it is not iterable.
Author
Owner

@peteeckel commented on GitHub (Mar 6, 2024):

I found a workaround for the problem, applied to the ZoneSerializer class:

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.nested:
            self.validators = []

This could also be the solution for NetBoxModelSerializer in general, as it emulates the situation with WritableNestedSerializer, which does not have any validators.

@peteeckel commented on GitHub (Mar 6, 2024): I found a workaround for the problem, applied to the `ZoneSerializer` class: ``` def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) if self.nested: self.validators = [] ``` This could also be the solution for `NetBoxModelSerializer` in general, as it emulates the situation with `WritableNestedSerializer`, which does not have any validators.
Author
Owner

@jeffgdotorg commented on GitHub (Apr 1, 2024):

Working with your original steps to reproduce on NetBox 4.0.0-dev 9fd23b4 and netbox_dns v1.0.0-dev 1214c1d, I bumped into a separate problem: NetBox raises a ValueError when I try to create the custom field in the UI.

Marking as revisions needed for the time being. I trust you'll get back to it and find steps to reproduce that don't require the installation of a plugin nightly in a NetBox nightly.

Edit: Can't markdown late in the day on <6h sleep.

@jeffgdotorg commented on GitHub (Apr 1, 2024): Working with your original steps to reproduce on NetBox 4.0.0-dev [`9fd23b4`](https://github.com/netbox-community/netbox/commit/3ab2f25ee13c07647984d1caaa5e2b15d9fd23b4) and netbox_dns v1.0.0-dev [`1214c1d`](), I bumped into a separate problem: NetBox raises a `ValueError` when I try to create the custom field in the UI. Marking as revisions needed for the time being. I trust you'll get back to it and find steps to reproduce that don't require the installation of a plugin nightly in a NetBox nightly. Edit: Can't markdown late in the day on <6h sleep.
Author
Owner

@peteeckel commented on GitHub (Apr 2, 2024):

Hi @jeffgdotorg, thanks for trying to reproduce it.

I guess your problem is a result of database migration having been run before you switched back to the older NetBox commit SHA, which is often a problem and very definitely so after c8d9d9358. Never mind, though, tonight, if everything works well, the first beta is due to be released for NetBox 4 and NetBox DNS, and then we'll have a fixed point.

The steps to reproduce will be different: NetBox DNS contains a workaround for the problem, and the obvious way to reproduce it is to remove the workarond:

diff --git a/netbox_dns/api/serializers_/zone.py b/netbox_dns/api/serializers_/zone.py
index 8acc97b..4e9abd1 100644
--- a/netbox_dns/api/serializers_/zone.py
+++ b/netbox_dns/api/serializers_/zone.py
@@ -19,10 +19,10 @@ class ZoneSerializer(NetBoxModelSerializer):
     #
     # See https://github.com/netbox-community/netbox/issues/15351 for details.
     # -
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        if self.nested:
-            self.validators = []
+#    def __init__(self, *args, **kwargs):
+#        super().__init__(*args, **kwargs)
+#        if self.nested:
+#            self.validators = []
@peteeckel commented on GitHub (Apr 2, 2024): Hi @jeffgdotorg, thanks for trying to reproduce it. I guess your problem is a result of database migration having been run before you switched back to the older NetBox commit SHA, which is often a problem and very definitely so after c8d9d9358. Never mind, though, tonight, if everything works well, the first beta is due to be released for NetBox 4 and NetBox DNS, and then we'll have a fixed point. The steps to reproduce will be different: NetBox DNS contains a workaround for the problem, and the obvious way to reproduce it is to remove the workarond: ```python diff --git a/netbox_dns/api/serializers_/zone.py b/netbox_dns/api/serializers_/zone.py index 8acc97b..4e9abd1 100644 --- a/netbox_dns/api/serializers_/zone.py +++ b/netbox_dns/api/serializers_/zone.py @@ -19,10 +19,10 @@ class ZoneSerializer(NetBoxModelSerializer): # # See https://github.com/netbox-community/netbox/issues/15351 for details. # - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - if self.nested: - self.validators = [] +# def __init__(self, *args, **kwargs): +# super().__init__(*args, **kwargs) +# if self.nested: +# self.validators = [] ```
Author
Owner

@peteeckel commented on GitHub (Apr 3, 2024):

Hi @jeffgdotorg, good news: At some point in the last month's worth of commits the issue has been resolved :-)

To be precise - @jeremystretch fixed it here:

git blame netbox/netbox/api/serializers/base.py
[...]
3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400  30)         # Disable validators for nested objects (which already exist)
3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400  31)         if self.nested:
3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400  32)             self.validators = []

... which is pretty exactly what I added for a workaround. Everything is OK now!

@peteeckel commented on GitHub (Apr 3, 2024): Hi @jeffgdotorg, good news: At some point in the last month's worth of commits the issue has been resolved :-) To be precise - @jeremystretch fixed it here: ``` git blame netbox/netbox/api/serializers/base.py [...] 3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400 30) # Disable validators for nested objects (which already exist) 3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400 31) if self.nested: 3be3bbe534 (Jeremy Stretch 2024-03-29 13:13:41 -0400 32) self.validators = [] ``` ... which is pretty exactly what I added for a workaround. Everything is OK now!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#9324