GraphQL: Support for filtering on custom fields #5540

@candlerb commented on GitHub (Oct 21, 2021):

Another way might be via a graphql query on custom_fields. I note that you can ask for custom_fields in a response:

# curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \
  --data '{"query": "query {device_list(id:\"5\") {id custom_fields}}"}' | python3 -m json.tool
{
    "data": {
        "device_list": [
            {
                "id": "5",
                "custom_fields": {
                    "snmp_module": [
                        "mikrotik_secret"
                    ]
                }
            }
        ]
    }
}

(There's also custom_field_data which returns a string holding JSON). But as far as I can tell, you can't currently filter on custom_fields in a query:

# curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \
  --data '{"query": "query {device_list(custom_fields:{snmp_module:\"mikrotik_secret\"}) {id custom_fields}}"}' | python3 -m json.tool
{
    "errors": [
        {
            "message": "Unknown argument \"custom_fields\" on field \"device_list\" of type \"Query\".",
            "locations": [
                {
                    "line": 1,
                    "column": 20
                }
            ]
        }
    ]
}

It would of course be good if such a query could be pushed down to a SQL condition in postgres, rather than making Netbox retrieve all objects from the database and discard the ones which don't match.

@candlerb commented on GitHub (Oct 21, 2021): Another way might be via a graphql query on `custom_fields`. I note that you can ask for `custom_fields` in a response: ``` # curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \ --data '{"query": "query {device_list(id:\"5\") {id custom_fields}}"}' | python3 -m json.tool { "data": { "device_list": [ { "id": "5", "custom_fields": { "snmp_module": [ "mikrotik_secret" ] } } ] } } ``` (There's also `custom_field_data` which returns a string holding JSON). But as far as I can tell, you can't currently filter on `custom_fields` in a query: ``` # curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \ --data '{"query": "query {device_list(custom_fields:{snmp_module:\"mikrotik_secret\"}) {id custom_fields}}"}' | python3 -m json.tool { "errors": [ { "message": "Unknown argument \"custom_fields\" on field \"device_list\" of type \"Query\".", "locations": [ { "line": 1, "column": 20 } ] } ] } ``` It would of course be good if such a query could be pushed down to a SQL condition in postgres, rather than making Netbox retrieve all objects from the database and discard the ones which don't match.

adam commented

@github-actions[bot] commented on GitHub (Dec 21, 2021):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Dec 21, 2021): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).

adam commented

@jeremystretch commented on GitHub (Jul 27, 2022):

Blocked by #9856

@jeremystretch commented on GitHub (Jul 27, 2022): Blocked by #9856

adam commented

@github-actions[bot] commented on GitHub (Nov 27, 2022):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Nov 27, 2022): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).

adam commented

@arthanson commented on GitHub (Mar 13, 2023):

Blocked on #11949

@arthanson commented on GitHub (Mar 13, 2023): Blocked on #11949

adam commented

@arthanson commented on GitHub (Apr 21, 2023):

Adding blocked until #9856

@arthanson commented on GitHub (Apr 21, 2023): Adding blocked until #9856

adam commented

@einstux commented on GitHub (Apr 23, 2024):

@arthanson Since #9856 is done, can you unblock this issue? Any ideas if and when this is now going to happen?

@einstux commented on GitHub (Apr 23, 2024): @arthanson Since #9856 is done, can you unblock this issue? Any ideas if and when this is now going to happen?

adam commented

@jeremystretch commented on GitHub (Apr 23, 2024):

I've marked this as "needs owner" for anyone who would like to take a stab at implementing this for v4.0 (with the new Strawberry GraphQL backend). @arthanson anything in particular you'd like to point out for this?

@jeremystretch commented on GitHub (Apr 23, 2024): I've marked this as "needs owner" for anyone who would like to take a stab at implementing this for v4.0 (with the new Strawberry GraphQL backend). @arthanson anything in particular you'd like to point out for this?

adam commented

@christophbeberweil commented on GitHub (Aug 5, 2024):

I would like to take a stab at it :) Right now I am looking into extending extras.graphql.mixins.CustomFieldsMixin, which seems to be a promising candidate for this change. Does anyone have a concrete preference how this should be implemented? If so feel free to leave any pointers

@christophbeberweil commented on GitHub (Aug 5, 2024): I would like to take a stab at it :) Right now I am looking into extending `extras.graphql.mixins.CustomFieldsMixin`, which seems to be a promising candidate for this change. Does anyone have a concrete preference how this should be implemented? If so feel free to leave any pointers

adam commented

@jeremystretch commented on GitHub (Aug 6, 2024):

@christophbeberweil thanks, I've assigned this to you.

@jeremystretch commented on GitHub (Aug 6, 2024): @christophbeberweil thanks, I've assigned this to you.

adam commented

@christophbeberweil commented on GitHub (Sep 4, 2024):

Hey @jeremystretch,

I think I found a way to enable filtering of custom fields in GraphQL. My solution works by extending netbox.graphql.filter_mixins.autotype_decorator, in such a way that create_attribute_and_function is called for each applicable CustomField of the decrated filter.

This works for str fields (foo_txt) but unfortunately, the filter method seems to iterate over the querystring and search for each occurrence of the letter in the custom field. While this does search, it is obviously not usable. Could I use StrFilterLookup instead? I saw its definition in the Graph_i_QL frontend, but not in the source code. Or is there another way to define the actual filter behaviour that I missed?

Speaking of the Graph_i_QL frontend, while the backend detects my new filter cf_foo_txt, but the lint in GraphiQL does not know of cf_foo_txt.

When I create a new custom field a_int for dcim.Device with type integer it is not recognized in the graphql api unless the server is restarted. Do you know of a way how to make graphql aware of changing custom fields at runtime, without the need for a server restart?

After I restart the server, resolve_field raises an exception (AttributeError: 'NoneType' object has no attribute 'model') in netbox.filtersets.BaseFilterSet.get_additional_lookups because the field_name custom_field_data__a_int queryied by get_model_field can not be found on the model Device. Interestingly, exactly the same code works fine for a CustomField of type str. What could be the reason for this?

Since these changes are not ready for merging, I did not open a merge request. Please consider this commit: ad78d04bb6

Thanks :)

@christophbeberweil commented on GitHub (Sep 4, 2024): Hey @jeremystretch, I think I found a way to enable filtering of custom fields in GraphQL. My solution works by extending `netbox.graphql.filter_mixins.autotype_decorator`, in such a way that `create_attribute_and_function` is called for each applicable CustomField of the decrated filter. This works for `str` fields (foo_txt) but unfortunately, the filter method seems to iterate over the querystring and search for each occurrence of the letter in the custom field. While this does search, it is obviously not usable. Could I use StrFilterLookup instead? I saw its definition in the Graph_i_QL frontend, but not in the source code. Or is there another way to define the actual filter behaviour that I missed? Speaking of the Graph_i_QL frontend, while the backend detects my new filter `cf_foo_txt`, but the lint in GraphiQL does not know of `cf_foo_txt`. When I create a new custom field `a_int` for `dcim.Device` with type integer it is not recognized in the graphql api unless the server is restarted. Do you know of a way how to make graphql aware of changing custom fields at runtime, without the need for a server restart? After I restart the server, resolve_field raises an exception (AttributeError: 'NoneType' object has no attribute 'model') in `netbox.filtersets.BaseFilterSet.get_additional_lookups` because the field_name `custom_field_data__a_int` queryied by get_model_field can not be found on the model Device. Interestingly, exactly the same code works fine for a CustomField of type str. What could be the reason for this? Since these changes are not ready for merging, I did not open a merge request. Please consider this commit: https://github.com/netbox-community/netbox/commit/ad78d04bb689c6d8136ec3e22b8794e7b9ba8d13 Thanks :)

adam commented

@christophbeberweil commented on GitHub (Sep 17, 2024):

Hi @jeremystretch, did you have a chance to look at my questions yet? Maybe you could indicate if my changes go in the right directopn from your point of view?

Thanks

@christophbeberweil commented on GitHub (Sep 17, 2024): Hi @jeremystretch, did you have a chance to look at my questions yet? Maybe you could indicate if my changes go in the right directopn from your point of view? Thanks

adam commented

@jeremystretch commented on GitHub (Sep 17, 2024):

@christophbeberweil I am not able to spend time on this, hence it being tagged for a volunteer. If you need help, I recommend reaching out to others in the community to see if they're able to assist.

@jeremystretch commented on GitHub (Sep 17, 2024): @christophbeberweil I am not able to spend time on this, hence it being tagged for a volunteer. If you need help, I recommend reaching out to others in the community to see if they're able to assist.

adam commented

@jeremypng commented on GitHub (Jan 8, 2025):

I worked on this today, building on @christophbeberweil's work and have it working for exact string queries against custom fields so far. I'm still working on the integer and other field types.

A couple of things:

I did use the autotype_decorator as well but used an instance of the filterset instead of the class. This allows us to use the generated filters for custom fields that the NetBoxModelFilterSet generates on init.
It appears that the deprecated Strawberry filters are in place because of the autotype_decorator to generate the filter fields. I think this means we can't use the updated StrFilterLookup. See here.
The GraphiQL web interface doesn't recognize my new filters either. I haven't chased that down yet.
It does not appear that the schema can be modified after the strawberry URL is loaded by Django. I tried really hard to modify it at runtime (ie: on a post_save signal for a custom field definition), but it is not designed for that.
The error from netbox.filtersets.BaseFilterSet.get_additional_lookups appears to be an issue in django_filters.utils.get_field_parts. It tries to follow relationships. However, the custom_field_data__cust_id filter field_name refers to the custom_field_data JSON field on the actual model. And the ORM uses JSON filtering to look for cust_id in that JSON structure. But get_field_parts only checks for RelatedField and ForeignObjectRel fields before giving up.
To resolve this, I added some exception handling to BaseFilterMixin to fall back to ORM filtering if the filterset is not valid. This seems to be working at the moment.
Another way to solve this would be to get away from the Filter field definitions being generated by the autotype_decorator and instead have it generate resolver functions. I went a little ways down this path, but I think it would require making more changes to the overall GraphQL implementation in Netbox.

My code is here for the moment. Happy to discuss further.

@jeremypng commented on GitHub (Jan 8, 2025): I worked on this today, building on @christophbeberweil's work and have it working for exact string queries against custom fields so far. I'm still working on the integer and other field types. A couple of things: - I did use the autotype_decorator as well but used an instance of the filterset instead of the class. This allows us to use the generated filters for custom fields that the NetBoxModelFilterSet generates on __init__. - It appears that the deprecated Strawberry filters are in place because of the autotype_decorator to generate the filter fields. I think this means we can't use the updated StrFilterLookup. See [here](https://strawberry.rocks/docs/django/guide/filters#legacy-filtering). - The GraphiQL web interface doesn't recognize my new filters either. I haven't chased that down yet. - It does not appear that the schema can be modified after the strawberry URL is loaded by Django. I tried really hard to modify it at runtime (ie: on a post_save signal for a custom field definition), but it is not designed for that. - The error from netbox.filtersets.BaseFilterSet.get_additional_lookups appears to be an issue in django_filters.utils.get_field_parts. It tries to follow relationships. However, the custom_field_data__cust_id filter field_name refers to the custom_field_data JSON field on the actual model. And the ORM uses JSON filtering to look for cust_id in that JSON structure. But get_field_parts only checks for RelatedField and ForeignObjectRel fields before giving up. - To resolve this, I added some exception handling to BaseFilterMixin to fall back to ORM filtering if the filterset is not valid. This seems to be working at the moment. - Another way to solve this would be to get away from the Filter field definitions being generated by the autotype_decorator and instead have it generate resolver functions. I went a little ways down this path, but I think it would require making more changes to the overall GraphQL implementation in Netbox. My code is [here](https://github.com/jeremypng/netbox/commit/a6cc2b7c3318bf7d4861967b08e1ff591fc1ecb4#diff-e0a3a07242c663b8a99ee18564f780d5ca797f1979c9bd22833255aa45e77342R44) for the moment. Happy to discuss further.

adam commented

@jeremypng commented on GitHub (Jan 10, 2025):

I've got this ready to submit a PR. If someone (@jeremystretch or @arthanson) can assign it to me, I'll submit it. I ended up making a few more changes to get this working smoothly.

The GraphiQL interface works fine with the custom schema entries now.
A restart of netbox is required for the Strawberry schema to recognize the new field. I have not found a way to modify the GraphQL schema at runtime.
Moved away from the deprecated filtering in Strawberry. This allows using the newer filter types and I expanded the map_strawberry_type function in filter_mixins.py.
Created a resolver function in netbox/graphql/resolvers.py to handle custom field filtering on list queries.
Modified each folder's schema.py list entries to use the new list resolver.
BaseFilterMixin in the netbox/graphql/schema.py no longer does any filtering. This is handled through the newer Strawberry to Django coupling since deprecated filters are disabled. It is still in place to apply the strawberry.input decorator to all of filters derived from this class.
Since BaseFilterMixin doesn't have a "filter_by_filterset" method anymore, we don't need the "should_create_function" logic in the autotype_decorator or map_strawberry_type functions.
Autotype_decorator creates filter.annotations for model fields, declared filters, and then instantiates a filterset instance to get the custom field filters.
The filterset instance requires database access and the migrations to be complete. Because the decorator loads at module import time, this requires a try/except pattern here to avoid crashing on startup if the migrations are missing or the database is not present (like in a testing only scenario). I tried delaying this until a later point, but the schema for graphql cannot be modified once it is loaded and it loads when the urls.py gets initialized, which is before the database is fully initialized. This is only a problem on first time startup and testing scenarios, but the try/except takes care of it. If there is a better way to solve this, I'm happy to make an adjustment.
The custom field annotations are augmented with a netbox_field_map on the filter class. This is used by the resolver to transform the GraphQL filter name (ie: cf_cust_id) into the Django field name (ie: custom_field_data__cust_id).
If a query contains a custom field, it is processed by the resolver function before being handed to Strawberry for normal processing. The resolver unsets the custom filter field entries because the normal Strawberry processing cannot map the custom fields to the custom_field_data field on the model. Permissions are enforced in the resolver, and the query syntax processing still happens using the Strawberry code. I'm just using that generated query and applying it against a queryset.filter() at that point.
At this point all of the tests are passing. It could use some more testing, but it looks like the functionality I was looking for is there without breaking anything else.

Here is the link to my code at this point.

@jeremypng commented on GitHub (Jan 10, 2025): I've got this ready to submit a PR. If someone (@jeremystretch or @arthanson) can assign it to me, I'll submit it. I ended up making a few more changes to get this working smoothly. - The GraphiQL interface works fine with the custom schema entries now. - A restart of netbox is required for the Strawberry schema to recognize the new field. I have not found a way to modify the GraphQL schema at runtime. - Moved away from the deprecated filtering in Strawberry. This allows using the newer filter types and I expanded the map_strawberry_type function in filter_mixins.py. - Created a resolver function in netbox/graphql/resolvers.py to handle custom field filtering on list queries. - Modified each folder's schema.py list entries to use the new list resolver. - BaseFilterMixin in the netbox/graphql/schema.py no longer does any filtering. This is handled through the newer Strawberry to Django coupling since deprecated filters are disabled. It is still in place to apply the strawberry.input decorator to all of filters derived from this class. - Since BaseFilterMixin doesn't have a "filter_by_filterset" method anymore, we don't need the "should_create_function" logic in the autotype_decorator or map_strawberry_type functions. - Autotype_decorator creates filter.__annotations__ for model fields, declared filters, and then instantiates a filterset instance to get the custom field filters. - The filterset instance requires database access and the migrations to be complete. Because the decorator loads at module import time, this requires a try/except pattern here to avoid crashing on startup if the migrations are missing or the database is not present (like in a testing only scenario). I tried delaying this until a later point, but the schema for graphql cannot be modified once it is loaded and it loads when the urls.py gets initialized, which is before the database is fully initialized. This is only a problem on first time startup and testing scenarios, but the try/except takes care of it. If there is a better way to solve this, I'm happy to make an adjustment. - The custom field annotations are augmented with a __netbox_field_map__ on the filter class. This is used by the resolver to transform the GraphQL filter name (ie: cf_cust_id) into the Django field name (ie: custom_field_data__cust_id). - If a query contains a custom field, it is processed by the resolver function before being handed to Strawberry for normal processing. The resolver unsets the custom filter field entries because the normal Strawberry processing cannot map the custom fields to the custom_field_data field on the model. Permissions are enforced in the resolver, and the query syntax processing still happens using the Strawberry code. I'm just using that generated query and applying it against a queryset.filter() at that point. - At this point all of the tests are passing. It could use some more testing, but it looks like the functionality I was looking for is there without breaking anything else. [Here is the link ](https://github.com/jeremypng/netbox/commit/a75056e76ac9fd3abda7703fd2633589913a8000) to my code at this point.

adam commented

@jeremypng commented on GitHub (Jan 18, 2025):

I have the custom field filtering itself working pretty well. However, the rest of the graphql implementation is in such bad shape, I'm going a different route. Introspecting the django fitlers and models to try to infer a graphql schema at import time is very complex and requires gutting Strawberry of most of its power.

I've cycled through most of the options outlined by @arthanson in Issue #17688 and come to some conclusions that there is a better way to solve this. I'll start a discussion around this.

@jeremypng commented on GitHub (Jan 18, 2025): I have the custom field filtering itself working pretty well. However, the rest of the graphql implementation is in such bad shape, I'm going a different route. Introspecting the django fitlers and models to try to infer a graphql schema at import time is very complex and requires gutting Strawberry of most of its power. I've cycled through most of the options outlined by @arthanson in Issue #17688 and come to some conclusions that there is a better way to solve this. I'll start a discussion around this.

adam commented

@jeremystretch commented on GitHub (Jan 20, 2025):

@jeremypng thanks for starting this discussion. I haven't had any time to dedicate to the theme of GraphQL recently and @arthanson is currently loaned out for another initiative, but I hope we can prioritize this work soon as it's clearly a pain point for NetBox users.

@jeremystretch commented on GitHub (Jan 20, 2025): @jeremypng thanks for starting [this discussion](https://github.com/netbox-community/netbox/discussions/18431). I haven't had any time to dedicate to the theme of GraphQL recently and @arthanson is currently loaned out for another initiative, but I hope we can prioritize this work soon as it's clearly a pain point for NetBox users.

adam commented