starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Leverage database triggers to generate object change records #8750

New Issue
Closed
opened 2025-12-29 20:40:45 +01:00 by adam · 6 comments
adam commented 2025-12-29 20:40:45 +01:00
Owner
Copy Link

Originally created by @jeremystretch on GitHub (Oct 17, 2023).

NetBox version

v3.6.4

Feature type

Change to existing functionality

Proposed functionality

NetBox currently records the creation, modification, and deletion of objects using application-level logic; namely the change_logging() context manager and receiver functions for Django's post_save and post_delete signals. This FR proposes replacing this logic with PostgreSQL triggers using django-pgtrigger. (This is similar in nature to how django-pghistory works, however we'll keep our general-purpose ObjectChange model in place.)

At a high level, this will entail the following:

  1. Remove the ObjectChange creation logic from the handle_changed_object() and handle_deleted_object() signal receivers. (The receivers themselves must remain to handle webhooks and metrics.)
  2. Enable or replicate django-pghistory's context manager to expose request details to the database.
  3. Define the appropriate triggers (TBD) under ChangeLoggingMixin to effect the automatic creation of a new changelog record whenever an object is created, modified, or deleted.

While much more exploration is needed, I believe all the necessary components have been identified.

Use case

Moving the change logging logic from the Python application layer to the database layer has several major benefits:

  1. Provide a much more robust change logging solution
  2. Eliminate the need to manually invoke the change_logging() context manager when working in the Python shell
  3. Reduce request processing overhead
  4. Simplify the core middleware

Database changes

While this implementation would obviously effect the creation of PostgreSQL triggers, I don't predict that any substantial schema changes to existing models will be necessary.

External dependencies

Originally created by @jeremystretch on GitHub (Oct 17, 2023). ### NetBox version v3.6.4 ### Feature type Change to existing functionality ### Proposed functionality NetBox currently records the creation, modification, and deletion of objects using application-level logic; namely the `change_logging()` context manager and receiver functions for Django's `post_save` and `post_delete` signals. This FR proposes replacing this logic with [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) using [`django-pgtrigger`](https://github.com/Opus10/django-pgtrigger). (This is similar in nature to how [`django-pghistory`](https://github.com/Opus10/django-pghistory) works, however we'll keep our general-purpose ObjectChange model in place.) At a high level, this will entail the following: 1. Remove the ObjectChange creation logic from the `handle_changed_object()` and `handle_deleted_object()` signal receivers. (The receivers themselves must remain to handle webhooks and metrics.) 2. Enable or replicate `django-pghistory`'s [context manager](https://django-pghistory.readthedocs.io/en/3.0.0/context/) to expose request details to the database. 3. Define the appropriate triggers (TBD) under `ChangeLoggingMixin` to effect the automatic creation of a new changelog record whenever an object is created, modified, or deleted. While much more exploration is needed, I believe all the necessary components have been identified. ### Use case Moving the change logging logic from the Python application layer to the database layer has several major benefits: 1. Provide a much more robust change logging solution 2. Eliminate the need to manually invoke the `change_logging()` context manager when working in the Python shell 3. Reduce request processing overhead 4. Simplify the core middleware ### Database changes While this implementation would obviously effect the creation of PostgreSQL triggers, I don't predict that any substantial schema changes to existing models will be necessary. ### External dependencies * [`django-pgtrigger`](https://github.com/Opus10/django-pgtrigger) * [`django-pghistory`](https://github.com/Opus10/django-pghistory) (potentially)
adam added the type: feature label 2025-12-29 20:40:45 +01:00
adam closed this issue 2025-12-29 20:40:46 +01:00
adam commented 2025-12-29 20:40:46 +01:00
Author
Owner
Copy Link

@ITJamie commented on GitHub (Oct 25, 2023):

one thing to consider. It's currently possible to have models that don't have changelog entries.

I recently ended up making an internal plugin for documenting "last seen" timestamps for inventory items. we did not want changelogs created every time we wanted to update the "last seen" time so a custom field was not an option, instead we made a plugin with a relationship model to inventory items insert /update last seen timestamps there.

I would hope by going down this path we can have some kind of flag on models where changelogs should or should not be created

@ITJamie commented on GitHub (Oct 25, 2023): one thing to consider. It's currently possible to have models that don't have changelog entries. I recently ended up making an internal plugin for documenting "last seen" timestamps for inventory items. we did not want changelogs created every time we wanted to update the "last seen" time so a custom field was not an option, instead we made a plugin with a relationship model to inventory items insert /update last seen timestamps there. I would hope by going down this path we can have some kind of flag on models where changelogs should or should not be created
adam commented 2025-12-29 20:40:47 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Oct 25, 2023):

one thing to consider. It's currently possible to have models that don't have changelog entries.

I recently ended up making an internal plugin for documenting "last seen" timestamps for inventory items. we did not want changelogs created every time we wanted to update the "last seen" time so a custom field was not an option, instead we made a plugin with a relationship model to inventory items insert /update last seen timestamps there.

I would hope by going down this path we can have some kind of flag on models where changelogs should or should not be created

It sounds like currently you are ommitting the ObjectChange mixin.

If I am understanding the proposal, the ObjectChange mixin will still exist but now it will instead generate the triggers for each table that needs to be change logged instead of having the middleware intercept the request and process it.

@DanSheps commented on GitHub (Oct 25, 2023): > one thing to consider. It's currently possible to have models that don't have changelog entries. > > I recently ended up making an internal plugin for documenting "last seen" timestamps for inventory items. we did not want changelogs created every time we wanted to update the "last seen" time so a custom field was not an option, instead we made a plugin with a relationship model to inventory items insert /update last seen timestamps there. > > I would hope by going down this path we can have some kind of flag on models where changelogs should or should not be created It sounds like currently you are ommitting the ObjectChange mixin. If I am understanding the proposal, the ObjectChange mixin will still exist but now it will instead generate the triggers for each table that needs to be change logged instead of having the middleware intercept the request and process it.
adam commented 2025-12-29 20:40:47 +01:00
Author
Owner
Copy Link

@ITJamie commented on GitHub (Oct 25, 2023):

It sounds like currently you are ommitting the ObjectChange mixin.

correct

If I am understanding the proposal, the ObjectChange mixin will still exist but now it will instead generate the triggers for each table that needs to be change logged instead of having the middleware intercept the request and process it.

Hopefully, otherwise ill have ~15k changelogs every night when airflow inventorys our ssd's and update the last_seen objects

@ITJamie commented on GitHub (Oct 25, 2023): > It sounds like currently you are ommitting the ObjectChange mixin. correct > If I am understanding the proposal, the ObjectChange mixin will still exist but now it will instead generate the triggers for each table that needs to be change logged instead of having the middleware intercept the request and process it. Hopefully, otherwise ill have ~15k changelogs every night when airflow inventorys our ssd's and update the last_seen objects
adam commented 2025-12-29 20:40:47 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Nov 6, 2023):

After digging further into this, I'm not sure it's feasible. All the logic for serializing object representations exists in Python; migrating this to a native database function would be challenging if not impossible. It would also preclude models from overriding their own serialization logic.

@jeremystretch commented on GitHub (Nov 6, 2023): After digging further into this, I'm not sure it's feasible. All the logic for serializing object representations exists in Python; migrating this to a native database function would be challenging if not impossible. It would also preclude models from overriding their own serialization logic.
adam commented 2025-12-29 20:40:47 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Feb 5, 2024):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Feb 5, 2024): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 20:40:47 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Feb 23, 2024):

We might revisit this idea in the future, but currently it does not seem feasible without undue effort.

@jeremystretch commented on GitHub (Feb 23, 2024): We might revisit this idea in the future, but currently it does not seem feasible without undue effort.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8750
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?