Add background jobs to plugin API #9468

New Issue

adam · 2025-12-29T20:50:16+01:00

adam commented

2025-12-29 20:50:16 +01:00

Originally created by @alehaa on GitHub (Apr 10, 2024).

Originally assigned to: @alehaa on GitHub.

NetBox version

v3.7.5

Feature type

Change to existing functionality

Proposed functionality

NetBox already includes the functionality to schedule jobs using either the django-rq library or the core.Job model combined with the JobsMixin class. Making the abstract job functionality publicly available to plugins allows decoupling from the backend of django-rq and a consistent experience in NetBox across all functionality. For this I propose:

Add netbox.models.JobsMixin to the list of API available model mixins in the documentation. This allows plugins to implement new models with Jobs enabled.

Add a new BackgroundJob class to implement the execution of the job's code, i.e. for consistency in calling start, terminate and setting error messages. This class should also be used in existing NetBox functionality to run background jobs for consistency. Below is a sample implementation from a plugin of mine for demonstration purposes.

import logging
from abc import ABC, abstractmethod

from rq.timeouts import JobTimeoutException

from core.choices import JobStatusChoices
from core.models import ObjectType, Job


class BackgroundJob(ABC):
    """
    Background Job helper class.

    This class handles the execution of a background job. It is responsible for
    maintaining its state, reporting errors, and scheduling recurring jobs.
    """

    @classmethod
    @abstractmethod
    def run(cls, *args, **kwargs) -> None:
        """
        Run the job.

        A `BackgroundJob` class needs to implement this method to execute all
        commands of the job.
        """
        pass

    @classmethod
    def handle(cls, job: Job, *args, **kwargs) -> None:
        """
        Handle the execution of a `BackgroundJob`.

        This method is called by the Job Scheduler to handle the execution of
        all job commands. It will maintain the job's metadata and handle errors.
        For periodic jobs, a new job is automatically scheduled using its
        `interval'.


        :param job: The job to be executed.
        """
        try:
            job.start()
            cls.run(job, *args, **kwargs)
            job.terminate()

        except Exception as e:
            job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e))
            if type(e) == JobTimeoutException:
                logging.error(e)

        # If the executed job is a periodic job, schedule its next execution at
        # the specified interval.
        finally:
            if job.interval:
                next_schedule_at = (job.scheduled or job.started) + timedelta(
                    minutes=job.interval
                )
                cls.enqueue(
                    instance=job.object,
                    name=job.name,
                    user=job.user,
                    schedule_at=next_schedule_at,
                    interval=job.interval,
                    **kwargs,
                )

    @classmethod
    def enqueue(cls, *args, **kwargs) -> None:
        """
        Enqueue a new `BackgroundJob`.

        This method is a wrapper of `Job.enqueue` using :py:meth:`handle` as
        function callback. See its documentation for parameters.
        """
        Job.enqueue(cls.handle, *args, **kwargs)

Optional: Enhance the core.models.Job.enqueue() method with a queue parameter to schedule jobs for specific queues (e.g. low, high). If not provided the default queue will be used, so there's no change in API compatibility.

Optional: Add the ability to schedule system background tasks by plugins, e.g. for periodic synchronization with other systems. Below is a sample implementation from a plugin of mine for demonstration purposes.

class ScheduledJob(BackgroundJob):
    """
    A periodic `BackgroundJob` used for system tasks.

    This class can be used to schedule system background tasks, e.g. to
    periodically perform data synchronization from other systems to NetBox or to
    perform housekeeping tasks.
    """

    ENQUEUED_STATUS = [
        JobStatusChoices.STATUS_PENDING,
        JobStatusChoices.STATUS_SCHEDULED,
        JobStatusChoices.STATUS_RUNNING,
    ]

    @classmethod
    def schedule(
        cls,
        instance: models.Model,
        name: str = "",
        interval: int = None,
        *args,
        **kwargs,
    ) -> None:
        """
        Schedule a `BackgroundJob`.

        This method adds a new `BackgroundJob` to the job queue. If the job
        schedule identified by its `instance` and `name` is already active,
        scheduling a second will be skipped. For additional parameters see
        :py:meth:`Job.enqueue`.

        The main use case for this method is to schedule jobs programmatically
        instead of using user events, e.g. to start jobs when the plugin is
        loaded in NetBox instead of when a user performs an event. It can be
        called from the plugin's `ready()` function to safely setup schedules.


        :param instance: The instance the job is attached to.
        :param name: Name of the job schedule.
        :param interval: Interval in which the job should be scheduled.
        """
        object_type = ObjectType.objects.get_for_model(
            instance,
            for_concrete_model=False,
        )
        job = Job.objects.filter(
            object_type=object_type,
            object_id=instance.pk,
            name=name,
            interval__isnull=(interval is None),
            status__in=cls.ENQUEUED_STATUS,
        ).first()
        if job:
            # If the job parameters haven't changed, don't schedule a new job
            # and reuse the current schedule. Otherwise, delete the existing job
            # and schedule a new job instead.
            if job.interval == interval:
                return
            job.delete()

        cls.enqueue(name=name, interval=interval, *args, **kwargs)


class SystemJob(ScheduledJob):
    """
    A `ScheduledJob` not being bound to any particular NetBox object.

    This class can be used to schedule system background tasks that are not
    specific to a particular NetBox object, but a general task.

    A typical use case for this class is to implement a general synchronization
    of NetBox objects from another system. If the configuration of the other
    system isn't stored in the database, but the NetBox configuration instead,
    there is no object to bind the `Job` object to. This class therefore allows
    unbound jobs to be scheduled for system background tasks.
    """

    @classmethod
    def enqueue(cls, *args, **kwargs) -> None:
        kwargs.pop("instance", None)
        super().enqueue(instance=Job(), *args, **kwargs)

    @classmethod
    def schedule(cls, *args, **kwargs) -> None:
        kwargs.pop("instance", None)
        super().schedule(instance=Job(), *args, **kwargs)

    @classmethod
    def handle(cls, job: Job, *args, **kwargs) -> None:
        # A job requires a related object to be handled, or internal methods
        # will fail. To avoid adding an extra model for this, the existing job
        # object is used as a reference. This is not ideal, but it works for
        # this purpose.
        job.object = job
        job.object_id = None  # Hide changes from UI

        super().handle(job, *args, **kwargs)

Use case

Plugins get a standardized interface for adding models with jobs enabled using the JobsMixin, just like native NetBox models. This provides a consistent experience.
The environment for running background jobs will be standardized, as startup, termination, and error handling will be the same for all jobs. Individual jobs don't have to worry about rescheduling, but can rely on well-tested and managed code.
Using the SystemJob interface, plugins could schedule system tasks such as periodic synchronization with other systems (e.g. virtualization clusters) or perform housekeeping. These jobs are usually not bound to a specific NetBox object and currently require either direct access to the django-rq library or use of an external cronjob and management commands.

Database changes

None

External dependencies

None

For the functionality described above I can share my existing code, add test cases and provide a PR for review. Special thanks goes to @wouterdebruijn for sharing his ideas and feedback in the NetBox discussions.

Originally created by @alehaa on GitHub (Apr 10, 2024). Originally assigned to: @alehaa on GitHub. ### NetBox version v3.7.5 ### Feature type Change to existing functionality ### Proposed functionality NetBox already includes the functionality to schedule jobs using either the `django-rq` library or the `core.Job` model combined with the `JobsMixin` class. Making the abstract job functionality publicly available to plugins allows decoupling from the backend of `django-rq` and a consistent experience in NetBox across all functionality. For this I propose: 1. Add `netbox.models.JobsMixin` to the list of API available model mixins [in the documentation](https://github.com/netbox-community/netbox/blob/b7668fbfc3a81abf2c6a8ace98047647fd9244c9/docs/plugins/development/models.md). This allows plugins to implement new models with Jobs enabled. 2. Add a new `BackgroundJob` class to implement the execution of the job's code, i.e. for consistency in calling `start`, `terminate` and setting error messages. This class should also be used in existing NetBox functionality to run background jobs for consistency. Below is a sample implementation from a plugin of mine for demonstration purposes. <details> ```Python import logging from abc import ABC, abstractmethod from rq.timeouts import JobTimeoutException from core.choices import JobStatusChoices from core.models import ObjectType, Job class BackgroundJob(ABC): """ Background Job helper class. This class handles the execution of a background job. It is responsible for maintaining its state, reporting errors, and scheduling recurring jobs. """ @classmethod @abstractmethod def run(cls, *args, **kwargs) -> None: """ Run the job. A `BackgroundJob` class needs to implement this method to execute all commands of the job. """ pass @classmethod def handle(cls, job: Job, *args, **kwargs) -> None: """ Handle the execution of a `BackgroundJob`. This method is called by the Job Scheduler to handle the execution of all job commands. It will maintain the job's metadata and handle errors. For periodic jobs, a new job is automatically scheduled using its `interval'. :param job: The job to be executed. """ try: job.start() cls.run(job, *args, **kwargs) job.terminate() except Exception as e: job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e)) if type(e) == JobTimeoutException: logging.error(e) # If the executed job is a periodic job, schedule its next execution at # the specified interval. finally: if job.interval: next_schedule_at = (job.scheduled or job.started) + timedelta( minutes=job.interval ) cls.enqueue( instance=job.object, name=job.name, user=job.user, schedule_at=next_schedule_at, interval=job.interval, **kwargs, ) @classmethod def enqueue(cls, *args, **kwargs) -> None: """ Enqueue a new `BackgroundJob`. This method is a wrapper of `Job.enqueue` using :py:meth:`handle` as function callback. See its documentation for parameters. """ Job.enqueue(cls.handle, *args, **kwargs) ``` </details> 3. Optional: Enhance the `core.models.Job.enqueue()` method with a `queue` parameter to schedule jobs for specific queues (e.g. `low`, `high`). If not provided the default queue will be used, so there's no change in API compatibility. 4. Optional: Add the ability to schedule system background tasks by plugins, e.g. for periodic synchronization with other systems. Below is a sample implementation from a plugin of mine for demonstration purposes. <details> ```Python class ScheduledJob(BackgroundJob): """ A periodic `BackgroundJob` used for system tasks. This class can be used to schedule system background tasks, e.g. to periodically perform data synchronization from other systems to NetBox or to perform housekeeping tasks. """ ENQUEUED_STATUS = [ JobStatusChoices.STATUS_PENDING, JobStatusChoices.STATUS_SCHEDULED, JobStatusChoices.STATUS_RUNNING, ] @classmethod def schedule( cls, instance: models.Model, name: str = "", interval: int = None, *args, **kwargs, ) -> None: """ Schedule a `BackgroundJob`. This method adds a new `BackgroundJob` to the job queue. If the job schedule identified by its `instance` and `name` is already active, scheduling a second will be skipped. For additional parameters see :py:meth:`Job.enqueue`. The main use case for this method is to schedule jobs programmatically instead of using user events, e.g. to start jobs when the plugin is loaded in NetBox instead of when a user performs an event. It can be called from the plugin's `ready()` function to safely setup schedules. :param instance: The instance the job is attached to. :param name: Name of the job schedule. :param interval: Interval in which the job should be scheduled. """ object_type = ObjectType.objects.get_for_model( instance, for_concrete_model=False, ) job = Job.objects.filter( object_type=object_type, object_id=instance.pk, name=name, interval__isnull=(interval is None), status__in=cls.ENQUEUED_STATUS, ).first() if job: # If the job parameters haven't changed, don't schedule a new job # and reuse the current schedule. Otherwise, delete the existing job # and schedule a new job instead. if job.interval == interval: return job.delete() cls.enqueue(name=name, interval=interval, *args, **kwargs) class SystemJob(ScheduledJob): """ A `ScheduledJob` not being bound to any particular NetBox object. This class can be used to schedule system background tasks that are not specific to a particular NetBox object, but a general task. A typical use case for this class is to implement a general synchronization of NetBox objects from another system. If the configuration of the other system isn't stored in the database, but the NetBox configuration instead, there is no object to bind the `Job` object to. This class therefore allows unbound jobs to be scheduled for system background tasks. """ @classmethod def enqueue(cls, *args, **kwargs) -> None: kwargs.pop("instance", None) super().enqueue(instance=Job(), *args, **kwargs) @classmethod def schedule(cls, *args, **kwargs) -> None: kwargs.pop("instance", None) super().schedule(instance=Job(), *args, **kwargs) @classmethod def handle(cls, job: Job, *args, **kwargs) -> None: # A job requires a related object to be handled, or internal methods # will fail. To avoid adding an extra model for this, the existing job # object is used as a reference. This is not ideal, but it works for # this purpose. job.object = job job.object_id = None # Hide changes from UI super().handle(job, *args, **kwargs) ``` </details> ### Use case 1. Plugins get a standardized interface for adding models with jobs enabled using the `JobsMixin`, just like native NetBox models. This provides a consistent experience. 2. The environment for running background jobs will be standardized, as startup, termination, and error handling will be the same for all jobs. Individual jobs don't have to worry about rescheduling, but can rely on well-tested and managed code. 3. Using the `SystemJob` interface, plugins could schedule system tasks such as periodic synchronization with other systems (e.g. virtualization clusters) or perform housekeeping. These jobs are usually not bound to a specific NetBox object and currently require either direct access to the `django-rq` library or use of an external cronjob and management commands. ### Database changes None ### External dependencies None For the functionality described above I can share my existing code, add test cases and provide a PR for review. Special thanks goes to @wouterdebruijn for sharing his ideas and feedback [in the NetBox discussions](https://github.com/netbox-community/netbox/discussions/14040#discussioncomment-8913862).

adam added the status: accepted type: feature topic: plugins complexity: medium labels 2025-12-29 20:50:16 +01:00

adam closed this issue

2025-12-29 20:50:16 +01:00

adam commented

2025-12-29 20:50:17 +01:00

@jonasnieberle commented on GitHub (May 21, 2024):

I think it would be very useful if netbox support easier usage of the jobs for plugins.
Im actually trying to implement a job for a plugin, it's very tricky to find how it works.

A lot of plugins implement a custom solution with the big disadvantage that the jobs not listed in the Jobs Tab.

@jonasnieberle commented on GitHub (May 21, 2024): I think it would be very useful if netbox support easier usage of the jobs for plugins. Im actually trying to implement a job for a plugin, it's very tricky to find how it works. A lot of plugins implement a custom solution with the big disadvantage that the jobs not listed in the Jobs Tab.

adam commented

2025-12-29 20:50:17 +01:00

@jeremystretch commented on GitHub (Jun 5, 2024):

@alehaa thanks for the detailed FR and example code! We've selected this as a milestone initiative for NetBox v4.1. I was set to work on it myself, but would you like to contribute a PR for it?

@jeremystretch commented on GitHub (Jun 5, 2024): @alehaa thanks for the detailed FR and example code! We've selected this as a milestone initiative for NetBox v4.1. I was set to work on it myself, but would you like to contribute a PR for it?

adam commented

2025-12-29 20:50:17 +01:00

@alehaa commented on GitHub (Jun 10, 2024):

@jeremystretch sure I can work on it.

@alehaa commented on GitHub (Jun 10, 2024): @jeremystretch sure I can work on it.

adam commented

2025-12-29 20:50:17 +01:00

@jeremystretch commented on GitHub (Jun 10, 2024):

@alehaa great! Please be sure to base your PR off of the feature branch as this will be going into NetBox v4.1.

@jeremystretch commented on GitHub (Jun 10, 2024): @alehaa great! Please be sure to base your PR off of the `feature` branch as this will be going into NetBox v4.1.

adam commented

2025-12-29 20:50:17 +01:00

@jsenecal commented on GitHub (Jun 10, 2024):

@alehaa I have done similar things in my plugins to achieve similar results. If you want we can collaborate on this feature in your fork. Just let me know. I'm also on slack with the same username if you want to reach out.

Thanks for your time :)

@jsenecal commented on GitHub (Jun 10, 2024): @alehaa I have done similar things in my plugins to achieve similar results. If you want we can collaborate on this feature in your fork. Just let me know. I'm also on slack with the same username if you want to reach out. Thanks for your time :)

adam commented

2025-12-29 20:50:18 +01:00

@jeremystretch commented on GitHub (Jun 17, 2024):

@alehaa can you share what progress you've made on this?

@jeremystretch commented on GitHub (Jun 17, 2024): @alehaa can you share what progress you've made on this?

adam commented

2025-12-29 20:50:18 +01:00

@alehaa commented on GitHub (Jun 17, 2024):

@jeremystretch Unfortunately I didn't start yet but plan on submitting a PR this week.

@alehaa commented on GitHub (Jun 17, 2024): @jeremystretch Unfortunately I didn't start yet but plan on submitting a PR this week.

adam commented

2025-12-29 20:50:18 +01:00

@alehaa commented on GitHub (Jun 23, 2024):

Despite the missing documentation pages, the BackgroundJob implementation is ready for merge. The jobs for synchronizing data sources and running scripts (background and interactive) have also been migrated to this new feature.

However, for adding SystemJob I encountered a problem: While my implementation documents to use the plugin's ready function, Django discourages this behavior because the database may not be ready. @jsenecal have you encountered a similar problem and can share some ideas to solve this? Otherwise I could share my current efforts in a PR and submit a second one for background system jobs at a later time @jeremystretch?

@alehaa commented on GitHub (Jun 23, 2024): Despite the missing documentation pages, the `BackgroundJob` implementation is ready for merge. The jobs for synchronizing data sources and running scripts (background and interactive) have also been migrated to this new feature. However, for adding `SystemJob` I encountered a problem: While my implementation documents to use the plugin's `ready` function, Django discourages this behavior because the database may not be ready. @jsenecal have you encountered a similar problem and can share some ideas to solve this? Otherwise I could share my current efforts in a PR and submit a second one for background system jobs at a later time @jeremystretch?

adam commented

2025-12-29 20:50:18 +01:00

@jeremystretch commented on GitHub (Jun 24, 2024):

@alehaa please submit what you have for now as a draft PR and I'll take a look. Thanks!

@jeremystretch commented on GitHub (Jun 24, 2024): @alehaa please submit what you have for now as a draft PR and I'll take a look. Thanks!

adam commented

2025-12-29 20:50:19 +01:00

@alehaa commented on GitHub (Jun 24, 2024):

@jeremystretch I've submitted a new draft PR related to this issue.

Additional ScheduledJob and SystemJob implementations can be found in the original issue body under item 4. However, I haven't committed these because I haven't found a way to use them in practice yet. I didn't want to commit something that could not be used.

@alehaa commented on GitHub (Jun 24, 2024): @jeremystretch I've submitted a new draft PR related to this issue. Additional `ScheduledJob` and `SystemJob` implementations can be found in the original issue body under item 4. However, I haven't committed these because I haven't found a way to use them in practice yet. I didn't want to commit something that could not be used.

adam commented

2025-12-29 20:50:19 +01:00

@alehaa commented on GitHub (Jun 28, 2024):

After giving it some thought, I see no built-in option for this in Django and some custom logic will need to be used to solve this. I could imagine sending a signal just before the WSGI application accepts requests to set up background jobs. ScheduledJob and SystemJob will then need to register for this signal to be scheduled when needed. In addition, a management command can be used to manually schedule all registered background jobs interactively. This way, the registration is done only once, instead of every time a ready function is executed.

Would this be acceptable and in line with the general NetBox codebase?

@alehaa commented on GitHub (Jun 28, 2024): After giving it some thought, I see no built-in option for this in Django and some custom logic will need to be used to solve this. I could imagine sending a signal just before the WSGI application accepts requests to set up background jobs. `ScheduledJob` and `SystemJob` will then need to register for this signal to be scheduled when needed. In addition, a management command can be used to manually schedule all registered background jobs interactively. This way, the registration is done only once, instead of every time a `ready` function is executed. Would this be acceptable and in line with the general NetBox codebase?

adam commented

2025-12-29 20:50:19 +01:00

@alehaa commented on GitHub (Jun 30, 2024):

I've found a solution that works without introducing new methods or custom signals by reusing Django's connection_created signal. This allows the database to be accessed on application startup, but without any interaction. This signal will be issued after all plugins have been loaded so no warning is issued in the logs. I'll continue working on my PR and submit results ASAP.

@alehaa commented on GitHub (Jun 30, 2024): I've found a solution that works without introducing new methods or custom signals by reusing Django's `connection_created` signal. This allows the database to be accessed on application startup, but without any interaction. This signal will be issued after all plugins have been loaded so no warning is issued in the logs. I'll continue working on my PR and submit results ASAP.

Sign in to join this conversation.

Branches Tags

main

update-changelog-comments-docs

feature-removal-issue-type

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#9468