Add a plugin API for allocating primary keys #23426

NetBox version

v4.7.3

Feature type

New functionality

Proposed functionality

Add a hook that lets a plugin supply the primary key for a new object before it is inserted. When no allocator is registered, or every allocator declines, nothing changes: PostgreSQL assigns the ID from the table's sequence as it does today.

1. An allocator-aware primary key field. Add NetBoxAutoField, a subclass of BigAutoField, to utilities/fields.py, and point DEFAULT_AUTO_FIELD at it.

class NetBoxAutoField(models.BigAutoField):
    """
    A BigAutoField whose value may be supplied by a registered primary key allocator.
    """
    def get_pk_value_on_save(self, instance):
        if pks := allocate_pks(instance._meta.model):
            return pks[0]
        return None

    def has_default(self):
        # Insert new objects directly, rather than attempting an UPDATE first
        return True

    def get_default(self):
        return None

    def deconstruct(self):
        # Present as a plain BigAutoField so that no migrations are generated
        name, path, args, kwargs = super().deconstruct()
        return name, 'django.db.models.BigAutoField', args, kwargs

• Django calls get_pk_value_on_save() from both Model.save() and QuerySet.bulk_create() whenever an object has no primary key. This makes it the single point where an allocator can step in.
• has_default() returns True because, when a primary key is set before save() on a field with no default, Django runs an UPDATE before the INSERT. A default makes Django insert new objects directly, as it already does for UUID keys with a default.
• get_default() returns None, so new instances still start with no primary key.

2. A registration API. Add two functions to netbox/utils.py, alongside register_request_processor():

def register_pk_allocator(func):
    """
    Register a callable which may supply primary keys for new objects.

    Signature: func(model, count) -> list[int] | None
    """
    registry['pk_allocators'].append(func)
    return func


def allocate_pks(model, count=1):
    """
    Return `count` primary keys for `model` from the first allocator which supplies them,
    or None if the database should assign them.
    """
    for allocator in registry['pk_allocators']:
        if (pks := allocator(model, count)) is not None:
            return pks
    return None

• Allocators are consulted in registration order. Returning None defers to the next allocator, and finally to the database.
• allocate_pks() is public, so code that inserts rows with raw SQL or COPY can reserve IDs the same way the ORM does.
• The hook is not told which database the insert targets, because Django doesn't pass it to get_pk_value_on_save(). An allocator decides from its own context; netbox-branching, for instance, uses its active-branch context.
• Django calls the hook once per object, including in bulk_create(). An allocator that talks to a remote sequence can fetch blocks of IDs and hand them out from a local cache.

3. Wiring.

• Switch models that declare their own BigAutoField (currently only Tag.id) to NetBoxAutoField.
• Register NetBoxAutoField in strawberry-django's field type maps. It looks up model fields by exact class, so without this the GraphQL schema fails to build.

4. Documentation. Add a section on primary key allocators to the plugin development docs. Plugin models pick up the hook through DEFAULT_AUTO_FIELD. Plugins that set default_auto_field themselves, or declare their own primary key, should use NetBoxAutoField so their models work with allocators. Code that inserts rows outside the ORM should call allocate_pks().

Verified in a prototype against v4.7.3:

• makemigrations --check reports no changes, and system checks pass.
• The OpenAPI schema is identical to today's.
• Without an allocator, save() issues a single INSERT and the database assigns the ID.
• With an allocator, save() issues a single INSERT using the allocated ID, and bulk_create() inserts all objects with their allocated IDs in one statement.

One behavior change. Because the primary key now reports a default, saving a new, unsaved instance whose primary key is set to an existing row's (Tag(pk=5, ...).save()) attempts an INSERT and fails on the duplicate key, instead of updating row 5. This matches Django's behavior for any primary key with a default. Raw saves are unaffected; that covers fixtures, deserialized objects, and netbox-branching's merge, which applies creates through deserialization. Fetching the object first, or using save(force_update=True), still updates as before.

Open questions

• Should has_default() return True unconditionally, or only while an allocator is registered? The latter avoids the behavior change on installs without one, but makes behavior depend on which plugins are installed.
• Should bulk_create() request a whole batch of IDs in one call, rather than one per object? That would mean overriding bulk_create() on NetBox's querysets, which doesn't cover auto-created many-to-many through models. Caching blocks inside the allocator avoids that.

Use case

The primary use case here is the netbox-branching plugin, which replays each change's object ID verbatim when it merges a branch into main. An object created in a branch must therefore never get an ID that main or another branch also hands out.

The plugin's schema-based backend guarantees this by pointing each branch table's ID default at main's sequence. However, a hypothetical future backend that provisions each branch as a separate PostgreSQL database (such as a copy-on-write clone) can't do that. A database can't call another database's sequences, so every clone starts with its own copy of each sequence at the same position as main's.

With this hook in place, netbox-branching can register an allocator that draws IDs from main's sequences for any insert made in a branch. That is the schema backend's behavior, carried across databases.

• IDs stay exactly as they are today: small, contiguous and in creation order.
• There is no limit on the number of branches, and no change to the schema, the REST API or GraphQL.
• The cost is a round trip to main for each allocation. Measured on PostgreSQL 16, that adds about 170 µs per row one at a time, or about 31 µs per row when IDs are fetched in blocks of 100. That's small next to a typical API write.

A minimal example allocator for this case:

@register_pk_allocator
def allocate_from_main(model, count):
    if not (active_branch.get() and supports_branching(model)):
        return None
    with connections[DEFAULT_DB_ALIAS].cursor() as cursor:
        cursor.execute(
            "SELECT nextval(pg_get_serial_sequence(%s, 'id')) FROM generate_series(1, %s)",
            [model._meta.db_table, count]
        )
        return [row[0] for row in cursor.fetchall()]

Database changes

None. Column types, sequences and identity columns are unchanged. Because NetBoxAutoField presents itself to migrations as BigAutoField, switching DEFAULT_AUTO_FIELD generates no migrations; this was confirmed with makemigrations --check.

External dependencies

N/A