Skip to content

[BF] Improve large IRRDB imports and prefix listings - #1187

Open
repstein2 wants to merge 3 commits into
inex:mainfrom
repstein2:fix/irrdb-large-import-performance
Open

repstein2 wants to merge 3 commits into
inex:mainfrom
repstein2:fix/irrdb-large-import-performance

Conversation

@repstein2

@repstein2 repstein2 commented Nov 7, 2026 •

Copy link
Copy Markdown

Large manual IRRDB refreshes can exceed HTTP request limits. Repeated requests can start competing imports and cause database lock timeouts. Prefix listings also load and render the entire result set.

This patch adds:

  • An optional dedicated background queue without changing the global queue default.
  • Duplicate-request protection and serialized member/type imports.
  • Batched database inserts and deletes.
  • Database pagination with 100 entries per page.
  • Bounded change previews with exact totals.
  • Updates limited to the requested address family.
  • Failure reporting with transaction rollback.

Worker setup is documented in IRRDB-QUEUE.md.

Validation:

  • PHP 8.4 syntax checks and isolated regression checks passed, including batching, duplicate dispatch, rollback, requested-family handling, and pagination/template rendering.
  • A 10,000-prefix fixture used 10 insert statements and completed in approximately 0.16 seconds.
  • Deployed at FL-IX, CIX-ATL, and CIX-NoVA, with successful live refreshes in all three markets.
  • A large IPv4 listing containing 862,942 prefixes now returns 100 rows per page.
  • No BGP reloads were performed during validation.

Outstanding before acceptance:

Ready for maintainer review and feedback on queue configuration, locking, and pagination/search behavior. Existing authorization checks are retained. The existing GET update endpoint is unchanged.

@repstein2
repstein2 marked this pull request as draft November 7, 2026 21:51
@repstein2 repstein2 closed this Nov 7, 2026
@repstein2 repstein2 changed the title [BF] Improve large IRRDB imports and prefix listings.Bound IRRDB listing and import costs; support dedicated background re… [BF] Improve large IRRDB imports and prefix listings Nov 7, 2026
@repstein2 repstein2 reopened this Nov 7, 2026
@repstein2
repstein2 marked this pull request as ready for review November 7, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant