[JiandiAnNVIDIA]

Problem

Container churn on NVIDIA Vera is roughly 5x slower with udev running than with it
stopped. On a 352-CPU NVL72 node, throughput is capped at 3,100-3,850 container
cycles/h from 8 concurrent containers upward and does not improve with more
load; stopping systemd-udevd raises it to 14,000-19,500. Across the loaded
concurrency levels the penalty is +276% to +514%. Kernel CPU sits at 19.9-26.0%
while user CPU is ~1%, so the machine is mostly idle while delivering a fifth
of its achievable throughput.

Root cause

Docker's default bridge networking creates a veth pair per container, and its
netlink library sets neither IFLA_NUM_TX_QUEUES nor IFLA_NUM_RX_QUEUES. In
that case veth allocates one rx and one tx queue per possible CPU and then
reduces the real count to one — but veth_newlink() does the reduction after
register_netdevice():

	err = register_netdevice(dev);
	...
	err = veth_init_queues(dev, tb);

register_queue_kobjects() publishes sysfs objects for real_num_rx_queues and
real_num_tx_queues, so every queue it just created is destroyed again a moment
later, each creation and destruction emitting a uevent. On this node that is
2816 uevents on the queues subsystem per veth pair — 704 adds plus 702
immediate removals per device, times two devices, plus 2 removals at delete.
udev's 720 workers then contend on the per-kernfs-root semaphores while
processing them, which throttles all subsequent sysfs activity on the host.

The cost scales with num_possible_cpus(), which is why this node is hit harder
than the 176-CPU node in the original report.

Fix

Set the real queue counts before registering each device. For a device that
is not yet registered, netif_set_real_num_tx_queues() and
netif_set_real_num_rx_queues() only assign the counter — they skip all kobject
work and the expedited RCU grace period the tx path otherwise takes — so the
extra queues never appear in sysfs at all, and the error label that unwound a
registration is no longer needed.

Two commits:

Commit	Upstream status
veth: decide peer queue count from the peer's own attributes	merged, 267bc3f113d8c441f57014c502443686f5d33d57, in v7.4-rc1
NVIDIA: SAUCE: veth: set the real number of queues before registering the devices	posted to netdev 2026-10-16, reviewed and tested, repost after the merge window requested — carried as SAUCE until it lands

Both are context-adjusted for this tree, which predates the netdev_hold() peer
tracker in veth_newlink() and therefore has one error label where upstream has
two.

Verification

Measured on a Vera NVL72 node: 2 sockets x 88 cores x 2 threads = 352 logical
CPUs, 64k pages, Ubuntu 26.04, udev 259.5-0ubuntu3.4 with the default 720
workers, Docker 29.7.2 on the default bridge.

Uevents per veth pair

	before	after
queues adds	1408	4
queues removes	1408	4

A 352x reduction, exactly num_possible_cpus(). No removals occur during
creation at all — the storm does not happen rather than being made cheaper.

200 veth pairs created and deleted, no Docker

phase	before	after
create	5.44 s	0.64 s
create drain (udevadm settle)	107.01 s	0.01 s
delete	4.43 s	4.37 s
total	116.89 s	5.03 s

Deletion is unaffected, as upstream's own numbers also show; this fixes
creation.

Container churn, cycles/h, two runs per kernel

containers	stock on	stock off	patched on	patched off
8	3,309–3,839	14,360–14,449	15,381–15,612	15,373–15,507
16	3,304–3,560	17,882–18,621	18,471–18,670	19,713–20,310
32	3,185–3,553	19,033–19,550	12,863–15,099	14,739–15,025
64	3,099–3,578	17,827–18,663	14,077–14,521	14,541–14,803
128	3,558–3,747	17,088–18,026	12,947–13,379	13,776–13,874

The udev penalty goes from +276%..+514% to −9%..+6%, straddling zero in both
patched runs. Kernel CPU with udev running drops from 19.9–26.0% to 0.6–7.9%.
Mean container cycle time at 32 concurrent containers goes from 30.75–33.01 s
to 7.52–7.81 s.

Profile — udev workers account for 0.96% of cycles on the patched kernel,
and rwsem_read_trylock plus up_read together total ~0.45% across all
processes.

No functional change — ethtool -l on a freshly created veth still reports
pre-set maximums of 352 RX and 352 TX with current settings of 1, so
num_tx_queues remains nr_possible_cpus() and ethtool -L headroom is
preserved. Container networking verified working.

LP: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-bos/+bug/2173592
NvBug: https://nvbugspro.nvidia.com/bug/6962526

[nirmoy]

BaseOS Kernel Review

Tip

✅ Review passed

No issues found across the reviewed commits.

Findings: none

🔍 Review artifacts

• 🟢 Boro review: no findings
• 🔵 Test plan: ready for validation
• 🟡 KTS Report: unavailable

📦 Build checks — 🟢 4/4 passed

• 🟢 amd64: build report and 4 debs
• 🟢 arm64: build report and 8 debs
• 🟢 amd64: NVIDIA driver and DOCA build report
• 🟢 arm64: NVIDIA driver and DOCA build report

Install the PR kernel with APT

Published for reviewed head 30af1de94314: noble/amd64. Requires the NVIDIA network.

curl -fsS https://baseos-reviewer-66d7b9.gitlab-master-pages.nvidia.com/install.sh | bash -s -- install pr-697

Manual setup · signing fingerprint DF52E10C47F28AC221DA19ED6D25A5DFC6686DFC

Manual commands for Ubuntu 24.04 Noble / amd64, on the NVIDIA network:

(
set -euo pipefail
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
curl -qfsS --proto '=https' -o "$work/archive-keyring.gpg" https://baseos-reviewer-66d7b9.gitlab-master-pages.nvidia.com/archive-keyring.gpg
curl -qfsS --proto '=https' -o "$work/baseos-internal-tools.crt" https://baseos-reviewer-66d7b9.gitlab-master-pages.nvidia.com/baseos-internal-tools.crt
sudo install -d -m 0755 /etc/apt/keyrings
sudo install -m 0644 "$work/archive-keyring.gpg" /usr/share/keyrings/baseos-pr-kernels.gpg
sudo install -m 0644 "$work/baseos-internal-tools.crt" /etc/apt/keyrings/baseos-pr-https.crt
printf '%s\n' \
  'Acquire::https::baseos-internal-tools.nvidia.com::CaInfo "/etc/apt/keyrings/baseos-pr-https.crt";' \
  'Acquire::https::baseos-internal-tools.nvidia.com::Verify-Peer "true";' \
  'Acquire::https::baseos-internal-tools.nvidia.com::Verify-Host "true";' | \
  sudo tee /etc/apt/apt.conf.d/90baseos-pr-kernels >/dev/null
printf '%s\n' 'deb [arch=amd64 signed-by=/usr/share/keyrings/baseos-pr-kernels.gpg] https://baseos-internal-tools.nvidia.com:4001/reviews/apt/NVIDIA/NV-Kernels/697/noble/amd64 noble main' | \
  sudo tee /etc/apt/sources.list.d/nv-kernels-pr697.list >/dev/null
sudo apt update --error-on=any
sudo apt install --no-remove nv-kernels-pr-697
)

The repository remains configured. Later apt update and apt install nv-kernels-pr-697 pick up newly published builds; APT verifies signed metadata with the shown Signed-By key.

Note

Build reports and debs are retained for 10 days after the PR closes.

• 🟢 PR Illustrated: ready

Review metadata

• Reviewed head: 30af1de94314
• Overall status: kernel validation inconclusive
• Build checks: 4/4 passed

This comment is maintained by BaseOS Reviewer and updated when the GitHub watcher publishes a newer review.

[github-actions]

PR Validation Report

Patchscan ✅ No Missing Fixes

All cherry-picked commits checked — no missing upstream fixes found.

PR Lint ❌ Errors found

Details

Checking 2 commits...

Cherry-pick digest:
E: 67f8106eba0a ("veth: decide peer queue count from the p"): patch-ID mismatch with upstream 267bc3f113d8
┌──────────────┬──────────────────────────────────────────────────────────────────┬────────────┬─────────┬───────────────────────────┐
│ Local        │ Referenced upstream / Patch subject                              │ Patch-ID   │ Subject │ SoB chain                 │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 30af1de94314 │ veth: set the real number of queues before registering the devic │ noted      │ found   │ ok, backporter: jan       │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 67f8106eba0a │ 267bc3f113d8 veth: decide peer queue count from the peer's own a │ MISMATCH   │ match   │ preserved + jan added     │
└──────────────┴──────────────────────────────────────────────────────────────────┴────────────┴─────────┴───────────────────────────┘

Lint: all checks passed.

[nirmoy]

Acked-by: Nirmoy Das <nirmoyd@nvidia.com>

[jamieNguyenNVIDIA]

Acked-by: Jamie Nguyen <jamien@nvidia.com>

[clsotog]

Acked-by: Carol L Soto <csoto@nvidia.com>

[nvmochs]

Acked-by: Matthew R. Ochs <mochs@nvidia.com>