Repository navigation
docs: operate sipnab: service, install, listener security, troubleshooting - #638
Merged
Merged
Conversation
POST /v1/tfps/ban against a TFPS whose tfps_ctl wrote `expires` as an RFC 3339 string: TFPS applied the ban, sipnab answered 502 "TFPS could not be asked: ... cannot read", and journaled the outcome as `failed`. Every later unban answered 409 "sipnab holds no ban on this address", admin token included. sipnab had lost track of a ban it caused, and no surface could lift it. Root cause: once tfps_ctl has run, TFPS may have acted whatever becomes of its answer, but action_reply() turned every TfpsError into the same Err, and finish() journaled every Err as `failed`. The ledger reads `failed` as "did not happen": it drops the intent from in-doubt and takes no effect. So a timeout, a non-zero exit with no result, too much output, or an answer outside the JSON contract all recorded an unknown effect as no effect. The parse error only exposed it. Only a tfps_ctl that never started (TfpsError::Spawn) is known to have done nothing. Fix: - TfpsError::ran(): every error but Spawn. action_reply() maps an error from a peer that ran to TfpsReply::Unanswered. - outcome_of(), one rule for action and revert records: Unanswered is journaled as `unknown`; `failed` is kept for a peer that never ran. - The ledger keeps an action whose outcome is `unknown` in doubt. It carries its address, caller and time, and `sipnab --journal-show` lists it. - unban(): an action in doubt on the address being unbanned no longer refuses the unban (one on any other address still does). The caller of a ban in doubt may lift it, as may an admin token; anyone else gets 403 naming the owner. TFPS's answer settles the doubt with `reconciled` records: an applied unban means the address was banned; not-blocked means it was not; any other answer leaves the doubt in place. - ActionError::Unanswered: still 502 (REST) and refusal `tfps` (MCP), now worded "TFPS was asked, and sipnab could not read what it did", and saying the action is held in doubt and how to settle it. InDoubt's wording covers both causes. - An unparseable answer names the contract sipnab reads: sippulse/tfps e7ee57a (2026-09-16) and later, where every time is epoch seconds; the draft of that contract (before 54dc12d on json-output-contract-history) wrote `expires` as RFC 3339. sipnab does not parse RFC 3339. Other mutating actions: TFPS ban and unban are the only action target (ActionTarget::Tfps); revert sends an unban and now records a lost answer as `unknown` too. The kill-scanner packet expects no answer. Tests (written first, seen failing for the right reason): - tests/actions_tfps_ctl_test.rs: a fake tfps_ctl that bans and then answers garbage, a pre-contract RFC 3339 answer, a non-zero exit, or hangs past the timeout gives Unanswered; one that cannot start is still an error. - tests/actions_service_test.rs: the lost ban is in doubt with address, caller and time; the caller and an admin can lift it and TFPS settles it; another caller gets NotYours and TFPS is not asked; not-blocked settles it as never applied; a lost unban answer keeps the doubt; other actions wait; a restart resolves it against TFPS's list; a revert whose answer is lost is held in doubt too. - sipnab --journal-show and the InDoubt refusal now name both causes of doubt and the unban that settles one. - tests/actions_journal_rest_test.rs: over REST with the real binary, for garbage, a non-zero exit and a timeout, each lifted by its caller and by an admin: 502 saying in doubt, outcome `unknown`, the unban reaches TFPS, and a new ban runs afterwards. - tests/support/server.rs gains post_json_bearer_waiting: the hang case answers only after sipnab's own 10 s tfps_ctl timeout, which the shared client's 10 s read timeout could not outlast (it failed with EAGAIN under load). The ban step waits 30 s; under the ledger mutation the hang cases now fail on the 409 the report describes. - tfps_failing_is_a_bad_gateway_and_the_outcome_is_journaled_as_failed asserted the defect itself (a tfps_ctl that ran and exited 1 with no result journaled as `failed`); it now asserts `unknown` and is renamed. Found against a real TFPS during the 2026-11-03 documentation review. Ratchet (tests/rust_style_known.tsv), per file: - AssertWithoutMessage tests/actions_journal_rest_test.rs 15 -> 14: the corrected `unknown` assertion carries a message; the `failed` one it replaces did not. (cherry picked from commit d032017)
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
NormB
force-pushed
the
docs/howto-ops-service
branch
from
November 4, 2026 05:32
450e94b to
83ab49a
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
NormB
force-pushed
the
docs/howto-ops-service
branch
from
November 4, 2026 06:48
83ab49a to
5e0f784
Compare
…eet inventory
Fleet management phase P0, slice 1 of docs/design/fleet-management.local.md:
read-only inventory, pulled from a static hosts file, behind a new
least-privilege token scope.
What it adds:
- GET /v1/inventory and the MCP tool fleet_inventory, built by one
function (output::inventory::current): build, SHA-256 of the running
binary, signature state (always unverified), configuration digests,
privilege, uptime, interface, capture and HEP counters, HEP senders.
- Configuration digests only: file_sha256 of the bytes read,
parsed_sha256 of the parsed configuration with sorted keys,
argv_sha256 of the command line, and effective_sha256 over the last
two. No value appears, and a secret given as a value changes
effective_sha256.
- Token scope inventory: reaches GET /v1/inventory and nothing else.
full, actions and admin reach it too. Refused at mint time for MCP.
- sipnab fleet inventory --hosts FILE [--json] [--timeout SECS]: asks
every listed instance at once, marks unreachable, refused and
incompatible instances, reports version_mismatch and
unlisted_hep_sender, exits 1 when any instance did not answer or there
is a finding.
The how-to is docs/fleet.md ("Manage many sipnab instances"): purpose,
prerequisites (API TLS, a token per instance, the hosts file, network
reachability, no change on other systems), steps, verification and
troubleshooting (unreachable, untrusted CA, 401, 404 from an older build,
a token file other users can read, a mixed HEP HMAC version,
unlisted_hep_sender, a newer schema_version). Its commands and output come
from a run on 2026-11-03: four instances in containers on 192.0.2.0/24,
three on this build over TLS and one on release 0.5.130, with the
controller on the host. The newer schema_version output came from a test
server, since no build answers schema_version 2. It is linked from
docs/README.md, the "Manage the fleet" section of docs/mcp-estate.md, and
the reference rows in docs/rest-api.md, docs/cli-reference.md,
docs/auth.md, docs/mcp-tools-server.md and docs/errors.md.
Changes from the earlier draft of this slice:
- effective_sha256 is new. The design names an effective configuration
digest; the draft compared parsed_sha256 only, so two instances with
one file and different flags showed one CONFIG. The table's CONFIG
column now shows effective_sha256.
- fleet_inventory joins SOURCE_INDEPENDENT_TOOLS. The MCP dispatch
stamped source_exhausted and source_stopped_early onto its answer, so
MCP and REST returned different documents. The draft's parity test
called the tool method directly and never passed through the dispatch;
the_mcp_dispatch_answers_the_rest_document drives a real sipnab --mcp
and failed on exactly those two members before the change.
- The tool description is shorter: the server bundle measured 6,347
bytes with the draft's description and 5,976 with this one; the output
schema already lists every field.
- LoadedConfig's new field was joined onto the line of the field before
it; split.
Tests (each seen failing for its fault):
- effective digest, REST/MCP dispatch parity, secret rotation and
build-versus---version checks were written before their code and
failed for the missing behavior first.
- 24 guarded mutations, one per fault in the design's test table and the
extra faults above, each turned at least one test red; each restore
was proved with cmp -s.
- tests/fleet_inventory_test.rs spawns two sipnab --api instances with
different node names and config files; sipnab fleet inventory lists
both with different effective digests, marks a third address
unreachable, still lists the other two and exits 1.
Ratchets moved, attributed per file:
- tests/docs_drift_test.rs registered MCP tools 76 -> 77:
fleet_inventory (src/mcp/tools/inventory.rs).
- tests/docs_drift_test.rs EXPECTED_MARKDOWN_FILES 322 -> 324:
docs/fleet.md and website/content/docs/fleet.md.
- tests/docs_drift_test.rs EXPECTED_TABLES 1311 -> 1318: docs/fleet.md
+3, website/content/docs/fleet.md +3 and
docs/design/surface-capability-matrix.md +1 (the "Fleet inventory"
row); no other changed page gained or lost a table against origin/main.
- src/mcp/server.rs byte budget server 5,500 -> 6,000 (measured 5,976)
and full 90,000 -> 90,500 (measured 90,456): fleet_inventory.
- tests/link_integrity_test.rs EXPECTED_WIKI_LINKS 2079 -> 2097:
docs/fleet.md +4; docs/README.md and docs/auth.md +1 each;
docs/cli-reference.md, docs/errors.md, docs/mcp-estate.md,
docs/mcp-tools-server.md, docs/mcp-tools.md and docs/rest-api.md +2 each.
Counted per page against origin/main.
- tests/link_integrity_test.rs EXPECTED_DOCS_PAGES 100 -> 101:
docs/fleet.md.
- tests/error_reference_test.rs EXPECTED_TEMPLATES 542 -> 546: four
messages in src/app/fleet.rs, each with a docs/errors.md entry.
- tests/mcp_stdio_test.rs tool count base 69 -> 70: fleet_inventory.
Counts in prose moved for the 77th tool, 64 of them read-only: README.md,
docs/mcp.md, docs/mcp-protocol.md, docs/mcp-tools.md,
docs/design/mcp-write-back.md and website/templates/index.html.
docs/fleet.md is registered as a site page (weight 66), a wiki page, a
DOCS_TO_SITE entry and a docs nav entry. sipnab fleet's own flags are
registered for the two documented-flag gates the way sipnab g729's are,
its documented commands are never run by doc_commands_run_test (it
contacts other hosts; tests/fleet_inventory_test.rs runs it), its hosts
file is foreign to the sipnab-config sample gate and is parsed by its own
parser in the_documented_hosts_files_parse, /v1/inventory is exempt from
capture_identity with its MCP counterpart checked over the wire,
fleet_inventory has an output-schema probe and a row in
docs/design/surface-capability-matrix.md, and doc_jq_fields_test checks a
sipnab fleet recipe against a report built by the command's own merge over
a real inventory record. Two new bounds are documented:
MAX_HEP_PEERS (rest-api.md) and MAX_HEAD_BYTES (fleet.md).
docs/design/implementation-plan-phases-8-10.md's citation of get_stats is
re-pointed by scripts/check-line-drift.py. man/sipnab.1 and
docs/design/surface-capability-inventory.md are regenerated.
docs/design/testing-matrix.md, website/static/openapi.json and the site
mirror are regenerated by their scripts.
Not covered: a secret held in a file (a key, token or password file
named by path) is digested by its path only, so rotating its contents
changes no digest. docs/fleet.md says so.
(cherry picked from commit 4a59fe6)
ones. Each one reads a function the others share.
- Each correlated leg carries its own th_callid. It is filled from
DialogStore::th_callid_status in two places that every surface reads:
output::query::correlated for --correlate, /correlated and
find_correlated, and DialogStore::correlation_tree for --call-tree,
/tree and get_call_tree. Before, only the source of a /correlated
answer had one. The --correlate and --call-tree tables gain a last
column, TH.
- The hidden legs held, by status, are on REST /v1/stats and MCP
capture_status as th_callid, and in Prometheus as
sipnab_th_callid_legs{status} and
sipnab_th_callid_undecoded_with_peer, with no per-call label. All of
them come from DialogStore::th_callid_summary. Before, only the end of
a run reported these counts, and only when legs had not joined.
- The mappings themselves are listed, one row each with state
(conflict, pending, joined) and which legs are captured, so an
operator can see why one call did not join. The listing is
output::query::th_mappings::th_mappings, served by GET
/v1/th-mappings, the MCP tool th_mappings and --th-mappings at the end
of a headless --hep-listen run. The route and the tool are in hep
builds.
Ratchets, by file:
- CLI flags 396 -> 397: src/cli.rs adds --th-mappings.
- REST routes 47 -> 48: src/output/api.rs adds /v1/th-mappings.
- MCP tools 76 -> 77: src/mcp/tools/hep.rs adds th_mappings.
- The signaling bundle grows from 20 to 21: src/mcp_profile.rs.
- docs/design/surface-capability-inventory.md is regenerated with
scripts/capability-matrix.py --write to match these counts.
- MCP tool counts 76 -> 77, and read-only tools 63 -> 64. th_mappings is
read-only, and the thirteen tools that write or act are unchanged.
The counts move in tests/docs_drift_test.rs, tests/mcp_stdio_test.rs and
tests/homepage_link_landing_test.rs,
in README.md, website/templates/index.html, docs/mcp.md,
docs/mcp-protocol.md and docs/design/mcp-write-back.md.
- MCP bundle byte ceilings, in src/mcp/server.rs: signaling
22_500 -> 24_000 and full 90_000 -> 91_500, from th_mappings' 1,102
bytes on tools/list. Measured: signaling 23,576 bytes over 21 tools,
full 91,074 bytes over 77. docs/mcp-tools.md's table is re-measured.
- Prometheus metrics 49 -> 51: sipnab_th_callid_legs and
sipnab_th_callid_undecoded_with_peer. The prose count moves in
docs/rest-api.md, docs/mcp-tools-server.md, src/output/runtime.rs and
src/mcp/server.rs.
- Error templates 542 -> 543: src/app/batch.rs adds
"--th-mappings: the mappings did not serialize: {e}", with an entry in
docs/errors.md.
- Documentation tables 1311 -> 1318, counted on top of #629:
- docs/rest-api.md +2 and its mirror +2;
- docs/mcp-tools-signaling.md +1 and its mirror +1;
- docs/design/surface-capability-matrix.md +1.
- Wiki links 2079 -> 2095, counted on top of #629:
- README.md +1;
- cli-reference.md +1;
- mcp-tools-core.md +2;
- mcp-tools-signaling.md +4;
- mcp-tools.md +2;
- prometheus-metrics.md +1;
- rest-api.md +5.
- Line citations in docs/design re-pointed: the inserted code moved
export_vcon, process_message, generation, get, get_stats,
build_router and parse_bind_addr. Two were re-pointed by
scripts/check-line-drift.py --apply, the rest by hand to the lines it
reported.
- The matrix gap ledger gains one line:
tests/surface_capability_matrix_test.rs records the TUI cell of the
new "Topology-hiding mappings" row as a gap. The TUI statistics view
shows the counts but does not list the mappings.
How-to: docs/rest-api.md "Find why a call through topology hiding did
not join" covers purpose, the OpenSIPS and sipnab prerequisites, steps,
verification and troubleshooting. Its output comes from a live listener
that received the committed OpenSIPS 4.0.2 capture, and a second live run
added a conflicting and a pending event. It is linked from docs/README.md,
the --th-mappings row and the th_mappings tool section.
Documentation defect fixed: 0.5.232 added the th_callid values mapped
and conflict, the method mapping and the strategy th_callid_mapping.
The doc comments that feed the OpenAPI and MCP schemas, the REST and
MCP references and the CLI reference still listed the older four
values each, and docs/mcp-estate.md listed eight strategies instead of
nine.
Test changes beyond the new tests:
- tests/cli_correlate_test.rs and tests/cli_call_tree_test.rs expect the
new TH column.
- The schema-description comparison in tests/cli_correlate_test.rs reads
a nullable reference's description from inside oneOf. utoipa puts the
text there for Option<enum>. This is the first leg field of that kind.
- /v1/th-mappings carries capture_identity, as the MCP tool does, and
tests/api_capture_identity_test.rs checks the two against each other.
- The tool is probed in tests/mcp_completeness_test.rs and in
tests/mcp_protocol_features_test.rs.
- tests/population_claim_test.rs names th_mappings as not probed, because
its file-replay session cannot hold a mapping.
a_page_size_never_moves_the_mapping_counts drives that property over a
real table instead.
Tests:
- Unit tests cover each surface.
- tests/th_mapping_test.rs spawns the binary with --hep-listen to cover
GET /v1/th-mappings and --th-mappings, in text and --json form.
- Each new test was seen failing first.
- Each was mutation-checked, and every restore was proved with cmp.
(cherry picked from commit 1cdd8a3)
Three parts, combined into one change to save a hook run on a shared
build queue.
== 1. Stale statements fixed at their source ==
Five statements no longer matched the code. Each is fixed at its
source, and the three that live in the binary have a test.
- sipnab sends rtpengine `statistics` as well as `list` and `query`
since --relay-stats (0.5.169). docs/rtpengine.md, the cookbook, the
CLI reference, the --rtpengine-control help, the features page and
one note said two commands only. A test reads the help against the
ReadOnlyCommand type; its match is exhaustive, so a new command
fails it until the help names it.
- The multi-relay REST recipe in docs/rtpengine.md built
H="-H 'Authorization: Bearer $KEY'" and expanded $H unquoted. Bash
does not re-read quotes in an expansion, so curl received
'Authorization:, Bearer and the key as three arguments. Run against
sipnab 0.5.232 with --api-key, the old form answered 401 and the new
-H "$AUTH" form answered 200 with the same capture_identity shape the
page shows.
- The cookbook said no command-line flag writes a WAV file
(--export-audio exists since 0.5.231) and that a G.729 stream cannot
be recovered (decoded since 0.5.222). The recipe now shows
--export-audio and GET /v1/dialogs/{call_id}/audio with real output,
exit codes and the file-name rule.
- Shift+P playback decodes G.711 and Opus; the F1 help, the
walkthrough, the key reference and the features page said G.711.
Playback's codecs are now one PlaybackCodec list, and a test holds
the help line to it.
- The log line a run writes when it keeps audio named the MCP
export_audio tool on every run, including --export-audio and
--retain-audio without --mcp.
Root cause for the prose: each surface added after a page was written
(statistics, --export-audio, G.729 decoding, Opus playback) changed
what the page described, and nothing tied the page to it.
Every command and output in the changed docs ran against the binary
built from this branch: relay commands against rtpengine 12.5.1 in the
repository harness, audio exports against tests/pcap-samples.
Multi-relay: the curl header mechanics ran against one node; two
relays were not available.
The cookbook's --export-audio recipe is pinned by a new golden,
tests/cli/cookbook/bc0fdabae7.trycmd. scripts/check-cookbook.py gains
--export-audio-out in OUTPUT_FLAGS: it names a file sipnab writes, and
without it the golden run wrote into the shared directory, which
cli_goldens refuses.
== 2. Ask a media relay about itself ==
The relay surfaces on main had reference rows and no task page:
--relay-stats, --relay-stats-call, --relay-stats-list,
--relay-stats-interval, --relay-compare, --relay-holdings, the six
/v1/relay/ REST routes, the MCP tools query_relay, relay_stats and
relay_compare, and the TUI relay view (S, ?, H, K).
docs/relay-statistics.md shows each in use, with what to check first
(rtpengine only, no change to rtpengine, a live source, privileges,
build features), the counters to read first, and a troubleshooting list
of the refusals with their exit codes. Its anchors are stable for
other pages to link: #ask-the-relay-for-its-counters,
Every command and output on the page ran against sipnab 0.5.232 built
from main at 0d8eea1, and rtpengine 12.5.1 (the Debian 13 package) in
the repository harness. The commits between 0d8eea1 and 9124a6f
touch no relay code (the one source change is src/app/vcon_scitt.rs). The relay's address is written as
127.0.0.1 and the SIP test client's as 192.0.2.21. K's comparison was
run from a capture that sees the SIP and not the media, so the page
shows that case.
The relay rows of the CLI reference, the REST and MCP references, the
key reference, docs/rtpengine.md and docs/rtpengine-sipnab.md link to
the page, and docs/README.md lists it under How-to guides.
The cookbook's relay recipes captured an interface without sudo and
showed no output, and "Ask a relay what it is still holding" read
created and destroyed counters rtpengine 12.5.1 does not publish. They
now use sudo, show real output, and that recipe uses --relay-holdings
and the session counters the relay reports.
The page is registered in build-site-pages.py (weight 68),
build-site-internals.py, build-wiki.py and the "capture" group of the
docs nav, beside docs/rtpengine.md.
== 3. Measure call quality ==
Call quality had reference rows and scattered mentions, and no task
page: the R-factor, the quality intervals and --quality-interval, the
TUI quality dashboard (D) and packet loss map (L), GET
/v1/streams?mos_below=, the far end's RTCP and RTCP XR figures, the
[quality] thresholds and --on-quality-exec.
docs/call-quality.md covers each, with real output from sipnab 0.5.232
(built from main at 0d8eea1; the commits to 9124a6f change one
source file, src/app/vcon_scitt.rs). Anchors: #read-each-streams-score,
and #when-something-does-not-work.
No committed capture carried RTCP XR, so tests/fixtures/rtcp-xr-call.pcap
is generated by rtcp_xr_call in tests/support/synthetic_captures.rs and
listed in OWNED and tests/PROVENANCE.md: a ten-second PCMU call whose
callee sends a sender report and an XR VoIP Metrics block about the
caller's stream. media_diagnostics_reports_the_far_ends_rtcp_xr_apart
checks the answer the page shows: the far end's figures under
endpoint_reported, and the MOS delay taken from half the XR round trip.
The test failed first because the fixture did not exist. With
record_extended_report made to drop every block, it fails on the
missing r_factor; restored and compared with cmp.
Corrections found while running the page:
- docs/mos-and-codecs.md said MCP rtp_stats carries endpoint_reported
and the MOS delay. media_diagnostics carries both; rtp_stats carries
round_trip_ms and round_trip_source.
- The TUI section is titled "Reported by far end (RTCP XR)", not "Far
End", in the walkthrough and the filter reference.
- The cookbook's media_diagnostics sample lacked the qos `expedited`
member the tool returns.
Also: docs/rtpproxy-sipnab.md shows the retried-command warning, run
against rtpproxy 2.1.1 in the harness with one command sent twice by
hand; docs/g729.md says which builds carry the g729 feature; the
troubleshooting quality section names the IMPOSSIBLE RATE warning.
The page's TUI screens came from a capture file. The active-stream
marker was checked on a live capture of a harness call through
rtpengine.
== Ratchets ==
Attributed per file against origin/main 9124a6f:
- EXPECTED_DOCS_PAGES 100 -> 102: docs/relay-statistics.md and
docs/call-quality.md.
- EXPECTED_WIKI_LINKS 2079 -> 2132: relay-statistics.md +8,
call-quality.md +7, cli-reference.md +22, examples.md +4, README.md +2,
rtpengine.md +2, troubleshooting.md +2, and keybindings.md,
mcp-tools-relay.md, mos-and-codecs.md, rest-api.md, rtpengine-sipnab.md
and tui-walkthrough.md +1 each.
- EXPECTED_MARKDOWN_FILES 322 -> 326: the two pages and their site
mirrors.
- EXPECTED_TABLES 1311 -> 1321: relay-statistics.md +4, call-quality.md
+1, and the same on their mirrors. No other changed page gained or lost
a table.
(cherry picked from commit bff27f9)
"Collect captures from several SIP servers in one place" is the fleet
guide's setup page. Each step now ran against OpenSIPS 3.6.9, Kamailio
6.1.4 and sipnab 0.5.232 in throwaway containers on one Docker network.
- The OpenSIPS tracer line was wrong. OpenSIPS 3.6.9 refuses
"[sipnab]uri=hep:...;version=3;transport=udp" with "Invalid key
<version> in trace id!" and does not start. The page now gives the
form homer.md uses: a hep_udp socket (without it OpenSIPS stops with
"No HEP listener defined!"), the address in proto_hep hep_id, and
trace_id naming it. Two packets per OPTIONS reached sipnab, with a
host name in hep_id.
- The Kamailio step said "the handful of modparams" without them. It now
lists them. Kamailio did not resolve a duplicate_uri host name that
only /etc/hosts carried ("could not resolve hostname"), so the example
uses an address and says so. With two duplicate_uri lines, only the
last listener received anything, as deployments.md states.
- The firewall step had no command. It now gives the nftables rule,
measured with the input policy set to drop: the allowed sender's 10
packets arrived, and a sender outside the range never reached the
roster. ufw could not be exercised in a container and is not named.
- "Verify HEP is arriving" sent an MCP initialize, which proves nothing
about HEP. It now calls the hep_senders tool in a session and prints
one line per admitted sender (real output from the Kamailio and
OpenSIPS above), and says the capture id there is the sender's, not
--node-name.
- A sizing step lists the three limits a busy estate meets first, with
their defaults and the counters that show them. No throughput number
was measured, so none is given.
Site mirror and llms-full.txt regenerated.
Ratchets, attributed per file against origin/main 9124a6f:
- tests/docs_drift_test.rs EXPECTED_TABLES 1311 -> 1313: docs/mcp-estate.md
+1 (the listener limits table) and website/content/docs/mcp-estate.md +1.
- tests/link_integrity_test.rs EXPECTED_WIKI_LINKS 2079 -> 2081:
docs/mcp-estate.md +2 (homer.md, and the cookbook's memory budget recipe).
(cherry picked from commit e9f0399)
…oting Add docs/run-as-a-service.md, a task page for operating sipnab under systemd: the packaged unit and how to check it, a drop-in that turns on the REST API with its key in an EnvironmentFile, a unit of your own for installer and tarball installs, where the log goes and what --syslog sends, the signals sipnab handles, crash reports, audit file rotation, the refusals a service meets, and health checks. Every output on the page was measured with the v0.5.232 arm64 .deb on Debian 13 with systemd 257, in a throwaway container. src/, Cargo.toml, Cargo.lock, packaging/ and Dockerfile differ between v0.5.232 and this branch's base 4d85bd7 in the content-persistence gate's config handling (e637453) and a doc comment in src/app/vcon_scitt.rs, none of which these pages describe. Corrections, each measured: - examples.md "Run sipnab as a service" had no User=, so the unit ran as root (sipnab then dropped to nobody), while the page said AmbientCapabilities let it run without root. It also said --syslog makes the journal useful; --syslog sends security alerts only, and the log goes to stderr. The unit now runs as User=sipnab with ambient capabilities, verified to capture eth0 as sipnab. - mcp-deploy.md granted capture with setcap for units that set NoNewPrivileges=true. The capture fails there ("Permission denied on 'eth0'"). The units now grant AmbientCapabilities. - mcp-deploy.md made /etc/sipnab/mcp.token root-owned mode 600 and then ran the unit as User=sipnab, which stops with "--mcp-token-file '/etc/sipnab/mcp.token': Permission denied (os error 13)", exit 2. The file is now root:sipnab mode 640. - mcp-deploy.md said the packaged unit starts REST and metrics; it starts metrics only. - examples.md "Run a live capture without giving sipnab root" ran `sudo sipnab ... --user sipnab --chroot /var/empty` after telling the reader to run without sudo. --chroot stops every root run at the privilege drop (the user lookup happens inside the new root), and Debian 13 has no /var/empty. The recipe now documents both, and no longer says sipnab opens output files before the drop. - config-reference.md [crash] said release builds abort without unwinding (they unwind since 0.5.228), and its example report_dir is not writable under the packaged unit. It now names /var/lib/sipnab. - contrib/observability/sipnab-hep.service said privilege drop happens internally next to --no-priv-drop, which keeps root for the run. - mcp-deploy.md ran `sudo sipnab ... --mcp-token-file` and the MCP TLS example as root with root-only files. A root run drops to nobody before it reads the token, the certificate and the key, and stops with exit 2 (for the key the message is the bare path). Both now pass --user sipnab with root:sipnab 640 files, verified in a container. Registration of the new page: scripts/build-site-pages.py (weight 66), scripts/build-site-internals.py, scripts/build-wiki.py, website/config.toml and docs/README.md. Site mirrors and llms files regenerated. Every command and output below was run with the v0.5.232 release artifacts or a binary built from this branch's base, in throwaway containers or on loopback. install.md: - The one-line installer: a table of its own failure messages (no GitHub API, a version with no release, a checksum mismatch, no checksum tool, an unwritable directory without sudo), and that it installs the -gnu build without libpcap, whose first run then fails to load libpcap.so.0.8. - The .rpm recipes use `dnf install ./<file>`. `rpm -i` refuses on a host without libpcap ("Failed dependencies"), measured on Fedora 42; dnf warns that the packages carry no OpenPGP signature. - The .deb unit "logs to syslog" was wrong: --syslog sends security alerts. - The audio plugin: since 0.5.118 the Linux gnu tarballs, standard .deb and .rpm hold no libsipnab_audio and no libasound Recommends (0.5.117's arm64 .deb had both). The artifact table and package text now say so. - Homebrew is marked as not run for the page: no Mac was available. - Verify a download: real gh attestation output, that gh 2.102.0 prints nothing to a pipe so a script must read the exit status, the failure for a modified file, cosign 3.1.4 working as written and 2.4.1 needing --new-bundle-format, and cosign's two failure messages. - Docker: the image is linux/amd64 only; `latest` follows main, not releases (on 2026-11-03 it was main commit 0d8eea1, attested from refs/heads/main), so every recipe pins a release tag; live capture needs --user root, because the image's sipnab user gets no capabilities even with --cap-add; image provenance with gh attestation verify oci://. - Uninstall: apt and dnf remove neither stop nor disable the service, and leave the user, /etc/sipnab, /var/lib/sipnab and drop-ins. - Building from source: the two late failures (no libpcap headers, no cmake for --features full), from building v0.5.232 in rust:1.99. auth.md, rest-api.md, prometheus-metrics.md: - A "Secure every listener" section: the four listeners' credentials, TLS flags, config keys and refusal exit codes; making a private CA and a certificate; one run serving all four over TLS with its checks; what a client sees when something is missing; the HEP sender's TLS failure; a Prometheus 3.5.0 scrape of the HTTPS metrics endpoint with Basic auth, reported up. - A root run reads the REST and MCP keys and tokens, the metrics credential, the REST and MCP certificates and keys, and a HEP sender's secret AFTER dropping to nobody, so root-only files stop it with exit 2. The REST and metrics TLS examples now pass --user sipnab with 640 files, verified in a container. - --mcp-revoked-file, measured; the 503 row names --api-rate-limit-per-peer; MCP's /health needs the token. troubleshooting.md: - Operations sections: a service that does not start, live capture Permission denied by cause, key files a root run cannot read after the drop, the answers a listener gives a client, TLS handshake failures, a setting with no effect, a long run filling disk or memory, crash reports under a service, and contained parser panics. Test changes: - tests/docs_drift_test.rs: the rpm version markers and the upgrade-route check follow the `dnf install ./sipnab-...` recipes; a marker gates the pinned image tag against published_version; FOREIGN_FLAGS scopes curl's --cacert and --data-urlencode, useradd's flags, docker's --cap-add and cosign's --new-bundle-format to the pages that use them. - tests/docs_drift_test.rs FOREIGN_FLAGS also scopes useradd's --system, --home and --shell and systemctl's --now, --no-pager and --value to the service page and docs/examples.md. - The release download markers in docs/install.md change form. Phase two of a release rewrites them by pattern, so every consumer moves with them: docs_current_version_markers_match_cargo's published list, release_completeness_test's every_download_instruction_names_the_ advertised_version, no_cut_marker_pattern_also_matches_a_published_marker (now also checks the dnf and image forms), and the marker list in docs/internals/build-ci-release.md. Markers on main: SIPNAB_VERSION= 1, VERSION= 2, "e.g." 2, `rpm -i sipnab-` 4. Here: SIPNAB_VERSION= 2, VERSION= 2, "e.g." 2, `dnf install ./sipnab-` 4, `ghcr.io/normb/sipnab:` 6. Each new line carries the version. Mutating one dnf line, and one image tag, to 0.5.231 failed docs_current_version_markers_match_cargo with "names 0.5.231 but the last PUBLISHED release is 0.5.232"; both restores were checked with cmp. No script and no site template reads the rpm lines (grep of tests/, scripts/, docs/internals/, website/, .github/). The reason for the rpm change, on Fedora 42 against the 0.5.232 rpm: $ rpm -i sipnab-0.5.232-1.aarch64.rpm error: Failed dependencies: libpcap is needed by sipnab-0.5.232-1.aarch64 libpcap.so.1()(64bit) is needed by sipnab-0.5.232-1.aarch64 exit=1 $ dnf install ./sipnab-0.5.232-1.aarch64.rpm Installing dependencies: libibverbs, libnl3, libpcap 14:1.10.6-1.fc42 Complete! exit=0 The image tag is pinned because `latest` follows main, not releases. Ratchets, attributed per file against origin/main 9124a6f (the rebase onto 4d85bd7 moved none of them): - EXPECTED_MARKDOWN_FILES 322 -> 324: docs/run-as-a-service.md and its mirror website/content/docs/run-as-a-service.md. - EXPECTED_DOCS_PAGES 100 -> 101: docs/run-as-a-service.md. - EXPECTED_TABLES 1311 -> 1335: docs/run-as-a-service.md +4, docs/install.md +4, docs/auth.md +2, docs/troubleshooting.md +2, and the site mirror of each +4, +4, +2, +2. - EXPECTED_WIKI_LINKS 2079 -> 2145: docs/run-as-a-service.md +22, docs/troubleshooting.md +20, docs/install.md +10, docs/auth.md +3, docs/examples.md +2, docs/mcp-deploy.md +2, docs/prometheus-metrics.md +2, docs/rest-api.md +2, docs/README.md +1, docs/config-reference.md +1, docs/mcp-estate.md +1. (cherry picked from commit 450e94b)
NormB
force-pushed
the
docs/howto-ops-service
branch
from
November 4, 2026 07:55
5e0f784 to
e76c324
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task pages for operating sipnab, from the documentation gap analysis (ops area). Commands and outputs come from runs against the v0.5.232 release artifacts. Those ran in throwaway containers (systemd on Debian 13, Fedora 42, Prometheus 3.5.0) or on loopback. The source differs from v0.5.232 only in the content-persistence gate's config handling (e637453) and one doc comment, none of which these pages describe.
Release procedure: the download marker forms changed
Phase two of a release rewrites the download markers in
docs/install.mdby pattern. This PR changes them:SIPNAB_VERSION=VERSION=e.g. <version>rpm -i sipnab-<v>dnf install ./sipnab-<v>ghcr.io/normb/sipnab:<v>Why
dnf install, on Fedora 42 against the 0.5.232 rpm:The image tags are new markers because the Docker recipes now pin a release:
latestfollows main, not releases. On 2026-11-03 it was main commit 0d8eea1, attested fromrefs/heads/main.Every consumer of the marker set moves with this change:
docs_current_version_markers_match_cargo;every_download_instruction_names_the_advertised_version;no_cut_marker_pattern_also_matches_a_published_marker, which now also checks the dnf and image forms;docs/internals/build-ci-release.md.Mutating one dnf line, and then one image tag, to 0.5.231 failed
docs_current_version_markers_match_cargo("names 0.5.231 but the last PUBLISHED release is 0.5.232"). Both restores were checked withcmp. No script and no site template reads the rpm lines.Pages
docs/run-as-a-service.md:EnvironmentFile=;--syslogsends;copytruncate;docs/examples.mdanddocs/mcp-deploy.md. These were measured failing:setcapunderNoNewPrivileges=truecannot capture;User=sipnabstops the run with exit 2;User=, yet the page said it ran without root.docs/install.md:dnfrecipes;--user rootfor live capture, and image provenance;docs/auth.md: "Secure every listener":rest-api.mdandprometheus-metrics.mdexamples now pass--user sipnab.docs/troubleshooting.md: operations sections.docs/config-reference.md[crash], andcontrib/observability/sipnab-hep.service's privilege comment.Product defects found (docs state the measured behavior; no code or packaging changed)
The packaged unit cannot write a crash report:
failed to write crash report: Read-only file system (os error 30).ghcr.io/normb/sipnab:latestfollows main, not releases.The image is
linux/amd64only.The image's
sipnabuser cannot capture, even with--cap-add.SIGHUP is unhandled. systemd records
Result=success, soRestart=on-failureleaves the service down.Log lines carry ANSI color codes when stderr is not a terminal.
NO_COLOR=1removes them.The
Permission denied on <dev>hint suggests--setup-caps,sudoandsetcapeven underNoNewPrivilegesor in a container.--chrootbreaks every root run. The drop user is looked up inside the new root:User 'nobody' not found, exit 1.Since 0.5.118 the Linux gnu tarballs,
.deband.rpmship no audio plugin and nolibasoundRecommends.release.ymladds,bpfto the gnu feature set and then testsfeatures = "full".apt removeanddnf removeneither stop nor disable the service.A root run reads these AFTER dropping privileges, so root-only files stop it with exit 2:
--api-signing-key-file,--mcp-signing-key-file,--mcp-token-file,--metrics-auth-file;--hep-auth-file.Many
sudo sipnab ... /etc/sipnab/...examples incli-reference.mdfail this way.An unreadable MCP TLS key reports only the bare path.
The HEP listener's routable-bind refusal exits 1, while the REST, MCP and metrics refusals exit 2.
--mcp --mcp-transport httpwith--jsonis refused with the stdio message.The runtime image has no CA bundle.
The installer installs the
-gnubuild without libpcap present.Ratchets
Attributed per file in the commit message.
Gates
The full pre-commit hook passed on this commit before the rebase onto 4d85bd7. After the rebase these passed in the quick slot:
release_completeness_test;link_integrity_test;docs_drift_test;link_relevance_test;doc_ellipsis_quote_test.The pre-push hook runs on push.
After the rebase, the commit message's statement about the base was out of date, so the message was replaced with
git commit-tree -Son the same tested tree and parent.bash .githooks/commit-msgpassed on that exact message:Commit message names no private identity... OK.Least-privilege confinement (
--sandbox,--seccomp) is the capture area's page. This PR links its reference rows: that page is not on main yet.