docs: operate sipnab: service, install, listener security, troubleshooting

CHANGELOG.md

@@ -10,10 +10,47 @@ entry that carries them.
 
 ## [Unreleased]
 
+### Added
+
+- **A how-to for asking a media relay about itself: [docs/relay-statistics.md](docs/relay-statistics.md).** The relay surfaces on `main` had only reference rows: `--relay-stats`, `--relay-stats-call`, `--relay-stats-list`, `--relay-stats-interval`, `--relay-compare`, `--relay-holdings`, the six `/v1/relay/` routes, the MCP tools `query_relay`, `relay_stats` and `relay_compare`, and the TUI relay view (`S`, `?`, `H`, `K`). The page shows each in use against rtpengine 12.5.1, with what to check first and a troubleshooting list of the refusals and their exit codes. The relay rows of the CLI reference, the REST and MCP references, the key reference, `docs/rtpengine.md` and `docs/rtpengine-sipnab.md` link to it.
+- **A how-to for measuring call quality: [docs/call-quality.md](docs/call-quality.md).** Each stream's MOS, R-factor, jitter and loss; the quality intervals and `--quality-interval`; the TUI quality dashboard (`D`) and packet loss map (`L`); `GET /v1/streams?mos_below=`; what the far end reported over RTCP and RTCP XR; the `[quality]` thresholds and how to check them; and `--on-quality-exec` with the variables it passes. Every output on the page ran against sipnab. `docs/README.md`, the CLI reference rows of those flags, the troubleshooting quality section and the TUI walkthrough link to it.
+- **A capture whose far end reports over RTCP, for the documentation and a test: `tests/fixtures/rtcp-xr-call.pcap`.** A generated ten-second PCMU call whose callee sends a sender report and an RTCP XR VoIP Metrics block about the caller's stream. `media_diagnostics_reports_the_far_ends_rtcp_xr_apart` checks that sipnab keeps the far end's figures apart and scores the MOS on the XR round trip.
+- **`docs/rtpproxy-sipnab.md` shows the warning for retried rtpproxy commands, and `docs/g729.md` says which builds decode G.729.**
+
+### Changed
+
+- **The cookbook's relay recipes run as written.** They captured an interface without `sudo`, showed no output, and the recipe "Ask a relay what it is still holding" read `created` and `destroyed` counters rtpengine 12.5.1 does not publish. That recipe now uses `--relay-holdings` and the session counters the relay reports.
+
 ### Fixed
 
 - **A `[[vcon.rule]]` writes its containers on a run that also serves `--api` or `--mcp`.** The persistence gate both doors share took its ceiling from the export flags alone (`--export-vcon`, `--export-vcon-when`), so on a run whose only writer was a configured rule the gate was never authorized. The exporter consults that gate, so the rule wrote nothing and printed nothing, and `GET /v1/persistence` answered `{"authorized":false,"enabled":false}`; the same config and capture without a door wrote the call. `Cli::persists_content` now reads the config file as well as the flags, and it is the one answer the gate and the `--redact` refusal both use. A rule authorizes the gate as a flag does, so `POST /v1/persistence` can close and reopen it. Found during the 2026-11-03 documentation review.
+- **A ban TFPS applied but whose answer sipnab could not read is held in doubt, and can be lifted.** Once `tfps_ctl` has run, TFPS may have acted, whatever became of its answer: a timeout, a non-zero exit with no result, or an answer that is not the JSON contract. sipnab journaled each of those as `failed`, which its ledger reads as "did not happen". So the ban stood in TFPS, the journal held nothing to act on, and every unban answered `409` "sipnab holds no ban on this address", with an `admin` token too. The outcome is now journaled as `unknown`, and the ledger keeps the action in doubt. `sipnab --journal-show` lists it with its address, caller and time. The caller who asked, or a token with the `admin` scope, can unban the address over REST or MCP. The unban goes to TFPS, and TFPS's answer settles the doubt: it lifted a ban, or it held none. Only a `tfps_ctl` that never started is still `failed`. The `502` still says what went wrong, now as "TFPS was asked, and sipnab could not read what it did" rather than "TFPS could not be asked", and an answer sipnab cannot read names the JSON contract it reads: sippulse/tfps `e7ee57a` and later, where every time is epoch seconds. A revert whose answer is lost is held in doubt the same way. Found against a real TFPS whose `tfps_ctl` gave `expires` as an [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) string.
+- **`docs/mos-and-codecs.md` names the MCP tool that carries `endpoint_reported` and the MOS delay.** It said `rtp_stats`. `media_diagnostics` carries both, and `rtp_stats` carries `round_trip_ms` and `round_trip_source`.
+- **The TUI walkthrough and the filter reference name the RTCP XR section as the TUI shows it, `Reported by far end (RTCP XR)`.**
+- **The `media_diagnostics` sample in the cookbook carries the `expedited` member the tool returns.**
+- **The docs and `--help` name all three commands sipnab sends rtpengine.** `docs/rtpengine.md`, the cookbook, the CLI reference, the `--rtpengine-control` help, the features page and one note said sipnab sends only `list` and `query`. Since `--relay-stats` (0.5.169) the same control client also sends `statistics`. A test now reads the help against the command type, so a new command fails it until the help names it.
+- **The multi-relay REST recipe in `docs/rtpengine.md` sends its bearer token.** It built `H="-H 'Authorization: Bearer $KEY'"` and expanded `$H` unquoted. Bash does not re-read quotes in an expansion, so `curl` got `'Authorization:`, `Bearer` and the key as separate arguments, and every query answered `401`. The recipe now passes `-H "$AUTH"`.
+- **The cookbook's audio recipe covers `--export-audio` and `GET /v1/dialogs/{call_id}/audio`.** It said no command-line flag writes a WAV file, which stopped being true in 0.5.231, and that a G.729 stream cannot be recovered, which stopped being true in 0.5.222. The `--export-audio-out` default in the CLI reference now says how the Call-ID becomes a file name.
+- **The TUI help, the walkthrough, the key reference and the features page say `Shift+P` plays Opus as well as G.711.** Playback has decoded Opus all along. The codecs playback decodes are now one list, and a test holds the `F1` help to it.
+- **The log line a run writes when it keeps audio no longer names the MCP `export_audio` tool.** It said retention was on "so the MCP export_audio tool can decode it" on every run that kept audio, including `--export-audio`, `--retain-audio` without `--mcp`, and vCon audio rules. It now says the run can decode the audio it captures.
+- **`docs/rtpengine-sipnab.md` no longer says sipnab never asks rtpengine on a timer.** `--relay-stats-interval` polls for statistics. The page now says that naming calls is never on a timer.
 - **A release build stays reproducible on a host that configures a compiler wrapper.** With sccache set as `build.rustc-wrapper`, the `cc` crate routed mimalloc's C compile through the sccache server, whose clock, not `SOURCE_DATE_EPOCH`, set `__DATE__` and `__TIME__`, so two builds of one commit differed (run 41514782899). `scripts/reproducible-build.sh build` now runs cargo with `RUSTC_WRAPPER`, `RUSTC_WORKSPACE_WRAPPER` and their `CARGO_BUILD_` forms set to the empty string, which overrides a configured wrapper.
+### Added
+
+- **A fleet inventory: which build, binary and configuration each sipnab instance runs.** `GET /v1/inventory` on the REST API and the MCP tool `fleet_inventory` return one record, from one builder: the build (version, git commit and tag, dirty flag, target, compiled features, the `--version` line), SHA-256 of the running binary (`/proc/self/exe` on Linux), the configuration as digests only (`file_sha256` of the bytes read, `parsed_sha256` of the parsed configuration with its keys sorted, `argv_sha256` of the command line, and `effective_sha256` over the last two), the user id and privilege drop, pid, start time and uptime, the capture interfaces' error and drop counters, capture and HEP export counters, and the HEP listener's sender addresses. No configuration value and no command-line argument is in the answer, so a secret in either changes a digest and never appears. `signature.state` is always `unverified`: a process cannot prove which binary it is, so compare `binary.sha256` with the release's attested `SHA256SUMS.txt`. The MCP answer carries no `source_exhausted` or `source_stopped_early`, so the two surfaces return the same document.
+- **`sipnab fleet inventory` reads the inventory of every instance in a hosts file.** `--hosts <FILE>` names one `[[host]]` table per instance (`name`, `url`, `token_file`, optional `ca_file`), `--json` prints the whole report, and `--timeout <SECS>` (1 to 300, default 10) bounds each instance. It asks the instances at once, prints one row per instance with its state (`ok`, `refused`, `unreachable` or `incompatible`), and reports the findings `version_mismatch` and `unlisted_hep_sender` (a HEP sender a listed collector receives from whose address no listed instance has). It exits `0` when every instance answered and there is no finding, `1` otherwise, and `2` when the hosts file cannot be used. An instance that is down is listed as `unreachable`, and the others are still reported. [Manage many sipnab instances](docs/fleet.md) is the how-to.
+- **A token scope for the inventory: `--token-scope inventory`.** It reaches `GET /v1/inventory` and nothing else, so a fleet controller does not hold a `full` token, which reads every call. `full`, `actions` and `admin` reach the inventory too. It is a REST API scope; sipnab refuses to mint it for MCP, where a `read` token reaches `fleet_inventory`.
+- **The OpenSIPS topology-hiding mappings can be listed, one row each, to find why a call did not join.** The counts under `th_mappings` said how many mappings were pending or in conflict, but not which. `GET /v1/th-mappings`, the MCP tool `th_mappings` and the new flag `--th-mappings` (at the end of a headless `--hep-listen` run; one JSON object on the last line of stdout with `--json`) return every mapping sipnab holds from `E_SIPNAB_TH_MAP` events: its caller-side and callee-side Call-IDs, its state (`conflict`, `pending` or `joined`) and whether each leg is captured, conflicts first. REST and MCP narrow it by `call_id` (either leg) and `state`. Beside the rows are the event counts and the hidden legs by status. All three are built by one function. The route and the tool exist in a build with the `hep` feature, where the events can arrive.
+- **The topology-hiding legs held are counted by status on every status surface.** REST `/v1/stats` and MCP `capture_status` carry `th_callid`: the hidden legs held now that are `decoded`, `no_secret`, `undecodable`, `unmatched`, `mapped` or `conflict`, and `undecoded_with_peer`, the undecoded ones whose caller leg was captured, which points at the secret or the prefix. Prometheus carries the same counts as `sipnab_th_callid_legs{status}` and `sipnab_th_callid_undecoded_with_peer`, with no per-call label. Before, only the end of a run said this, and only when legs did not join.
+- **Each correlated leg says whether it is a topology-hiding leg and how that leg stands.** Every leg on `--correlate`, `--call-tree`, REST `/v1/dialogs/{call_id}/correlated` and `/tree`, and MCP `find_correlated` and `get_call_tree` carries its own `th_callid`. Before, only the source of a `/correlated` answer had one, so an answer asked from the caller side did not say which leg was hidden or whether it had joined.
+
+### Changed
+
+- **`--correlate` and `--call-tree` print a `TH` column.** It is the last column and holds each leg's `th_callid`, `-` for a leg that is not hidden. A script that splits those tables by column position and reads the last column needs updating. The JSON adds `th_callid` to each leg and keeps every other field.
+
+### Fixed
+
+- **The documented values of `th_callid` and `method` include the ones 0.5.232 added.** The REST and MCP references, the OpenAPI and MCP schema descriptions and the CLI reference listed `th_callid` as `decoded`, `no_secret`, `undecodable` or `unmatched`, and `method` as `decoded`, `header`, `sdp` or `timing`. They now list `mapped` and `conflict`, and `mapping`, which the code already returned, and the strategy lists name `th_callid_mapping`.
 
 ## [0.5.232] - 2026-11-03
 

Cargo.toml

@@ -478,6 +478,14 @@ wasm-opt = false
 [profile.dev]
 debug = "line-tables-only"
 
+# SHA-256 at opt-level 0 is slow enough to fail a test: `/v1/inventory` and
+# `fleet_inventory` hash the running binary on first use, and a debug sipnab
+# is about 200 MB. Measured on the aarch64 build host: the first request took
+# 1.97 s unoptimized; on CI's ubuntu runner it outlasted the tests' 10 s read
+# timeout. Only sha2 is raised; tests/config_wiring_test.rs holds the setting.
+[profile.dev.package.sha2]
+opt-level = 3
+
 # `strip = true` holds for every build, the release included. The release
 # workflow appends `-C strip=none` to RUSTFLAGS (from
 # `scripts/split-debuginfo.sh --rustflags`) so the line tables below survive

README.md

@@ -218,7 +218,7 @@ sipnab's analysis as tools.
 sipnab --mcp -N -I sip-problem-call.pcap --mcp-tools core
 ```
 
-The server offers 76 tools. Every tool costs the agent context before it asks
+The server offers 78 tools. Every tool costs the agent context before it asks
 anything.
 
 - `--mcp-tools core` loads a small set that still answers a whole call.
@@ -388,8 +388,8 @@ Next:
   - A capture started as root drops to `--user` after opening its devices.
   - It can confine itself with a Landlock sandbox (`--sandbox`) and a seccomp
     allowlist (`--seccomp`).
-- **MCP server.** 76 tools over stdio or HTTP.
-  - 63 only read.
+- **MCP server.** 78 tools over stdio or HTTP.
+  - 65 only read.
   - The 13 that write each say what they change.
   - File writes stay inside `--mcp-file-root`.
   - These stay off until you enable them:

contrib/observability/sipnab-hep.service

@@ -54,7 +54,8 @@ ExecStart=/usr/local/bin/sipnab \
 User=root
 Group=root
 
-# Privilege drop happens internally; HEP listen on :9060 (>1024) needs no caps.
+# --no-priv-drop above keeps root for the whole run: sipnab does not drop to
+# another user. HEP listen on :9060 (>1024) needs no capabilities.
 NoNewPrivileges=true
 ProtectSystem=strict
 ProtectHome=true

docs/README.md

@@ -39,6 +39,12 @@ pages use.
 
 Each answers one "how do I" question and assumes you already know what you want.
 
+**Run sipnab in production**
+
+- **[Run sipnab as a service](run-as-a-service.md)**: the packaged systemd
+  unit and one of your own, drop-ins, the log, signals, crash reports, audit
+  file rotation, and a service that does not start.
+
 **Find out what went wrong**
 
 - **[Troubleshooting](troubleshooting.md)**: symptom to command. Failed calls,
@@ -49,6 +55,9 @@ Each answers one "how do I" question and assumes you already know what you want.
 - **[Worked examples from real captures](real-world-captures.md)**: twelve
   findings read out of live carrier and PBX traffic, each with the command,
   the output and what to do next.
+- **[Measure call quality](call-quality.md)**: each stream's MOS, R-factor,
+  jitter and loss, when in the call they changed, what the far end reported
+  over RTCP, and the thresholds and alerts that act on them.
 - **[Narrow to the calls that matter](filter-dsl.md)**: the filter language
   (`method == 'INVITE' and rtp.mos < 3.5`) and the diagnostic aliases
   (`--filter codec-asym`), with its complete grammar and field list.
@@ -80,6 +89,9 @@ Each answers one "how do I" question and assumes you already know what you want.
   means. [API TLS](rest-api.md#api-tls) serves it over HTTPS, and
   [Call the REST API from code](https://sipnab.com/docs/api-clients/) has
   clients in Python, TypeScript, Rust and Go, and curl one-liners.
+- **[Manage many sipnab instances](fleet.md)**: one table of every
+  instance's build, binary and configuration digests, privilege and uptime,
+  and which instances did not answer.
 - **[Set up authentication](auth.md)**: signed bearer tokens for the API and
   MCP, with lifetimes, key rotation and revocation.
 - **[Export a call as a vCon](vcon.md)**: write one observed dialog as a
@@ -95,6 +107,9 @@ Each answers one "how do I" question and assumes you already know what you want.
   guides below build on. No sipnab involved.
 - **[Run sipnab beside OpenSIPS](opensips-sipnab.md)**: watch a call through
   OpenSIPS as one call, from both its legs.
+- **[Find why a call through topology hiding did not join](rest-api.md#find-why-a-call-through-topology-hiding-did-not-join)**:
+  list the OpenSIPS `E_SIPNAB_TH_MAP` mappings sipnab holds, and see which
+  leg of which call is missing.
 - **[Use Kamailio as your voice stack's SIP server](kamailio.md)**: the same
   with Kamailio 6.1, alone or beside OpenSIPS. No sipnab involved.
 - **[Run sipnab beside Kamailio](kamailio-sipnab.md)**: watch a call through
@@ -133,6 +148,9 @@ Each answers one "how do I" question and assumes you already know what you want.
   sipnab involved.
 - **[Let sipnab name rtpengine's media](rtpengine-sipnab.md)**: tie the media
   on a relay to its call, from the relay's control plane or by asking it.
+- **[Ask a media relay about itself](relay-statistics.md)**: rtpengine's own
+  counters, the calls it holds, and its count for one call beside sipnab's,
+  from the command line, REST, MCP or the TUI.
 - **[Add rtpproxy to your voice stack](rtpproxy-relay.md)**: anchor every
   call's media on an rtpproxy relay instead, from OpenSIPS or Kamailio. No
   sipnab involved.