Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
What happens
The whole stack runs and data flows. Then we stop and start only the dApp. After that the dApp gets no data: it stays in WAITING and never receives an indication. The controller drops every uplink slot the gNB produces (about 401 per second), and its "published" counter stays at the value it had before the restart (5243).
How to reproduce
- Start the E3Controller (JSON encoding,
shm.writer: gnb), then the ocudu-e3 gNB with jbpf, then the adaptive_cpu dApp.
Radio setup: Foxconn RU, TDD 7/6/2/4 with the sensing config (allow_request_on_empty_uplink_slot: true, no uplink in the special slot). With this config the gNB produces about 400 uplink slots per second even when no UE is attached. - Wait until the dApp shows data. In our run the controller had published 5243 slots.
- Stop the dApp only. Leave the controller and the gNB running.
- Start the dApp again.
- The dApp shows no indications. Every second, the controller prints
dropped 401 slot(s) in the last second.
Logs
dApp status after restarting only the dApp:
──────────────── dApp status · 03:01:53 · _2.0s ────────────────
liveness _ WAITING · last ind never · uptime 8s
indications OCUDU_L1 0 · OCUDU_L2 0 (+0/s)
rx_link gNB 0/s │ rx 0/s 0.00 Gb/s │ drop 0/s (0.0%) │ life: rx 0 drop 0
latency age _ (no IQ this window)
shed transport 0/s · stale 0/s · no_range 0/s │ snap skip 0/win
compute idle (no slots this window)
spectrum _ (no compute yet)
downstream snap 0/s │ publish 0/s · skip 0/s │ v0 p0 _s · busy 0% │ subsample 1/1
────────────────────────────────────────────────────────────────
E3Controller log at the same time:
[E3SMLayer1] dropped 1 slot(s) in the last second (2 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (403 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (804 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (1205 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (1606 total, 5243 published)
401 drops per second is the full slot rate of the gNB. So this is not a partial loss: every slot is dropped.
What we know
- The controller that showed this was built from
84a3828("configs: bake demo e3_controller.yaml …", branchtests/native-memcpy-libe3-latest). - That build does not have
417d0a4("Resend the helper config after every codelet load"):git merge-base --is-ancestor 417d0a4 84a3828fails. The fix is onmain, where it was squash-merged with Publish UL slots through a native gNB helper instead of a jbpf copy #8 (fbdf325). 417d0a4fixed this same symptom. When the last dApp unsubscribes, the controller unloads the codelet, and the next subscription loads a new one with empty maps. The controller sent the shared-memory config and the slot selector only once per process, so after a dApp restart the gNB helper refused every slot. The warning for that is printed only once, and the first session had already used it, so the log shows only the drop counts.- So the most likely cause is that the demo branch does not have this fix. This is not confirmed yet.
- Related, but a different bug: Restarting the controller while the gNB runs leaves the dApp reading stale rows #10 (restarting the controller while the gNB runs leaves the dApp reading stale rows).
To check tomorrow
- Reproduce with
main, which has the fix. If the problem is gone, the fix is to move the demo branch ontomain. Close this issue with that. - If it still happens on
main, find which drop reason grows. The drop CSV (logging.drops_log_path) has one column per reason (for exampleno_subscribers,ran_published_nothing,blob_too_small), and the shutdown summary lists them.ran_published_nothing: the gNB helper is not attached after the codelet reload.no_subscribers: the new dApp subscription does not reach the Service Model.
- Read the controller log at the restart. Does it print
Codeletset 'uplink_slot_samples' loaded successfullyagain? DoesSent e3_shm_cfg …follow it? - Check whether a full stack restart (controller, then gNB, then dApp) brings the data back.
Workaround
Restart the whole stack in this order: controller, then gNB, then dApp. This is not confirmed yet (see point 4 above). On this build, restarting only the dApp does not bring the data back.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
aferaudo commentedon Oct 15, 2026
Confirmed: the demo build (
84a3828) does not have the fix. Fixed by 417d0a4 ("Resend the helper config after every codelet load"), which is onmainas part of fbdf325 (#8).Tested over the air with the sensing config (400 uplink slots/s, no UE):
main: after each of 3 dApp-only restarts the data comes back at 399 slots/s. One slot is dropped per codelet load.mainwith 417d0a4 reverted: this restart code is identical to84a3828. The log matches this issue: every slot is dropped asran_published_nothing, "published" stays frozen and the dApp staysWAITING. A full stack restart brings the data back.