Skip to content

Restarting only the dApp leaves it with no data (the controller drops every slot) #12

Closed

Description

@aferaudo

What happens

The whole stack runs and data flows. Then we stop and start only the dApp. After that the dApp gets no data: it stays in WAITING and never receives an indication. The controller drops every uplink slot the gNB produces (about 401 per second), and its "published" counter stays at the value it had before the restart (5243).

How to reproduce

  1. Start the E3Controller (JSON encoding, shm.writer: gnb), then the ocudu-e3 gNB with jbpf, then the adaptive_cpu dApp.
    Radio setup: Foxconn RU, TDD 7/6/2/4 with the sensing config (allow_request_on_empty_uplink_slot: true, no uplink in the special slot). With this config the gNB produces about 400 uplink slots per second even when no UE is attached.
  2. Wait until the dApp shows data. In our run the controller had published 5243 slots.
  3. Stop the dApp only. Leave the controller and the gNB running.
  4. Start the dApp again.
  5. The dApp shows no indications. Every second, the controller prints dropped 401 slot(s) in the last second.

Logs

dApp status after restarting only the dApp:

──────────────── dApp status · 03:01:53 · _2.0s ────────────────
 liveness     _ WAITING · last ind never · uptime 8s
 indications  OCUDU_L1 0 · OCUDU_L2 0 (+0/s)
 rx_link      gNB 0/s │ rx 0/s 0.00 Gb/s │ drop 0/s (0.0%) │ life: rx 0 drop 0
 latency      age _ (no IQ this window)
 shed         transport 0/s · stale 0/s · no_range 0/s │ snap skip 0/win
 compute      idle (no slots this window)
 spectrum     _ (no compute yet)
 downstream   snap 0/s │ publish 0/s · skip 0/s │ v0 p0 _s · busy 0% │ subsample 1/1
────────────────────────────────────────────────────────────────

E3Controller log at the same time:

[E3SMLayer1] dropped 1 slot(s) in the last second (2 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (403 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (804 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (1205 total, 5243 published)
[E3SMLayer1] dropped 401 slot(s) in the last second (1606 total, 5243 published)

401 drops per second is the full slot rate of the gNB. So this is not a partial loss: every slot is dropped.

What we know

  • The controller that showed this was built from 84a3828 ("configs: bake demo e3_controller.yaml …", branch tests/native-memcpy-libe3-latest).
  • That build does not have 417d0a4 ("Resend the helper config after every codelet load"): git merge-base --is-ancestor 417d0a4 84a3828 fails. The fix is on main, where it was squash-merged with Publish UL slots through a native gNB helper instead of a jbpf copy #8 (fbdf325).
  • 417d0a4 fixed this same symptom. When the last dApp unsubscribes, the controller unloads the codelet, and the next subscription loads a new one with empty maps. The controller sent the shared-memory config and the slot selector only once per process, so after a dApp restart the gNB helper refused every slot. The warning for that is printed only once, and the first session had already used it, so the log shows only the drop counts.
  • So the most likely cause is that the demo branch does not have this fix. This is not confirmed yet.
  • Related, but a different bug: Restarting the controller while the gNB runs leaves the dApp reading stale rows #10 (restarting the controller while the gNB runs leaves the dApp reading stale rows).

To check tomorrow

  1. Reproduce with main, which has the fix. If the problem is gone, the fix is to move the demo branch onto main. Close this issue with that.
  2. If it still happens on main, find which drop reason grows. The drop CSV (logging.drops_log_path) has one column per reason (for example no_subscribers, ran_published_nothing, blob_too_small), and the shutdown summary lists them.
    • ran_published_nothing: the gNB helper is not attached after the codelet reload.
    • no_subscribers: the new dApp subscription does not reach the Service Model.
  3. Read the controller log at the restart. Does it print Codeletset 'uplink_slot_samples' loaded successfully again? Does Sent e3_shm_cfg … follow it?
  4. Check whether a full stack restart (controller, then gNB, then dApp) brings the data back.

Workaround

Restart the whole stack in this order: controller, then gNB, then dApp. This is not confirmed yet (see point 4 above). On this build, restarting only the dApp does not bring the data back.

Activity

  1. aferaudo commented on Oct 15, 2026

    @aferaudo
    CollaboratorAuthor

    Confirmed: the demo build (84a3828) does not have the fix. Fixed by 417d0a4 ("Resend the helper config after every codelet load"), which is on main as part of fbdf325 (#8).

    Tested over the air with the sensing config (400 uplink slots/s, no UE):

    • main: after each of 3 dApp-only restarts the data comes back at 399 slots/s. One slot is dropped per codelet load.
    • main with 417d0a4 reverted: this restart code is identical to 84a3828. The log matches this issue: every slot is dropped as ran_published_nothing, "published" stays frozen and the dApp stays WAITING. A full stack restart brings the data back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions