[alex-zaitsev]

Summary

Incrementally introduce an Observe → Decide → Act Raft safety planner for CHK.

• Extract pure raftplan.PlanAdmission and raftplan.CheckDisruption, with unit and parity tests.
• Observe live Keeper StatefulSets and published common-ConfigMap raft_configuration before staged admissions.
• Use the planner as the authoritative admission and quorum-disruption decision-maker by default; the normal e2e operator installation requires no environment variables.
• Retain the existing ConfigMap/StatefulSet reconcilers, preliminary staging, quorum wait/retry budget, serialization, and settle delays. Remove the superseded pre-create membership rollback.
• Treat unknown or inconsistent state conservatively (fail closed). Keep shadow comparison and the optional legacy decision predicate available for diagnostics/fallback.
• Stage 1→N / 2→3 joiners with a host-local Raft bootstrap configuration; publish into the established members only after Pod Started, then verify live membership and current Kubernetes readiness before completing.
• Integration from #2115 was merged into this branch (not into 0.28.0).

How to test

Build the operator from this PR's head branch refactor/chk-raftplan-shadow-20261022 and run the usual e2e suite; the planner is on by default. To run unit tests:

go test -mod=vendor -race -count=1 ./pkg/controller/chk/...

Prioritize staged 1→3 admission, intermediate unready joiner, interrupted Create/operator restart, 2→3, 3→5, degraded quorum during rollout, stop/resume, and scale-down.

The staged-join and readiness changes passed the owner's local suite, but CI for the later cleanup exposed a separate intermittent watcher test failure and a unit-test fixture panic. Commit a5b78bd corrects the fixture, tests the watcher using its recorded no-endpoint-change event and unchanged CHI ZooKeeper XML instead of an unrelated live session timestamp, and adds failure diagnostics for a distinct 0-2 scale-up stall. Re-run CI before marking the PR ready.

Not in this change

• No elimination of existing blocking membership settle delays.
• No CHK CRD or status-format change.

Review notes

The authoritative implementation was developed as stacked PR #2115 and merged into this branch for a single end-to-end review. This PR still targets 0.28.0; it has not been merged into the release branch.

Broken-image scale-up regression

The uncommitted test_020003_4 scenario found that a 1→3 scale-up with a broken image could remain indefinitely InProgress rather than reporting Aborted. Two startup paths were corrected:

• PodHasAllContainersStarted now requires reported Started=true status for every declared container; empty/partial container statuses are not successful startup.
• For an unchanged (ObjectStatusSame), unready staged joiner on a later reconciliation, the StatefulSet reconciler performs the normal bounded Started, not Ready, wait and uses CHK's existing ErrCRUDAbort fallback on timeout.

The regression motivated a two-phase join flow: boot with host-local membership, wait for Pod Started, publish the new peer to the leader, then verify the leader's live Raft membership and the joiner's fresh StatefulSet/Pod readiness. A failed startup may abort and resume after a spec edit. No existing Keeper needs to restart for admission.

Cross-reconcile stalled-joiner timeout

After the initial broken-image startup fix, e2e still remained InProgress: the next joiner was repeatedly returning a soft ErrCRUDDeferred for a previously published, unready member. The admission executor now inspects the blocking member's live Pod: if all containers never started and its Pod age exceeds reconcile.statefulSet.update.timeout, it returns hard ErrCRUDAbort instead of repeating the deferral. This deadline is based on Kubernetes timestamps and survives reconcile retries/operator restarts; Pods that have started but are not yet Keeper Ready continue to defer safely. A missing Pod uses the blocking StatefulSet's age. Failed API observations remain errors, not absence.

Regression tests cover pure deadline cases, abort propagation, resume, staged Raft publication after Pod Started, and the final committed-but-not-Ready join state. The owner previously confirmed the targeted join scenario passed. The broader e2e watch/scale-up scenario is not yet fully green on CI: two retries stalled creating the final Keeper Pod. The latest patch adds StatefulSet and Kubernetes event diagnostics to investigate any recurrence; no admission logic has been changed.

Cleanup after the working refactor

Removed the unused rollbackStagedRaftAdmit path and its obsolete tests, the empty includeAllHostsIntoCluster hook, the duplicate staged-join verification branch, and the unused boolean return from publishStagedAdmit. Replaced pre-create publication test coverage with a test of the current Started → publish ordering. Updated stale inline comments. Legacy decision fallback, shadow diagnostics, quorum safety, and settle delays remain intentionally intact.