Repository navigation
Fix OOMs in long sequences and optimize performance - #104
Merged
Merged
Conversation
alexkorovko
commented
Sep 9, 2026
alexkorovko
commented
Collaborator
- GPU native solver for BA boosts its performance
- Optimized correlation volume lookup kernel
- Support of infinitely long sequences
- MoGEv2 depth prior model support
Ports the MoGe-2 depth model from vipe_v2: vendors its inference code under vipe/priors/depth/moge2/ (no pip `moge`/`utils3d` dependency, same pattern as the existing dav2/dap vendoring), extends MogeModel to support version=2 alongside the existing v1 path unchanged, and wires up "moge2-l" in make_depth_model(). Checkpoint resolves via VIPE_MOGE2_CHECKPOINT env, an offline torch-hub cache, or HuggingFace. Existing "moge"/v1 and all other depth models (unidepth, metric3d, dav3, etc.) are untouched. Verified: full pytest suite, ruff, mypy, doc-gen, mkdocs build, and the existing dog-example smoke test all pass; additionally ran the default pipeline end-to-end with keyframe_depth=moge2-l on dog-example.mp4 to confirm the new model loads and integrates with the SLAM depth-alignment path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports vipe_v2's correlation-lookup kernel rewrite: 4x8 (32-thread, single-warp) tiles become 8x8 (64-thread) tiles for higher GA10x occupancy, plus a new thread-per-pixel fp16 fast path (altcorr_index_forward_kernel_pix) for the common C<=128 case, with a VIPE_ALTCORR_PIX=0 escape hatch back to the staged kernel. Porting this surfaced a real bug, not present in vipe_v2's source either: altcorr_forward_kernel and altcorr_backward_kernel write x2s[tid]/y2s[tid] to shared memory and then read x2s[k1]/y2s[k1] for k1 belonging to *other* threads, with no __syncthreads() between the write and that cross-thread read. This was latently present at the original 32-thread (one warp) tile size, where SIMT lockstep made the ordering accidentally safe, but retiling to 64 threads spans two independently-scheduled warps and turns it into a genuine race: repeated runs of test_indexed_altcorr_matches_corr_layer_reference on a fixed seed produced different wrong pixels each time. Added the missing barrier in both kernels; verified over 10+ repeated runs (both the default fp16 pix path and the legacy path forced via VIPE_ALTCORR_PIX=0) that the race is gone. Also ported the fixed-seed golden-value regression test from vipe_v2, plus a new repeated-run determinism test guarding specifically against this class of bug recurring. Verified: full pytest suite, ruff, mypy, doc-gen, mkdocs build, and the dog-example smoke test all pass; altcorr tests re-run 10x to confirm determinism. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports vipe_v2's GpuBlockSystem: a device-resident replacement for the legacy fused-BA path's host round-trip through Eigen's CPU SimplicialLLT solver. Symbolic index patterns (AccumPattern, SchurPattern) are precomputed once per BA call on the host from the (fixed, across Gauss-Newton iterations) edge graph; the per-iteration solve assembles a dense normal-equations system via GPU scatter-add and factorizes it on-device (float64 up to VIPE_BA_GPU_FP64_MAX_DIM, else float32 + iterative refinement), eliminating ~35 host syncs per BA call. Reachable only via the new BAConfig.solver == "fused_v2" (default stays "legacy", i.e. this is strictly opt-in); the existing ba/ba_extended bindings, kernels, and code paths are untouched. New files: csrc/slam_ext/gpu_block_system.cuh (ATen-only, no custom CUDA kernels of its own). geom_kernels.cu/slam.cpp gain the additive ba_extended_v2 binding; buffer.py dispatches to it only when ba_config.solver == "fused_v2". Verified: ported vipe_v2's BA-v2-vs-legacy parity tests (tests/test_slam_ext_ba_extended_v2.py), added a 10x-repeated determinism variant given the sync-race lesson from the correlation kernel port; full pytest suite, ruff, mypy, doc-gen, mkdocs all pass. Ran the dog-example pipeline end-to-end with ba.solver=fused_v2: completes successfully, and the resulting 64-frame pose trajectory matches the legacy solver to within 2e-5 max per-element difference. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gtsam is required for the upcoming long-sequence SLAM mode's final Sim3 pose-graph optimization (vipe.slam.longseq.pose_graph). Its only PyPI release (4.2.2, same version vipe_v2 uses) ships wheels for Python 3.11-3.14 only (no 3.10) and hard-pins numpy<2.0.0. vipe_v2 itself declares requires-python>=3.10 with this same unconditional gtsam dependency, which is actually inconsistent (gtsam cannot install on a real 3.10 interpreter) - its CI still targets 3.10, so this looks like a pre-existing, unnoticed gap over there. We resolve it properly here rather than carrying the same inconsistency: bump requires-python to >=3.11, drop the 3.10 classifier, move CI/dev tooling (.python-version, ci.yml/pages.yml/publish.yml) to 3.12, and accept the numpy 2.x -> 1.26.4 downgrade gtsam forces (along with the opencv-python and rerun-sdk downgrades numpy<2 drags in), matching vipe_v2's own locked versions exactly. Verified this migration in isolation, before any long-sequence code lands: full pytest suite, ruff, mypy, doc-gen check, mkdocs build, and the dog-example smoke test all pass on the rebuilt Python 3.12 / numpy 1.26.4 environment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports vipe_v2's sliding-window SLAM recipe: vipe/slam/longseq/ adds LongSequenceSLAMSystem (subclasses SLAMSystem unmodified, re-orchestrates the frontend/backend/filler around a retire/infill/pose-graph cycle so GPU memory stays O(window) instead of O(sequence length)), a CPU-side TrajectoryLedger recording retired keyframe poses/edges/map chunks, a RollingFrameCache bounding host memory to the retiring chunk's span, and a Sim3PoseGraph doing a final global-consistency pass. Companion buffer.py/factor_graph.py additions (GraphBuffer.retire_head, FactorGraph.shift_indices) compact the keyframe buffer and re-index the factor graph when a chunk retires. New SlamWindowConfig (slam.window, default null) selects it; two new pipelines (pose_only, pose_only_long) and their configs/pipeline/*.yaml presets wire it up end-to-end, reusing the existing -p/--pipeline CLI flag -- no new CLI code needed. All additive: default.yaml/panorama.yaml and existing pipeline classes are untouched. Dependency note: the pose graph needs gtsam's Similarity3 between/prior factors. The only stable gtsam release on PyPI (4.2.2, what vipe_v2 itself depends on) does not wrap them for Python at all (confirmed by running the ported code directly) -- they exist only on gtsam's unreleased develop branch. Switched to gtsam-develop (nightly builds of that branch) instead of reimplementing the pose graph on gtsam.Custom Factor, since the latter needs its own from-scratch Sim3 Jacobian derivation with real correctness risk for marginal benefit over a pinned nightly snapshot. Bonus: gtsam-develop has no numpy upper bound, so this also undoes the numpy 2.x->1.x (and opencv-python/rerun-sdk) downgrade the stable gtsam dependency had forced earlier in this port; gtsam-develop's own pytest>=9 requirement is why the dev group's pytest pin moved to >=9. opencv-python is capped at <5 to avoid an unrelated major-version jump surfacing from the same relock. Verified: ported vipe_v2's long-sequence unit tests (tests/test_long_sequence.py: GraphBuffer.retire_head, FactorGraph.shift_indices, TrajectoryLedger Sim3 deformation, the gtsam pose graph incl. fixed-point-chain and noisy-loop-closure convergence, LongSequenceSLAMSystem construction/validation) -- all pass, repeated 3x for determinism. Full pytest suite (80 tests), ruff, mypy, doc-gen, mkdocs build, and the dog-example smoke test all pass on the gtsam-develop environment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
alexkorovko
changed the title
alexkorovko
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.