Skip to content

Commit 73b4e23

Browse files
Sweep dead start bits in slices instead of a full reclaim
A dead object keeps its start bit until something clears it, and its mark id comes round again after 16 collections, so the collector runs a full reclaim every 15. That is a stop of 5 ms on gc_pauses and 17 ms on a million-entity heap, once in 16 cycles, in a collector built around 0.5 ms slices. The benchmarks never saw it: the harness collects between samples, which restarts the count. Each collection now owes a twelfth of the blocks to a sweep, which runs as jobs on the mark queue. A block is swept every 12 collections, and an object marked by neither this cycle nor the last has its start bit cleared. A full reclaim is left as the fallback: if the sweep falls more than a collection behind, on a failed cycle, for an explicit collection, and to give memory back from a heap well over its target. Nothing else needed the periodic one, and live bytes now come from the row count. A job is much more work than one object, and IncrementalExpired reads the clock once in 64 calls, so 64 sweep jobs could run past a deadline: a first slice took 1.3 ms. A slice now looks at the clock after each job. First slices are 0.6 ms. Without the harness's collections, 4 interleaved runs against the default collector, before and after: gc_frames +34% to +11%, longest slice 3.9 to 1.2 ms; gc_pauses +78% to +5%, 5.6 to 2.5 ms, p99.9 2.8 to 0.9 ms; million-entity heap +24% to +8%, 17.0 to 3.1 ms. Test: IncrementalGcStartSweep runs 40 cycles on a heap fragmented into small holes, and checks no start bit outlives 14 collections and no full reclaim ran.
1 parent ac1a5fa commit 73b4e23

11 files changed

Lines changed: 264 additions & 6 deletions

File tree

‎.github/workflows/main.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ jobs:
229229
if: ${{ !cancelled() && steps.setup.outcome == 'success' }}
230230
working-directory: test/regression
231231
run: |
232-
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcPacing IncrementalGcFinishBoundary GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
232+
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcPacing IncrementalGcStartSweep IncrementalGcFinishBoundary GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
233233
(cd "$test" && haxe -main Main -cpp bin -lib hxcpp -D HXCPP_M64 -D HXCPP_GC_INCREMENTAL -D HXCPP_GC_INCREMENTAL_VERIFY -D HXCPP_GC_INCREMENTAL_TEST -D HXCPP_SANITIZE=${{ matrix.sanitize }} && bin/Main > out.txt && diff <(tr -d '\r' < stdout.txt) <(tr -d '\r' < out.txt))
234234
done
235235

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,7 @@ jobs:
402402
working-directory: test/regression
403403
run: |
404404
failed=""
405-
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcPacing IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalWindowsHeaders GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
405+
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcPacing IncrementalGcStartSweep IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalWindowsHeaders GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
406406
echo "::group::incremental gc: $test"
407407
# The lifecycle fixtures' hxml selects generational mode; compile
408408
# the same Main with incremental mode here instead.

‎Changes.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22
libhxcpp 6.0.0 (unreleased)
33
------------------------------------------------------------
44

5+
* The incremental collector no longer stops for a full reclaim every 15 collections, which took 5 ms on a 120 MB heap and 17 ms on an 800 MB one. Slices sweep a share of the blocks each collection instead, a finished cycle counts rows and leaves the sweep to the zeroing thread and the allocator, and a slice looks at the clock after each large job (libhxcpp #372)
56
* Fixed the incremental collector doing no incremental work for a program that allocates mostly array storage, such as Bytes: a slice was skipped inside a mutation scope and a cycle could not finish in one, so every collection was a full stop. Slices run there now, and the stop that finishes a cycle scans the holders of open scopes again (libhxcpp #372)
6-
* The incremental collector (HXCPP_GC_INCREMENTAL) costs far less and stops for less. Each cycle is paced from the last one, starts as late as it can finish inside the free space the stop-the-world collector would spend, and spaces its slices by allocation as well as time. The collecting thread marks too, a slice no longer waits for a marker the scheduler has not run or for the zeroing thread, and an array's chunks are marked in parallel. On gc_pauses it went from 74% slower than the default collector to 5%, and the slowest 0.1% of work units from 2.7 ms to 1.4 ms (libhxcpp #372)
7+
* The incremental collector (HXCPP_GC_INCREMENTAL) costs far less and stops for less. Each cycle is paced from the last one, starts as late as it can finish inside the free space the stop-the-world collector would spend, and spaces its slices by allocation as well as time. The collecting thread marks too, a slice no longer waits for a marker the scheduler has not run or for the zeroing thread, and an array's chunks are marked in parallel. On gc_pauses it went from 78% slower than the default collector to 9%, and the slowest 0.1% of work units from 2.8 ms to 0.9 ms (libhxcpp #372)
78
* Fixed a thread that throws while the process exits crashing it on musl, as one with a failing cpp.Lib.load did about once in 800 exits on Alpine: libgcc's unwinder frees the table it finds frames in when the library finalizers run, and other threads still ran Haxe code then. The exit now parks every other thread that runs Haxe code, at its next safe point or as it leaves a GC-free zone, waiting up to 100 ms for one in a loop that does not allocate. A thread let go during the exit stays parked instead of ending (libhxcpp #423)
89
* Fixed programs that use sys.ssl without sockets failing to link on Linux with an undefined reference to hxcpp_alloc_kind: hxcpp_std, linked a second time after hxcpp_ssl, came after the runtime's second copy. The runtime is linked once more after all the libraries linked twice, which keep their order
910
* Fixed closing, reading or polling a socket handle that a library made in stock hxcpp's layout, as CrossByte's accept does, reading past the handle: the runtime took its own wrapper's later fields from it, and a close wrote through what followed it in memory. Only the descriptor is read from such a handle, and the runtime's wrapper no longer shares its name, and with it its vtable, with a library's

‎bench/results/incremental-gc/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,34 @@ All six `gc_frames` processes returned checksum 1106626610; all six `gc_pauses`
1919

2020
The largest budgeted collector invocation was 1.346 ms in `gc_frames` and 1.783 ms in `gc_pauses`, against a requested 0.5 ms. Cumulative unbudgeted maxima were 13.553 ms and 9.971 ms, respectively, including explicit collections outside measured frames. These results do not establish a sub-millisecond collector or eliminate synchronous interruptions.
2121

22+
## Pacing, sweep and slices
23+
24+
These results are for the pacer, the slice changes and the sweep, on main at `346c7e52` against the same source with them. Elapsed time is the median of each process's median sample, and the change is the median of its ratio to the existing collector in the same round. p99.9 is the median across processes. Maximum pause is the longest budgeted slice in any process, from the `maxSlicePauseMs` counter.
25+
26+
| Workload | Collector | Elapsed, ms | vs existing | p99.9, ms | Worst unit, ms | Maximum slice, ms | Peak RSS, MiB |
27+
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
28+
| `gc_frames` | Existing | 217.6 | | 7.66 | 11.78 | | 127 |
29+
| `gc_frames` | Incremental at `346c7e52` | 289.0 | +36.5% | 1.06 | 4.72 | 3.17 | 134 |
30+
| `gc_frames` | Incremental now | 257.1 | +15.6% | 0.81 | 5.06 | 2.23 | 140 |
31+
| `gc_pauses` | Existing | 150.0 | | 4.98 | 8.62 | | 133 |
32+
| `gc_pauses` | Incremental at `346c7e52` | 259.4 | +78.3% | 2.84 | 6.33 | 4.43 | 133 |
33+
| `gc_pauses` | Incremental now | 160.5 | +9.0% | 0.87 | 4.29 | 3.26 | 134 |
34+
35+
The harness collects between samples, and that restarts the count to the existing collector's periodic full reclaim, which comes every 15 collections. No sample reaches it, though an application that does not collect explicitly does. With those collections taken out of the harness:
36+
37+
| Workload | Collector | Elapsed, ms | vs existing | p99.9, ms | Worst unit, ms | Maximum slice, ms | Peak RSS, MiB |
38+
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
39+
| `gc_frames` | Existing | 225.6 | | 7.81 | 12.56 | | 129 |
40+
| `gc_frames` | Incremental at `346c7e52` | 300.1 | +33.6% | 1.06 | 4.08 | 3.88 | 134 |
41+
| `gc_frames` | Incremental now | 249.3 | +10.5% | 0.78 | 3.78 | 1.22 | 140 |
42+
| `gc_pauses` | Existing | 153.9 | | 5.33 | 11.49 | | 133 |
43+
| `gc_pauses` | Incremental at `346c7e52` | 278.1 | +78.0% | 2.79 | 6.91 | 5.55 | 132 |
44+
| `gc_pauses` | Incremental now | 164.9 | +5.1% | 0.86 | 3.92 | 2.53 | 134 |
45+
46+
The same changes took the longest slice from 17.0 ms to 3.1 ms on a heap of a million entities and about 800 MiB, a `gc_frames` world with 8,000 frames per sample, where a full reclaim costs most. That workload and a port of another frame loop are not in the harness.
47+
48+
Windows x64 as below, Haxe 4.3.7. Eight rounds of two warmups and ten samples with the harness as it is, and four rounds of two warmups and 24 samples without its collections, the order of the collectors rotating each round, on a machine 8 to 18% busy with other work. Checksums agreed in every round. The differences of a few percent between the two tables are within what a rebuild or the other load can move.
49+
2250
## Protocol
2351

2452
- Windows x64, AMD Ryzen 9 9950X, Haxe 4.3.7 and MSVC 19.44.35213.

‎docs/IncrementalGc.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ haxe build.hxml -lib hxcpp -D HXCPP_GC_INCREMENTAL
1414

1515
The library macro requests the existing Haxe generator's setter barriers. It enables `HXCPP_GC_GENERATIONAL` internally for code generation, while the runtime keeps ordinary allocation headers and does not enable the nursery. No compiler or standard library patch is needed. Do not add the generational flag yourself or reuse C++ generated without the incremental flag.
1616

17-
Automatic collection needs no engine calls. It accounts for newly acquired allocation blocks and large buffers, and uses a 0.5 ms slice target and a preferred 1 ms interval. Each finished cycle teaches the pacer how many slices a cycle takes and how fast the app allocates during one. The next cycle then starts as late as it can still finish inside the reserve, the space the existing collector would spend before collecting: every free block, and growth up to the memory target. A later start means fewer marks for the same allocation. During a cycle a slice is due at the interval, or sooner once the app has allocated its share of the reserve, with at least 0.25 ms for the app between slices. Past three quarters of what was left at the start, slices stop waiting. A cycle the allocator cuts short counts as a longer one, so the next starts earlier, and the first cycle, with nothing to learn from, starts at half the reserve. Deadlines and pause counters use a monotonic clock. Exhausting available space can still force synchronous completion. `Gc.enable(false)` disables automatic collection as before; explicit collection and scheduling calls still work.
17+
Automatic collection needs no engine calls. It accounts for newly acquired allocation blocks and large buffers, and uses a 0.5 ms slice target and a preferred 1 ms interval. Each finished cycle teaches the pacer how many slices a cycle takes and how fast the app allocates during one. The next cycle then starts as late as it can still finish inside the reserve, the space the existing collector would spend before collecting: the free list, leaving out holes too small to use, and growth up to the memory target. A later start means fewer marks for the same allocation. During a cycle a slice is due at the interval, or sooner once the app has allocated its share of the reserve, with at least 0.25 ms for the app between slices. Past three quarters of what was left at the start, slices stop waiting. A cycle the allocator cuts short counts as a longer one, so the next starts earlier, and the first cycle, with nothing to learn from, starts at half the reserve. Deadlines and pause counters use a monotonic clock. Exhausting available space can still force synchronous completion. `Gc.enable(false)` disables automatic collection as before; explicit collection and scheduling calls still work.
1818

1919
Optional engine hooks are library-owned:
2020

@@ -45,7 +45,11 @@ Native bulk operations use a mutation scope that records the holder before and a
4545

4646
Arrays and maps longer than 256 elements register jobs containing a managed owner and a cursor. Each chunk reacquires current storage from that owner. Structural mutation resets the cursor. Scalar insertion shades its target without restarting the whole container. An array's chunks go onto the shared mark queue as jobs, so every marker scans them in parallel, and a job not reached in one slice waits for the next. A map keeps one cursor and is scanned by the collecting thread. Several chunks feed one marker round to avoid waking the pool for every small chunk. Traditional raw array-buffer jobs cannot survive an incremental pause.
4747

48-
No heap storage is reclaimed while marking is unfinished. Completed collection free ranges are materialized before starting a new partial mark, so lazy allocator reclamation cannot interpret partial row marks as a completed collection. Detached threads transfer their unread mutation entries before their contexts are pooled or destroyed.
48+
No heap storage is reclaimed while marking is unfinished. Completed collection free ranges are materialized before starting a new partial mark, so lazy allocator reclamation cannot interpret partial row marks as a completed collection. The stop that finishes a cycle only counts rows and builds the free list. A block is reclaimed when it is first used, or before that by the zeroing thread, which has time to spare while the allocator catches up, so a cycle's first stop rarely finds any left to materialize. A block outside the free list waits for a later collection.
49+
50+
A dead object keeps its start bit until something clears it, and its mark id comes round again after 16 collections. The existing collector clears them in a full reclaim every 15 collections, a stop that grows with the heap. Slices sweep a twelfth of the blocks per collection instead, as jobs on the mark queue, so steady allocation needs no full reclaim. One still runs if the sweep falls more than a collection behind, for an explicit collection, and to give memory back from a heap well over its target.
51+
52+
Detached threads transfer their unread mutation entries before their contexts are pooled or destroyed.
4953

5054
If mutation metadata cannot be retained, the partial cycle is discarded before any reclamation. Recovery rewinds headers changed by the partial epoch before restarting a full mark. Advancing directly to another epoch would make unvisited objects appear marked to the alternating-bit fast path. Tests cover that failure transition. Log storage is capped at 1,048,576 entries per thread, and range registration at 1,048,576 owners per cycle. Native allocation failure requests recovery; this is not a guarantee that a process with exhausted system memory can continue.
5155

@@ -65,6 +69,7 @@ Conventional fixtures under `test/regression` cover:
6569
| `IncrementalGcContainers` | Resize and reorder after a partially traversed array range, with payload checks after completion |
6670
| `IncrementalGcArrayChunks` | Array chunks still queued at a stop, through shrinking and reallocation, and dropped with a failed cycle |
6771
| `IncrementalGcPacing` | Cycles spending the free space, and finishing when all allocation is in mutation scopes |
72+
| `IncrementalGcStartSweep` | Dead start bits cleared before their mark id returns, and no full reclaim, on a heap fragmented into small holes |
6873
| `IncrementalGcFinishBoundary` | New roots, descendants and container stores published between strong marking and finishing |
6974
| `IncrementalGcVerifier` | Independent detection of an omitted descendant |
7075
| `GcWeakMapChain` | Weak-map reachability closure |
@@ -99,7 +104,7 @@ Counters include setup and explicit collections from process startup:
99104

100105
These counters describe the collecting thread's successful invocations. They are not a distribution of every application thread's wait. Several assists can occur in one allocation burst, and explicit APIs can finish synchronously. Measure frame or request latency as well. The benchmark executable includes `incrementalGc` counters in its JSON when built with the flag.
101106

102-
Development measurements must compare the same workload, source, toolchain and hardware against the existing libhxcpp collector, with verification and test controls omitted. Record checksums, frame or request distributions, throughput, RSS and reserved memory. The game frame workload has shown shorter tail frames locally, while the allocation-heavy `gc_pauses` workload still shows a material throughput cost. Do not treat either result as a universal speedup or a hard pause guarantee. The p99.9, maximum interruption, throughput and memory gates in #372 remain the acceptance criteria for promoting this mode beyond experimental use.
107+
Development measurements must compare the same workload, source, toolchain and hardware against the existing libhxcpp collector, with verification and test controls omitted. Record checksums, frame or request distributions, throughput, RSS and reserved memory. On the development machine `gc_frames` runs 10 to 16% slower than the existing collector and `gc_pauses` 5 to 9%, with the slowest 0.1% of work units under 1 ms against 5 to 8 ms, and up to 10% more peak memory. The harness's full collection between samples resets the existing collector's periodic full reclaim, so it hides that stop: compare with those collections removed as well. Do not treat either result as a universal speedup or a hard pause guarantee. The p99.9, maximum interruption, throughput and memory gates in #372 remain the acceptance criteria for promoting this mode beyond experimental use.
103108

104109
## Supported configurations
105110

0 commit comments

Comments
 (0)