Skip to content

Commit a125493

Browse files
Give the memory of freed large buffers back to the system with glibc (#434)
glibc keeps a freed buffer's memory in its heap, unless it had mapped the buffer on its own. It maps buffers of 128 KB or more on their own at first, but each time one is freed it raises that size to the freed buffer's, up to 32 MB on 64 bits, so later buffers of those sizes come from the heap and stay there once freed. After a burst of large allocations the process stayed as large as the burst, although the collector had freed the buffers and MEM_INFO_RESERVED no longer counted them. In an OpenFL game that reads 17 MB texture files while a level loads, glibc held 722 MB of free memory once the level was loaded, out of 1740 MB of anonymous memory. With this change the same point is at 1160 MB. The collector now counts what it frees to the C heap, the large buffers and the block groups, and calls malloc_trim once that reaches 16 MB. The trim runs on the thread that frees dropped buffers, after its batch, so no other thread waits for it: it took up to 26 ms in the game, for 360 MB. A collection that frees its buffers itself, as a compacting one does, trims before it returns. A collection with no buffers to drop still wakes the thread when a trim is due, for what the sweep freed. Tests: test/regression/LargeFreedMemoryReturned drops 384 MB of 4 MB buffers, after freeing some 8 MB ones so that glibc takes them from its heap. It stayed at 396 MB resident and now returns to 12 MB. It reads the resident size from VmRSS in /proc/self/status, and passes without measuring where that is missing and under ASan, whose allocator keeps freed memory in its quarantine. Co-authored-by: Chris Speciale <christopher.gaf@gmail.com>
1 parent 7842182 commit a125493

5 files changed

Lines changed: 158 additions & 12 deletions

File tree

‎Changes.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
libhxcpp 6.0.0 (unreleased)
33
------------------------------------------------------------
44

5+
* Fixed a program on Linux with glibc staying as large as its largest burst of large allocations, such as the files a game reads while it loads: glibc keeps a freed buffer's memory in its heap unless it had mapped the buffer on its own, and once a buffer mapped on its own has been freed it stops mapping those of that size or less, up to 32 MB. The memory of the large buffers a collection freed then stayed the process's. The collector now trims the C heap once 16 MB has been freed to it, on the thread that frees the buffers. A program that dropped 384 MB of 4 MB buffers stayed at 396 MB resident and now returns to 12 MB
56
* Fixed a native extension that includes hx/CFFI.h before hx/CFFIPrime.h, as SteamWrap does, failing to compile with hxs_encoding, hxs_utf16 and alloc_hxs_utf8 not declared: CFFI.h included first declares the CFFI API without HXCPP_PRIME, so without the Prime string functions that the ordinary wrapper of a DEFINE_PRIME function now uses. CFFIPrime.h declares them then
67
* Made the collector's thread context on Linux a __thread pointer, read with one load from the thread pointer, instead of a call to pthread_getspecific each time an allocation leaves the inline path or a String or number is boxed. It was 3 to 4% of an OpenFL game's main thread. HXCPP_PTHREAD_FAST_TLS keeps pthread_getspecific (libhxcpp #429)
78
* Fixed a call on a null Array<Dynamic> in cppia ending the process under HXCPP_CHECK_POINTER: push, contains, remove, indexOf, lastIndexOf, join, toString, sort, insert and unshift used the array without checking it, interpreted and under the JIT. They throw Null Object Reference, as a typed array's calls and the other Array<Dynamic> calls do

‎src/hx/gc/Immix.cpp‎

Lines changed: 76 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,16 @@
3232
#include <string>
3333
#include <stdlib.h>
3434

35+
// glibc keeps the memory of a freed buffer in its heap, unless the buffer was
36+
// mapped on its own, and it stops mapping buffers on their own up to the size
37+
// of the largest one freed so far, 32 MB at most on 64 bits. The memory of the
38+
// large buffers a collection frees then stays the process's, for the next
39+
// allocation of a size that fits, which may never come
40+
#if defined(__GLIBC__) && !defined(__EMSCRIPTEN__)
41+
#include <malloc.h>
42+
#define HX_GC_TRIM_FREED
43+
#endif
44+
3545
#if defined(__SANITIZE_ADDRESS__)
3646
#define HX_ADDRESS_SANITIZER
3747
#elif defined(__has_feature)
@@ -3840,6 +3850,11 @@ class AllocCounter : public hx::VisitContext
38403850

38413851

38423852
static const size_t sLargeAllocSpaceFloor = 40 << 20;
3853+
#ifdef HX_GC_TRIM_FREED
3854+
// The heap is trimmed once this much has been freed to it: a trim walks every
3855+
// free chunk of the heap, which is not worth it for less
3856+
static const size_t sTrimFreedBytes = 16 << 20;
3857+
#endif
38433858

38443859
class GlobalAllocator *sGlobalAlloc = 0;
38453860

@@ -4223,6 +4238,37 @@ class GlobalAllocator
42234238
return bytes;
42244239
}
42254240

4241+
// Counts memory handed to free(), for trimFreed
4242+
void noteFreed(size_t inBytes)
4243+
{
4244+
#ifdef HX_GC_TRIM_FREED
4245+
mFreedSinceTrim.fetch_add(inBytes, std::memory_order_relaxed);
4246+
#endif
4247+
}
4248+
4249+
bool trimDue()
4250+
{
4251+
#ifdef HX_GC_TRIM_FREED
4252+
return mFreedSinceTrim.load(std::memory_order_relaxed) >= sTrimFreedBytes;
4253+
#else
4254+
return false;
4255+
#endif
4256+
}
4257+
4258+
// Hands the free memory of the C heap back to the system once enough has
4259+
// been freed to it. Not called while the other threads wait: the trim
4260+
// takes time in the memory it gives back
4261+
void trimFreed()
4262+
{
4263+
#ifdef HX_GC_TRIM_FREED
4264+
if (trimDue())
4265+
{
4266+
mFreedSinceTrim.store(0, std::memory_order_relaxed);
4267+
malloc_trim(0);
4268+
}
4269+
#endif
4270+
}
4271+
42264272
// A collection drops the recycle cache it found, and frees it once every
42274273
// thread runs again: freeing tens of thousands of buffers took tens of
42284274
// milliseconds while they all waited (libhxcpp #332). A thread of its
@@ -4231,7 +4277,8 @@ class GlobalAllocator
42314277
// before it returns
42324278
void releaseDropped(RecycleLists &ioDropped, bool inNow)
42334279
{
4234-
if (ioDropped.empty() && !inNow)
4280+
// With nothing to free, the releaser still trims what the sweep freed
4281+
if (ioDropped.empty() && !inNow && !trimDue())
42354282
return;
42364283
#if !defined(HXCPP_SINGLE_THREADED_APP) && !defined(__EMSCRIPTEN__)
42374284
if (!inNow)
@@ -4264,21 +4311,31 @@ class GlobalAllocator
42644311
}
42654312
#endif
42664313

4267-
mDroppedBytes.fetch_sub(freeRecycle(ioDropped), std::memory_order_relaxed);
4314+
releaseBatch(ioDropped);
42684315

42694316
#if !defined(HXCPP_SINGLE_THREADED_APP) && !defined(__EMSCRIPTEN__)
4270-
std::unique_lock<std::mutex> lock(mReleaseLock);
4271-
while(!mReleaseQueue.empty())
42724317
{
4273-
RecycleLists waiting;
4274-
waiting.swap(mReleaseQueue.back());
4275-
mReleaseQueue.pop_back();
4276-
lock.unlock();
4277-
mDroppedBytes.fetch_sub(freeRecycle(waiting), std::memory_order_relaxed);
4278-
lock.lock();
4318+
std::unique_lock<std::mutex> lock(mReleaseLock);
4319+
while(!mReleaseQueue.empty())
4320+
{
4321+
RecycleLists waiting;
4322+
waiting.swap(mReleaseQueue.back());
4323+
mReleaseQueue.pop_back();
4324+
lock.unlock();
4325+
releaseBatch(waiting);
4326+
lock.lock();
4327+
}
4328+
mReleaseIdle.wait(lock, [this]() { return !mReleaseBusy; });
42794329
}
4280-
mReleaseIdle.wait(lock, [this]() { return !mReleaseBusy; });
42814330
#endif
4331+
trimFreed();
4332+
}
4333+
4334+
void releaseBatch(RecycleLists &ioBatch)
4335+
{
4336+
size_t bytes = freeRecycle(ioBatch);
4337+
mDroppedBytes.fetch_sub(bytes, std::memory_order_relaxed);
4338+
noteFreed(bytes);
42824339
}
42834340

42844341
#if !defined(HXCPP_SINGLE_THREADED_APP) && !defined(__EMSCRIPTEN__)
@@ -4295,7 +4352,8 @@ class GlobalAllocator
42954352
mReleaseQueue.pop_back();
42964353
mReleaseBusy = true;
42974354
lock.unlock();
4298-
mDroppedBytes.fetch_sub(freeRecycle(batch), std::memory_order_relaxed);
4355+
releaseBatch(batch);
4356+
trimFreed();
42994357
lock.lock();
43004358
mReleaseBusy = false;
43014359
mReleaseIdle.notify_all();
@@ -5168,6 +5226,7 @@ class GlobalAllocator
51685226
#endif
51695227

51705228
HxFreeGCBlock(g.alloc);
5229+
noteFreed(groupBytes);
51715230
g.alloc = 0;
51725231

51735232
groups++;
@@ -6754,6 +6813,7 @@ class GlobalAllocator
67546813
else
67556814
{
67566815
HxFree(blob);
6816+
noteFreed(size);
67576817
}
67586818

67596819
qeraseLarge(idx);
@@ -7628,6 +7688,10 @@ class GlobalAllocator
76287688
std::atomic_size_t mRecycleBytes;
76297689
// The data bytes of dropped caches not freed yet
76307690
std::atomic_size_t mDroppedBytes;
7691+
#ifdef HX_GC_TRIM_FREED
7692+
// Bytes freed to the C heap since it was last trimmed
7693+
std::atomic_size_t mFreedSinceTrim{0};
7694+
#endif
76317695
#if !defined(HXCPP_SINGLE_THREADED_APP) && !defined(__EMSCRIPTEN__)
76327696
std::mutex mReleaseLock;
76337697
std::condition_variable mReleaseWake;
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
import cpp.vm.Gc;
2+
import haxe.io.Bytes;
3+
4+
// glibc keeps a freed buffer's memory in its heap unless the buffer was mapped
5+
// on its own, and it stops mapping buffers on their own once one that size has
6+
// been freed. The collector hands the memory of the large buffers it frees back
7+
// to the system.
8+
class Main {
9+
static inline var MB = 1024 * 1024;
10+
11+
static var kept:Array<Bytes> = [];
12+
static var temporary:Array<Bytes>;
13+
14+
// Megabytes of the process in memory, or -1 where it cannot be measured:
15+
// without /proc, and under ASan, whose allocator replaces glibc's and keeps
16+
// freed memory in its quarantine
17+
static function resident():Float {
18+
#if (HXCPP_SANITIZE == "address")
19+
return -1;
20+
#else
21+
if (Sys.systemName() != "Linux" || !sys.FileSystem.exists("/proc/self/status"))
22+
return -1;
23+
// VmRSS is in kB whatever the page size, which is 16 or 64 KB on some arm64 kernels
24+
for (line in sys.io.File.getContent("/proc/self/status").split("\n")) {
25+
if (StringTools.startsWith(line, "VmRSS:"))
26+
return Std.parseFloat(StringTools.trim(line.substr(6)).split(" ")[0]) / 1024;
27+
}
28+
return -1;
29+
#end
30+
}
31+
32+
// A small buffer that lives follows each large one, so that none of the
33+
// freed memory is at the end of the heap, which glibc gives back by itself
34+
static function burst(count:Int, size:Int) {
35+
temporary = [];
36+
for (i in 0...count) {
37+
var bytes = Bytes.alloc(size);
38+
bytes.fill(0, size, 1);
39+
temporary.push(bytes);
40+
kept.push(Bytes.alloc(64 * 1024));
41+
}
42+
}
43+
44+
static function release() {
45+
temporary = null;
46+
// The first collection keeps the dead buffers for reuse, the second frees them
47+
Gc.run(true);
48+
Gc.run(true);
49+
}
50+
51+
static function main() {
52+
if (resident() < 0) {
53+
Sys.println("the burst was in memory: true");
54+
Sys.println("freed large buffers go back to the system: true");
55+
return;
56+
}
57+
// Freeing buffers mapped on their own raises the size from which glibc maps them
58+
burst(4, 8 * MB);
59+
release();
60+
var before = resident();
61+
burst(96, 4 * MB);
62+
var during = resident();
63+
release();
64+
// A thread of the collector's frees them
65+
var after = resident();
66+
var waited = 0.0;
67+
while (after - before > 96 && waited < 10) {
68+
Sys.sleep(0.05);
69+
waited += 0.05;
70+
after = resident();
71+
}
72+
Sys.println("the burst was in memory: " + (during - before > 300));
73+
Sys.println("freed large buffers go back to the system: " + (after - before <= 96));
74+
if (after - before > 96)
75+
Sys.println("resident before " + Math.round(before) + " MB, during " + Math.round(during) + " MB, after " + Math.round(after) + " MB");
76+
}
77+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
-m Main
2+
--cpp bin
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
the burst was in memory: true
2+
freed large buffers go back to the system: true

0 commit comments

Comments
 (0)