Skip to content

Commit 36d4828

Browse files
Merge the fixes for CI on Linux, the sanitizers and Haxe nightly
The Linux link of programs that use sys.ssl, a socket handle in stock hxcpp's layout as CrossByte makes one, the map walk counter that mark threads raced on, the moved-object address stored misaligned, two fuzz finds in the cppia loader and Date, and the runtime at API level 500. The TSan suppressions and the Alpine runner cover the debug regressions, and Run.hx shows a failing test's stderr, for the failures CI does not explain yet.
2 parents 1afcf1e + 18894ea commit 36d4828

24 files changed

Lines changed: 198 additions & 19 deletions

File tree

‎Changes.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22
libhxcpp 6.0.0 (unreleased)
33
------------------------------------------------------------
44

5+
* Fixed programs that use sys.ssl without sockets failing to link on Linux with an undefined reference to hxcpp_alloc_kind: hxcpp_std, linked a second time after hxcpp_ssl, came after the runtime's second copy. Libraries linked twice are now added again last group first, so the runtime's copy follows the libraries that call into it
6+
* Fixed closing, reading or polling a socket handle that a library made in stock hxcpp's layout, as CrossByte's accept does, reading past the handle: the runtime took its own wrapper's later fields from it, and a close wrote through what followed it in memory. Only the descriptor is read from such a handle, and the runtime's wrapper no longer shares its name, and with it its vtable, with a library's
7+
* Fixed a map that two marking threads marked at once being left unable to fold its buckets: the marker counted itself as a walk over the map without a lock, so the count could stay above zero. Marking no longer counts as a walk, as folding only happens after it (follow-up to libhxcpp #204)
8+
* Fixed the moving collector storing a moved object's new address as a pointer at an address only 4-byte aligned, which is undefined and which UBSan reported once string data stopped being 8-byte aligned. The address is copied in and out
9+
* Fixed a malformed cppia file whose function body was some other expression being run as a function. It fails to load with "Function body is not a function" (libhxcpp #257)
10+
* Fixed the fields and text of a Date whose time is not a number, or past what a year in a tm holds, converting the time to whole seconds, which is undefined. Such a time gives the fields of the epoch (libhxcpp #257)
11+
* Fixed the runtime not compiling at API level 500, which Haxe nightly builds use: a cppia module's release function defined only the __run that level 500 removed, and VirtualArray's closures compared arrays with a != that C++20 finds ambiguous
512
* Fixed converting a weak map to another value type losing live keys, about one in five in a map whose keys had mostly died: a collection during the conversion folded the map's buckets (#204), moving entries into buckets the conversion had already passed. A collection now leaves the fold to a later one while anything walks the map, which keys(), values(), toString and conversion do (libhxcpp #204)
613
* Fixed the moving collector (HXCPP_GC_MOVING) giving the whole heap back on Gc.run(true) and whenever the allocator's free list filled, as the other modes stopped doing in #230 and #231: a heap with a small live set stayed near 9 MB and collected far more often, and Gc.run(true) shrank it to 1 MB. Only Gc.compact() releases everything now; the collector still moves objects on those collections (libhxcpp #231)
714
* Sped up converting a Float to a String, which rounded it with printf to 15, 16 and 17 digits and read each back with strtod to find the shortest digits that read back. Grisu3 finds them with integer arithmetic, and the old search does the doubles it cannot decide, about 0.5% in Loitsch's measurements. The text is the same for every value: Std.string on Floats went from 996k to 8.8M conversions a second (libhxcpp #390)

‎src/Array.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1040,7 +1040,7 @@ namespace cpp
10401040
{ \
10411041
auto casted = dynamic_cast<const _hx_virtualarray_##name *>(inRhs); \
10421042
if (!casted) return 1; \
1043-
if (mThis != casted->mThis) return -1; \
1043+
if (mThis.GetPtr() != casted->mThis.GetPtr()) return -1; \
10441044
return 0; \
10451045
} \
10461046
}; \

‎src/hx/Date.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,13 @@ static bool hx_time_t_holds(double inSeconds)
123123
return sizeof(time_t) >= 8 || (inSeconds >= -2147483648.0 && inSeconds < 2147483648.0);
124124
}
125125

126+
// A time that is not a number, or whose year a tm cannot hold, gives the
127+
// fields of the epoch: converting it to whole seconds is undefined
128+
static double hx_field_seconds(double inSeconds)
129+
{
130+
return fabs(inSeconds) < 6.0e16 ? inSeconds : 0.0;
131+
}
132+
126133
static void hx_civil_from_days(long long z, long long *outY, int *outM, int *outD)
127134
{
128135
z += 719468;
@@ -311,6 +318,7 @@ static void hx_read_zone(hx_zone_state *outState)
311318
*/
312319
void __internal_localtime(double inSeconds, struct tm* time)
313320
{
321+
inSeconds = hx_field_seconds(inSeconds);
314322
// Date field getters (getHours, getMinutes, getFullYear, ...) each call this
315323
// on the SAME timestamp, so formatting one date would do N localtime() calls.
316324
// localtime is expensive (timezone lookup, CRT lock on Windows); a 1-entry
@@ -351,6 +359,7 @@ void __internal_localtime(double inSeconds, struct tm* time)
351359
*/
352360
void __internal_gmtime(double inSeconds, struct tm* time)
353361
{
362+
inSeconds = hx_field_seconds(inSeconds);
354363
// Same rationale as __internal_localtime: cache the last conversion so the
355364
// UTC field getters don't each re-run gmtime for one timestamp.
356365
static thread_local double sLastSeconds = 0;

‎src/hx/Hash.h‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -948,6 +948,14 @@ struct Hash : public HashBase< typename ELEMENT::Key >
948948
void iterate(F &inFunc)
949949
{
950950
AutoWalk walk(this);
951+
iterateBuckets(inFunc);
952+
}
953+
954+
// Without counting a walk, for the collector: it folds only after marking,
955+
// and mark threads can mark one map at once, so their ++ and -- would race
956+
template<typename F>
957+
void iterateBuckets(F &inFunc)
958+
{
951959
for(int b=0;b<bucketCount;b++)
952960
{
953961
Element *el = bucket[b];
@@ -1120,7 +1128,7 @@ struct Hash : public HashBase< typename ELEMENT::Key >
11201128
#endif
11211129

11221130
HashMarker marker(__inCtx);
1123-
iterate(marker);
1131+
iterateBuckets(marker);
11241132
}
11251133

11261134
#ifdef HXCPP_GC_INCREMENTAL

‎src/hx/cppia/CppiaFunction.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1167,7 +1167,12 @@ void CppiaFunction::load(CppiaStream &stream,bool inExpectBody)
11671167
DBGLOG(" arg %c%s:%s\n", arg.optional?'?':' ', cppia.identStr(arg.nameId), cppia.typeStr(arg.typeId) );
11681168
}
11691169
if (inExpectBody)
1170-
funExpr = (ScriptCallable *)createCppiaExpr(stream);
1170+
{
1171+
// A malformed file can give any expression as the body
1172+
funExpr = dynamic_cast<ScriptCallable *>(createCppiaExpr(stream));
1173+
if (!funExpr)
1174+
throw "Function body is not a function";
1175+
}
11711176
stream.module->creatingFunction = 0;
11721177
}
11731178

‎src/hx/cppia/CppiaModule.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -555,11 +555,14 @@ class CppiaObject : public hx::CppiaLoadedModule_obj
555555
{
556556
hx::Object *module;
557557
ReleaseFunc(hx::Object *inModule) : module(inModule) { }
558-
Dynamic __run() HXCPP_OVERRIDE
558+
Dynamic __Run(const Array<Dynamic> &inArgs) HXCPP_OVERRIDE
559559
{
560560
static_cast<CppiaObject *>(module)->release();
561561
return null();
562562
}
563+
#if (HXCPP_API_LEVEL<500)
564+
Dynamic __run() HXCPP_OVERRIDE { return __Run(null()); }
565+
#endif
563566
int __ArgCount() const HXCPP_OVERRIDE { return 0; }
564567
void __Mark(hx::MarkContext *__inCtx) HXCPP_OVERRIDE { HX_MARK_MEMBER(module); }
565568
#ifdef HXCPP_VISIT_ALLOCS

‎src/hx/gc/Immix.cpp‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4911,7 +4911,9 @@ class GlobalAllocator
49114911
memcpy(buffer, src, size);
49124912
//GCLOG(" move %p -> %p %d (%08x %08x )\n", src, buffer, size, buffer[-1], buffer[1] );
49134913

4914-
*(unsigned int **)src = buffer;
4914+
// An object, string data among them, need only be 4-byte
4915+
// aligned, so the pointer is copied rather than stored
4916+
memcpy(src, &buffer, sizeof(buffer));
49154917
header = IMMIX_OBJECT_HAS_MOVED;
49164918
moveObjs++;
49174919

@@ -4971,7 +4973,8 @@ class GlobalAllocator
49714973
if ( ((*(unsigned int **)ioPtr)[-1]) == IMMIX_OBJECT_HAS_MOVED )
49724974
{
49734975
//GCLOG(" patch object to %p -> %p\n", *ioPtr, (*(hx::Object ***)ioPtr)[0]);
4974-
*ioPtr = (*(hx::Object ***)ioPtr)[0];
4976+
// Copied: the moved object may be only 4-byte aligned
4977+
memcpy(ioPtr, *ioPtr, sizeof(*ioPtr));
49754978
//GCLOG(" %08x %08x ...\n", ((int *)(*ioPtr))[0], ((int *)(*ioPtr))[1] );
49764979
}
49774980
}
@@ -4981,7 +4984,7 @@ class GlobalAllocator
49814984
if ( ((*(unsigned int **)ioPtr)[-1]) == IMMIX_OBJECT_HAS_MOVED )
49824985
{
49834986
//GCLOG(" patch reference to %p -> %p\n", *ioPtr, (*(void ***)ioPtr)[0]);
4984-
*ioPtr = (*(void ***)ioPtr)[0];
4987+
memcpy(ioPtr, *ioPtr, sizeof(*ioPtr));
49854988
//GCLOG(" %08x %08x ...\n", ((int *)(*ioPtr))[0], ((int *)(*ioPtr))[1] );
49864989
}
49874990
}

‎src/hx/libs/std/Socket.cpp‎

Lines changed: 33 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,13 @@ struct ReadAheadLock
256256

257257
static void forgetHanded(ReadAhead *inAhead);
258258

259+
// The vtable of the wrappers made here
260+
static std::atomic<void *> sWrapperVtable(0);
261+
262+
// In an unnamed namespace: libraries that make socket handles of their own,
263+
// as CrossByte's accept does, declare a SocketWrapper too, and two classes of
264+
// one name would share a vtable
265+
namespace {
259266
struct SocketWrapper : public hx::Object
260267
{
261268
HX_IS_INSTANCE_OF enum { _hx_ClassId = hx::clsIdSocket };
@@ -270,8 +277,23 @@ struct SocketWrapper : public hx::Object
270277

271278
int __GetType() const HXCPP_OVERRIDE { return socketType; }
272279

280+
// A handle of the socket kind that a library made has stock hxcpp's
281+
// layout, the descriptor after the object header and nothing more, so
282+
// only the descriptor is read from it
283+
bool own() const
284+
{
285+
return *(void *const *)this == sWrapperVtable.load(std::memory_order_relaxed);
286+
}
287+
288+
ReadAhead *currentAhead()
289+
{
290+
return own() ? ahead.load(std::memory_order_acquire) : 0;
291+
}
292+
273293
ReadAhead *readAhead()
274294
{
295+
if (!own())
296+
return 0;
275297
ReadAhead *a = ahead.load(std::memory_order_acquire);
276298
if (!a && stream)
277299
{
@@ -286,10 +308,12 @@ struct SocketWrapper : public hx::Object
286308

287309
bool holdsBytes()
288310
{
289-
ReadAhead *a = ahead.load(std::memory_order_acquire);
311+
ReadAhead *a = currentAhead();
290312
return a && a->held.load(std::memory_order_relaxed);
291313
}
292314

315+
unsigned int currentSerial() const { return own() ? serial : 0; }
316+
293317
// A socket that is dropped without close still releases its descriptor
294318
static void finalize(Dynamic inObj)
295319
{
@@ -314,14 +338,17 @@ struct SocketWrapper : public hx::Object
314338
wrap->stream = inStream;
315339
wrap->serial = ++sNextSocketSerial;
316340
wrap->ahead = 0;
341+
sWrapperVtable.store(*(void **)wrap, std::memory_order_relaxed);
317342
_hx_set_finalizer(wrap, finalize);
318343
return wrap;
319344
}
320345
};
346+
}
321347

322348

323-
// The wrapper of a handle, or null for a sys.net.Socket without one
324-
SocketWrapper *val_wrapper(Dynamic inValue)
349+
// The wrapper of a handle, or null for a sys.net.Socket without one. A
350+
// library's handle of the socket kind comes back as one too, for its socket
351+
static SocketWrapper *val_wrapper(Dynamic inValue)
325352
{
326353
if (inValue.mPtr)
327354
{
@@ -581,7 +608,7 @@ void _hx_std_socket_close( Dynamic handle )
581608
sSocketsClosed++;
582609
}
583610
reset_sock(handle);
584-
ReadAhead *ahead = wrap ? wrap->ahead.load() : 0;
611+
ReadAhead *ahead = wrap ? wrap->currentAhead() : 0;
585612
if (ahead)
586613
{
587614
ahead->discard();
@@ -1761,7 +1788,7 @@ void _hx_std_socket_shutdown( Dynamic o, bool r, bool w )
17611788
// none, or fail on Windows, so the bytes read ahead go too: Windows still
17621789
// reports them as readable. Linux goes on giving what it holds, and the
17631790
// bytes read ahead come first, as they did.
1764-
ReadAhead *ahead = r && wrap ? wrap->ahead.load() : 0;
1791+
ReadAhead *ahead = r && wrap ? wrap->currentAhead() : 0;
17651792
if (ahead)
17661793
#ifdef NEKO_WINDOWS
17671794
ahead->discard(true);
@@ -1988,7 +2015,7 @@ struct PollQueue
19882015
FdSlots &at = slotOf[sock];
19892016
if (!at.pending)
19902017
{
1991-
Slot slot = { sock, inWrap->serial, 0, 0, -1, -1 };
2018+
Slot slot = { sock, inWrap->currentSerial(), 0, 0, -1, -1 };
19922019
slots.push_back(slot);
19932020
at.pending = (int)slots.size();
19942021
}
505 Bytes
Binary file not shown.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Av

0 commit comments

Comments
 (0)