Skip to content

Commit 48038ad

Browse files
Merge main into feature/cppia-adopt-instances
# Conflicts: # Changes.md
2 parents cce2fed + 31ccfc1 commit 48038ad

36 files changed

Lines changed: 699 additions & 81 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -451,7 +451,7 @@ jobs:
451451
working-directory: test/regression
452452
run: |
453453
failed=""
454-
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcPacing IncrementalGcStartSweep IncrementalGcStats IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalWindowsHeaders GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
454+
for test in GcMutationRoots GcContainerMutation IncrementalGcSlices IncrementalGcContainers IncrementalGcArrayChunks IncrementalGcDequeMoves IncrementalGcPacing IncrementalGcStartSweep IncrementalGcStats IncrementalGcFinishBoundary IncrementalGcVerifier IncrementalWindowsHeaders GcWeakMapChain GenerationalFinalizerLifecycle GenerationalLargeBufferLifecycle; do
455455
echo "::group::incremental gc: $test"
456456
# The lifecycle fixtures' hxml selects generational mode; compile
457457
# the same Main with incremental mode here instead.

‎Changes.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,15 @@ libhxcpp 6.0.0 (unreleased)
33
------------------------------------------------------------
44

55
* Added adoptInstances to a loaded cppia module: called with the module it replaces, before that one is released, it moves the live instances of each of the old module's classes onto the new class of the same name when the two lay out their fields the same way, so code reloaded into a running process reaches objects made before the reload. Calls on them run the new code, through interfaces and dynamic calls too, and Type.getClass gives the new class. It returns a row for each old class with live instances: how many moved, and for a class whose fields changed, which field differs. Bound methods and closures made before keep the old code. See docs/CppiaModuleRelease.md (libhxcpp #265)
6+
* Fixed the incremental collector (HXCPP_GC_INCREMENTAL) freeing a thread object's arrays while the thread was being created, as the gc stress test's verifier caught on Windows. A slice could find the new object on the stack while its first array was being allocated, and the cycle could end before the constructor's barrier, with the arrays stored after that unscanned. The arrays are now made in a mutation scope, which the stop that finishes a cycle scans again
7+
* cpp.encoding.Utf8.decode replaces each malformed UTF-8 sequence with U+FFFD instead of throwing, with smart strings or without. Haxe 5's Bytes.toString and getString decode with it, so they threw for bytes that Haxe 4's, which go through String::create, decode with U+FFFD in the same places. Valid input still takes the decoder that counts first; only malformed input takes the replacing one. Utf8.codepoint still throws
8+
* Fixed String::utf8_str into a view, at API level 500, writing the bytes and then throwing when the view had room for them but not the terminator. It returns false before writing anything. It also stopped a UTF-16 string at its first NUL, which is a character, and reported that length (libhxcpp #408)
9+
* Fixed String::create from a UTF-16 view, at API level 500, completing a high surrogate at the end of the view with the unit after it. The view bounds it: a high surrogate the view does not complete throws Invalid UTF16, as a broken pair inside it did (libhxcpp #412)
10+
* Fixed cpp.encoding.Utf8.encode returning a null array for the empty string, at API level 500 (libhxcpp #406)
11+
* Fixed cpp.encoding.Utf8.encode writing bytes for a code point that is no Unicode scalar value, a surrogate or one past U+10FFFF, which UTF-8 cannot hold and the UTF-16 encoder refused. Both encoders, and both getByteCount, throw for one before writing anything, at API level 500 (libhxcpp #407)
12+
* Fixed Haxe 5's Bytes.ofString keeping an array of three bytes for each unit of a string that was not all ASCII: cpp.encoding.Utf8.encode allocated for the worst case and only shortened it, so a megabyte of ASCII with one other character held 3 MB. It counts the bytes and allocates them exactly (libhxcpp #405)
13+
* Fixed a malformed cppia file that calls a static function on a host interface crashing the loader: it looked the function up in the interface's script glue, which an interface does not have. The fuzz job found it
14+
* Added timedWait to hx::thread::ConditionVariable, which Haxe 5's sys.thread.Condition.timedWait calls: with the Haxe nightly, every program that used Condition failed to compile. It returns true once a signal or broadcast came and false when the time ran out, and it waits out the wakeups MSVC's timed waits give just before the deadline with nothing signalled
615
* Fixed an exception from blit, setSize or resize under the cppia JIT ending the process instead of reaching the script's catch, on typed arrays and on Array<Dynamic>: their native calls had no exception bridge. The script catches it, as it does interpreted and in compiled code (libhxcpp #404)
716
* Fixed resize() and cpp.NativeArray.setSize() on an Array<Dynamic> under the cppia JIT evaluating the array expression twice, so a call in it ran twice and the array it returned the second time was the one resized. The JIT evaluates it once (libhxcpp #411)
817
* Fixed indexOf and lastIndexOf on an Array<Dynamic> under the cppia JIT ignoring fromIndex: the JIT never generated it, so a call or exception in it was lost too. The search starts from it, as it does interpreted and in compiled code (libhxcpp #410)
@@ -19,6 +28,7 @@ libhxcpp 6.0.0 (unreleased)
1928
* Added HXCPP_PGO, profile-guided optimisation with MSVC, GCC and Clang: -D HXCPP_PGO=generate builds a program that counts what it runs, and -D HXCPP_PGO=use builds it again from what a run counted, with link-time optimisation. On a replication benchmark it took a further 4 to 29% off with MSVC, 19 to 43% with GCC and 12 to 46% with Clang. See docs/ProfileGuidedOptimisation.md
2029
* Fixed HXCPP_OPTIMIZE_LINK builds with Clang on Linux failing to link ("file format not recognized"): objects were compiled with -flto and linked without it, which GCC's linker plugin covered for, on one thread, and Clang's bitcode objects did not. Programs and shared libraries now link with -flto
2130
* Fixed an MSVC build with HXCPP_OPTIMIZE_LINK failing after any change (C1382, then LNK1257) until its objects were deleted: a group's PCH is rebuilt whenever one of its files compiles, and the link checks each object against the PCH it was compiled with. When the PCH is rebuilt under link-time code generation, the rest of the group is compiled against it too
31+
* Fixed the incremental collector (HXCPP_GC_INCREMENTAL) freeing items still in a sys.thread.Deque of more than 256 items, as the gc stress test's verifier caught about once in 75 runs: a cycle scans a long array in chunks, and the runtime's Deque moves its items itself, when a pop compacts the queue and when a push at the front makes room, which could carry items no chunk had reached into chunks already scanned. Those moves now have the queue scanned again, as Array's own moves do (libhxcpp #424)
2232
* Fixed hxcpp.gc.Incremental.getStats() returning null for six of its twelve counters when read by name, such as active and requestedSlices: they were fixed fields out of hash order, which a read by name searches. Printing the object and Json showed them, as those use the field's own name (libhxcpp #372)
2333
* The incremental collector no longer stops for a full reclaim every 15 collections, which took 5 ms on a 120 MB heap and 17 ms on an 800 MB one. Slices sweep a share of the blocks each collection instead, a finished cycle counts rows and leaves the sweep to the zeroing thread and the allocator, and a slice looks at the clock after each large job (libhxcpp #372)
2434
* Fixed the incremental collector doing no incremental work for a program that allocates mostly array storage, such as Bytes: a slice was skipped inside a mutation scope and a cycle could not finish in one, so every collection was a full stop. Slices run there now, and the stop that finishes a cycle scans the holders of open scopes again (libhxcpp #372)

‎include/hx/thread/ConditionVariable.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ namespace hx
1818
bool tryAcquire();
1919
void release();
2020
void wait();
21+
// True when woken, false when the time ran out
22+
bool timedWait(double seconds);
2123
void signal();
2224
void broadcast();
2325

‎src/String.cpp‎

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -965,21 +965,27 @@ String String::create(const::cpp::marshal::View<char>& buffer)
965965

966966
String String::create(const cpp::marshal::View<char16_t>& buffer)
967967
{
968-
auto start = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
969-
auto end = start + buffer.length;
970-
auto extra = 0;
968+
auto units = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
969+
auto length = int64_t{ 0 };
971970

972-
while (start < end) {
973-
if (Char16Advance(start) == false)
971+
// Up to a terminator, inside the view: a high surrogate is completed only
972+
// by a low one the view holds, and Char16Advance would read past its end
973+
while (length < buffer.length && units[length] != 0)
974+
{
975+
auto unit = units[length++];
976+
977+
if (IsUtf16Surrogate(unit))
974978
{
975-
// set extra to 1 so we don't include the null terminating character in the calculated length.
976-
extra = 1;
979+
if (IsUtf16LowSurrogate(unit) || length >= buffer.length || !IsUtf16LowSurrogate(units[length]))
980+
{
981+
hx::Throw(HX_CSTRING("Invalid UTF16"));
982+
}
977983

978-
break;
984+
length++;
979985
}
980986
}
981987

982-
return String::create(buffer.ptr.ptr, buffer.length - (end - start) - extra);
988+
return String::create(buffer.ptr.ptr, length);
983989
}
984990

985991
#endif
@@ -2592,6 +2598,27 @@ String _hx_utf8_to_utf16(const unsigned char *ptr, int inUtf8Len, bool addHash)
25922598

25932599
return String(str, char16Count);
25942600
}
2601+
#elif (HXCPP_API_LEVEL>=500)
2602+
// For cpp.encoding.Utf8.decode without smart strings, where a String keeps
2603+
// UTF-8: the bytes, with each malformed sequence replaced by U+FFFD as
2604+
// _hx_utf8_to_utf16 replaces it, so both give the same text
2605+
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len)
2606+
{
2607+
int bytes = 0;
2608+
const unsigned char *u = ptr;
2609+
const unsigned char *end = ptr + inUtf8Len;
2610+
while(u<end)
2611+
bytes += UTF8Bytes(DecodeAdvanceUTF8(u,end));
2612+
2613+
char *str = hx::NewString(bytes);
2614+
char *o = str;
2615+
u = ptr;
2616+
while(u<end)
2617+
UTF8EncodeAdvance(o,DecodeAdvanceUTF8(u,end));
2618+
str[bytes] = 0;
2619+
2620+
return String(str, bytes);
2621+
}
25952622
#endif
25962623

25972624

@@ -2886,16 +2913,10 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
28862913
#ifdef HX_SMART_STRINGS
28872914
if (isUTF16Encoded())
28882915
{
2916+
// All of it: a NUL in the string is a character, not its end
28892917
auto cursor = __w;
2890-
2891-
while (Char16Advance(cursor,false)) {}
2892-
2893-
auto calculated = cursor - __w - 1;
2894-
2895-
cursor = __w;
2896-
2897-
auto end = cursor + calculated;
2898-
auto chars = 0;
2918+
auto end = __w + length;
2919+
auto chars = 0;
28992920

29002921
while (cursor < end)
29012922
{
@@ -2926,7 +2947,8 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
29262947
}
29272948
#endif
29282949

2929-
if (buffer.length < length)
2950+
// The terminator too, before anything is written
2951+
if (buffer.length < length + 1)
29302952
{
29312953
return false;
29322954
}

‎src/cpp/encoding/Encodings.cpp‎

Lines changed: 78 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@
55

66
using namespace cpp::marshal;
77

8+
#if !defined(HX_SMART_STRINGS)
9+
// In String.cpp, beside _hx_utf8_to_utf16
10+
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len);
11+
#endif
12+
813
bool cpp::encoding::Ascii::isEncoded(const String& string)
914
{
1015
if (null() == string)
@@ -142,6 +147,17 @@ namespace
142147
return 0xFFFD;
143148
}
144149
#endif
150+
151+
// A code point that is not a Unicode scalar value, a surrogate or past
152+
// U+10FFFF, has no UTF-8 or UTF-16 form. The UTF-16 encoder refused them
153+
// and the UTF-8 one wrote invalid bytes
154+
void checkScalar(char32_t codepoint)
155+
{
156+
if ((codepoint >= 0xD800 && codepoint < 0xE000) || codepoint > 0x10FFFF)
157+
{
158+
hx::Throw(HX_CSTRING("Invalid codepoint"));
159+
}
160+
}
145161
}
146162

147163
int cpp::encoding::Utf8::getByteCount(const null&)
@@ -152,6 +168,8 @@ int cpp::encoding::Utf8::getByteCount(const null&)
152168

153169
int cpp::encoding::Utf8::getByteCount(char32_t codepoint)
154170
{
171+
checkScalar(codepoint);
172+
155173
if (codepoint <= 0x7F)
156174
{
157175
return 1;
@@ -281,7 +299,8 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
281299

282300
if (0 == string.length)
283301
{
284-
return 0;
302+
// An empty array: 0 would be a null one
303+
return Array<uint8_t>(0, 0);
285304
}
286305

287306
if (string.isAsciiEncoded())
@@ -297,8 +316,17 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
297316
}
298317

299318
#if defined(HX_SMART_STRINGS)
300-
// estimate the utf8 length with an upper bound
301-
Array<uint8_t> out(string.length * 3, 0);
319+
// Counted first: an array of three bytes a unit, which resize only made
320+
// shorter, kept that capacity as long as the bytes lived, three times
321+
// what a mostly ASCII string needs (libhxcpp #405)
322+
auto bytes = getByteCount(string);
323+
324+
if (bytes > 0x7fffffff)
325+
{
326+
return hx::Throw(HX_CSTRING("String too long"));
327+
}
328+
329+
Array<uint8_t> out(static_cast<int>(bytes), 0);
302330
View<uint8_t> buffer(out->Pointer(), out->length);
303331

304332
auto units = string.raw_wptr();
@@ -310,8 +338,6 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
310338
k += encode(stringCodepoint(units, string.length, i), buffer.slice(k));
311339
}
312340

313-
out->resize(k);
314-
315341
return out;
316342
#else
317343
return hx::Throw(HX_CSTRING("Unexpected encoding error"));
@@ -320,6 +346,9 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
320346

321347
int cpp::encoding::Utf8::encode(char32_t codepoint, const cpp::marshal::View<uint8_t>& buffer)
322348
{
349+
// Before anything is written
350+
checkScalar(codepoint);
351+
323352
if (codepoint <= 0x7F)
324353
{
325354
buffer[0] = static_cast<uint8_t>(codepoint);
@@ -383,42 +412,58 @@ String cpp::encoding::Utf8::decode(const cpp::marshal::View<uint8_t>& buffer)
383412
{
384413
return Ascii::decode(buffer);
385414
}
386-
#if defined(HX_SMART_STRINGS)
387-
auto bytes = utf16Units(buffer) * sizeof(char16_t);
388-
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
389-
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
390-
int64_t k = 0;
391-
int64_t i = 0;
415+
if (buffer.length > 0x7fffffff)
416+
{
417+
return hx::Throw(HX_CSTRING("String too long"));
418+
}
392419

393-
while (i < buffer.length)
420+
// Malformed input decodes as Haxe 4's String::create and Bytes.toString
421+
// decode it, each malformed sequence as one U+FFFD, since Haxe 5's
422+
// Bytes.toString and getString come here. The strict decoder below
423+
// throws for one, and is kept for valid input, which it decodes faster
424+
auto ptr = static_cast<const unsigned char*>(buffer.ptr.ptr);
425+
auto len = static_cast<int>(buffer.length);
426+
#if defined(HX_SMART_STRINGS)
427+
try
394428
{
395-
auto p = codepoint(buffer.slice(i));
429+
auto bytes = utf16Units(buffer) * sizeof(char16_t);
430+
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
431+
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
432+
int64_t k = 0;
433+
int64_t i = 0;
396434

397-
i += getByteCount(p);
398-
k += Utf16::encode(p, output.slice(k));
399-
}
435+
while (i < buffer.length)
436+
{
437+
auto p = codepoint(buffer.slice(i));
438+
439+
i += getByteCount(p);
440+
k += Utf16::encode(p, output.slice(k));
441+
}
400442

401-
auto chars = static_cast<int>(k / sizeof(char16_t));
443+
auto chars = static_cast<int>(k / sizeof(char16_t));
402444

403-
((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
404-
exact[chars] = 0;
445+
((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
446+
exact[chars] = 0;
405447

406-
return String(exact, chars);
448+
return String(exact, chars);
449+
}
450+
catch (Dynamic&)
451+
{
452+
return _hx_utf8_to_utf16(ptr, len, false);
453+
}
407454
#else
408-
// The bytes are kept as they are, but malformed input still throws, as
409-
// it does with smart strings. codepoint takes an encoded surrogate
410-
// (ED A0 80 to ED BF BF), which Utf16::encode rejects there, so it is
411-
// rejected here too
412-
for (int64_t i = 0; i < buffer.length;)
455+
// The bytes are kept as they are when they are valid. codepoint takes an
456+
// encoded surrogate (ED A0 80 to ED BF BF), which getByteCount refuses
457+
try
413458
{
414-
auto p = codepoint(buffer.slice(i));
415-
416-
if (0xD800 <= p && p < 0xE000)
459+
for (int64_t i = 0; i < buffer.length;)
417460
{
418-
hx::Throw(HX_CSTRING("Failed to read codepoint"));
461+
i += getByteCount(codepoint(buffer.slice(i)));
419462
}
420-
421-
i += getByteCount(p);
463+
}
464+
catch (Dynamic&)
465+
{
466+
return _hx_utf8_replace_malformed(ptr, len);
422467
}
423468

424469
// +1 for the terminator - the allocation is not zeroed and hxcpp
@@ -557,6 +602,8 @@ int cpp::encoding::Utf16::getByteCount(const null&)
557602

558603
int cpp::encoding::Utf16::getByteCount(char32_t codepoint)
559604
{
605+
checkScalar(codepoint);
606+
560607
return codepoint <= 0xFFFF ? 2 : 4;
561608
}
562609

‎src/hx/Thread.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,10 @@ struct Deque : public Array_obj<Dynamic>
5050
}
5151
else if (head > 64 && head >= length - head)
5252
{
53-
// Compact once the consumed prefix dominates - O(1) amortized
53+
// Compact once the consumed prefix dominates - O(1) amortized.
54+
// An incremental cycle may have scanned the slots the items move
55+
// to and not those they come from, so it scans the queue again
56+
HX_INCREMENTAL_MUTATION(this);
5457
int remaining = length - head;
5558
::memmove(items, items + head, remaining * sizeof(Dynamic));
5659
::memset(items + remaining, 0, head * sizeof(Dynamic));
@@ -79,6 +82,8 @@ struct Deque : public Array_obj<Dynamic>
7982
// building an n-deep queue that way was O(n^2) while holding the lock
8083
void openFront()
8184
{
85+
// The items move, as for a compaction in popHead
86+
HX_INCREMENTAL_MUTATION(this);
8287
int count = length;
8388
int gap = count >> 1;
8489
if (gap < 16)

‎src/hx/cppia/Cppia.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2141,7 +2141,9 @@ struct CallStatic : public CppiaExpr
21412141

21422142
if (!replace && type->haxeClass.mPtr)
21432143
{
2144-
ScriptNamedFunction func = type->haxeBase->findStaticFunction(field);
2144+
// A host interface has a class but no script glue, and only a
2145+
// malformed file calls a static on one
2146+
ScriptNamedFunction func = type->haxeBase ? type->haxeBase->findStaticFunction(field) : ScriptNamedFunction();
21452147
if (func.signature)
21462148
{
21472149
//CPPIALOG(" found function %s\n", func.signature );

0 commit comments

Comments
 (0)