Skip to content

Commit 919b735

Browse files
Fix API level 500 string and UTF-8 faults (#439)
cpp.encoding.Utf8.encode returns an empty array for the empty string (#406), refuses a code point that is no Unicode scalar value before writing (#407) and allocates exactly what it writes (#405). String::utf8_str into a view leaves room for the terminator and converts a UTF-16 string's whole length (#408), and String::create from a UTF-16 view stays inside it (#412). Utf8.decode gives U+FFFD for malformed input, as Haxe 4 does. Fixes #405, fixes #406, fixes #407, fixes #408, fixes #412. # Conflicts: # Changes.md
2 parents 360bdca + a1b84e9 commit 919b735

19 files changed

Lines changed: 387 additions & 52 deletions

File tree

‎Changes.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,12 @@ libhxcpp 6.0.0 (unreleased)
33
------------------------------------------------------------
44

55
* Fixed the incremental collector (HXCPP_GC_INCREMENTAL) freeing a thread object's arrays while the thread was being created, as the gc stress test's verifier caught on Windows. A slice could find the new object on the stack while its first array was being allocated, and the cycle could end before the constructor's barrier, with the arrays stored after that unscanned. The arrays are now made in a mutation scope, which the stop that finishes a cycle scans again
6+
* cpp.encoding.Utf8.decode replaces each malformed UTF-8 sequence with U+FFFD instead of throwing, with smart strings or without. Haxe 5's Bytes.toString and getString decode with it, so they threw for bytes that Haxe 4's, which go through String::create, decode with U+FFFD in the same places. Valid input still takes the decoder that counts first; only malformed input takes the replacing one. Utf8.codepoint still throws
7+
* Fixed String::utf8_str into a view, at API level 500, writing the bytes and then throwing when the view had room for them but not the terminator. It returns false before writing anything. It also stopped a UTF-16 string at its first NUL, which is a character, and reported that length (libhxcpp #408)
8+
* Fixed String::create from a UTF-16 view, at API level 500, completing a high surrogate at the end of the view with the unit after it. The view bounds it: a high surrogate the view does not complete throws Invalid UTF16, as a broken pair inside it did (libhxcpp #412)
9+
* Fixed cpp.encoding.Utf8.encode returning a null array for the empty string, at API level 500 (libhxcpp #406)
10+
* Fixed cpp.encoding.Utf8.encode writing bytes for a code point that is no Unicode scalar value, a surrogate or one past U+10FFFF, which UTF-8 cannot hold and the UTF-16 encoder refused. Both encoders, and both getByteCount, throw for one before writing anything, at API level 500 (libhxcpp #407)
11+
* Fixed Haxe 5's Bytes.ofString keeping an array of three bytes for each unit of a string that was not all ASCII: cpp.encoding.Utf8.encode allocated for the worst case and only shortened it, so a megabyte of ASCII with one other character held 3 MB. It counts the bytes and allocates them exactly (libhxcpp #405)
612
* Added timedWait to hx::thread::ConditionVariable, which Haxe 5's sys.thread.Condition.timedWait calls: with the Haxe nightly, every program that used Condition failed to compile. It returns true once a signal or broadcast came and false when the time ran out, and it waits out the wakeups MSVC's timed waits give just before the deadline with nothing signalled
713
* Fixed an exception from blit, setSize or resize under the cppia JIT ending the process instead of reaching the script's catch, on typed arrays and on Array<Dynamic>: their native calls had no exception bridge. The script catches it, as it does interpreted and in compiled code (libhxcpp #404)
814
* Fixed resize() and cpp.NativeArray.setSize() on an Array<Dynamic> under the cppia JIT evaluating the array expression twice, so a call in it ran twice and the array it returned the second time was the one resized. The JIT evaluates it once (libhxcpp #411)

‎src/String.cpp‎

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -965,21 +965,27 @@ String String::create(const::cpp::marshal::View<char>& buffer)
965965

966966
String String::create(const cpp::marshal::View<char16_t>& buffer)
967967
{
968-
auto start = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
969-
auto end = start + buffer.length;
970-
auto extra = 0;
968+
auto units = reinterpret_cast<const char16_t*>(buffer.ptr.ptr);
969+
auto length = int64_t{ 0 };
971970

972-
while (start < end) {
973-
if (Char16Advance(start) == false)
971+
// Up to a terminator, inside the view: a high surrogate is completed only
972+
// by a low one the view holds, and Char16Advance would read past its end
973+
while (length < buffer.length && units[length] != 0)
974+
{
975+
auto unit = units[length++];
976+
977+
if (IsUtf16Surrogate(unit))
974978
{
975-
// set extra to 1 so we don't include the null terminating character in the calculated length.
976-
extra = 1;
979+
if (IsUtf16LowSurrogate(unit) || length >= buffer.length || !IsUtf16LowSurrogate(units[length]))
980+
{
981+
hx::Throw(HX_CSTRING("Invalid UTF16"));
982+
}
977983

978-
break;
984+
length++;
979985
}
980986
}
981987

982-
return String::create(buffer.ptr.ptr, buffer.length - (end - start) - extra);
988+
return String::create(buffer.ptr.ptr, length);
983989
}
984990

985991
#endif
@@ -2592,6 +2598,27 @@ String _hx_utf8_to_utf16(const unsigned char *ptr, int inUtf8Len, bool addHash)
25922598

25932599
return String(str, char16Count);
25942600
}
2601+
#elif (HXCPP_API_LEVEL>=500)
2602+
// For cpp.encoding.Utf8.decode without smart strings, where a String keeps
2603+
// UTF-8: the bytes, with each malformed sequence replaced by U+FFFD as
2604+
// _hx_utf8_to_utf16 replaces it, so both give the same text
2605+
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len)
2606+
{
2607+
int bytes = 0;
2608+
const unsigned char *u = ptr;
2609+
const unsigned char *end = ptr + inUtf8Len;
2610+
while(u<end)
2611+
bytes += UTF8Bytes(DecodeAdvanceUTF8(u,end));
2612+
2613+
char *str = hx::NewString(bytes);
2614+
char *o = str;
2615+
u = ptr;
2616+
while(u<end)
2617+
UTF8EncodeAdvance(o,DecodeAdvanceUTF8(u,end));
2618+
str[bytes] = 0;
2619+
2620+
return String(str, bytes);
2621+
}
25952622
#endif
25962623

25972624

@@ -2886,16 +2913,10 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
28862913
#ifdef HX_SMART_STRINGS
28872914
if (isUTF16Encoded())
28882915
{
2916+
// All of it: a NUL in the string is a character, not its end
28892917
auto cursor = __w;
2890-
2891-
while (Char16Advance(cursor,false)) {}
2892-
2893-
auto calculated = cursor - __w - 1;
2894-
2895-
cursor = __w;
2896-
2897-
auto end = cursor + calculated;
2898-
auto chars = 0;
2918+
auto end = __w + length;
2919+
auto chars = 0;
28992920

29002921
while (cursor < end)
29012922
{
@@ -2926,7 +2947,8 @@ bool String::utf8_str(::cpp::marshal::View<char> buffer, int* outByteLength) con
29262947
}
29272948
#endif
29282949

2929-
if (buffer.length < length)
2950+
// The terminator too, before anything is written
2951+
if (buffer.length < length + 1)
29302952
{
29312953
return false;
29322954
}

‎src/cpp/encoding/Encodings.cpp‎

Lines changed: 78 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@
55

66
using namespace cpp::marshal;
77

8+
#if !defined(HX_SMART_STRINGS)
9+
// In String.cpp, beside _hx_utf8_to_utf16
10+
String _hx_utf8_replace_malformed(const unsigned char *ptr, int inUtf8Len);
11+
#endif
12+
813
bool cpp::encoding::Ascii::isEncoded(const String& string)
914
{
1015
if (null() == string)
@@ -142,6 +147,17 @@ namespace
142147
return 0xFFFD;
143148
}
144149
#endif
150+
151+
// A code point that is not a Unicode scalar value, a surrogate or past
152+
// U+10FFFF, has no UTF-8 or UTF-16 form. The UTF-16 encoder refused them
153+
// and the UTF-8 one wrote invalid bytes
154+
void checkScalar(char32_t codepoint)
155+
{
156+
if ((codepoint >= 0xD800 && codepoint < 0xE000) || codepoint > 0x10FFFF)
157+
{
158+
hx::Throw(HX_CSTRING("Invalid codepoint"));
159+
}
160+
}
145161
}
146162

147163
int cpp::encoding::Utf8::getByteCount(const null&)
@@ -152,6 +168,8 @@ int cpp::encoding::Utf8::getByteCount(const null&)
152168

153169
int cpp::encoding::Utf8::getByteCount(char32_t codepoint)
154170
{
171+
checkScalar(codepoint);
172+
155173
if (codepoint <= 0x7F)
156174
{
157175
return 1;
@@ -281,7 +299,8 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
281299

282300
if (0 == string.length)
283301
{
284-
return 0;
302+
// An empty array: 0 would be a null one
303+
return Array<uint8_t>(0, 0);
285304
}
286305

287306
if (string.isAsciiEncoded())
@@ -297,8 +316,17 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
297316
}
298317

299318
#if defined(HX_SMART_STRINGS)
300-
// estimate the utf8 length with an upper bound
301-
Array<uint8_t> out(string.length * 3, 0);
319+
// Counted first: an array of three bytes a unit, which resize only made
320+
// shorter, kept that capacity as long as the bytes lived, three times
321+
// what a mostly ASCII string needs (libhxcpp #405)
322+
auto bytes = getByteCount(string);
323+
324+
if (bytes > 0x7fffffff)
325+
{
326+
return hx::Throw(HX_CSTRING("String too long"));
327+
}
328+
329+
Array<uint8_t> out(static_cast<int>(bytes), 0);
302330
View<uint8_t> buffer(out->Pointer(), out->length);
303331

304332
auto units = string.raw_wptr();
@@ -310,8 +338,6 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
310338
k += encode(stringCodepoint(units, string.length, i), buffer.slice(k));
311339
}
312340

313-
out->resize(k);
314-
315341
return out;
316342
#else
317343
return hx::Throw(HX_CSTRING("Unexpected encoding error"));
@@ -320,6 +346,9 @@ Array<uint8_t> cpp::encoding::Utf8::encode(const String& string)
320346

321347
int cpp::encoding::Utf8::encode(char32_t codepoint, const cpp::marshal::View<uint8_t>& buffer)
322348
{
349+
// Before anything is written
350+
checkScalar(codepoint);
351+
323352
if (codepoint <= 0x7F)
324353
{
325354
buffer[0] = static_cast<uint8_t>(codepoint);
@@ -383,42 +412,58 @@ String cpp::encoding::Utf8::decode(const cpp::marshal::View<uint8_t>& buffer)
383412
{
384413
return Ascii::decode(buffer);
385414
}
386-
#if defined(HX_SMART_STRINGS)
387-
auto bytes = utf16Units(buffer) * sizeof(char16_t);
388-
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
389-
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
390-
int64_t k = 0;
391-
int64_t i = 0;
415+
if (buffer.length > 0x7fffffff)
416+
{
417+
return hx::Throw(HX_CSTRING("String too long"));
418+
}
392419

393-
while (i < buffer.length)
420+
// Malformed input decodes as Haxe 4's String::create and Bytes.toString
421+
// decode it, each malformed sequence as one U+FFFD, since Haxe 5's
422+
// Bytes.toString and getString come here. The strict decoder below
423+
// throws for one, and is kept for valid input, which it decodes faster
424+
auto ptr = static_cast<const unsigned char*>(buffer.ptr.ptr);
425+
auto len = static_cast<int>(buffer.length);
426+
#if defined(HX_SMART_STRINGS)
427+
try
394428
{
395-
auto p = codepoint(buffer.slice(i));
429+
auto bytes = utf16Units(buffer) * sizeof(char16_t);
430+
auto exact = static_cast<char16_t*>(::hx::InternalNew(bytes + sizeof(char16_t), false));
431+
auto output = View<uint8_t>(reinterpret_cast<uint8_t*>(exact), bytes);
432+
int64_t k = 0;
433+
int64_t i = 0;
396434

397-
i += getByteCount(p);
398-
k += Utf16::encode(p, output.slice(k));
399-
}
435+
while (i < buffer.length)
436+
{
437+
auto p = codepoint(buffer.slice(i));
438+
439+
i += getByteCount(p);
440+
k += Utf16::encode(p, output.slice(k));
441+
}
400442

401-
auto chars = static_cast<int>(k / sizeof(char16_t));
443+
auto chars = static_cast<int>(k / sizeof(char16_t));
402444

403-
((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
404-
exact[chars] = 0;
445+
((unsigned int*)exact)[-1] |= HX_GC_STRING_CHAR16_T;
446+
exact[chars] = 0;
405447

406-
return String(exact, chars);
448+
return String(exact, chars);
449+
}
450+
catch (Dynamic&)
451+
{
452+
return _hx_utf8_to_utf16(ptr, len, false);
453+
}
407454
#else
408-
// The bytes are kept as they are, but malformed input still throws, as
409-
// it does with smart strings. codepoint takes an encoded surrogate
410-
// (ED A0 80 to ED BF BF), which Utf16::encode rejects there, so it is
411-
// rejected here too
412-
for (int64_t i = 0; i < buffer.length;)
455+
// The bytes are kept as they are when they are valid. codepoint takes an
456+
// encoded surrogate (ED A0 80 to ED BF BF), which getByteCount refuses
457+
try
413458
{
414-
auto p = codepoint(buffer.slice(i));
415-
416-
if (0xD800 <= p && p < 0xE000)
459+
for (int64_t i = 0; i < buffer.length;)
417460
{
418-
hx::Throw(HX_CSTRING("Failed to read codepoint"));
461+
i += getByteCount(codepoint(buffer.slice(i)));
419462
}
420-
421-
i += getByteCount(p);
463+
}
464+
catch (Dynamic&)
465+
{
466+
return _hx_utf8_replace_malformed(ptr, len);
422467
}
423468

424469
// +1 for the terminator - the allocation is not zeroed and hxcpp
@@ -557,6 +602,8 @@ int cpp::encoding::Utf16::getByteCount(const null&)
557602

558603
int cpp::encoding::Utf16::getByteCount(char32_t codepoint)
559604
{
605+
checkScalar(codepoint);
606+
560607
return codepoint <= 0xFFFF ? 2 : 4;
561608
}
562609

‎test/native/tests/encoding/TestUtf8.hx‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -157,8 +157,18 @@ class TestUtf8 extends Test {
157157
}
158158

159159
public function test_decode_malformed_string() {
160-
for (hex in ['61626364656667f09d84', '6162636465666768c3', 'e3833061', '616263646566676880', 'eda080', 'c0af']) {
161-
Assert.raises(() -> Utf8.decode(Bytes.ofHex(hex).asView()));
160+
// Each malformed sequence becomes one U+FFFD, as Haxe 4's Bytes.toString
161+
// gives, since Haxe 5's decodes here
162+
var expected = [
163+
'61626364656667f09d84' => 'abcdefg\u{FFFD}',
164+
'6162636465666768c3' => 'abcdefgh\u{FFFD}',
165+
'e3833061' => '\u{FFFD}0a',
166+
'616263646566676880' => 'abcdefgh\u{FFFD}',
167+
'eda080' => '\u{FFFD}\u{FFFD}\u{FFFD}',
168+
'c0af' => '\u{FFFD}\u{FFFD}'
169+
];
170+
for (hex => text in expected) {
171+
Assert.equals(text, Utf8.decode(Bytes.ofHex(hex).asView()));
162172
}
163173
}
164174
}
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
// String::create from a UTF-16 view completed a high surrogate at the end of
2+
// the view with the unit past it (openfl/libhxcpp#412). It is API level 500,
3+
// so the checks are native code that passes below it.
4+
@:cppFileCode('
5+
#if (HXCPP_API_LEVEL>=500)
6+
#include <cpp/marshal/View.hpp>
7+
static bool boundedUtf16Prefix() {
8+
char16_t data[3] = {0xD83D, 0xDE00, 0};
9+
try {
10+
String value = String::create(cpp::marshal::View<char16_t>(data, 1));
11+
return value.length <= 1;
12+
} catch (...) { return true; }
13+
}
14+
static bool completeUtf16Pair() {
15+
char16_t data[3] = {0xD83D, 0xDE00, 0};
16+
String value = String::create(cpp::marshal::View<char16_t>(data, 2));
17+
// fromCharCode gives the string either way, smart strings or not
18+
return value == String::fromCharCode(0x1F600);
19+
}
20+
static bool terminatedUtf16View() {
21+
char16_t data[4] = {0x0100, 0, 65, 0};
22+
String value = String::create(cpp::marshal::View<char16_t>(data, 3));
23+
return value == String::fromCharCode(0x0100);
24+
}
25+
#else
26+
static bool boundedUtf16Prefix() { return true; }
27+
static bool completeUtf16Pair() { return true; }
28+
static bool terminatedUtf16View() { return true; }
29+
#endif
30+
')
31+
class Main {
32+
static function main() {
33+
var bounded = true;
34+
var pair = true;
35+
var terminated = true;
36+
bounded = untyped __cpp__("boundedUtf16Prefix()");
37+
pair = untyped __cpp__("completeUtf16Pair()");
38+
terminated = untyped __cpp__("terminatedUtf16View()");
39+
Sys.println("UTF16 construction stays inside the supplied view: " + bounded);
40+
Sys.println("a complete surrogate pair is preserved: " + pair);
41+
Sys.println("a NUL terminator ends native string construction: " + terminated);
42+
}
43+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
-m Main
2+
--cpp bin
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
UTF16 construction stays inside the supplied view: true
2+
a complete surrogate pair is preserved: true
3+
a NUL terminator ends native string construction: true

0 commit comments

Comments
 (0)