[nlohmann]

unflatten() takes quadratic time in the number of reference tokens of one key on current develop (3.13.0 work in progress; 3.12.0 is linear). An object with one key of 100,000 /0 tokens (200 KB) takes 2.4 s; 3.12.0 needs 5 ms. The time quadruples with each doubling of the key. An attacker who controls one key of an object passed to unflatten() can make the process busy for minutes with a few megabytes of input. unflatten.md promises "Linear in the size of the JSON value".

Cause

3.13.0 decides in a first pass which prefixes of the keys denote arrays (those that have a reference token 0 among their children), so that the result no longer depends on the order of the keys, and looks the prefixes up again in the second pass. A prefix was identified by hashing it and by comparing it byte by byte with the set entry, from its beginning, so a key with n tokens cost n²/2 bytes. For ordered_json (and any unsorted object type) the shortcut for keys that share a beginning with the previous key does not help: four keys were enough to be quadratic.

Change

• Prefixes shorter than 64 bytes stay in the hash table and are compared as before (a bounded number of bytes).
• Longer prefixes are kept in a tree whose nodes are looked up by their parent and the text of one token. The node of a prefix is found by walking the tokens of the key from the root, so all prefixes of a key together cost one pass over the key and no prefix is compared as a whole. The nodes along the key walked last are kept, so keys with a common beginning do not walk it again. No recursion.
• The hash function is seeded with a value derived from the address of a local variable (ASLR changes it between processes). With the previous fixed constants, crafted keys could be made to collide: 20,000 keys of 18 bytes with equal hash took 0.5 s instead of 7 ms. Where addresses are not randomized, or if the seed is known, k short prefixes (< 64 bytes) can still collide and cost k²/2 comparisons of fewer than 64 bytes. Results never depend on the seed.
• The array/object decision and the independence from the key order are unchanged. Only private helpers of detail::json_pointer changed.

Measurements (Apple clang -O2, arm64)

unflatten() of one key with n tokens /0:

	n=25k	n=50k	n=100k	n=200k
before	175 ms	617 ms	2437 ms	9729 ms
after	4 ms	6 ms	12 ms	27 ms
3.12.0	1 ms	3 ms	5 ms	10 ms

ordered_json, 4 keys (/0×m /a0, /10, /0×m /a1, /11): before 397 / 1402 / 5191 / 20723 ms for m = 25k / 50k / 100k / 200k, after 4 / 7 / 15 / 33 ms, 3.12.0 2 / 3 / 8 / 17 ms.

Ordinary documents (flatten() then unflatten(), instructions retired): within 0.3 % of before for canada, citm_catalog, twitter, jeopardy, mesh, update-center and random. A realistic deep document (100 levels, 100k leaves): +1.5 % instructions, +5-12 % wall time vs develop, and about 9× faster than 3.12.0.

Known trade-off: two thousand different long keys (2000 keys × 2000 unique tokens, 12 MB of keys) take 1.4 s instead of 3.1 s on develop, but still about 6× longer than 3.12.0 (0.2 s) and use about 300 MiB more memory for the tree (about 78 bytes per token). 3.13.0 needs a set entry per prefix for the order independence anyway; path compression of unique tails would reduce this further and is left for a follow-up.

Tests

unit-json_pointer.cpp: semantics of the array/object decision (prefixes around the 64-byte hand-over, different token splits of one prefix, escapes, tokens longer than a word), a random round trip in shuffled key order with ordered_json, and a time-ratio check (best of 3 CPU-time runs, the same keys with an array-hitting token vs. token a; seven shapes including the 4-key ordered_json shape; ratio about 3 now, 25 to 300 on the previous code, bound 6 or 10). The test fails on develop and on the intermediate version that still had the ordered_json problem. It was run 50× 8 in parallel and 64× 16 in parallel at -O0 and -O1 without a failure.

Differential tests against develop (about 150,000 randomized flattened objects, json and shuffled ordered_json, with JSON_DIAGNOSTICS) gave identical results and error messages. Allocation failure was injected at every allocation of one unflatten() call: no leaks. The suite passes under ASan and UBSan; clang-tidy 22, clang -Weverything and g++ 16 report nothing new.

Also: a key that is a prefix of an earlier key (#6105)

This PR also fixes #6105, which is in the same function. With an object type that keeps the insertion order (ordered_json), a key such as /a can come after /a/b. The final assignment then replaced the object built for /a/b without an error: {"/a/b":2,"/a":1} gave {"a":1}, {"/a/0/b":1,"/a/0":2} gave {"a":[2]} and {"/x":1,"":2} gave 2. json never sees this, because std::map visits the shorter key first and the walk of the longer key throws type_error.313. 3.12.0 has the same bug.

unflatten() now throws type_error.313 when the target of the final assignment is no longer null, so the result no longer depends on the key order. A null value keeps the container, as it does for json when the shorter key comes first: {"/a/b":2,"/a":null} gives {"a":{"b":2}} for both object types. This case is not exotic, because flatten() turns empty objects and arrays into null.

The branch is not reached for json or any sorted object type. continue skips only the assignment, so the cached prefix nodes stay valid. Tests in unit-json_pointer.cpp cover the three inputs above, the reverse order, and the null cases for json and ordered_json; 5 of the new assertions fail without the change. unflatten.md documents the new type_error.313 case and has a version-history note.

Fixes #6105.

Breaking changes

No breaking changes to the API. The speed-up changes only private helpers. With an object type that keeps the insertion order, an input with a key that is a prefix of an earlier key now throws the documented type_error.313 instead of silently dropping data (bug fix, #6105); json behaves as before.

🤖 Generated with Claude Code