Keep only the beginning and the end of a long token for error messages by nlohmann · Pull Request #6121 · nlohmann/json

nlohmann · 2026-11-13T13:39:50Z

Problem

For error messages (last read: '...'), the lexer keeps the text of the token it is reading. The token start moves only at strings and numbers, so after a string or number everything that follows (whitespace, structural characters, literals) still belongs to that token. Input that cannot be read again copies it while parsing: the std::istream and FILE* adapters in their carry buffer on every buffer refill, and the lexer in token_string for iterator and other input. Valid input such as 1 followed by 100,000,000 newlines therefore peaked at about 225 MB for std::istream, FILE* and input iterators (3.12.0: the same), and the raw text of a long string was kept a second time next to the parsed value.

Change

The text is only used for error messages, so it does not need to be kept in full:

token_text_buffer<CharType> (in input_adapters.hpp, outside the JSON_NO_IO guard) replaces the char-only token_carry_buffer (now an alias of token_text_buffer<char>) and the lexer's std::vector<char_type> token_string. It keeps the first 4096 and at least the last 4112 characters of a token. When it is full and holds more than 12,320 characters, it drops the characters in between instead of growing. That happens in make_room(), the out-of-line path that previously reallocated, so appending a character is still one comparison and a store. The capacity never exceeds 12,320 characters.
get_token_string() shows a token of more than 8192 characters as its first 4096 characters, ..., and its last 4096 characters. For contiguous input, which rebuilds the token from positions, it copies only these two parts instead of the whole range.
exceptions.md (parse_error.101) describes the shortened token.

Tokens of up to 8192 characters are shown exactly as before. Messages for longer tokens are the same for every kind of input; the test below checks this for std::string, std::istream, FILE* and std::list<char>.

Measurements

Peak resident memory for valid input of 1 followed by 100,000,000 newlines (reproducer in #6120, Apple clang 21 -O2):

input	develop	this PR
`std::istream`	224.8 MB	1.8 MB
`FILE*`	225.3 MB	1.8 MB
input iterator	225.3 MB	1.7 MB

The same input with a ? at the end (a parse error) peaks at 2.4 to 2.7 MB, and its what() has 65,684 bytes for all three inputs.

Parsing speed, instructions retired (develop vs. this PR, 20 parses each; std::list<char> 5), Apple clang 21 -O2 -DNDEBUG, arm64:

input	canada	citm_catalog	twitter
`std::string`	-0.06 %	+0.02 %	-0.12 %
`std::istream`	+0.03 %	-0.08 %	+0.06 %
`FILE*`	-0.03 %	-0.07 %	-0.01 %
`std::list<char>`	-16.97 %	-12.89 %	-14.11 %

The iterator path gets cheaper because the buffer's push_back does less than std::vector::push_back. A first version that kept std::vector and added a capacity check before each character cost +5 % to +22 % there, so it was dropped.

On Linux (GCC 15.2, -O2 -DNDEBUG, x86-64), exact instruction counts with callgrind (3 parses each):

input	canada	citm_catalog	twitter
`std::string`	+39	−42	−5
`std::istream`	+92	+105	+99
`FILE*`	−3,175	−2,247	−657
`std::list<char>`	−4.42 %	−3.19 %	−1.11 %

(for std::string, std::istream and FILE* these are instructions out of 70 to 420 million, i.e. below 0.001 %). Wall times on the same machine varied by ±4 % in both directions between identical builds, so they are not given.

Tests

unit-class_lexer.cpp: token_text_buffer directly, with characters added one by one and in pieces of different sizes (around the 4112-character boundary and larger than the capacity): the first 4096 and the last 4112 characters are kept, the size stays within the limit, pop_back() after dropping works, and clear() starts a new token.
unit-class_parser.cpp: the message of a long string, of a long whitespace run after a number, and of a token just below the limit, each for std::string, std::istream, FILE* (tmpfile) and std::list<char>; valid input with a long whitespace run. 8 of its assertions fail on develop.
With ASan and UBSan, C++11 and C++20: class_parser, class_lexer, stream-bulk-scan, file_input, user_defined_input, deserialization, wstring, diagnostics, diagnostic-positions, guard-page, precise-stream-position, ubjson, bjdata, contiguous_parser, testsuites, unicode, alt-string, noexcept, regression1, regression2, json_view. It also builds with JSON_NO_IO (C++11 -O0) and with GCC 16 without warnings. clang-tidy (CI configuration) is clean.

Breaking changes

No API change. The text of a parse error message changes for a token of more than 8192 characters: the middle of the token is replaced by .... The SAX parse_error() last_token argument is shortened in the same way. Message texts are not part of the API.

🤖 Generated with Claude Code

For "last read" in parse errors, the lexer keeps the text of the token it reads. The token start moves only at strings and numbers, so everything after a string or number belongs to it until the next one starts. Input that cannot be read again copies that text while parsing (the istream and FILE* adapters in their carry buffer, the lexer in token_string for iterators), so valid input of 1 followed by 100,000,000 newlines peaked at about 225 MB, and a long string was kept a second time. Replace the carry buffer and token_string by token_text_buffer, which keeps the first 4096 and at least the last 4112 characters of a token and drops the characters in between where it would otherwise grow, so appending stays as cheap as before. get_token_string() shows a token of more than 8192 characters as its first and last 4096 characters around "...", the same for every kind of input; contiguous input copies only these parts. The example input now peaks at 1.8 MB. Instruction counts of parsing from std::string, std::istream and FILE* are unchanged (clang 21 and GCC 15); parsing from std::list<char> needs 1 to 17 % fewer. Fixes #6120. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

        if (n != 0)
        {
-            std::memcpy(m_end, first, n);
+            std::memcpy(m_end, first, n * sizeof(CharType));


+        {
+            const std::size_t piece = n < tail() ? n : tail();
+            make_room(piece);
+            std::memcpy(m_end, first, piece * sizeof(CharType));


+            std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+            if (size != 0)
+            {
+                std::memcpy(data.get(), m_data.get(), size * sizeof(CharType));


+            }
+            else if (kind == 2)
+            {
+                FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)


nlohmann added this to the Release 3.13.0 milestone Nov 13, 2026

github-advanced-security Bot found potential problems Nov 13, 2026

View reviewed changes

github-actions Bot added documentation L tests labels Nov 13, 2026

Navigation Menu

Repository navigation

Keep only the beginning and the end of a long token for error messages - #6121

Keep only the beginning and the end of a long token for error messages#6121
nlohmann wants to merge 1 commit into
developfrom
fix/token-text-bounded

nlohmann commented Nov 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nlohmann commented Nov 13, 2026

Problem

Change

Measurements

Tests

Breaking changes

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants