number suffix type annotations

[zkat]

Fixes: #510

[zkat]

@tabatkins honestly the most heartbreaking thing about all this to me is that I can't have 123u32 and such :( Gotta do 123#u32

[zkat]

@bgotink does this align with your understanding/experience/what you did in your lib?

[zkat]

alright, tests added. This is ready for final review.

[clarfonthey]

Just for readability purposes, I think it's worth mentioning the # escape hatch here with a reference to the "Explicit Suffix Type Annotation" section, at least when it comes to types like u32. For example, maybe:

To remove ambiguity, some suffixes must be prefixed with #: for example, 10.0u8 is invalid, but 10.0#u8 is. The full list of rules for invalid suffixes is clarified in the "Explicit Suffix Type Annotation" section.

[tabatkins]

I dropped a suggestion for this.

[clarfonthey]

Ah, GitHub collapsed it as "outdated" because they're bad

[tabatkins]

r+ after review of suggested changes

[tabatkins]

I dropped a suggestion for this.

[bgotink]

This matches what I implemented apart from one test that appears to be wrong and one mistake on my end where the parser skips certain validations on # suffixes which makes 123#123 equivalent to ("123")123 which is wrong.

[bgotink]

integer can end on an underscore so this is actually valid and equivalent to (abc)123_

[tabatkins]

correct. this is why we can't start a bare suffix with _, becuase the syntax would parse differently than intended.

[bgotink]

The following tests that previously failed now run successfully:

test name	test	equivalent document
bare_ident_numeric_fail.kdl	node 0n	node (n)0
bare_ident_numeric_sign_fail.kdl	node +0n	node (n)+0
illegal_char_in_binary_fail.kdl	node 0bx01	node (bx01)0
multiple_x_in_hex_fail.kdl	node 0xx10	node (xx10)0
no_digits_in_hex_fail.kdl	node 0x	node (x)0

[zkat]

Uggghhhhh. That makes sense. Looks like we’re gonna need to be more specific what order things run in. I have an idea for the grammar.

[zkat]

@bgotink

The following tests that previously failed now run successfully:

Looking at this...

These should be dropped because they're ok now:

test name test equivalent document
bare_ident_numeric_fail.kdl node 0n node (n)0
bare_ident_numeric_sign_fail.kdl node +0n node (n)+0

These tests should stay and continue to fail. Once you've hit a 0b/0o/0x number prefix, you SHOULD only be able to parse their related number formats:

test name test equivalent document
illegal_char_in_binary_fail.kdl node 0bx01 node (bx01)0
multiple_x_in_hex_fail.kdl node 0xx10 node (xx10)0
no_digits_in_hex_fail.kdl node 0x node (x)0

[zkat]

uggghhhh I understand why those last 3 tests pass now. I want to do something about it, though. I don't like that. I wonder if there's a good way around it.

[zkat]

@bgotink @tabatkins I've... kind of changed the rules a bit. They're simpler now. And most importantly: we have 0u64 available now!

I checked #510 and I didn't see any discussion about us tackling these rules, because we were focusing so much on what the rules for the suffix should be that we didn't take a step back and think of these numbers as a whole, and how the number syntax in general could be addressed.

Please lmk what you think. I think with these changes, we'll get the results I expected in #513 (comment)

[zkat]

Some key changes:

• I reorganized things a bit in general.
• The grammar itself now blocks simultaneous suffix and prefix annotations
• The underscore _ initial character restriction was removed: a bare suffix CAN'T start with one because the integer will slurp it up first
• The complex rules meant to disambiguate from non-decimals were removed. The fact that we only allow bare prefixes on decimals is sufficient to protect us here imo. I think these complex rules formed because things were moving fast and we didn't take a step back and really rethink the implications of only doing decimals. This has been clarified in the spec. With the new rules, there is no dangerous ambiguity, just potential syntax errors on certain zero values, which is notably NOT an unexpected parsing success, which I think is the dangerous bit.
• The rules around exponential-likes have been changed a bit to guard against small typos if you miss the digit part (so 1e+ is illegal)

[zkat]

I'm also wondering: could we just drop the exponent restriction (but keep the e+ protection, in case someone fails to write that digit and ends up with an unfortunate parse)

[mwh]

I don't think the specification prose as currently written does require or even allow a trailing _ to be consumed by the number ("digits ... may be separated by _"), but the authoritative grammar does include arbitrary underscores in place of digits anywhere except the beginning.

I think either the description of the grammatical language needs tightening, or the grammar may now allow some undesirable constructions with underscores and suffixes. Specifically, I'm not sure whether * is expected to commit a parser to whatever it finds the first time, or whether it can backtrack to allow later productions to match. Here, it would be backtracking to an underscore after the input didn't match when the underscore was consumed as part of a number.

Consider 12_3,x. The overall match would fail when (digit | '_')* from the integer production consumes _3; does it then backtrack and try consuming less of the input? If it stops short after "12" and leaves "_3,x", the whole input matches suffixed-decimal successfully with significand accepting "12" and bare-type-suffix accepting "_3,x" — so the result would be equivalent to (_3,x)12, though I think it should be an error. 12_3.4 and 12_3.4.5 differ at the same point, or consider 1_234,567. All of these can be produced from the grammar, at least.

I'm not sure whether I am misreading the description of the grammar language. * consumes "as many instances as possible without failing the match" — is that failing the match of the whole input, or does it just mean as many instances as are present at this point in the input and failure isn't really part of it? The comparison to standard regex semantics and the existence of cut points makes me think it can shorten if needed. If it does commit early to the longest sequence it finds, this issue doesn't come up. Otherwise, for integer to definitively slurp the underscore up and make this an error, I think there would need to be a cut point suffixed-decimal := significand ¶ (bare-type-suffix | (exponent? explicit-type-suffix)).

If there is an issue, either banning _ as the initial character of a suffix again or solidifying the grammatical handling of integers could address it. I am not personally a fan of baking parsing and backtracking rules into a grammar and would probably just block _, but there are reasons to have that kind of grammar too, particularly to constrain where an error is detected.

Something shaped like 1_234,567 seems like the primary case where this is realistic and actually matters, or someone trying to write a list 1_234, 5_678, .... I do think the grammar should rule these out, but if I were implementing a parser directly off this grammar right now, I would end up with this backtracking and unknowingly accepting these cases because nothing commits the parse to the path that produces an error. If I built the parser with a lexer in the front, I think the lexer would probably consume the whole number as expected and then I wouldn't encounter the issue. Clearly one of those is wrong, but it's not good for compatibility if both seem reasonable to make.

If nothing else, test cases including both underscores within the number and invalid suffixes will be good to ensure that the incorrect readings are detected.

---

I also wonder in a similar way about 0xaz. The prose does rule this out. This time it's the grammar semantics of - in significand-initial I'm not certain about: "any digit except something that matches the literal '0x'" seems like it'd be the same as just digit alone, because "0x" is not a digit and is not matched by digit. The intention here is clear, just the formalism may not match up.

[tabatkins]

This use of - isn't quite right - it's not subtracting from the LHS (the digit), it's imposing a wider restriction on the digit and its following character (which in this grammar would come from the start of bare-type-suffix).

I think this should instead be handled as a cut point on the hex/octal/binary productions, so that if you see their prefix you're committed. (We'll also have to specify that the ordering of | alternates is significant when cuts are involved, so normal-value is parsed first and gets to claim that syntax space with a cut before suffixed-decimal can run and claim it.)

[tabatkins]

Consider 12_3,x [snip discussion about why this successfully matches the current grammar]

Yup, the _ prefix restriction was actually load-bearing unless we introduce more cut-points, which I'd like to avoid except when necessary. (They're easy to forget about and then accidentally have side-effects.) We need to bring it back for the grammar (even if it might not be worth calling out in the prose, tho I actually do think some discussion about prefixes that won't work as expected due to parsing rules is worthwhile in the prose).

[tabatkins]

I think these complex rules formed because things were moving fast and we didn't take a step back and really rethink the implications of only doing decimals. This has been clarified in the spec. With the new rules, there is no dangerous ambiguity, just potential syntax errors on certain zero values, which is notably NOT an unexpected parsing success, which I think is the dangerous bit.

This isn't correct, there's still an unexpected parsing success in some cases. See 20x1, for example - totally valid under these rules (as expected), equivalent to (x1)20. But if the value becomes 0, it misparses - 0x1 parses equivalent to 1, not (x1)0. That's the scenario we were afraid of, and the reason those restrictions existed.

If the part following the x doesn't look like hex digits (20xfoo), then sure, when the value is 0 it'll just be a parse failure rather than a misparse. (We were somewhat worried about that as a usability issue, but I agree it can be a reasonable bullet to bite.) But the misparse when the following does look like hex digits is the real problem.

[tmke8]

This seems like a very useful feature but appears stuck at the moment. Would something like the following work?

• Bare Suffixes may not contain digits (this should remove most ambiguities, including exponents)
• additionally, a Bare Suffix may not start with an x followed by [a-fA-F] (i.e., hex letters)
  • this is to prevent the surprise from having 20xdecaf interpreted as (xdecaf)20 but 0xdecaf interpreted as hex
• let trailing underscores be swallowed by the number
• Explicit Suffixes just need to be valid Identifier Strings
• optional but probably useful restrictions (can all be worked around by using Explicit Suffixes with #):
  • also ban . and , as the first character of a Bare Suffix, because that's most likely a typo
  • a typo guard for incomplete exponents: 1e+ would be disallowed

These would then just work:

• 1e3em = (em)1000
• 0bytes = (bytes)0
• 0ohm = (ohm)0
• 123_abc = (abc)123

These would need an Explicit Suffix with #:

• 3#u8
• 10#km2
• 20#xdecaf
• 20#x1
• 1#.abc
• 1#e+

In the test cases mentioned above, node 0n and node +0n would be allowed, while:

• node 0bx01 -> invalid because the suffix contains digits
• node 0xx10 -> invalid, same reason
• node 0x -> valid, equivalent to node (x)0; I think there's no reason to make it invalid because there's no surprise with 20x and the similar 0b is also allowed

[zkat]

honestly it's less that it's stuck on specification, and more that it's stuck on at least one implementation. I started it in kdl-rs but dropped that work for now. If anyone ends up picking that up, it would help us figure out a lot of issues.

[bgotink]

My js implementation has support for one of the many versions discussed in this thread (honestly don't remember the exact details, it's been a while, the tests reference 07788d1 so probably the version as described in that commit)

in the console on https://github.bram.dev/kdl/

> kdl.parse('10u32', {as: "value", flags: {experimentalSuffixedNumbers: true}})
< e {
    type: 'value',
    representation: '10',
    tag: e {type: 'tag', representation: 'u32', leading: undefined, trailing: undefined, suffix: true, …},
    betweenTagAndValue: undefined,
    value: 10
  }

If there's interest to toy with this I could update the implementation to the latest version in this PR, and we could then easily set this up under a flag on the playground so people can get a feel for what's being proposed.

[zkat]

getting this on the playground would be awesome tbh

[mwh]

I also have an implementation of this in https://github.com/mwh/kdly. It's gated behind a hard-coded boolean switch in the lexer (line 7, experimental_suffix_type_annotations = False) but with that toggled will accept them.

I believe it implements the latest version as discussed here. It accepts and rejects the examples from the specification. Of the new examples above, it does not permit 1e3em, 0bytes, or 1e+, but 123_abc is (abc)123 and 1ohm is (ohm)1, but it does allow 3u8, 20xdecaf. It treats 1e_10 as (e_10)1.

[shimeoki]

#513 (comment):

I also have an implementation of this in https://github.com/mwh/kdly.

i don't see it in the official list of implementations. is there a reason why you haven't submitted it?

[shimeoki]

i don't think my implementation in go will be helpful here, because two projects in js and python are available, but i still can give some input from an implementer perspective.

my project is aimed to be fully format-preserving. entries and annotations are represented like this:

//	Annotation = "(" Leading? Name Trailing? ")" Following?
type Annotation struct {
	Leading   Whitespace
	Name      String
	Trailing  Whitespace
	Following Whitespace
}

//	Entry = Slashdash? Key? Annotation? Value Trailing?
type Entry struct {
	Slashdash  *Slashdash
	Key        *Key
	Annotation *Annotation
	Value      Value
	Trailing   Whitespace
}

Value is the actual value stored in an entry: string, boolean, number or null.

the issue here is that Annotation is an optional element that represents a type annotation as the whole thing with the parentheses. even if suffix annotations mean semantically the same thing as the prefix annotations, i cannot just keep them here.

however, it's a pretty funny thing that my number type is a string:

type Number string // implements Value

i did that to store the original number format as in the source. and that also helps to resolve this situation, because the suffix annotation can just be stored as a part of this string.

while that works, it blends the separation between values and annotations. previously it was only in the element tree, and now it lives with the number. it can be convenient to, for example, represent 10% as 0.1, but doesn't exactly feel right to me.

[shimeoki]

kdl-rs was mentioned previously, and it is also a format-preserving library. based on the code here:

pub struct KdlEntry {
    pub(crate) ty: Option<KdlIdentifier>,
    pub(crate) value: KdlValue,
    pub(crate) name: Option<KdlIdentifier>,
    pub(crate) format: Option<KdlEntryFormat>,
    #[cfg(feature = "span")]
    pub(crate) span: SourceSpan,
}

the same layout is used in kdl-rs: annotations are represented by an optional string. if it is available, then it is referred to as the whole block with the parentheses.

however, values are represented as primitives (i don't know whether String is a built-in symbol or not):

pub enum KdlValue {
    String(String),
    Integer(i128),
    Float(f64),
    Bool(bool),
    Null,
}

that leads to this, because the library cannot store numbers as strings for arbitrary large numbers. also, probably library is unable to store the source format of numbers in the first place.

if suffix type annotations for numbers will be added, then there are two solutions:

1. try to fit it into the KdlEntry struct
2. rework KdlValue to store numbers as strings

the first solution is backwards compatible, but is pretty messy. just a flag like suffix in an entry can be added, however:

• it is a boolean flag in an element like this, which is, imo, not good
• introduces "invalid" entry states, where suffix annotation is used not just for number values

the second solution is not backwards compatible and is a bit of a stretch for the end user (the number is a string and needs to be converted to the needed data type before use), but removes the limit on parseable numbers, stores the format and is able to store the annotation.

i am not a kdl-rs contributor or even a rust developer at all, so i don't know about the policy of breaking changes in kdl-rs or in rust projects. but the library is already a v6.x.x, so i suppose there already were quite a bit.

i saw that someone wants to rewrite parser internals, so i thought it's better to mention this now.

[shimeoki]

also some comments for things i was involved in:

in compressed format these annotations should probably always be stored in the prefix format anyway. while suffix format is able to be more compact, parsing the annotation once again to figure out whether it can be used as a suffix or not probably is not worth it. also probably many implementations won't support suffix format for a while, so storing all annotations as prefixes is better for compatibility.

for kdl-in-json, all annotations should be considered as prefixed for serialization (from json to kdl). for deserialization, it is parsed and stored in the same field, because semantically prefix and suffix are identical.

[tmke8]

I fuzzed bgotink/kdl's vs kdly's implementation of number suffixes to see if there are differences. I found these:

#	Input	bgotink/kdl	kdly
1	-5px	(-5px)NaN	(px)-5
2	-0x10	NaN	(x10)0
3	0x-10	error	-16
4	1-1	(-1)1	error
5	1+1	(+1)1	error
6	1e_1	error	(e_1)1
7	1#	1	error
8	1#.5	error	(.5)1
9	1._x	error	(x)1.0

I think bgotink/kdl is spec-compliant on 3, 8, and 9. kdly is compliant on 1 and 7. Neither is right on 2. And I'm not sure about 4, 5 and 6.

[shimeoki]

#513 (comment):

And I'm not sure about 4, 5 and 6.

i think, according to the current spec description and grammar:

• 4 and 5 are not valid (must not parse), because -1 and +1 are suffixes, and they are not valid identifier strings
• 6 is valid: 1 is the decimal part, and then e_1 is the suffix. it is an identifier string, and it doesn't match [eE]([-+]|[0-9])

i don't know how i feel about 6. i am going to say that it looks confusing enough to disallow it, so the regex above should include an underscore with the plus and minus

[mwh]

I've fixed the four kdly cases there (2, 3, 8, 9). 4 & 5 should be errors because -1/+1 are not valid identifier strings ("the + and - characters can only be used as an initial character if the second character is not a digit") and so can't be suffixes. 6 is currently spec-correct, since what is ruled out is just type suffixes matching ^[eE]([-+]|[0-9]) that could be the start of legitimate exponential suffixes, so e_ is not different to ex there as it stands. It would be fine to rule it out, but it's not the level of problem that the others are.

[shimeoki]

wanted to give it a try and implement it, but bumped into an interesting case: what we should do about keyword numbers?

right now, according to the specification, bare suffix type annotations cannot be used with keyword numbers, because they are not decimals. but explicit suffixes say "any number". and spec says that numbers have 5 valid syntax variations, including keywords. e.g. this should be valid:

node #inf#bytes

which is not the case for both bgotink/kdl and mwh/kdly right now.

is this an oversight or is this intended? personally, i don't see an issue here (looks okay and doesn't break anything), but thought i should mention it anyway

[shimeoki]

as a follow-up, i checked the grammar and right now it doesn't even permit hex/octal/binary numbers with explicit suffixes: only decimals. however, the text says that 0x1234#u32 should be valid. is grammar WIP (because i think i saw some unfinished/unresolved reviews) or is this intentional? should implementers use the text or grammar right now?

[bgotink]

6 1e_1 error (e_1)1

6 is a conscious decision on my part, see #513 (comment): 1e_1 is currently a parser error because the exponent mustn't start with an underscore, imo we should keep it that way.