Expand description
Tokens of JSON5 source.
Whitespace and comments are tokens too, so the parser can keep them.
Identifier characters are approximated with char::is_alphabetic and
char::is_alphanumeric plus $, _, U+200C and U+200D. That covers the
ECMAScript letter categories and decimal digits, but misses some combining
marks and connector punctuation. Whitespace is exact: the ECMAScript list
and the Zs category.