Advance the lexer to the next token.

A nested Block Collection

[200]	s-l+block-collection(n,c)	::=	( [[separate s-separate(n+1,c)]] c-ns-properties(n+1,c) )?
[[multilineComment s-l-comments]]
( [[blockSequence l+block-sequence(seq-spaces(n,c))]]
| [[blockMapping]] l+block-mapping(n) )

[201]	seq-spaces(n,c)	::=
c = block-out ⇒ n-1
c = block-in  ⇒ n

Returns the new defined indentation or MAX_VALUE if autodetect

[162] c-b-block-header(m,t)	::=
( ( c-indentation-indicator(m) c-chomping-indicator(t) )
| ( c-chomping-indicator(t) c-indentation-indicator(m) )
)
s-b-comment

[163] c-indentation-indicator(m) ::=	ns-dec-digit ⇒ m = ns-dec-digit - #x30
| Empty

[164] c-chomping-indicator(t) ::=
“-” ⇒ t = strip
|   “+” ⇒ t = keep
|   “ ” ⇒ t = clip

[188] ns-l-block-map-entry(n) ::= c-l-block-map-explicit-entry(n)

| ns-l-block-map-implicit-entry(n)

A Block mapping is a series of entries, each presenting a key: value pair.

[187] l+block-mapping(n) ::= ( s-indent(n+m) ns-l-block-map-entry(n+m) )+

For some fixed auto-detected m greater than 0

A block sequence is simply a series of nodes, each denoted by a leading “-” indicator. The “-” indicator must be separated from the node by white space. This allows “-” to be used as the first character in a plain scalar if followed by a non-space character (e.g. “-1”).

[183]	l+block-sequence(n)	::=	( [[indent s-indent(n+m)]] [[blockSeqEntry c-l-block-seq-entry(n+m)]] )+
(For some fixed auto-detected m > 0)

Line Break normalized to lineFeed

[29] b-as-line-feed ::= b-break

Convert a lineBreak to an space (fold it)

[72] b-as-space ::= b-break

[76] b-comment ::= b-non-content | EofChar

Outside scalar content, YAML allows any line break to be used to terminate lines.

[30] b-non-content ::= b-break

Compare with the specified String and return 0 or the length of the string if all characters are equal

[166]	l-chomped-empty(n,t) ::=
t = strip(-) ⇒ l-strip-empty(n)
t = clip     ⇒ l-strip-empty(n)
t = keep (+) ⇒ l-keep-empty(n)

[167]	l-strip-empty(n)	::=	([[indentLessOrEqual s-indent(≤n)]] [[breakNonContent b-non-content]])*
[[trailComments l-trail-comments(n)]]?

[168]	l-keep-empty(n)	    ::=	[[emptyLine l-empty(n,block-in)]]*
[[trailComments l-trail-comments(n)]]?

Process the last block of an Scalar based on the chomp indicator

[165]	b-chomped-last(t) ::=
t = strip(-) ⇒ b-non-content  | EOF
t = clip     ⇒ b-as-line-feed | EOF
t = keep (+) ⇒ b-as-line-feed | EOF

[75] c-nb-comment-text ::= “#” nb-char*

Actually it is doing: commentText ::= '#' nb-char* b-comment

Compare with the specified String and consume if all characters are equal

Compare with the specified char and consume if they are equal

Emit 2 Tokens

Emit a Token

Emit 2 Tokens with same mark

[67]	s-line-prefix(n,c)	::=
c = block-out ⇒ s-block-line-prefix(n)
c = block-in  ⇒ s-block-line-prefix(n)
c = flow-out  ⇒ s-flow-line-prefix(n)
c = flow-in   ⇒ s-flow-line-prefix(n)
[68]	s-block-line-prefix(n)	::=	s-indent(n)
[69]	s-flow-line-prefix(n)	::=	s-indent(n) s-separate-in-line?
[70]	l-empty(n,c)	::=	( s-line-prefix(n,c) | s-indent(<n) ) b-as-line-feed

JSON-like flow styles all have explicit start and end indicators.

[157] c-flow-json-content(n,c)	::=
[[flowSequence c-flow-sequence(n,c)]]
| [[flowMapping c-flow-mapping(n,c)]]
| [[quotedScalar c-single-quoted(n,c)]]
| [[quotedScalar c-double-quoted(n,c)]]

Flow sequence content is denoted by surrounding “[” and “]” characters.

[137] c-flow-sequence(n,c) ::= “[” s-separate(n,c)? [\[flowSequenceEntries ns-s-flow-seq-entries(n,in-flow(c))]\]? “]”

Any flow node may be used as a flow sequence entry. In addition, YAML provides a compact notation for the case where a flow sequence entry is a mapping with a single key: value pair.

[139] ns-flow-seq-entry(n,c) ::= [\[flowPair ns-flow-pair(n,c)]\]

| [\[flowNode ns-flow-node(n,c)]\]

[159] ns-flow-yaml-node(n,c) ::= [\[aliasNode c-ns-alias-node]\]

| [\[flowYamlContent ns-flow-yaml-content(n,c)]\]

| ( [\[nodeProperties c-ns-properties(n,c)]\] ( ( s-separate(n,c) [\[flowYamlContent ns-flow-yaml-cont(n,c)]\] )

| e-scalar ) )

The folded style is denoted by the “>” indicator. It is similar to the literal style; however, folded scalars are subject to line folding.

[174] c-l+folded(n) ::= “>” c-b-block-header(m,t) [\[foldedContent l-folded-content(n+m,t)]\]

Process an indentation exactly as the current one

[63] s-indent(n) ::= s-space × n

Detect an indentation lower than the current one

[64] s-indent(<n) ::= s-space × m (Where m < n)

Detect an indentation lower or equal to the current one

[64] s-indent(≤n) ::= s-space × m (Where m ≤ n)

Check that the current char is the specified one and if so emit it as an Indicator

Init must initialize the current _tokenData (may be invoking advance)

[130] ns-plain-char(c) ::= ( ns-plain-safe(c) - “:” - “#” )

| ( /* An ns-char preceding */ “#” )

| ( “:” /* Followed by an ns-plain-safe(c) */ )

Recognition of lineBreak Sequence

[28] b-break ::= ( b-carriage-return b-line-feed ) | b-line-feed

[78]	l-comment	::=	s-separate-in-line c-nb-comment-text? b-comment

Actually refactored to:
[[separateInLine s-separate-in-line]] ([[breakComment b-non-content]] | [[commentText]])

[173]	l-literal-content(n,t)	::=
( [[literalContentLine l-nb-literal-text(n)]]
b-nb-literal-next(n) *
[[chompedLast b-chomped-last(t)]]
)?
[[chompedEmpty l-chomped-empty(n,t)]]

[172]	b-nb-literal-next(n) ::= [[breakAsLineFeed b-as-line-feed]] [[literalContentLine l-nb-literal-text(n)]]

Explicit map entries are denoted by the “?” mapping key indicator

[189]	c-l-block-map-explicit-entry(n)	::=	c-l-block-map-explicit-key(n)
( l-block-map-explicit-value(n) | [[emptyNode e-node]] )

[190]	c-l-block-map-explicit-key(n)	::=	“?” [[blockIndented s-l+block-indented(n,block-out)]]

Detect a Map implicit Entry

[192]	ns-l-block-map-implicit-entry(n) ::=
( ns-s-block-map-implicit-key | [[emptyNode e-node]] )
[[mapImplicitValue c-l-block-map-implicit-value(n)]]

[193]	ns-s-block-map-implicit-key	::=	  [[implicitJsonKey c-s-implicit-json-key(block-key)]]
|   [[implicitYamlKey ns-s-implicit-yaml-key(block-key)]]

[194] c-l-block-map-implicit-value(n) ::= “:” ( s-l+block-node(n,block-out)

| ( [\[emptyNode e-node]\] s-l-comments ) )

[79] s-l-comments ::= ( s-b-comment | /* Start of line */ ) l-comment*

We're not at the Eof

Check if there are emitted tokens

Process all pending tokens. Trivial implementation just emit the EofToken More complex ones can continue returning pending tokens until they emit the EofToken

[77] s-b-comment ::= ( s-separate-in-line c-nb-comment-text? )? b-comment* Actually refactored to: b-comment | l-comment

get the current token in the input stream.

All the token data.

Get the current Token String.

Get the current Token Char Sequence.

Get the specified Token Char Sequence.

Explicit comment lines may follow the trailing empty lines. To prevent ambiguity, the first such comment line must be less indented than the block scalar content. Additional comment lines, if any, are not so restricted. This is the only case where the indentation of comment lines is constrained.

[169] l-trail-comments(n) ::= s-indent(<n) c-nb-comment-text b-comment l-comment*

A YAML stream consists of zero or more documents. Subsequent documents require some sort of separation marker line. If a document is not terminated by a document end marker line, then the following document must begin with a directives end marker line. The stream format is intentionally “sloppy” to better support common use cases, such as stream concatenation.

[211]	l-yaml-stream	::=
[[documentPrefix l-document-prefix]]*
[[anyDocument l-any-document]]?
(
l-document-suffix+ l-document-prefix* l-any-document?
| l-document-prefix* l-explicit-document?
)*