Tokenizer line numbers incorrect with Windows line endings

[cedws]

When source containing Windows line endings is tokenized, the line numbers can be incorrect.

Given this file:

# a
# b
# c

This list of tokens is generated:

- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" a" [ORG]:"# a\r" [POS(line:column:level:offset)]: 1:1:0:1
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" b" [ORG]:"\n# b\r" [POS(line:column:level:offset)]: 3:1:0:5
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" c" [ORG]:"\n# c\r" [POS(line:column:level:offset)]: 5:1:0:9

Note that each line is numbered 1,3, and 5 respectively.

With Windows line endings replaced with Unix line endings, the lines are numbered correctly:

- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" a" [ORG]:"# a\n" [POS(line:column:level:offset)]: 1:1:0:1
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" b" [ORG]:"# b\n" [POS(line:column:level:offset)]: 2:1:0:4
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" c" [ORG]:"# c\n" [POS(line:column:level:offset)]: 3:1:0:7

You can add Windows line endings by running this sed expression against the input file:

sed -i -e 's/$/\r/' file

And remove them with this:

sed -i -e 's/\r$//' file

Code to reproduce:

tokens := lexer.Tokenize(src)
tokens.Dump()

[cedws]

Upon further investigation it looks like this only occurs due to the way comments are scanned.

go-yaml/scanner/scanner.go

Lines 660 to 675 in f4ccce9

	for idx, c := range ctx.src[ctx.idx:] {
	ctx.addOriginBuf(c)
	switch c {
	case '\n', '\r':
	if ctx.previousChar() == '\\' {
	continue
	}
	value := ctx.source(ctx.idx, ctx.idx+idx)
	progress := len([]rune(value))
	ctx.addToken(token.Comment(value, string(ctx.obuf), s.pos()))
	s.progressColumn(ctx, progress)
	s.progressLine(ctx)
	ctx.clear()
	return true
	}
	}

When a \r rune is encountered, it is immediately considered a newline and the line is progressed, even though the following \n is also part of the CRLF sequence. Removing \r from the case resolves this bug.

- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" a\r" [ORG]:"# a\r\n" [POS(line:column:level:offset)]: 1:1:0:1
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" b\r" [ORG]:"# b\r\n" [POS(line:column:level:offset)]: 2:1:0:5
- [TYPE]:"Comment" [CHARTYPE]:"Indicator" [INDICATOR]:"Comment" [VALUE]:" c\r" [ORG]:"# c\r\n" [POS(line:column:level:offset)]: 3:1:0:9