Wrong Token offset with CLRF and LF

[mih-kopylov]

Tests to reproduce

import (
	"github.com/goccy/go-yaml/lexer"
	"github.com/stretchr/testify/assert"
	"testing"
)

func TestCrlfDemo(t *testing.T) {
	content := "project:\r\n  version: 1.2.3\r\n"
	tokens := lexer.Tokenize(content)
	assert.Len(t, tokens, 5)
	assert.Equal(t, "1.2.3", tokens[4].Value)
	assert.Equal(t, 21, tokens[4].Position.Offset)
}

func TestLfDemo(t *testing.T) {
	content := "project:\n  version: 1.2.3\n"
	tokens := lexer.Tokenize(content)
	assert.Len(t, tokens, 5)
	assert.Equal(t, "1.2.3", tokens[4].Value)
	assert.Equal(t, 20, tokens[4].Position.Offset)
}

=== RUN   TestCrlfDemo
    demo_test.go:14: 
        	Error Trace:	demo_test.go:14
        	Error:      	Not equal: 
        	            	expected: 21
        	            	actual  : 26
        	Test:       	TestCrlfDemo
--- FAIL: TestCrlfDemo (0.00s)

=== RUN   TestLfDemo
    demo_test.go:22: 
        	Error Trace:	demo_test.go:22
        	Error:      	Not equal: 
        	            	expected: 20
        	            	actual  : 26
        	Test:       	TestLfDemo
--- FAIL: TestLfDemo (0.00s)

Notes

• TestCrlfDemo has the version token at the position 21, but the token stores 26, like the end of the token
• TestLfDemo has the version token at the position 20, but the token stores 26 again, like the end of the token + 1

[goccy]

@mih-kopylov Thank you for your reports. The offset starts at 1, with CRLF counting as 2 characters and LF as 1 character, so the correct values are 22 and 21.

I've fixed this problem with #524