Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increased memory consumption when serializing long slices. #451

Open
RekGRpth opened this issue May 31, 2024 · 0 comments
Open

Increased memory consumption when serializing long slices. #451

RekGRpth opened this issue May 31, 2024 · 0 comments
Labels
enhancement New feature or request low priority

Comments

@RekGRpth
Copy link

RekGRpth commented May 31, 2024

When serializing long slices, increased memory consumption is possible, mine exceeds 3 gigabytes! Steps to reproduce:
save this

package main

import (
	"fmt"
	"os"

	"github.com/goccy/go-yaml"
)

type TOC struct {
	StatisticsEntries []MetadataEntry
}

type MetadataEntry struct {
	Name  string
	Value string
	Other string
}

func panicOnErr(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	toc := TOC{}
	size := 1000000
	for i := 0; i < size; i++ {
		name := fmt.Sprintf("Name %v", i)
		value := fmt.Sprintf("Value %v", i)
		toc.StatisticsEntries = append(toc.StatisticsEntries, MetadataEntry{Name: name, Value: value})
	}
	fmt.Println("press enter 1")
	var s string
	fmt.Scanf("%s", &s)
	file, _ := os.OpenFile("test.yaml", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
	defer file.Close()
	enc := yaml.NewEncoder(file)
	defer enc.Close()
	err := enc.Encode(toc)
	panicOnErr(err)
	fmt.Println("press enter 2")
	fmt.Scanf("%s", &s)
}

to file test.go and run it

GODEBUG=gctrace=1 go run test.go
gc 1 @0.002s 2%: 0.018+0.34+0.031 ms clock, 0.29+0.061/0.49/0.026+0.49 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 2 @0.006s 2%: 0.012+0.21+0.009 ms clock, 0.19+0.050/0.59/0.47+0.14 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 3 @0.008s 4%: 0.089+0.32+0.091 ms clock, 1.4+0.18/0.84/0.45+1.4 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 4 @0.011s 7%: 0.38+0.63+0.013 ms clock, 6.2+0.77/1.2/0.031+0.22 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 5 @0.013s 7%: 0.066+0.35+0.005 ms clock, 1.0+0.38/1.1/0.099+0.086 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 6 @0.014s 8%: 0.029+0.30+0.005 ms clock, 0.47+0.33/0.78/0.025+0.091 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 7 @0.014s 8%: 0.031+0.27+0.026 ms clock, 0.51+0.045/0.71/0.54+0.41 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 8 @0.016s 8%: 0.011+0.24+0.008 ms clock, 0.18+0.034/0.64/0.73+0.14 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 9 @0.019s 7%: 0.010+0.29+0.007 ms clock, 0.16+0.027/0.78/0.80+0.12 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 10 @0.020s 7%: 0.044+0.32+0.006 ms clock, 0.71+0.35/0.64/0+0.10 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 11 @0.021s 7%: 0.016+0.34+0.083 ms clock, 0.25+0.049/0.75/0.43+1.3 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 12 @0.022s 7%: 0.045+0.27+0.009 ms clock, 0.72+0/0.69/0+0.15 ms cpu, 3->4->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 13 @0.022s 7%: 0.011+0.23+0.009 ms clock, 0.18+0.031/0.55/0.24+0.14 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 14 @0.023s 8%: 0.048+0.40+0.061 ms clock, 0.77+0.057/0.65/0.48+0.99 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 15 @0.040s 5%: 0.026+0.29+0.008 ms clock, 0.42+0.040/0.82/0.73+0.13 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
# command-line-arguments
gc 1 @0.000s 5%: 0.007+0.27+0.004 ms clock, 0.11+0.084/0.55/0+0.077 ms cpu, 4->6->5 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 2 @0.002s 4%: 0.008+0.31+0.008 ms clock, 0.14+0.052/0.97/0.002+0.12 ms cpu, 14->14->12 MB, 14 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 1 @0.003s 1%: 0.009+0.40+0.005 ms clock, 0.15+0.30/0.10/0+0.089 ms cpu, 3->3->1 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 2 @0.005s 2%: 0.008+1.0+0.020 ms clock, 0.13+0/1.0/0.23+0.32 ms cpu, 4->4->2 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 3 @0.008s 2%: 0.009+1.8+0.005 ms clock, 0.15+0/2.2/0+0.090 ms cpu, 6->6->3 MB, 6 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 4 @0.013s 2%: 0.010+0.84+0.005 ms clock, 0.16+0.11/1.4/0+0.090 ms cpu, 6->6->4 MB, 7 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 5 @0.017s 3%: 0.008+2.6+0.036 ms clock, 0.14+0.19/3.1/0.24+0.58 ms cpu, 12->12->6 MB, 12 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 6 @0.024s 3%: 0.008+0.85+0.010 ms clock, 0.12+0.027/2.7/0.003+0.17 ms cpu, 12->13->8 MB, 13 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 7 @0.034s 3%: 0.022+5.8+0.005 ms clock, 0.35+0.75/5.6/1.0+0.090 ms cpu, 24->24->12 MB, 24 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 8 @0.047s 3%: 0.009+1.2+0.009 ms clock, 0.15+0.093/4.4/0+0.15 ms cpu, 24->24->16 MB, 25 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 9 @0.062s 2%: 0.018+5.8+0.008 ms clock, 0.28+0.040/5.9/0+0.12 ms cpu, 31->32->21 MB, 32 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 10 @0.082s 3%: 0.014+10+0.009 ms clock, 0.23+0.15/13/5.8+0.15 ms cpu, 59->60->30 MB, 60 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 11 @0.112s 2%: 0.016+15+0.010 ms clock, 0.27+5.8/7.0/0.004+0.16 ms cpu, 59->60->39 MB, 61 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 12 @0.156s 2%: 0.017+11+0.008 ms clock, 0.28+4.0/8.7/2.0+0.14 ms cpu, 77->78->51 MB, 79 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 13 @0.202s 2%: 0.018+24+0.006 ms clock, 0.29+1.5/31/7.4+0.098 ms cpu, 146->146->74 MB, 146 MB goal, 0 MB stacks, 0 MB globals, 16 P
press enter 1

gc 14 @33.816s 0%: 0.017+10+0.009 ms clock, 0.27+1.4/39/45+0.14 ms cpu, 144->149->122 MB, 148 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 15 @33.976s 0%: 0.018+19+0.010 ms clock, 0.29+0.061/66/113+0.16 ms cpu, 240->248->201 MB, 245 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 16 @34.232s 0%: 0.019+27+0.010 ms clock, 0.31+0.067/96/228+0.16 ms cpu, 386->395->322 MB, 404 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 17 @34.640s 0%: 0.037+39+0.022 ms clock, 0.59+0.088/155/425+0.35 ms cpu, 617->629->513 MB, 645 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 18 @35.280s 0%: 0.019+70+0.011 ms clock, 0.30+0.077/260/660+0.18 ms cpu, 984->1006->825 MB, 1028 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 19 @36.306s 0%: 0.020+114+0.012 ms clock, 0.32+0/415/1166+0.20 ms cpu, 1581->1607->1320 MB, 1650 MB goal, 0 MB stacks, 0 MB globals, 16 P
gc 20 @38.481s 0%: 0.020+140+0.011 ms clock, 0.33+0.040/561/1533+0.19 ms cpu, 2534->2553->1793 MB, 2641 MB goal, 0 MB stacks, 0 MB globals, 16 P
press enter 2
@RekGRpth RekGRpth added the bug Something isn't working label May 31, 2024
@goccy goccy added enhancement New feature or request and removed bug Something isn't working labels Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request low priority
Projects
None yet
Development

No branches or pull requests

2 participants