Sentry: syncing_file.go:67: write ×: bad address (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.com/cockroachdb/cockroach/pkg/storage.(...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6188779918/?referrer=webhooks_plugin

Panic Message:

syncing_file.go:67: write ×: bad address
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).SyncWait
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:605
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*SyncWaiterLoop).waitLoop
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/sync_waiter.go:95
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*SyncWaiterLoop).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/sync_waiter.go:84
  | [...repeated from below...]
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
  | 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:67
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
  | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
  | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:531
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
  | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:477
  | runtime/pprof.Do
  | 	GOROOT/src/runtime/pprof/runtime.go:40
  | github.com/cockroachdb/pebble/record.NewLogWriter.func2
  | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:360
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (3) write ×
Wraps: (4) bad address
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:67: *withstack.withStack (top exception)
panic.go:884: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 69a32fa

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

cockroach/pkg/kv/kvserver/logstore/sync_waiter.go

Lines 83 to 85 in 69a32fa

	func(ctx context.Context) {
	w.waitLoop(ctx, stopper)
	})

cockroach/pkg/kv/kvserver/logstore/sync_waiter.go

Lines 94 to 96 in 69a32fa

	case w := <-w.q:
	if err := w.wg.SyncWait(); err != nil {
	log.Fatalf(ctx, "SyncWait error: %+v", err)

cockroach/pkg/storage/pebble_batch.go

Lines 604 to 606 in 69a32fa

	// or don't have after they receive an error from this method.
	panic(err)
	}

GOROOT/src/runtime/panic.go#L883-L885
GOROOT/src/runtime/asm_amd64.s#L1593-L1595
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L359-L361
GOROOT/src/runtime/pprof/runtime.go#L39-L41
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L476-L478
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L530-L532
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L561-L563
github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L66-L68

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/logstore/sync_waiter.go in pkg/kv/kvserver/logstore.(*SyncWaiterLoop).Start.func1 at line 84
pkg/kv/kvserver/logstore/sync_waiter.go in pkg/kv/kvserver/logstore.(*SyncWaiterLoop).waitLoop at line 95
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).SyncWait at line 605
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 360
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 40
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 477
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 531
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 562
github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 67

Tags

Tag	Value
Command	server
Environment	v23.1.15
Go Version	go1.19.13
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.15
Cockroach SHA	69a32fa
# of CPUs	3
# of Goroutines	375

Jira issue: CRDB-45922

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/storage

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}