Skip to content

Commit

Permalink
Add pkg/pg with dialects.go & txdb.go (#910)
Browse files Browse the repository at this point in the history
* Add pkg/pg with dialects.go & txdb.go

Neither of these were in the actual pg package in chainlink repo.
dialects.go came from core/store/dialects and txdb.go from
core/internal/testutils/pgtest, but neither of these seem like they
deserve their own package in chainlink-common--we can lump all the
postgres specific common utilities under pkg/pg

* Add TestTxDBDriver, NewSqlxDB, SkipShort, SkipShortDB and SkipFlakey

* Add idempotency test of RegisterTxDb

* Create ctx from testing context, instead of using context.Background

* Only abort tx's when last connection is closed

Also: convert rest of panic isn't ordinary errors

* go mod tidy

* Split abort channel into one per connection object

All txdb connections share the same underlying connection to the
postgres db. Calling NewSqlxDB() or NewConnection() with dialect=txdb
doesn't create a new pg connection, it just creates a new tx with
BEGIN. Closing the connection with db.Close() issues ROLLBACK.

Both NewSqlxDB() and NewConneciton() choose random UUID's for their
dsn string, so we shouldn't have a case where the same dsn is opened
more than once. If that did happen, then these two different txdb
"connections" would be sharing the same transaction which would
mean closing the abort channel due to a query sent over one of them
would affect the other. Hopefully that's not a problem? If it is
I think our only option will be to go back to using context.Background
for all queries.

Before this commit, there was only one abort channel for the entire
txdb driver meaning that even two entirely different connections
opened with different dsn's could interfere with each other's queries.
This should fix that case, which is presumably the only case we
care about. Since each dsn corresponds to a different call to
NewSqlxDB() and the UUID's are generated randomly, there should no
longer be a conflict. Each txdb connection will have its own abort
channel.

* Errorf -> Fatalf on failure to register txdb driver

* Add in-memory DataSource using go-duckdb

* Fall back to testing txdb with in-memory backed db if CL_DATABASE_URL is not set

This allows us to test most of it in CI, and all locally

* Fix imports & fmt.Sprintf -> t.Log

* Add concurrency test for RegisterTxDb()

* Fix race condition

This showed up in some of the unit tests in the linked PR in chainlink repo

* Remove pg.SkipDB(), add DbUrlOrInMemory()

* pkg/pg -> pkg/sqlutil/pg

* NewSqlxDB -> NewTestDB, DbUrlOrInMemory -> TestURL
  • Loading branch information
reductionista authored Dec 13, 2024
1 parent dbebc0f commit edc5dee
Show file tree
Hide file tree
Showing 8 changed files with 905 additions and 29 deletions.
40 changes: 29 additions & 11 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/smartcontractkit/chainlink-common
go 1.23.3

require (
github.com/andybalholm/brotli v1.1.0
github.com/andybalholm/brotli v1.1.1
github.com/atombender/go-jsonschema v0.16.1-0.20240916205339-a74cd4e2851c
github.com/bytecodealliance/wasmtime-go/v23 v23.0.0
github.com/confluentinc/confluent-kafka-go/v2 v2.3.0
Expand All @@ -20,15 +20,18 @@ require (
github.com/hashicorp/go-plugin v1.6.2
github.com/iancoleman/strcase v0.3.0
github.com/invopop/jsonschema v0.12.0
github.com/jackc/pgx/v4 v4.18.3
github.com/jmoiron/sqlx v1.4.0
github.com/jonboulle/clockwork v0.4.0
github.com/jpillora/backoff v1.0.0
github.com/lib/pq v1.10.9
github.com/linkedin/goavro/v2 v2.12.0
github.com/marcboeker/go-duckdb v1.8.3
github.com/pelletier/go-toml/v2 v2.2.0
github.com/prometheus/client_golang v1.17.0
github.com/riferrei/srclient v0.5.4
github.com/santhosh-tekuri/jsonschema/v5 v5.2.0
github.com/scylladb/go-reflectx v1.0.1
github.com/shopspring/decimal v1.4.0
github.com/smartcontractkit/grpc-proxy v0.0.0-20240830132753-a7e17fec5ab7
github.com/smartcontractkit/libocr v0.0.0-20241007185508-adbe57025f12
Expand All @@ -51,17 +54,19 @@ require (
go.opentelemetry.io/otel/sdk/metric v1.30.0
go.opentelemetry.io/otel/trace v1.30.0
go.uber.org/goleak v1.3.0
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.27.0
golang.org/x/crypto v0.28.0
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0
golang.org/x/tools v0.25.0
gonum.org/v1/gonum v0.15.0
google.golang.org/grpc v1.66.1
google.golang.org/protobuf v1.34.2
golang.org/x/tools v0.26.0
gonum.org/v1/gonum v0.15.1
google.golang.org/grpc v1.67.1
google.golang.org/protobuf v1.35.1
sigs.k8s.io/yaml v1.4.0
)

require (
github.com/apache/arrow-go/v18 v18.0.0 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
Expand All @@ -73,20 +78,33 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.13.0 // indirect
github.com/go-playground/universal-translator v0.17.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/goccy/go-yaml v1.12.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect; indirec
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/yamux v0.1.1 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.14.3 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.3 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/leodido/go-urn v1.2.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/oklog/run v1.0.0 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16 // indirect
Expand All @@ -96,14 +114,14 @@ require (
github.com/stretchr/objx v0.5.2 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.30.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/net v0.29.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.25.0 // indirect
golang.org/x/text v0.18.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
Expand Down
Loading

0 comments on commit edc5dee

Please sign in to comment.