Releases: spotify/scio
Releases · spotify/scio
v0.14.11
🇫🇷 🇫🇷 🫡 @RustedBones 🫡 🇫🇷 🇫🇷
What's Changed
- Remove Zoltar by @kellen in #5540
- Deprecate annoy by @kellen in #5541
- Update sbt-typelevel to 0.7.5 by @RustedBones in #5543
- Update GHA by @RustedBones in #5545
- Update sbt-avro to 4.0.0 by @RustedBones in #5547
- Update to beam 2.62.0 by @RustedBones in #5546
- Some missing options in SnowflakeIO by @turb in #5550
- Update sbt to 1.10.7 by @RustedBones in #5552
- Ignore all jackson module update by @RustedBones in #5581
- Update scala-compiler, scala-library, ... to 2.13.16 by @RustedBones in #5571
- Pin tensorflow API to v4 by @RustedBones in #5582
- Update sbt-typelevel to 0.7.7 by @RustedBones in #5580
- Update sbt-bom to 1.0.25 by @RustedBones in #5563
- Update sbt-buildinfo to 0.13.1 by @RustedBones in #5557
- Update sbt-scalafix to 0.14.0 by @RustedBones in #5553
- Update sbt-assembly to 2.3.1 by @RustedBones in #5556
- Update sbt-mdoc to 2.6.2 by @RustedBones in #5575
- Update scalafmt-core to 3.8.6 by @RustedBones in #5576
- Update elasticsearch-java to 8.15.5 by @RustedBones in #5554
- Update testcontainers-scala-elasticsearch, ... to 0.41.8 by @RustedBones in #5555
- Update metrics-core to 4.2.30 by @RustedBones in #5567
- Update jna to 5.16.0 by @RustedBones in #5568
- Update neo4j-java-driver to 4.4.19 by @RustedBones in #5570
- Update scala-collection-compat to 2.13.0 by @RustedBones in #5572
- Update parquet-avro, parquet-column, ... to 1.14.4 by @RustedBones in #5569
- Update mysql-connector-j to 9.2.0 by @RustedBones in #5564
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.21.2 by @RustedBones in #5562
- Update algebra, cats-core, cats-kernel to 2.13.0 by @RustedBones in #5579
- Update munit to 1.0.4 by @RustedBones in #5574
- Update voyager to 2.1.0 by @RustedBones in #5566
- Update socco-ng to 0.1.14 by @RustedBones in #5583
- Update munit to 1.1.0 by @RustedBones in #5588
- Update elasticsearch-java to 8.17.1 by @RustedBones in #5585
- Switch BQ typed underlying API to avro by @RustedBones in #5529
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.23.0 by @RustedBones in #5586
- Add java priority queue, set, deque, collection coders by @kellen in #5520
- 10 minute timeout for futures by default to split the difference between Inf and 1 by @kellen in #5591
- Update parquet-avro, parquet-column, ... to 1.15.0 by @RustedBones in #5587
- Handle TableRow json specifically by @RustedBones in #5589
- Cleanup some options by @RustedBones in #5593
- Fix memory leak in avro SlowGenericRecordCoder by @RustedBones in #5592
- Supply gcs-connector options from default Configuration by @clairemcginty in #5549
- Prep for 0.14.11 by @kellen in #5594
Full Changelog: v0.14.10...v0.14.11
Contributors
kellen, turb, and 2 other contributors
Assets 2
v0.14.10
Includes Beam 2.61.0 support.
This release also comes with a better support of BQ types. New TableRow extension methods were added for those types.
To note the behavior change of (r: TableRow).getRecord("field-name") that now throws instead of returning null when field-name is not present in the row. Change for getRecordOpt(f: AnyRef): Option[TableRow] instead.
🚀 Enhancements
- Support Snowflake (#5500) by @turb (#5502)
- Refactor partitionByKey to get clearer error (#5527) @RustedBones
- Allow timeout for async DoFn (#5534) @RustedBones
- Update default Dataflow add-open opts for Java 17+ (#5537) @clairemcginty
- Include more default jdkAddOpenModules for Java 17+ (#5538) @clairemcginty
- Add Batched lookups for streaming GRPC endpoints and BigTable (#5521) @lofifnc
🐛 Bug Fixes
- BigQuery JSON column: encode as Jackson JsonNode (#5523) @turb
- Stable TableRow converted from BQ types (#5536) @RustedBones
📗 Documentation
- Prepare for Scio 0.14.10 (#5539) @clairemcginty
🏗️ Build Improvements
- Skip BQ avro typed integration test until 2.61 (#5526) @RustedBones
🧪 Test Improvements
- best effort cleanup (#5528) @RustedBones
🌱 Dependency Updates
- Update to BigQuery read API v1 interface (#5431) @kmjung
- Update beam to 2.61 (#5525) @RustedBones
- Bump codecov/codecov-action from 4 to 5 (#5524) @dependabot
- Update sbt-scalafix to 0.13.0 (#5506) @scala-steward
- Update tensorflow-metadata to 0.16.1 (#5535) @RustedBones
Contributors to this release
@RustedBones, @turb, @kmjung, @lofifnc, @clairemcginty and @scala-steward
New Contributors
Full Changelog: v0.14.9...v0.14.10
Contributors
turb, clairemcginty, and 5 other contributors
Assets 3
v0.14.9
What's changed
Includes Beam 2.60.0 support.
🚀 Enhancements
- Add missing fanout SCollection API (#5497) @RustedBones
- Add latest API to SCollection (#5495) @RustedBones
- Add top to collections with fanout (#5487) @RustedBones
🐛 Bug Fixes
- Fix RemoteFileUtil to download in parallel as expected. (#5515) @psobot
- Fix file batch operation (#5519) @RustedBones
📗 Documentation
- Prepare for release 0.14.9 (#5522) @RustedBones
🏗️ Build Improvements
- Remove deprecated usage of tlSonatypeUseLegacyHost in build (#5517) @RustedBones
- Update sbt-bom to 1.0.16 (#5501) @scala-steward
- Update sbt-mdoc to 2.6.1 (#5496) @scala-steward
- Update sbt, sbt-dependency-tree to 1.10.2 (#5490) @scala-steward
🔧 Refactorings
- Consistent implementation for reduce operation (#5498) @RustedBones
🌱 Dependency Updates
- Update beam to 2.60 (#5516) @RustedBones
- Update socco-ng to 0.1.13 (#5518) @RustedBones
- Update scala-compiler, scala-library, ... to 2.13.15 (#5504) @scala-steward
- Update elasticsearch-java to 8.15.2 (#5503) @scala-steward
- Update jedis to 5.2.0 (#5507) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.21.0 (#5509) @scala-steward
- Update metrics-core to 4.2.28 (#5511) @scala-steward
- Update scalacheck to 1.18.1 (#5491) @scala-steward
- Update jna to 5.15.0 (#5489) @scala-steward
- Update munit to 1.0.2 (#5492) @scala-steward
Contributors to this release
Contributors
psobot, RustedBones, and scala-steward
Assets 3
v0.14.8
What's changed
Includes Beam 2.59.0 support.
🐛 Bug Fixes
- (fix #5472) Set desiredBundleSizeBytes to Long.MaxValue BinaryIO reads to prevent file subrange-splitting (#5473) @clairemcginty
- use noOp cache instead of null when instantiating typedSparkeySideInput (#5469) @pgoggijr
📗 Documentation
- Prepare release 0.14.8 (#5486) @RustedBones
- Fix redis examples (#5482) @kellen
- Fixes a minor formatting issue in the FAQs (#5467) @luciomartinez
🧪 Test Improvements
- Add test for empty string arg (#5485) @RustedBones
- Support pubsub epoch timestamp attribute in test (#5484) @RustedBones
🏗️ Build Improvements
- Update sbt-typelevel to 0.7.3 (#5480) @scala-steward
- Update sbt-scoverage to 2.2.0 (#5479) @scala-steward
- Update sbt-avro to 3.5.0 (#5461) @scala-steward
🌱 Dependency Updates
- Update beam to 2.59 (#5471) @RustedBones
- Update elasticsearch-java to 8.15.1 (#5474) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.1 (#5475) @scala-steward
- Update circe-core, circe-generic, ... to 0.14.10 (#5476) @scala-steward
- Update scala-compiler, scala-library, ... to 2.12.20 (#5478) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.2 (#5477) @scala-steward
- Update metrics-core to 4.2.27 (#5463) @scala-steward
- Update mssql-jdbc to 12.8.1.jre11 (#5462) @scala-steward
- Update jedis to 5.1.5 (#5465) @scala-steward
Contributors to this release
@RustedBones, @clairemcginty, @kellen, @luciomartinez, @pgoggijr and @scala-steward
Contributors
kellen, clairemcginty, and 4 other contributors
Assets 3
v0.14.7
What's changed
Includes Beam 2.58.1 support.
🚀 Enhancements
🐛 Bug Fixes
- Fix Async DoFn to emit element with restored metadata (#5441) @RustedBones
📜 Scalafix Migrations
- Improve accuracy of FixQuery rule (#5438) @clairemcginty
📗 Documentation
- Update version matrix with scio 0.14.7 (#5460) @RustedBones
🏗️ Build Improvements
- Ignore updates from zstd-jni and sync with beam's version (#5457) @RustedBones
- Update scalac-options to 0.1.7 (#5454) @scala-steward
- Update scalafmt-core to 3.8.3 (#5453) @scala-steward
- Move files in folder matching package (#5445) @RustedBones
- Enforce javafmt on CI (#5442) @RustedBones
- Disable MiMa for scio-bom (#5439) @RustedBones
🌱 Dependency Updates
- Update beam to 2.58.1 (#5459) @RustedBones
- Update elasticsearch-java to 8.15.0 (#5446) @scala-steward
- Update jedis to 5.1.4 (#5455) @scala-steward
- Update munit to 1.0.1 (#5452) @scala-steward
- Update voyager to 2.0.9 (#5450) @scala-steward
- Update mssql-jdbc to 12.8.0.jre11 (#5449) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.0 (#5448) @scala-steward
- Update beam to 2.58 (#5436) @RustedBones
Contributors to this release
Contributors
clairemcginty, RustedBones, and 2 other contributors
Assets 3
v0.14.6
What's changed
Includes Beam 2.57.0 support.
🚀 Enhancements
- Publish scio-bom (#5423) @RustedBones
- Support BigQuery JSON column type (#5416) @turb
🐛 Bug Fixes
- fix: use fully qualified types in macros (#5434) @RustedBones
- [gcp] Fix BQ BIGNUMERIC type support (#5428) @RustedBones
- Use hash join when writing sparkey (#5402) @aslotnick
📗 Documentation
- Prepare release v0.14.6 (#5437) @RustedBones
- doc: Update doc regarding avro version and logical-type support (#5435) @RustedBones
🏗️ Build Improvements
- Update sbt, sbt-dependency-tree to 1.10.1 (#5426) @scala-steward
- Update sbt-mdoc to 2.5.4 (#5427) @scala-steward
- Update sbt-bom to 1.0.14 (#5425) @scala-steward
- Use sbt-bom (#5414) @RustedBones
- Update sbt-scoverage to 2.1.0 (#5422) @scala-steward
- Update scalafmt-core to 3.8.2 (#5399) @scala-steward
- Derive artifact dependent versions (#5395) @RustedBones
🌱 Dependency Updates
- Update magnolify to 0.7.4 (#5432) @RustedBones
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.19.1 (#5424) @scala-steward
- Update elasticsearch-java to 8.14.3 (#5429) @scala-steward
- Update beam to 2.57.0 (#5413) @RustedBones
- Use default beam avro version (#5415) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.9 (#5420) @scala-steward
- Update mysql-connector-j to 9.0.0 (#5419) @scala-steward
- Update neo4j-java-driver to 4.4.18 (#5412) @scala-steward
- Update scalatest to 3.2.19 (#5408) @scala-steward
- Update mssql-jdbc to 12.6.3.jre11 (#5403) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.1 (#5405) @scala-steward
- Update testcontainers-scala-elasticsearch, ... to 0.41.4 (#5398) @scala-steward
- Update metrics-core to 4.2.26 (#5394) @scala-steward
Contributors to this release
Contributors
turb, RustedBones, and 2 other contributors
Assets 3
v0.14.5
What's changed
Includes Beam 2.56.0 support.
warning: This release transitively pulls newer avro and jackson versions. If you need to depend on previous versions, you'll need to pin them in your build.
🚀 Enhancements
- (#5296) Support Parquet predicates/projections in tests (#5309) @clairemcginty
- Add support for Zstd coders (#5321) @kellen
- [scio-core](feature) Sample SCollection with max weight (#5352) @RustedBones
- [scio-core](feature) Add readFiles and readFilesWithPath apis (#5350) @RustedBones
🐛 Bug Fixes
- Fix sparkey loading issue for unknown keys (#5387) @RustedBones
- (fix)[avro] Use correct DatumWriter constructor (#5371) @RustedBones
- fix FixQuery scalafix rule for macro-generated classes (#5356) @clairemcginty
📜 Scalafix Migrations
- Refactor FixAvroCoder case application (#5353) @clairemcginty
- Update FixAvroCoder to match transformed Avro SCollections (#5351) @clairemcginty
📗 Documentation
- Update release table for 0.14.5 (#5392) @RustedBones
- Prefer waitUndilDone over waitUntilFinish in test and examples (#5349) @RustedBones
🧪 Test Improvements
- Drop slow test tag (#5391) @RustedBones
🏗️ Build Improvements
- Update scalac-options to 0.1.5 (#5385) @scala-steward
- Fix assembly conflict (#5382) @RustedBones
- Update munit to 1.0.0 (#5378) @scala-steward
- [avro] Fix CI test for avro latest (#5372) @RustedBones
- Update sbt, sbt-dependency-tree to 1.10.0 (#5365) @scala-steward
- Update sbt-typelevel to 0.7.1 (#5366) @scala-steward
- Update socco-ng to 0.11.12 (#5368) @RustedBones
- Update sbt-scalafix to 0.12.1 (#5357) @scala-steward
- Update sbt-scoverage to 2.0.12 (#5363) @scala-steward
- (chore) Update to codecov-action (#5354) @RustedBones
- Enable mima for test artifacts (#5343) @RustedBones
- Update scalacheck to 1.18.0 (#5344) @scala-steward
🌱 Dependency Updates
- Sync flink artifact (#5393) @RustedBones
- Update elasticsearch-java to 7.17.21 (#5390) @RustedBones
- Update elasticsearch-java to 8.14.0 (#5388) @scala-steward
- Update neo4j-java-driver to 4.4.17 (#5384) @scala-steward
- Update algebra, cats-core, cats-kernel to 2.12.0 (#5383) @scala-steward
- Update magnolify to 0.7.3 (#5381) @RustedBones
- Update mssql-jdbc to 12.6.2.jre11 (#5380) @scala-steward
- Update jedis to 5.1.3 (#5376) @scala-steward
- Update shapeless to 2.3.12 (#5374) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.1 (#5369) @scala-steward
- Update scala-compiler, scala-library, ... to 2.13.14 (#5362) @scala-steward
- Update mysql-connector-j to 8.4.0 (#5359) @scala-steward
- Update beam to 2.56 (#5346) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.7 (#5360) @scala-steward
Contributors to this release
Contributors
kellen, clairemcginty, and 2 other contributors
Assets 3
v0.14.4
Includes Beam 2.55.1 support.
🚀 Enhancements
- Robust throwable kryo coder (#5318) @RustedBones
- Use JacksonJsonpMapper as default for Elasticsearch (#5306) @kellen
- Allow String key type to transform SMB sources with CharSequence key (#5297) @clairemcginty
🐛 Bug Fixes
- [scio-avro] fix: allow conversions field in record (#5332) @RustedBones
- Add all avro logical type conversions to model (#5301) @RustedBones
- fix: safer implementation of distinctBy (#5299) @RustedBones
📜 Scalafix Migrations
- scalafix: Rule to migrate deprecated query to queryRaw (#5302) @RustedBones
📗 Documentation
- Prepare release v0.14.4 (#5342) @RustedBones
- [doc] Fix scaladoc link to class declared in package objects (#5316) @RustedBones
- Add staging dir to REPL docs (#5308) @kellen
🧪 Test Improvements
- Move test code in respective projects (#5330) @RustedBones
- [integration] Use literals for fromSchemaFile integration test (#5334) @RustedBones
- [integration] Fix avro integration test (#5333) @RustedBones
🏗️ Build Improvements
- Update sbt-site, sbt-site-paradox to 1.7.0 (#5324) @scala-steward
- Update sbt-paradox to 0.10.7 (#5325) @scala-steward
- Update scalafmt-core to 3.8.1 (#5320) @scala-steward
- Update sbt-buildinfo to 0.12.0 (#5310) @scala-steward
- Update sbt-assembly to 2.2.0 (#5304) @scala-steward
🌱 Dependency Updates
- Update scalacheck to 1.17.1 (#5341) @scala-steward
- Update cassandra-all to 3.11.17 (#5340) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.0 (#5339) @scala-steward
- Update pprint to 0.9.0 (#5336) @scala-steward
- Update scala-collection-compat to 2.12.0 (#5337) @scala-steward
- Update beam to 2.55.1 (#5322) @RustedBones
- Update elasticsearch-java to 8.13.2 (#5323) @scala-steward
- Update neo4j-java-driver to 4.4.15 (#5328) @scala-steward
- Update magnolify-avro, magnolify-bigtable, ... to 0.7.2 (#5319) @scala-steward
- Update elasticsearch v7 to 7.17.19 (#5317) @RustedBones
- Update beam to 2.55 (#5307) @RustedBones
- Update sbt-site, sbt-site-paradox to 1.6.0 (#5311) @scala-steward
- Update voyager to 2.0.6 (#5313) @scala-steward
Contributors to this release
Contributors
kellen, clairemcginty, and 2 other contributors
Assets 3
v0.14.3
🐛 Bug Fixes
- fix: Execute SmbIO output assertions (#5289) @RustedBones
- (fix #5290) Support empty input in TransformOverride.ofSource (#5293) @clairemcginty
- (Fix #5285) Allow String key type to read SMB sources written with CharSequence key type (#5291) @clairemcginty
🌱 Dependency Updates
- Update jedis to 5.1.2 (#5292) @scala-steward
- Update neo4j-java-driver to 4.4.14 (#5288) @scala-steward
🏗️ Build Improvements
- Update sbt-scalafix to 0.12.0 (#5287) @scala-steward
Contributors to this release
Contributors
clairemcginty, RustedBones, and scala-steward
Assets 3
v0.14.2
Contains a bugfix for duplicated SMB Transforms.
🐛 Bug Fixes
- (fix) Remove duplicated transform application (#5283) @RustedBones
🚀 Enhancements
- Make checkVersion more dynamic (#5282) @RustedBones
- Improve implicit coder not found message (#5281) @RustedBones
📗Documentation
- (doc) fix 0.14 migration guide broken links (#5278) @RustedBones
🏗️ Build Improvements
- Update scala to 2.13.13 and 2.12.19 (#5276) @RustedBones
Contributors to this release
Contributors
RustedBones