Vitess v16.0.5

Release of Vitess v16.0.5

The entire changelog for this release can be found here.

The release includes 29 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/github-actions, @app/vitess-bot, @arthurschreiber, @rohit-nayak-ps

Vitess v15.0.5

Release of Vitess v15.0.5

The entire changelog for this release can be found here.

The release includes 20 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @arthurschreiber, @frouioui, @shlomi-noach, @systay

Vitess v17.0.2

Changelog of Vitess v17.0.2

Bug fixes

Backup and Restore

• [release-17.0] Address vttablet memory usage with backups to Azure Blob Service (#13770) #13775
• [release-17.0] Do not drain tablet in incremental backup (#13773) #13789

Cluster management

• [release-17.0] Flaky tests: Fix race in memory topo (#13559) #13577

Evalengine

• [release-17.0] Fix a number of encoding issues when evaluating expressions with the evalengine (#13509) #13551
• [release-17.0] fastparse: Fix bug in overflow detection (#13702) #13705

Online DDL

• v17 backport: Fix closed channel panic in Online DDL cutover #13731
• v17 backport: Solve RevertMigration.Comment read/write concurrency issue #13734

Query Serving

• [release-17.0] Fix flaky vtgate test TestInconsistentStateDetectedBuffering (#13560) #13575
• [release-17.0] vtgate: fix race condition iterating tables and views from schema tracker (#13673) #13796

CI/Build

Backup and Restore

• [release-17.0] Fixing backup_pitr flaky tests via wait-for loop on topo reads (#13781) #13790

Online DDL

• [release-17.0] CI: fix onlineddl_scheduler flakiness (#13754) #13760

Release

General

• Back to dev mode after v17.0.1 #13663

Testing

Build/CI

• [release-17.0] Flakes: Delete VTDATAROOT files in reparent test teardown within CI (#13793) #13798

Vitess v16.0.4

Release of Vitess v16.0.4

The entire changelog for this release can be found here.

The release includes 11 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @dbussink, @rohit-nayak-ps, @shlomi-noach, @systay

Vitess v17.0.1

Release of Vitess v17.0.1

The entire changelog for this release can be found here.

The release includes 23 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @frouioui, @mattlord, @shlomi-noach

Vitess v16.0.3

Release of Vitess v16.0.3

The entire changelog for this release can be found here.

The release includes 38 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/github-actions, @app/vitess-bot, @frouioui, @harshit-gangal, @shlomi-noach, @systay

Vitess v15.0.4

Release of Vitess v15.0.4

The entire changelog for this release can be found here.

The release includes 33 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @frouioui, @harshit-gangal, @shlomi-noach, @systay

Vitess v17.0.0

Release of Vitess v17.0.0

Summary

Table of Contents

• Major Changes
  • Breaking Changes
    • VTTablet: Initializing all replicas with super_read_only
    • Default Local Cell Preference for TabletPicker
    • Dedicated stats for VTGate Prepare operations
    • VTAdmin web migrated from create-react-app to vite
    • Keyspace name validation in TopoServer
    • Shard name validation in TopoServer
    • Compression CLI flags removed from vtctld and vtctldclient binaries
    • VtctldClient command RestoreFromBackup will now use the correct context
    • VTTablet Restore Metrics
  • New command line flags and behavior
    • Builtin backup: read buffering flags
    • Manifest backup external decompressor command
    • Throttler config via topo enabled by default
  • New stats
    • Detailed backup and restore stats
    • VTtablet Error count with code
    • VReplication stream status for Prometheus
  • Online DDL
    • --cut-over-threshold DDL strategy flag
  • VReplication
    • Support for MySQL 8.0 binlog_transaction_compression
    • Support for the noblob binlog row image mode
  • VTTablet
    • VTTablet: Initializing all replicas with super_read_only
    • Vttablet Schema Reload Timeout
    • Settings pool enabled
  • VTGate
    • StreamExecute GRPC API
    • Insert Planner Gen4
  • Deprecations and Deletions
    • Deprecated Flags
    • Deprecated Stats
    • Deprecated vtgr
    • Deprecated k8stopo

Major Changes

Breaking Changes

Default Local Cell Preference for TabletPicker

We added options to the TabletPicker that allow for specifying a cell preference in addition to making the default behavior to give priority to the local cell and any alias it belongs to. We are also introducing a new way to select tablet type preference which should eventually replace the in_order: hint currently used as a prefix for tablet types. The signature for creating a new TabletPicker now looks like:

func NewTabletPicker(
	ctx context.Context,
	ts *topo.Server,
	cells []string,
	localCell, keyspace, shard, tabletTypesStr string,
	options TabletPickerOptions,
) (*TabletPicker, error) {...}

Where ctx, localCell, option are all new parameters.

option is of type TabletPickerOptions and includes two fields, CellPreference and TabletOrder.

• CellPreference: "PreferLocalWithAlias" (default) gives preference to vtgate's local cell, or "OnlySpecified" which only picks from the cells explicitly passed in by the client
• TabletOrder: "Any" (default) for no ordering or random, or "InOrder" to use the order specified by the client

See PR 12282 Description for examples on how this changes cell picking behavior.

Default TLS version changed for vtgr

When using TLS with vtgr, we now default to TLS 1.2 if no other explicit version is configured. Configuration flags are provided to explicitly configure the minimum TLS version to be used.

vtgr is now deprecated as part of v17.0.0, please see the deprecation notice.

Dedicated stats for VTGate Prepare operations

Prior to v17 Vitess incorrectly combined stats for VTGate Execute and Prepare operations under a single stats key (Execute). In v17 Execute and Prepare operations generate stats under independent stats keys.

Here is a (condensed) example of stats output:

{
  "VtgateApi": {
    "Histograms": {
      "Execute.src.primary": {
        "500000": 5
      },
      "Prepare.src.primary": {
        "100000000": 0
      }
    }
  },
  "VtgateApiErrorCounts": {
    "Execute.src.primary.INVALID_ARGUMENT": 3,
    "Execute.src.primary.ALREADY_EXISTS": 1
  }
}

VTAdmin web migrated to vite

Previously, VTAdmin web used the Create React App framework to test, build, and serve the application. In v17, Create React App has been removed, and Vite is used in its place. Some of the main changes include:

• Vite uses VITE_* environment variables instead of REACT_APP_* environment variables
• Vite uses import.meta.env in place of process.env
• Vitest is used in place of Jest for testing
• Our protobufjs generator now produces an es6 module instead of commonjs to better work with Vite's defaults
• public/index.html has been moved to root directory in web/vtadmin

Keyspace name validation in TopoServer

Prior to v17, it was possible to create a keyspace with invalid characters, which would then be inaccessible to various cluster management operations.

Keyspace names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetKeyspace and CreateKeyspace methods return an error if given an invalid name.

Shard name validation in TopoServer

Prior to v17, it was possible to create a shard name with invalid characters, which would then be inaccessible to various cluster management operations.

Shard names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetShard and CreateShard methods return an error if given an invalid name.

Compression CLI flags remove from vtctld and vtctldclient binaries

The CLI flags below were mistakenly added to vtctld and vtctldclient in v15. In v17, they are no longer present in those binaries.

• --compression-engine-name
• --compression-level
• --external-compressor
• --external-compressor-extension
• --external-decompressor

VtctldClient command RestoreFromBackup will now use the correct context

The VtctldClient command RestoreFromBackup initiates an asynchronous process on the specified tablet to restore data from either the latest backup or the closest one before the specified backup-timestamp.
Prior to v17, this asynchronous process could run indefinitely in the background since it was called using the background context. In v17 PR#12830,
this behavior was changed to use a context with a timeout of action_timeout. If you are using VtctldClient to initiate a restore, make sure you provide an appropriate value for action_timeout to give enough
time for the restore process to complete. Otherwise, the restore will throw an error if the context expires before it completes.

VTTablet Restore Metrics

As part of the VTTablet Sidecar Schema Maintenance Refactor in v16.0.0, we dropped the local_metadata table from the sidecar database schema. This table was storing a couple of metrics related to restores from backup.
They have now been re-introduced as metrics that can be accessed from /debug/vars.

Vttablet's transaction throttler now also throttles DML outside of BEGIN; ...; COMMIT; blocks

Prior to v17, vttablet's transaction throttler (enabled with --enable-tx-throttler) would only throttle requests done inside an explicit transaction, i.e., a BEGIN; ...; COMMIT; block.
In v17 PR#13040, this behavior was being changed so that it also throttles work outside of explicit transactions for INSERT/UPDATE/DELETE/LOAD queries.

New command line flags and behavior

Backup --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size

Prior to v17 the builtin Backup Engine does not use read buffering for restores, and for backups uses a hardcoded write buffer size of 2097152 bytes.

In v17 these defaults may be tuned with, respectively --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size.

• --builtinbackup-file-read-buffer-size: read files using an IO buffer of this many bytes. Golang defaults are used when set to 0.
• --builtinbackup-file-write-buffer-size: write files using an IO buffer of this many bytes. Golang defaults are used when set to 0. (default 2097152)

These flags are applicable to the following programs:

• vtbackup
• vtctld
• vttablet
• vttestserver

Manifest backup external decompressor command

Add a new builtin/xtrabackup flag --manifest-external-decompressor. When set the value of that flag is stored in the manifest field ExternalDecompressor. This manifest field may be consulted when decompressing a backup that was c...

Vitess v17.0.0-rc2

Release of Vitess v17.0.0-rc2

Summary

Table of Contents

• Major Changes
  • Breaking Changes
    • Default Local Cell Preference for TabletPicker
    • Dedicated stats for VTGate Prepare operations
    • VTAdmin web migrated from create-react-app to vite
    • Keyspace name validation in TopoServer
    • Shard name validation in TopoServer
    • Compression CLI flags removed from vtctld and vtctldclient binaries
    • VtctldClient command RestoreFromBackup will now use the correct context
  • New command line flags and behavior
    • Builtin backup: read buffering flags
    • Manifest backup external decompressor command
    • Throttler config via topo enabled by default
  • New stats
    • Detailed backup and restore stats
    • VTtablet Error count with code
    • VReplication stream status for Prometheus
  • Online DDL
    • --cut-over-threshold DDL strategy flag
  • VReplication
    • Support for MySQL 8.0 binlog_transaction_compression
    • Support for the noblob binlog row image mode
  • VTTablet
    • VTTablet: Initializing all replicas with super_read_only
    • Vttablet Schema Reload Timeout
    • Settings pool enabled
  • VTGate
    • StreamExecute GRPC API
    • Insert Planner Gen4
  • Deprecations and Deletions
    • Deprecated Flags
    • Deprecated Stats
    • Deprecated vtgr
    • Deprecated k8stopo

Major Changes

Breaking Changes

Default Local Cell Preference for TabletPicker

We added options to the TabletPicker that allow for specifying a cell preference in addition to making the default behavior to give priority to the local cell and any alias it belongs to. We are also introducing a new way to select tablet type preference which should eventually replace the in_order: hint currently used as a prefix for tablet types. The signature for creating a new TabletPicker now looks like:

func NewTabletPicker(
	ctx context.Context,
	ts *topo.Server,
	cells []string,
	localCell, keyspace, shard, tabletTypesStr string,
	options TabletPickerOptions,
) (*TabletPicker, error) {...}

Where ctx, localCell, option are all new parameters.

option is of type TabletPickerOptions and includes two fields, CellPreference and TabletOrder.
CellPreference: "PreferLocalWithAlias" (default) gives preference to vtgate's local cell, or "OnlySpecified" which only picks from the cells explicitly passed in by the client TabletOrder`: "Any" (default) for no ordering or random, or "InOrder" to use the order specified by the client

See PR 12282 Description for examples on how this changes cell picking behavior.

Default TLS version changed for vtgr

When using TLS with vtgr, we now default to TLS 1.2 if no other explicit version is configured. Configuration flags are provided to explicitly configure the minimum TLS version to be used.

Dedicated stats for VTGate Prepare operations

Prior to v17 Vitess incorrectly combined stats for VTGate Execute and Prepare operations under a single stats key (Execute). In v17 Execute and Prepare operations generate stats under independent stats keys.

Here is a (condensed) example of stats output:

{
  "VtgateApi": {
    "Histograms": {
      "Execute.src.primary": {
        "500000": 5
      },
      "Prepare.src.primary": {
        "100000000": 0
      }
    }
  },
  "VtgateApiErrorCounts": {
    "Execute.src.primary.INVALID_ARGUMENT": 3,
    "Execute.src.primary.ALREADY_EXISTS": 1
  }
}

VTAdmin web migrated to vite

Previously, VTAdmin web used the Create React App framework to test, build, and serve the application. In v17, Create React App has been removed, and Vite is used in its place. Some of the main changes include:

• Vite uses VITE_* environment variables instead of REACT_APP_* environment variables
• Vite uses import.meta.env in place of process.env
• Vitest is used in place of Jest for testing
• Our protobufjs generator now produces an es6 module instead of commonjs to better work with Vite's defaults
• public/index.html has been moved to root directory in web/vtadmin

Keyspace name validation in TopoServer

Prior to v17, it was possible to create a keyspace with invalid characters, which would then be inaccessible to various cluster management operations.

Keyspace names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetKeyspace and CreateKeyspace methods return an error if given an invalid name.

Shard name validation in TopoServer

Prior to v17, it was possible to create a shard name with invalid characters, which would then be inaccessible to various cluster management operations.

Shard names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetShard and CreateShard methods return an error if given an invalid name.

Compression CLI flags remove from vtctld and vtctldclient binaries

The CLI flags below were mistakenly added to vtctld and vtctldclient in v15. In v17, they are no longer present in those binaries.

• --compression-engine-name
• --compression-level
• --external-compressor
• --external-compressor-extension
• --external-decompressor

VtctldClient command RestoreFromBackup will now use the correct context

The VtctldClient command RestoreFromBackup initiates an asynchronous process on the specified tablet to restore data from either the latest backup or the closest one before the specified backup-timestamp.
Prior to v17, this asynchronous process could run indefinitely in the background since it was called using the background context. In v17 PR#12830,
this behavior was changed to use a context with a timeout of action_timeout. If you are using VtctldClient to initiate a restore, make sure you provide an appropriate value for action_timeout to give enough
time for the restore process to complete. Otherwise, the restore will throw an error if the context expires before it completes.

Vttablet's transaction throttler now also throttles DML outside of BEGIN; ...; COMMIT; blocks

Prior to v17, vttablet's transaction throttler (enabled with --enable-tx-throttler) would only throttle requests done inside an explicit transaction, i.e., a BEGIN; ...; COMMIT; block.
In v17 PR#13040, this behavior was being changed so that it also throttles work outside of explicit transactions for INSERT/UPDATE/DELETE/LOAD queries.

New command line flags and behavior

Backup --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size

Prior to v17 the builtin Backup Engine does not use read buffering for restores, and for backups uses a hardcoded write buffer size of 2097152 bytes.

In v17 these defaults may be tuned with, respectively --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size.

• --builtinbackup-file-read-buffer-size: read files using an IO buffer of this many bytes. Golang defaults are used when set to 0.
• --builtinbackup-file-write-buffer-size: write files using an IO buffer of this many bytes. Golang defaults are used when set to 0. (default 2097152)

These flags are applicable to the following programs:

• vtbackup
• vtctld
• vttablet
• vttestserver

Manifest backup external decompressor command

Add a new builtin/xtrabackup flag --manifest-external-decompressor. When set the value of that flag is stored in the manifest field ExternalDecompressor. This manifest field may be consulted when decompressing a backup that was compressed with an external command.

This feature enables the following flow:

1. Take a backup using an external compressor

    Backup --compression-engine=external \
           --external-compressor=zstd \
           --manifest-external-decompressor="zstd -d"
   

2. Restore that backup with a mere Restore command, without having to specify --external-decompressor.

vttablet --throttler-config-via-topo

This flag was introduced in v16 and defaulted to false. In v17 it defaults to true, and there is no need to supply it.

Note that this flag overrides --enable-lag-throttler and --throttle-threshold, ...

Vitess v17.0.0-rc1

Release of Vitess v17.0.0-rc1

Summary

Table of Contents

• Major Changes
  • Breaking Changes
    • Default Local Cell Preference for TabletPicker
    • Dedicated stats for VTGate Prepare operations
    • VTAdmin web migrated from create-react-app to vite
    • Keyspace name validation in TopoServer
    • Shard name validation in TopoServer
    • Compression CLI flags removed from vtctld and vtctldclient binaries
    • VtctldClient command RestoreFromBackup will now use the correct context
  • New command line flags and behavior
    • Builtin backup: read buffering flags
    • Manifest backup external decompressor command
    • Throttler config via topo enabled by default
  • New stats
    • Detailed backup and restore stats
    • VTtablet Error count with code
    • VReplication stream status for Prometheus
  • Online DDL
    • --cut-over-threshold DDL strategy flag
  • VReplication
    • Support for MySQL 8.0 binlog_transaction_compression
    • Support for the noblob binlog row image mode
  • VTTablet
    • VTTablet: Initializing all replicas with super_read_only
    • Vttablet Schema Reload Timeout
    • Settings pool enabled
  • VTGate
    • StreamExecute GRPC API
    • Insert Planner Gen4
  • Deprecations and Deletions
    • Deprecated Flags
    • Deprecated Stats

Major Changes

Breaking Changes

Default Local Cell Preference for TabletPicker

We added options to the TabletPicker that allow for specifying a cell preference in addition to making the default behavior to give priority to the local cell and any alias it belongs to. We are also introducing a new way to select tablet type preference which should eventually replace the in_order: hint currently used as a prefix for tablet types. The signature for creating a new TabletPicker now looks like:

func NewTabletPicker(
	ctx context.Context,
	ts *topo.Server,
	cells []string,
	localCell, keyspace, shard, tabletTypesStr string,
	options TabletPickerOptions,
) (*TabletPicker, error) {...}

Where ctx, localCell, option are all new parameters.

option is of type TabletPickerOptions and includes two fields, CellPreference and TabletOrder.
CellPreference: "PreferLocalWithAlias" (default) gives preference to vtgate's local cell, or "OnlySpecified" which only picks from the cells explicitly passed in by the client TabletOrder`: "Any" (default) for no ordering or random, or "InOrder" to use the order specified by the client

See PR 12282 Description for examples on how this changes cell picking behavior.

Default TLS version changed for vtgr

When using TLS with vtgr, we now default to TLS 1.2 if no other explicit version is configured. Configuration flags are provided to explicitly configure the minimum TLS version to be used.

Dedicated stats for VTGate Prepare operations

Prior to v17 Vitess incorrectly combined stats for VTGate Execute and Prepare operations under a single stats key (Execute). In v17 Execute and Prepare operations generate stats under independent stats keys.

Here is a (condensed) example of stats output:

{
  "VtgateApi": {
    "Histograms": {
      "Execute.src.primary": {
        "500000": 5
      },
      "Prepare.src.primary": {
        "100000000": 0
      }
    }
  },
  "VtgateApiErrorCounts": {
    "Execute.src.primary.INVALID_ARGUMENT": 3,
    "Execute.src.primary.ALREADY_EXISTS": 1
  }
}

VTAdmin web migrated to vite

Previously, VTAdmin web used the Create React App framework to test, build, and serve the application. In v17, Create React App has been removed, and Vite is used in its place. Some of the main changes include:

• Vite uses VITE_* environment variables instead of REACT_APP_* environment variables
• Vite uses import.meta.env in place of process.env
• Vitest is used in place of Jest for testing
• Our protobufjs generator now produces an es6 module instead of commonjs to better work with Vite's defaults
• public/index.html has been moved to root directory in web/vtadmin

Keyspace name validation in TopoServer

Prior to v17, it was possible to create a keyspace with invalid characters, which would then be inaccessible to various cluster management operations.

Keyspace names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetKeyspace and CreateKeyspace methods return an error if given an invalid name.

Shard name validation in TopoServer

Prior to v17, it was possible to create a shard name with invalid characters, which would then be inaccessible to various cluster management operations.

Shard names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetShard and CreateShard methods return an error if given an invalid name.

Compression CLI flags remove from vtctld and vtctldclient binaries

The CLI flags below were mistakenly added to vtctld and vtctldclient in v15. In v17, they are no longer present in those binaries.

• --compression-engine-name
• --compression-level
• --external-compressor
• --external-compressor-extension
• --external-decompressor

VtctldClient command RestoreFromBackup will now use the correct context

The VtctldClient command RestoreFromBackup initiates an asynchronous process on the specified tablet to restore data from either the latest backup or the closest one before the specified backup-timestamp.
Prior to v17, this asynchronous process could run indefinitely in the background since it was called using the background context. In v17 PR#12830,
this behavior was changed to use a context with a timeout of action_timeout. If you are using VtctldClient to initiate a restore, make sure you provide an appropriate value for action_timeout to give enough
time for the restore process to complete. Otherwise, the restore will throw an error if the context expires before it completes.

Vttablet's transaction throttler now also throttles DML outside of BEGIN; ...; COMMIT; blocks

Prior to v17, vttablet's transaction throttler (enabled with --enable-tx-throttler) would only throttle requests done inside an explicit transaction, i.e., a BEGIN; ...; COMMIT; block.
In v17 PR#13040, this behavior was being changed so that it also throttles work outside of explicit transactions for INSERT/UPDATE/DELETE/LOAD queries.

New command line flags and behavior

Backup --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size

Prior to v17 the builtin Backup Engine does not use read buffering for restores, and for backups uses a hardcoded write buffer size of 2097152 bytes.

In v17 these defaults may be tuned with, respectively --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size.

• --builtinbackup-file-read-buffer-size: read files using an IO buffer of this many bytes. Golang defaults are used when set to 0.
• --builtinbackup-file-write-buffer-size: write files using an IO buffer of this many bytes. Golang defaults are used when set to 0. (default 2097152)

These flags are applicable to the following programs:

• vtbackup
• vtctld
• vttablet
• vttestserver

Manifest backup external decompressor command

Add a new builtin/xtrabackup flag --manifest-external-decompressor. When set the value of that flag is stored in the manifest field ExternalDecompressor. This manifest field may be consulted when decompressing a backup that was compressed with an external command.

This feature enables the following flow:

1. Take a backup using an external compressor

    Backup --compression-engine=external \
           --external-compressor=zstd \
           --manifest-external-decompressor="zstd -d"
   

2. Restore that backup with a mere Restore command, without having to specify --external-decompressor.

vttablet --throttler-config-via-topo

This flag was introduced in v16 and defaulted to false. In v17 it defaults to true, and there is no need to supply it.

Note that this flag overrides --enable-lag-throttler and --throttle-threshold, which now give warnings, and will be removed in v18.

New stats

####...