Vitess v2.2.0-rc.1

This is the first release candidate of the 2.2 release.

We expect minimal changes between this and the final 2.2 release.

Download

• Server binaries are available in the Docker image "vitess/lite:v2.2.0-rc.1".
• Java client packages are published on Maven Central.
• Source code is available via the tag "v2.2.0-rc.1".

Upgrading

All Vitess 2.2 changes are backward compatible with the 2.1.1 release. Please make sure that you follow our recommended upgrade order.

Changes since the 2.1.1 Release

VTGate

• Query normalization
• Streamlog and plan stats
• Instant-commit for autocommit
• Smarter pass-through of SQL statements for unsharded keyspaces or single-shard targets.
• Pass-through support for information_schema and dual table
• Improved master detection (if multiple vttablets declare themselves to be master)
• Multi-column vindexes

VTTablet

• Heartbeat
• Can run on a separate machine from its MySQL
• Hot row protection for DMLs
• Per-user transaction limits

Improved RBR support

• Vttablet allows all ‘INSERT..ON DUPLICATE KEY’ constructs if MySQL is in RBR mode.
• JSON support for RBR filtered replication and update stream
• A separate flag to pass-through all DMLs without rewriting them.

New Session-based protocol with support for TargetString and autocommit

• Legacy mode flag for those transitioning to the new behavior
• Auth plugin

MySQL protocol behaves more like mysql

• Autocommit is on by default, and mimics MySQL for set autocommit, begin & commit commands.
• Support of use statement with the ability to target keyspace, shard, tablet type or a combination of those.
• Support for CLIENT_FOUND_ROWS connection flag
• Support streaming

New constructs:

• SET TRANSACTION_MODE=[single,multi,2pc]
• SELECT NEXT :n VALUES FROM seq
• SHOW VINDEXES

Messages

• Multi-column support
• ACL
• New STREAM sql syntax

More query constructs

• Scatter aggregates, order by and limit (count and offset)
• Cross-shard delete
• INSERT ON DUPLICATE KEY with lookup vindexes
• INSERT IGNORE with lookup vindexes
• Cross-shard sorting of date/time and text columns
• UPDATE of secondary lookup vindex columns

Vindexes

• Autocommit support for lookup vindexes
• Write_only support which can be used for migration
• Keyrange support
• Null vindex

GRPC

• Upgrade to 1.10.0, faster and more efficient
• Allow setting of max message size for GRPC

Misc

• VTExplain: new tool for previewing query plans end-to-end
• Vtqueryserver: a new mysql proxy that has all the guardrails of vttablet, but can be used directly from an app.
• New docker-compose example
• New kubernetes helm charts
• Apache V2 license
• Move to CNCF
• Java client moved under io.vitess root
• Query logging support to output as JSON and log to a file
• Improved caching of topology server state at vtgate to allow queries to continue when the topo service is unavailable.
• Optional flags to limit the time a query will wait for a connection from the pool and the total number of outstanding requests.
• Options to redact and/or truncate sensitive queries in the debug UIs

Deprecated

• Queryserver-config-strict-mode
• PHP client
• V1 toposerver scheme

Vitess 2.1.1

Bugfix release for Vitess 2.1.

Changes since 2.1:

• Fixed issue that auto inc columns for unsharded keyspaces were not working (#2804).

If you are upgrading from a version < 2.1.0, please read the "Upgrading" section from the 2.1 release notes: https://github.com/youtube/vitess/releases/tag/v2.1.0

Vitess 2.1

Vitess 2.1 brings many new features since the last major release 2.0.

Highlights are Two-Phase Commit (2PC) support and early support for the native MySQL protocol.

See below for full details.

Download

• Server binaries are available in the Docker image "vitess/lite:v2.1.0".
• Java client packages are published on Maven Central.
• Source code is available via the tag "v2.1.0".

Upgrading

All Vitess 2.1 changes are backward compatible with the 2.0 release. Please make sure that you follow our recommended upgrade order.

If you use custom start scripts, you will have to change the command line flags which specify the topology (zookeeper or etcd) addresses. New flags are:

-topo_implementation zookeeper|etcd|zk2|etcd2|consul
-topo_global_server_address global_server1,global_server2
-topo_global_root /vitess/global

Additionally, for the zk2/etcd2/consul implementations, you'll have to register cell local topology servers with the vtctl AddCellInfo command. See our documentation for details.

This version also adds new topology plugins ("zk2", "etcd2") with a different storage format. The legacy plugins "zookeeper" and "etcd" will continue to work in 2.1 but will be deprecated in the next major version. Therefore, we recommend to migrate to one of the new plugins once you updated to 2.1.

Changes since the 2.0 Release

• Our website vitess.io has a new logo! :)
• Kubernetes:
  • Added Helm chart and support for StatefulSet.
• all:
  • Topology:
    • Rewrote plugins for etcd and Zookeeper (etcd2, zk2). Please see the doc how to migrate to the new storage format.
      Command line flags have changed as well (shipped examples are updated).
    • New plugin: Consul.
  • Internal: We have changed the internal error representation in the RPC messages in a backward compatible way.
• clients:
  • Java:
    • TLS support in client and JDBC driver.
    • Unified package structure under new root io.vitess.
    • 2.1.0 packages are published on Maven Central e.g. our JDBC driver as artifact vitess-jdbc.
  • Map Reduce:
    • Faster execution on tables with a non-uniform distribution by sampling rows and splitting the table into equal-sized row ranges.
• vtctld:
  • New Angular 2 based UI.
  • Online Schema Swap: Apply complex schema changes without any downtime.
  • Resharding workflow. Automatically runs the resharding steps and shows them in the Workflow UI Tab. New tutorials added which use the workflow.
• vtgate:
  • 2PC: Support for distributed transactions.
  • Native MySQL server protocol (alpha).
  • Direct access to shards via "keyspace:shard" database name.
  • Buffer master traffic during failovers and retry to avoid client errors (disabled by default).
  • Support for SHOW statements to expose Vitess metadata.
  • Increase large installation scalability with (optional) two-layer vtgate pools (l2vtgate, applicable to 100+ shard installations).
• vttablet:
  • Native MySQL client protocol (instead of using libmysqlclient) (disabled by default).
  • Support for MySQL Row-based-Replication e.g. needed when resharding (alpha).
  • Update Stream: Applications can subscribe to a change stream (e.g. to invalidate an app level cache).
  • Messages: Native support for a queue processing framework (alpha).
  • Hot Row Protection: Serializes write transaction for the same row (range) to prevent that all transaction pool slots can be exhausted (disabled by default).
  • Transaction Throttler: Rejects new transactions with retryable error if replication lag on replicas is too high (disabled by default).

Other Noteworthy Changes

• Minimum required Go version changed to 1.8 (from 1.5).
• Updated to gRPC 1.0.4 (from 1.0.0).

Vitess v2.1.0-rc.1

This is the first release candidate of the 2.1 release.

We expect minimal changes between this and the final 2.1 release.

Download

• Server binaries are available in the Docker image "vitess/lite:v2.1.0-rc.1".
• Java client packages are published on Maven Central.
• Source code is available via the tag "v2.1.0-rc.1".

Upgrading

All Vitess 2.1 changes are backward compatible with the 2.0 release. Please make sure that you follow our recommended upgrade order.

Changes since the 2.0 Release

• Our website vitess.io has a new logo! :)
• Kubernetes:
  • Added Helm chart and support for StatefulSet.
• all:
  • Topology:
    • Rewrote plugins for etcd and Zookeeper (etcd2, zk2). Please see the doc how to migrate to the new storage format.
      Command line flags have changed as well (shipped examples are updated).
    • New plugin: Consul.
  • Internal: We have changed the internal error representation in the RPC messages in a backward compatible way.
• clients:
  • Java:
    • TLS support in client and JDBC driver.
    • Unified package structure under new root io.vitess.
    • 2.1.0-rc.1 packages are published on Maven Central e.g. our JDBC driver as artifact vitess-jdbc.
  • Map Reduce:
    • Faster execution on tables with a non-uniform distribution by sampling rows and splitting the table into equal-sized row ranges.
• vtctld:
  • New Angular 2 based UI.
  • Online Schema Swap: Apply complex schema changes without any downtime.
  • Resharding workflow. Automatically runs the resharding steps and shows them in the Workflow UI Tab. New tutorials added which use the workflow.
• vtgate:
  • 2PC: Support for distributed transactions.
  • Native MySQL server protocol (alpha).
  • Direct access to shards via "keyspace:shard" database name.
  • Buffer master traffic during failovers and retry to avoid client errors (disabled by default).
  • Support for SHOW statements to expose Vitess metadata.
  • Increase large installation scalability with (optional) two-layer vtgate pools (l2vtgate, applicable to 100+ shard installations).
• vttablet:
  • Native MySQL client protocol (instead of using libmysqlclient) (disabled by default).
  • Support for MySQL Row-based-Replication e.g. needed when resharding (alpha).
  • Update Stream: Applications can subscribe to a change stream (e.g. to invalidate an app level cache).
  • Messages: Native support for a queue processing framework (alpha).
  • Hot Row Protection: Serializes write transaction for the same row (range) to prevent that all transaction pool slots can be exhausted (disabled by default).
  • Transaction Throttler: Rejects new transactions with retryable error if replication lag on replicas is too high (disabled by default).

Other Noteworthy Changes

• Minimum required Go version changed to 1.8 (from 1.5).
• Updated to gRPC 1.0.4 (from 1.0.0).

Vitess v2.1.0-alpha.1

This is the first alpha release of the upcoming 2.1 version.

2.1 will be a feature release which will bring the following new features:

• Support for distributed transactions, using 2 phase commit.
• Resharding workflow improvements, to increase manageability of the process.
• Online schema swap, to apply complex schema changes without any downtime.
• New dynamic UI (vtctld), rewritten from scratch in angular2.
• Update Stream functionality, for applications to subscribe to a change stream (for cache invalidation, for instance).
• Improved Map-Reduce support, for tables with non-uniform distributions.
• Increase large installation scalability with two-layer vtgate pools (l2vtgate, applicable to 100+ shard installations).
• Better Kubernetes support (Helm support, better scripts, ...).

Note that some of these features are still in development and not fully documented. This will change before we release the final 2.1 version.

Other noteworthy changes:

• Minimum required Go version changed to 1.7 (from 1.5).
• Updated to gRPC 1.0.0 (from 0.13.0).
• Added Percona Docker Images (changes contributed by Stitch Labs).

Vitess v2.0.0

Release to Manufacturing

This is the code cut for what will become the first GA release of Vitess 2.0.
We're now focused on updating documentation in preparation for General Availability.

Changes since v2.0.0-rc.2

Code Cleanup

• vtctld: Remove experimental VSchema editor UI

Vitess v2.0.0-rc.2

Changes since v2.0.0-rc.1

Bug Fixes

• Automatically create /zk root if it doesn't exist
• Fix edge cases in MariaDB backup/restore + reparent scenarios
• Randomize MySQL server-id when initializing tablet from scratch
  • Fixes edge case of replicate-same-server-id=0 together with backup/restore of former master.
• Fix for panic during resharding with unicode_loose_md5 vindex
• Fix for ApplySchema fails on DDLS which don't result in a schema change
• Fix for using SplitQuery with VTGate High-Level API
• VSchema for vtcombo is per-keyspace
  • This better matches the per-keyspace nature of VSchema in topology.
• Verify that autocommit is ON
  • This is required for proper operation of connection pooling.
• ApplySchema now reloads schema on slaves
  • Previously, only the master would immediately reload schema.
• Enable admin commands on vtcombo

Plugins

• Add backup storage plugin for Amazon S3

Vitess v2.0.0-rc.1

Changes since v2.0.0-beta.2

This is the first release candidate for v2.0.0. We are now freezing the release-2.0 branch for anything except bug fixes.

Client API

• Allow specifying a connection-wide default keyspace for VTGateV3 queries.
  (#1725,
  #1727,
  #1738)
  • Similar to the concept of a default database for a MySQL client connection.

SQL Support

• Preserve case of column names in query result field info.
• Preserve trailing comments in queries rewritten by VTGateV3.
• Support string-based hash vindexes
  • Previously, the value to hash had to be numeric or binary.

Management

• Hide VtTablet* and VtGate* commands behind a flag in vtctl/vtctld
  (#1700).
  • These commands are mostly meant for development, and should usually not be
    allowed in production.
  • Launch vtctld with the -enable_queries flag to allow these commands.
• Restart replication automatically
  if a tablet is restarted (e.g. as part of a Vitess upgrade).
• Optionally allow the use of insecure caller ID.
• It's no longer required to set a "sharding column name" when using VTGateV3.
  • i.e. no more vtctl SetKeyspaceShardingInfo when using VTGateV3.

Schema and VSchema

• VSchema config format changed
  • Some fields have been renamed for consistency.
• VSchema updates are now denormalized into per-cell records.
  (#1750)
  • No longer need to restart VTGate to pick up VSchema for new/removed
    keyspaces.
  • VSchema is still available if global topology goes down.
  • It's now possible to atomically move tables between keyspaces, for example
    when doing a vertical split.
  • NOTE: Even if you don't use VSchema, it's now required to run
    RebuildVSchemaGraph
    to initialize these records for a new keyspace.
• Schema changes are tested as a group, rather than individually.
  (#1749)
  • This allows batches of schema changes that depend on one another (#1731).

Interoperability

• Support MySQL 5.7
• Orchestrator Integration
  • Along with some patches contributed to Orchestrator, these new Vitess
    features allow the two systems to work together automatically.
    We'll be updating the documentation soon to show how to use it.

Code Cleanup

• Remove deprecated "serving graph" system, which had already been replaced by
  streaming health checks.
  • #1679,
    #1683,
    #1685,
    #1698,
    #1699,
    #1706
• Remove deprecated rowcache
  • This will be replaced by a caching scheme that makes sense for both VTGateV2
    and VTGateV3.
• Remove deprecated -target_tablet_type flag
  • Use -init_tablet_type instead.

Vitess v2.0.0-beta.2

Changes since v2.0.0-beta.1

Client Libraries

• JDBC-compatible wrapper
  • Contributed by Flipkart
  • See JDBC Example
• Update Java client to better match semantics of JDBC ResultSet
  • columnIndex now starts at 1
  • columnLabel is now case-insensitive
• Java: Synchronous client wrapper
  • To simplify and enforce proper usage of Java client in blocking mode.
• Java: Support more bind variable types
• Python: Always use colon style for bind variables
  • Removes extra translation layer for % formatting from Python client.
  • This makes the bind variable style consistent across all Vitess clients.
  • See local example client changes

SQL Support

• Phase 2 of VTGate V3 API
  • Can now route more complex queries.
  • Can perform cross-shard joins.
  • See phase 2 of V3 design for what is now supported.
• Cross-shard sequences
  • Note that NEXT is now a reserved word, and must be back-quoted if used as a column name.
• Support INTERVAL construct
• Support keyspace-qualified names
  • i.e. SELECT * FROM keyspace.table, analogous to db.table outside Vitess.
• Support STRAIGHT_JOIN and UNION DISTINCT SELECT

Resharding

• Support merging shards
• Support resharding tables with no keyspace_id column
  • As long as there is a VSchema that allows Vitess to compute the keyspace_id.
• SplitDiff: support custom sharding column name

Deployment and Management

• Configure SSL and authentication for gRPC
  • See Transport Security Model docs.
• VTGate watches for VSchema updates and refreshes automatically
• V3: Allow duplicate table names in vschema
  • As long as they're in different keyspaces.
• V3: Simplify VSchema
  • Remove the Class concept
  • VSchema is now per-keyspace
• Removed deprecated gorpc-based protocols
  • All protocols are now based on gRPC.
• Ceph Backup Storage Plugin
  • Contributed by Flipkart

Build Dependencies

• Pre-built Docker images for Percona Server
• We now depend on Go 1.5+ for native vendoring

Analysis

• Pre-scan algorithm for MapReduce/Hadoop query splitting

Vitess v2.0.0-beta.1

Changes since v2.0.0-alpha5

Client Libraries

• VTGateV3 Go driver
  can now also support certain types of VTGateV2 queries
• Publish PHP client to Packagist
• Refactor PHP client for better standards compliance
• PHP client returns associative array rows
• Java client is asynchronous by default
• Java client now returns 0 when converting SQL NULL to primitive types
• Python client now defaults to gRPC
• More sample clients for examples/local
  • In addition to Python,
    there are now sample clients for Go,
    PHP,
    and Java.

Query Service

• query.proto revamp
  • Fix NULL handling in new proto3-based query result format.
  • Unify type system for bind variables and query results.
• query_rules improvements
  • Improved JSON syntax for bindvar constraints.
  • Debug URL for viewing rule sets.
• Support REGEXP and RLIKE in
  SQL parser

Build Dependencies

• Default ZooKeeper version upgraded
  from 3.3.5 to 3.4.6.
• Update to gRPC 0.13
• Support building on OS X

Deployment and Management

• Add vtctld server to vtcombo
• Kubernetes config updates
  • Use Kubernetes DNS for service discovery
    and etcd bootstrap.
  • Kubernetes example for MySQL 5.6.
  • Use Kubernetes proxy to link from vtctld to vttablet status pages.
• Make MySQL users and privileges configurable
  when bootstrapping MySQL data directory.
  • Uses mysql_install_db and init_db.sql
    instead of extracting the old mysql-db-dir.tbz archive.
• Support semi-sync without async fallback
• Remove old vtctld UI files
  • Deployments no longer need to include go/cmd/vtctld/templates folder or
    set -templates flag on vtctld.
• Add option to bypass sanity check in ApplySchema
• Print human-readable tables for CLI queries
  • e.g. for ExecuteFetchAsDba