v1.0.4

What's Changed

• Fix spurious timeout issues by increasing grace period for deadline propagation by @timostamm in #913
• Add test cases for handling empty (zero byte) messages by @jhump in #918
• Add test cases for Connect protocol when "message" property in errors is omitted by @timostamm in #922
• Add test cases for Connect protocol when "debug" property in errors is omitted or invalid by @timostamm in #921

Full Changelog: v1.0.3...v1.0.4

v1.0.3

What's Changed

Bugfixes

• Update test cases for consistency, backfill missing cancel tests by @jhump in #874
• Fix reference server to verify that client implementations properly enforce timeouts by @jhump in #877
• Use strict codecs in reference client and server to produce better error messages in the face of certain codec/format errors by @jhump in #878
• Fix the Connect GET test suite configuration so those tests are only run when a client/server supports GET by @jhump in #883
• Add missing gRPC tests for trailers-only responses by @jhump in #904

Other changes

• Update to use Go to 1.21 by @jhump in #866
• Deprecate the textpb codec by @jhump in #875

Full Changelog: v1.0.2...v1.0.3

v1.0.2

This release includes fixes for testing Connect clients that run in browser environments.

What's Changed

Bugfixes

• Fix reference server issues related to CORS by @jhump in #855 and @srikrsna-buf in #857
• When reference server is in HTTP 1.1 mode, don't allow client to negotiate HTTP/2 via TLS by @jhump in #856

New Contributors

• @srikrsna-buf made their first contribution in #857

Full Changelog: v1.0.1...v1.0.2

v1.0.1

This is a minor bugfix release. It fixes a bug in the reference server that impacts one test case and corrects two gRPC-Web test case definitions.

What's Changed

Bugfixes

• Reference server must percent-encode "grpc-message" trailer for gRPC and gRPC-Web protocols by @jhump in #847
  • Also fixes response header expectations for gRPC-Web "trailers-only" test cases
  • Also expands reference client checks to verify encoding of "grpc-message" and "grpc-status-details-bin" trailers and to verify encoding of all other binary metadata.

Full Changelog: v1.0.0...v1.0.1

v1.0.0

This marks the initial release of the Connect Conformance Test Suite! 🎉

We've been using the release candidates up to this point to improve the behavior of the various Connect implementations. As of this release, new Connect implementations have a valuable asset to use to make sure their new implementation's behavior is correct and will successfully inter-operate with other Connect and gRPC implementations.

What's Changed

Below are the changes since the last release candidate, v1.0.0-rc4.

Bugfixes

• Fix some gRPC test cases that failed to include "TE: Trailers" request header by @jhump in #839
• If connectconformance test runner fails to parse command-line flags, it should exit with non-zero code by @jhump in #843

Enhancements

• Reference and gRPC servers send independent header and trailer metadata in error responses for unary and client-stream endpoints by @jhump in #840
• Improve test runner behavior when client-under-test times out, crashes, or exits prematurely by @jhump in #844

Full Changelog: v1.0.0-rc4...v1.0.0

v1.0.0-rc4

This contains some minor compatibility issues with the previous release candidate:

1. The message previously known as connectrpc.conformance.v1.ClientCompatRequest.TLSCreds has been moved to a top-level message and is now named connectrpc.conformance.v1.TLSCreds. This will impact generated code and user code that refers to it.
2. Some test cases have been moved around and renamed, so patterns in older "known-failing" and "known-flaky" configurations may no longer be valid.

What's Changed

Bugfixes

• In referenceclient, inspecting wire details needs to support compressed error bodies by @jhump in #811
• Prevent errant 'content-length: 0' request header from being printed when --trace used by @jhump in #815
• Fix deflate algorithm to adhere to RFC 1950 by @jchadwick-buf in #824
• Fix naming and filtering of test cases that use the gRPC implementations by @jhump in #814
• Troubleshoot and fix issues in grpc-web JS conformance client by @jhump in #831
• Server response must include a certificate when using TLS by @jhump in #832
• Fix issues with test runner hanging by @jhump in #829
• Relax expectation for unspecified errors, fix expectation for cardinality violations by @jhump in #833

Enhancements

• Tighten up when responses are allowed to combine headers and trailers; add check for number of unsent requests by @jhump in #805
• Add support for --trace flag for test cases that use the gRPC implementations by @jhump in #823
• Test runner generates self-signed cert that servers-under-test can use by @jhump in #828
• Reconcile test cases with latest Connect spec changes and pull in fixes from latest connect-go release (v1.16.0) by @jhump in #826

Other Changes

• Add docs for configuring and running tests, testing servers, and testing clients by @jhump and @smaye81 in #804, #808, #818, and #819
• Add GitHub action workflow to automatically build/publish release artifacts after tag is created by @jhump in #835

Full Changelog: v1.0.0-rc3...v1.0.0-rc4

v1.0.0-rc3

What's Changed

Bugfixes

• Fix issue that prevented use of --known-failing patterns that could match the test cases run against the grpc-go reference implementations by @smaye81 in #754
• Fix use of raw request in test cases: test runner needs to add test case expectation headers to raw request, by @jhump in #758
• Fix assertions on query parameters that were overly strict, which prevented working implementations in connect-kotlin and connect-swift from passing the Connect GET test cases, by @jhump in #771
• Higher grace periods in test case timeouts to reduce flakiness, by @jhump in #782 (and small part of #761)
• Disable support for accepting gzip responses in referenceclient when no compression should be used by @jhump in #783

Enhancements

• Add command-line flags for more control over test cases to run and how they're handled by @jhump in #757
  • This adds new --run, --skip, and --known-flaky command-line flags, which all take a pattern and can be specified multiple times. To point to a text file containing multiple patterns (one per line), prefix the argument with @. The --run and --skip are mainly intended for interactive runs of tests. CI invocations should prefer --known-failing and --known-flaky.
  • This includes a backwards-incompatible change to the --known-failing flag so its behavior is consistent with the above new flags: its argument is now interpreted as a single pattern. To instead point to a file that contains patterns, prefix the argument with @.
• Add --trace option to print full trace of HTTP requests and responses for failed test cases by @jhump in #761
• Add tests for HTTP status code <-> RPC code mappings by @smaye81 in #762
• Add numerous protocol-specific test cases that use raw requests and raw responses, by @jhump in #770 and #794
• Add test cases that verify deadlines are correctly propagated by @jhump in #772
• The referenceserver checks that there are no request trailers by @jhump in #779
• The referenceclient validates raw wire details for gRPC-Web trailers, actual HTTP trailers, and Connect error and end-stream JSON by @smaye81 in #768 and @jhump in #780, #790
• Add tests for unexpected raw responses by @smaye81 in #774

Other Changes

• Test the referenceserver and grpcserver implementation of gRPC-web protocol using gRPC-web JS client by @smaye81 in #751
• Add docs for writing test cases by @smaye81 in #791
• Add additional documentation to the proto module and update error code in proto model to use an enum by @smaye81 in #792
  • This includes a backwards-incompatible change. The Error.code field is now an enum and not an int32. This will require corresponding code changes in conformance clients and servers.
• Rename test suites and test cases according to new, documented naming convention by @smaye81 in #786
  • This includes backwards-incompatible changes. Configuration for known failing tests may need to be updated to account for new names.

Full Changelog: v1.0.0-rc2...v1.0.0-rc3

v1.0.0-rc2

What's Changed

Bugfixes

• Newer version of connect-go fixes some bugs in the reference client related to Connect GET protocol and for interpreting errors as "deadline exceeded" for timeout cases by @smaye81 in #728 and #734.

Enhancements

• Adds new test cases for Connect GET(includes change to conformance service) by @smaye81 in #721.
• Adds new test cases for cancellations (includes change to ClientCompatRequest, for how to instruct client to cancel) by @smaye81 in #726.
• Adds more test cases for verifying all error codes by @smaye81 in #736.
• Adds new test cases for limits on receiving large messages by @smaye81 in #739.
• Adds new reference server for validating gRPC-Web protocol, based on grpc/grpc-go + improbable-eng/grpc-web by @jhump in #719.
• Reference client and server can support half-duplex bidi streams over HTTP 1.1 by @jhump in #741.
• Multiple servers are started in parallel. Reference client also runs multiple RPCs in parallel, greatly speeding up testing (especially since timeout tests are slow) by @jhump in #720.
• Adds new command-line flags to CLI for test runner: can now specify a particular YAML file with test cases to run, can specify a particular port and TLS cert for the reference server, and can specify number of concurrent servers to run and parallelism of reference client by @smaye81 and @jhump in #717, #720, and #735.
• Reference client can send "raw requests" and reference server can send "raw responses", allowing ultimate control over on the wire request and response encoding, allowing testing of more edge cases than just what the connect-go framework will produce by @jhump in #691, #737, and #740.

Other Changes

• Runs nearly all test cases using TLS, when client or server under test supports TLS (previously, only a handful of TLS-specific cases were run) by @jhump in #732.

v1.0.0-rc1

What's Changed

This is a release candidate for the initial release of the Connect Conformance Tests.

Prior to this release, this repo contained "cross tests" to verify interoperability
between various Connect client and server implementations and gRPC client and server
implementations. This previous structure required new client implementations to
manually implement all test cases in a way that could be run against a variety of
server implementations. Server implementations would run various client test
implementations against their server implementation.

The new model introduced with this release (hopefully culminating soon in a v1.0.0)
is data-driven and requires client implementations to write a generic client that
can read request definitions over stdin, issue RPCs, and then write the results to
stdout. Similarly, server implementations write a server that implements a generic
ConformanceService interface and reads configuration from stdin and writes status
to stdout. This allows new test cases to be written declaratively (via YAML files)
in this repo, and then to automatically run these cases against all clients and
servers without any bespoke code being written in the various implementations.

The new model also introduces new "reference" client and server implementations
which are used to verify other implementations. These implementations can identify
conformance issues that a typical RPC client or server implementation would not.

Instead of publishing Docker images for the variety of clients and servers that the
prior versions of this repo offered, there is a single statically-linked binary,
connectconformance, which is the test runner. This test runner program embeds the
reference client and server implementation as well as all test case data.

New Contributors

In alphabetical order, this release contains contributions from:

• @chrispine
• @jhump
• @smaye81

Full Changelog: 0d0d9b5...v1.0.0-rc1