Release v2.6.2

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.9: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to seal a stream through a stream update (#2584)
  • Ability to do roll ups (#2585, #2593, #2600, #2601)
  • Allow consumers to request only headers to be delivered with HeadersOnly consumer configuration (#2596)
  • New stream configuration properties to limit delete/purge/rollups (#2595)

Updated

• JWT library dependency to v2.1.0

Improved

• MQTT:
  • The client ID will now be part of the client log statements. Thanks to @imranrazakhan for the report (#2598)

Fixed

• JetStream:
  • Adjust cluster size on cold start in mixed mode (#2579)
  • Issue with the "get message" audit event (#2583)
  • Some data races (#2590, #2619)
  • More robust checking for decoding append entries (#2606)
  • Issues with RAFT around snapshots and recovery (#2615)
  • Starting a consumer with "deliver last per subject" with a filtered subject same as the stream's subject would possibly return wrong NumPending value (#2616)
  • Purging filtered stream resets all stream consumer offsets. Thanks to @minight for the report (#2616, #2617)
  • Honor max memory and max file settings of 0. Thanks to @biskit for the report (#2618)
• Websocket:
  • Issue with compression and Safari browsers using the experimental feature NSURLSession WebSocket. Thanks to @byazrail for the report (#2613)
• Handling of message headers in some system requests (#2580)
• Go doc fixes and some code cleanup. Thanks to @dtest11 for the contributions (#2610, #2614)

Complete Changes

v2.6.1...v2.6.2

Release v2.6.1

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Deadlock when using stream mirrors with source with non limit retention, which could manifest with increase memory usage (#2561)
  • Possible panic due when getting a slice of a message block cache buffer (#2565)
  • Internal "direct consumers" used for mirroring were incorrectly subject to the "max consumers" limit (#2564)
  • Storage limit of 0 (which means disabled) was not always enforced (#2563)
  • Handling when exceeding account resources (#2569)
  • Internal connections were counted toward the account's max connections (client or leafnode) (#2568)

Complete Changes

v2.6.0...v2.6.1

Release v2.6.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• Monitoring:
  • JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints (#2539, #2555)
• Hardened systemd service. Thanks to @c0deaddict for the contribution (#2318)

Changed

• JetStream:
  • The server store message headers with a uint 16, which limits to 64KB, however, it was not rejecting incoming messages with headers bigger than that but then was failing its recovery (#2526)
  • The server was accepting consumer creation requests with flow control enabled but without heartbeats, which could lead to the consumer stalling if the library missed the flow control message. The server will now reject such consumer creation request (#2533)
  • Trying to add a stream with a configuration that matched exactly the one of an existing stream would report an error in cluster mode, not in standalone mode. The cluster mode will now behave as standalone mode and treat the stream creation as idempotent (#2535)
  • The server will now reject negative duplicate window values (#2546)
• The server will now fail to start if a token is specified in cluster{authorization{}} or gateway{authorization{}}. Routes and Gateways have never supported token authentication, however, it was silently ignored. Thanks to @antong for the report (#2518)

Fixed

• JetStream:
  • Filestore compaction issue that could to message lookup to fail, possibly stalling consumers (#2522)
  • Reject messages with headers greater than 64KB (#2526)
  • Condition where assets could be created in multiple leafnode domains (#2529)
  • Reject consumer creation requests with a flow control but no heartbeat (#2533)
  • Stream creation request was idempotent in standalone mode, but not in clustered mode (#2535)
  • In clustered mode, getting consumer information or a just created consumer could sometimes return a "consumer not found" error (#2537)
  • Make large pull subscribe batch requests expire more efficiently (#2538)
  • Stabilize filestore implementation. Thanks to @izarraga for the report (#2545)
  • Reject duplicate window with negative value (#2546)
  • Memory store memory leak when running in cluster mode. Thanks to @anjmao for the report (#2550)
  • Some issues with RAFT WAL repair (#2549)
  • A consumer on mirror stream would fail to be recovered if its stream was recovered before the mirrored stream. Thanks to @tbeets for the report (#2554)
• Websocket:
  • Possible panic when decoding CLOSE frames. Thanks to @byazrail for the report (#2519)
• Memory leak when using Queue Subscriptions. Thanks to @andrey-shalamov for the report and fix! (#2517)
• Fail configuration that would have token specified in cluster{} or gateway{}'s authorization{} blocks. Authentication token were never supported there but the misconfiguration was silently ignored. Thanks to @antong for the report (#2518)

Complete Changes

v2.6.0...v2.5.0

Release v2.5.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See note in the Improved and Changed section.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• MQTT/Monitoring:
  • MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client. Thanks to @carr123 and @imranrazakhan for the suggestions (#2507)

Improved

• MQTT:
  • Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage (#2501)

Changed

• JetStream:
  • Using Nats-Expected-Last-Subject-Sequence header with a value of 0 now means that the server will reject the store command if there were messages on this subject (#2506)
• MQTT:
  • Due to improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_<xxxx> streams to a new $MQTT_sess stream for the user's account (#2501)

Fixed

• JetStream:
  • Possible deadlock due to lock inversion (#2479)
  • Possible consumer stall. Thanks to @carr123 for the report (#2480)
  • Don't send 408 status when pull request expires. This has no visible impact for users, but library implementers may want to be aware of it (#2482)
  • During peer removal, try to remap any stream or consumer assets (#2493)
  • Issues with remapping of stream and/or consumer assets during peer removal (#2493)
  • Issue that could lead to perceived message loss (#2490)
  • Message cleanup for interest stream and AckNone consumers in clustered mode (#2499)
  • Suppress duplicates on JS deny all for system account (#2502)
  • Consumers stopped receiving messages. Thanks to @anjmao, @izarraga and @tigrato for the report (#2505)
• Handle SIGTERM on Windows platform (#2481)
• Account resolver TLS connection may fail with x509: certificate signed by unknown authority errors. Thanks to @Ryner51, @ronaldslc for the report (#2483)

Complete Changes

v2.4.0...v2.5.0

Release v2.4.0

Changelog

Notice for JetStream Users

With the latest release of the NATS server we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.
• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.
• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message than the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.
• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeat are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this mis-behavior is still possible.
• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects an core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are update to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.
• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.
• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer.
  It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.
• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

Go Version

• 1.16.7: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Domain to the content of a PubAck protocol (#2432, #2434)
  • PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription (#2438)
  • DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for (#2438)
  • Warning log statement in situations where catchup for a stream resulted in an error (#2444)
• Monitoring:
  • Ability for normal accounts to access scoped connz information (#2437)
• Operator option resolver_pinned_accounts to ensure user are signed by certain accounts (#2461)

Changed

• JetStream:
  • ConsumerInfo's Delivered and AckFloor are now SequenceInfo instead of SequencePair. SequenceInfo contains Last which represents the last active time (in UTC) (#2462)
  • Delivery of messages for consumers will now be subject to proper group information. Older clients may fail to receive messages on a queue subscription if the library has not been updated to specify the DeliverGroup (#2438)

Improved

• Make error message actionable when adding operator and leaf nodes. Thanks to @alsuren for the contribution (#2449)
• JetStream:
  • File utilization when using a Jetstream stream as a KeyValue store (#2456)
  • Encryption at rest with message expiration on server restart (#2467)
  • Enable global account on non JetStream servers in mixed mode (#2473)

Fixed

• JetStream:
  • Stream delete can fail for non empty directory (#2418)
  • Possible panic for concurrent stream remove and consumer create (#2419)
  • Simplified flow control and avoid stalls due to message loss (#2425)
  • Consumer info max_msgs_per_subject defaults to 0, but should be -1 (#2426)
  • Creating a consumer with a max_waiting value and a deliver subject was returning the wrong error message (#2427)
  • Assign default to max_ack_pending when AckExplicit or AckAll (#2428)
  • Subscriptions for internal clients for JetStream consumers were not properly removed, resulting in possible subscriptions leak/high memory usage (#2439)
  • Expiration of messages during a server restart could lead to clients reporting errors after reconnect when trying to send new messages (#2452)
  • Added additional checks for failures during filestore encryption (#2453)
  • Processing of a publish ACK could cause a server panic (#2460)
  • Consumer's number of pending messages for multiple matches and merging (#2464)
  • Deadlock due to lock inversion when creating a RAFT group. Thanks to @lokiwins for the report (#2471)
• Leafnode:
  • User authorization issue when JetStream is involved. Thanks to @wchajl for the report (#2430)
  • A remote websocket connection with wss:// scheme but no tls{} configuration block would be attempted as non TLS connection, resulting on an "invalid websocket connection" in the log of the server attempting to create the remote connection (#2442)
  • Wrong permission checks prevented messages flow. Thanks to @rbboulton for the report (#2455, #2470)
  • Daisy chained subject propagation issue. Thanks to @KimonHoffmann for the report (#2468)
• Possible subscription leak with the use of "AutoUnsubscribe" (#2421)
• Prevent JWT claim updates from removing system imports (#2450, #2451)
• Error print when adding back existing system imports (#2466)
• Build on OpenBSD-6.9. Thanks to @miraculli for the contribution (#2472)

Complete Changes

v2.3.4...v2.4.0

Release v2.3.4

Changelog

Go Version

• 1.16.6: Both release executables and Docker images are built with this Go release.

Changed:

• Server will now reject a max_payload that is set higher than max_pending since no message would be able to be delivered to subscriptions. The server will also warn if the value is set above 8MB (and may enforce this limit in the future) (#2407, #2413)

Fixed

• JetStream:
  • A deadlock could happen when the server was removing messages. Thanks to @rwrz for the report (#2404)
  • Various fixes and improvements to clustered Filestore consumer stores (#2406)
  • Leafnodes with same domain and shared system account should behave like flat Jetstream network (#2410)
  • With stream with multi-subject or wildcard and max_msgs_per_subject set, if a consumer had deliver_last_per_subject set, the initial pending would be 1 higher than it actually was (#2412)
  • Memory store would sometimes incorrectly select the proper starting sequence (#2412)

Complete Changes

v2.3.3...v2.3.4

Release v2.3.3

Changelog

Go Version

• 1.16.6: Both release executables and Docker images are built with this Go release.

Added

• ReloadOptions API to support configuration reload without use of configuration file for embedded cases. Thanks to @taigrr for the contribution (#2341)
• Kind and ClientType to account CONNECT/DISCONNECT events. Thanks to @mullerch for the report (#2351)
• JetStream:
  • Streams and consumers now have a Description property (#2377)
  • New DeliverLastPerSubject delivery policy (#2381, #2390)

Changed

• Default account fetch timeout to be smaller than client timeout to increase probability of getting the Authorization Violation error instead of a i/o timeout error when credentials of an account has not yet been pushed to the account server (#2365)
• Executable symbol table no longer stripped. Thanks to @yzhao1012 and @justicezyx for the contribution (#2383)

Improved

• TLS timeout in configuration file parsing now accept units, such as "2s" for 2 seconds (#2364)
• JetStream:
  • Server restart time with many expired messages (#2387)

Fixed

• JetStream:
  • A data race on JetStream shutdown (#2353)
  • In clustered mode, the maximum consumers limit was not always applied for ephemeral consumers (#2354)
  • Consumer's NumPending may be stuck at 1 (#2357)
  • Removed a stack print that may appear in some cases since v2.3.1 (#2362)
  • Possible subscription leak when processing service imports and processing of pull subscribers (#2373)
  • Unique server name requirement across domains (#2378)
  • A clustered consumer on an interest retention policy could cause the server to panic when the consumer was being deleted (#2382)
  • Allow non-JS leafnode(s) to access a HUB transparently (#2393)
  • A stream with un-acknowledged messages would not redeliver new un-acknowledged messages following a purge. Thanks to @sloveridge for the report (#2394)
• Subscription on a subject that is not a subset of a wildcard import. Thanks to @DamianoChini for the report (#2369)
• OCSP issue in embedded cases when the TLS configuration did not set the certificate Leaf (#2376)

Complete Changes

v2.3.2...v2.3.3

Release v2.3.2

Changelog

Go Version

• 1.16.5: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Error codes for consumers creation errors (#2345)

Changed

• JetStream:
  • Creating an ephemeral consumer if there is not yet interest will no longer fail, that is, a JetStream consumer can be created prior to low level NATS subscription on the delivery subject (#2347)

Fixed

• JetStream:
  • Updates to a multi-subject tacked stream (#2334)
  • Possible publish timeout due to server sending messages to consumers on a slow connection (#2337)
  • Possible message corruption (#2344)
  • Peer info reports had a large last active values
• Headers handling in system services (#2338, #2348)

Complete Changes

v2.3.1...v2.3.2

Release v2.3.1

Changelog

Go Version

• 1.16.5: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to get a stream last message by subject (#2313)
  • Ability to match based on last expected sequence per subject (#2322)

Improved

• JetStream:
  • Large number of R1 consumers per stream (#2324, #2326)

Fixed

• JetStream:
  • Max consumers was not enforced when set on a stream (#2316)
  • Clustered streams can become broken with sequence mismatch state on low level store failures (#2317)
  • Do not log at [ERR] level some "normal" store failures (such as max messages, bytes, etc..) (#2321)
• Race when generated random service reply subject (#2325)

Complete Changes

v2.3.0...v2.3.1

Release v2.3.0

Changelog

Go Version

• 1.16.5: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Richer API errors. JetStream errors now contain an ErrCode that uniquely describes the error. Thanks to @jon-whit and others for the suggestion (#2168, #2255, #2266)
  • Ability to send more advanced Stream purge requests (#2296, #2297, #2303, #2306)
  • Stream can now be configured with a per-subject message limit (#2284)
  • Encryption at rest (#2302)
• Monitoring:
  • JetStream information into statsz (#2269, #2276)
• OCSP support (#2240, #2263, #2277)

Changed

• CPU and memory usage report on macOS (removed dependency on ps) (#2260)
• Throttle the number of maximum subscriptions exceeded log statements per account, to 1 every 2 seconds (#2304)

Improved

• JetStream:
  • Setting initial pending and selecting starting sequence number of streams with multiple subjects (#2284)
  • Filestore memory usage (#2306)

Fixed

• Gateways:
  • Handling of subject rewrites for subjects to a globally routed subject (#2275)
  • Message headers were lost (passed in the message payload) from a response across a Gateway and through a route (#2278)
• JetStream:
  • Better support for multiple domains where the hub is JetStream enabled but the hub account is not, and the Leafnode is (#2261)
  • Orphaned consumers on sourced or mirrored streams keep trying to create new ones (#2279)
  • CPU spikes in some catch-up situations (#2280)
  • Dynamic account limits would be applied based on single server limits (#2281)
  • Error description missing in some requests (#2293, #2294, #2295)
• LeafNode:
  • Hanging connection when account can't be found (#2267, #2288)
  • Configuration reload could fail even if there were no changes to the Leafnode configuration (#2274)
  • Service export interest was not propagated correctly (#2288)
• MQTT:
  • Panic when subjects cross accounts with import/export. Thanks to @mullerch for the report (#2268)
• Websocket:
  • Compression/Decompression issues with continuation frames. Thanks to @luan007 for the report (#2300)
• Clients disconnected on configuration reload when only $SYS account is configured (#2301)
• Failed route TLS handshake would leave the failed connection's lock in a locked state (#2305)

Complete Changes

v2.2.6...v2.3.0