New sequence clipping behavior seems to broke sourcing from memory streams

[AlexeiSosnov]

We currently have NATS configuration, when a server stream TST is sourced from a stream SST from a leaf-node. Our leaf-node is busy by i/o from other components of our product, so we are using in-memory stream storage on the leaf-node to avoid additional i/o load.

Previously (NATS v2.10.12) this configuration was very reliable to sourced memory stream TST recreations (e.g. due to leaf-node server restarts).
Indeed, when the leaf-node restarts, the sourcing stream TST stops to receive messages from the source consumer, so the sourcing stream TST recreates a source consumer on the sourced stream SST, starting from the last remembered offset in the sourced stream:
mset.setupSourceConsumer(si.iname, si.sseq+1, time.Time{})
When the sourced memory stream SST is recreated, it has FirstSeq = LastSeq = 0. In NATS v2.10.12, when a consumer start sequence was out of the stream, it was clipped to be inside the stream. So the source consumer sequence was clipped to zero, and the consumer started to deliver messages from the beginning of the recreated stream.
However, starting from NATS v2.10.19 (#5785) this behavior is changed, and consumer OptStartSequence is now not being clipped: ec54164
This behavior leads to temporary stalling and to lost messages when sourcing, because the source consumer on the sourced stream SST is now waiting until the sourced stream LastSeq catches up to the consumer sequence.

Moreover, the same behavior also applies not only for memory streams. For instance, when a sourced server unexpectedly shuts down, the last block of the sourced stream SST can be lost due to not being flushed on the disk. As a results, the source consumer is stalling again until the sourced stream catches up its sequence.

So, it seems that new clipping behavior may be not applicable for source consumers that should be reliable.
So, what should we do?
For example, one could introduce additional field OptReplicationStartSequence, use it for source consumers and clip a start sequence provided by the field to be always inside a sourced stream.

[neilalexander]

We've reverted the behaviour in #6014, which will go into 2.10.22, and will rethink how to satisfy both use cases in 2.11.

[pricelessrabbit]

Just for info, the reverted behavior solves also the similar issue i opened here in case a sourced stream is manually deleted and recreated. This can happen when you source stream for other accounts that you are not controlling directly