Review and Add Special Headers for Optimistic Concurrency Control

[arboleya]

Related:

• RPC Consistency Proposal fuel-core#1897

RPC Consistency Proposal

A common issue with querying distributed databases is ensuring each replica or shard is consistent. Without consistency, load-balancing strategies may cause unexpected 404s for users, where some requests for recently written data succeed and some fail, depending on how the requests are balanced and how up-to-date each database node is.

Since the SDK typically polls for additional receipts or data after a transaction is confirmed, subsequent requests may sporadically fail. Using L7 cookie-based sticky sessions at the load balancer level was considered, but it doesn't look like a great approach because it may lead to uneven load distribution and many complexities with cross-domain cookies in the web browser.

Proposed Solution (Optimistic Concurrency Control)

A better approach that wouldn’t require any significant architecture change from the RPC or networking layer is to allow requests to specify the minimum expected block height needed to satisfy the query. This will enable the client to quickly bail out of a request if the receiving node is behind and retry until a node with sufficient block height is reached. This technique is commonly known as optimistic concurrency control.

The following proposal specifics assume the behavior is implemented as middleware for Axum/Tower.

Request Header

To minimize breaking changes to the GQL interface used by our RPC, the user specifies the expected block height using an HTTP request header. Users may optionally set this request header if they want requests to fail fast if the node is not up to date:

• REQUIRED_FUEL_BLOCK_HEIGHT: 12345

Behavior

if `REQUIRED_FUEL_BLOCK_HEIGHT` > `CURRENT_FUEL_BLOCK_HEIGHT`
  - Return HTTP status code “[`[412 Precondition Failed](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412)`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412)"

if `REQUIRED_FUEL_BLOCK_HEIGHT` ≤ `CURRENT_FUEL_BLOCK_HEIGHT`
  - Process the query

Response Header

Name: CURRENT_FUEL_BLOCK_HEIGHT
Value: On every RPC response, include the value of the current block height

Every request returned from RPC should also include the current block height so that the user knows what the expected block height should be. If the SDK automatically sets its minimum expected block height for future requests based on the highest block height it’s observed from any previous response headers; then the user will never experience any consistency issues.

Potential Issues

• SDK may make a lot of retry requests and exceed rate limits more often.
  • The SDK should use a back-off strategy between requests to avoid too quickly using up the request quota.
    This may increase the UX's latency if a chain of requests (such as simulation/estimation) needs several retries.
  • In some situations, it may be acceptable for the SDK to omit the REQUIRED_FUEL_BLOCK_HEIGHT header and tolerate inconsistencies between responses.
    • For example: predicate estimation likely doesn’t need this level of sensitivity, as gas price upgrades are pretty rare.
  • Try to bundle sequential queries into a single one when possible. Ie. if many values are fetched from a contract and a dry-run query is used for each one, those dry-runs could be combined into a single script/query.