Substrate-RPC-Proxy

JSON-RPC/substrate aware reverse proxy in Go.

Project consists of two parts:

proxy - simple reverse proxy supports websocket/http

consumer (optional) - monitors extrinsics (seen by proxy) inclusion to blockchain, provides observability and retry mechanism to resist node failures after transaction was accepted by RPC node but before it was included into block

How it works

sequenceDiagram;
    actor client
    client-->>+sub-proxy: user send JSON-RPC request
    alt is extrinsic?
    sub-proxy-->redis: save to redis stream
    end
    sub-proxy-->>+nodes: proxy forward requests to upstream node
    nodes-->>-sub-proxy: upstream response
    sub-proxy-->>-client: client response
    Note over sub-proxy,client: typical interaction
    redis-->sub-consumer: retry/check inclusion
    Note over redis,sub-consumer: listen stream in loop

Loading

Proxy configuration

• SUB_LISTEN (default ":9944,:9933") - Comma-separated list of local addresses to listen
• SUB_HOSTS (default "http+ws://127.0.0.1:9944/") - Comma-separated list of upstream urls, each upstream schemes supported: ws, http, http+ws, https, wss, https+wss
• SUB_UPSTREAM_MAXGAP (default 5), SUB_UPSTREAM_MINPEERS (default 1) - Upstream healthness parameters, see "How Proxy choose upstream" section for more details
• SUB_TLS_CERT, SUB_TLS_KEY - path to TLS/https keypair if SUB_TLS_* specified proxy will start in https mode, consider changing SUB_LISTEN to 443. Disclaimer: although proxy can offload https it is still better to use CDN in production environments
• SUB_LOG_LEVEL (default "trace") - Will log each rpc request and server-side push possible values: "trace","debug","info","warning","error","fatal","panic"
• SUB_LOG_PARAMS (default false) - Toggle JSON frame "params" logging
• SUB_LOG_FORMAT (default "text") - Can be switched to "json"
• SUB_INSPECT_EXTRINISICS (default false) - Informs proxy to check incomming requests for extrinsics and post extrinsics to redis stream
• SUB_INSPECT_METHODS (default author_submitExtrinsic) - Comma-separated list of RPC methods to inspec extrinsics (does not make sense if SUB_INSPECT_EXTRINISICS: false)
• SUB_REDIS_STREAM_NAME, SUB_REDIS_ADDR, SUB_REDIS_PASSWORD - Redis connection settings (does not make sense if SUB_INSPECT_EXTRINISICS: false)
• SUB_METRICS_LISTEN (default ":8888") - Separate binding for exposing prometheus metrics
• SUB_PPROF_ENABLED (default false) - Enable /debug/pprof/ hadnler on SUB_METRICS_LISTEN
• SUB_THROTTLE_BACKLOG_SIZE (default 30000), SUB_THROTTLE_LIMIT (default 6000) - Rate-limiting parameters if SUB_THROTTLE_LIMIT reached requests proxy will queue 26k requests = SUB_THROTTLE_BACKLOG_SIZE - SUB_THROTTLE_LIMIT requests for 3 minutes maximum after that requests will be dropped and client will get error.
• SUB_IGNORE_HEALTHCHECKS - Removes upstream healthchecking logic considering that it's always alive
• SUB_DENY_METHODS (default "author_rotateKeys") - Comma-separated list of methods that proxy must not forward to upstream

How Proxy choose upstream

Only one load-balancing strategy supported: random

1. If SUB_HOSTS contains DNS records and upstream does not support TLS then hosts will be resolved to IP and future subrequests will be performed against IP, it will continue track DNS changes every 10 seconds. In other case (TLS+DNS name) each new connection IP will be resolved on OS level.
2. proxy polls system_health and system_syncState of each upstream to identify status
3. healthy upstream must meet following conditions:
   • system_syncState.highestBlock - system_syncState.currentBlock < SUB_UPSTREAM_MAXGAP
   • system_health.peers > SUB_UPSTREAM_MINPEERS
4. if these is no healthy upsteam, then upsteam will be picked from servers that able to handle websocket connection, does not considering block gap and number of peers.
5. if there is no server that can handle websocket request then proxy will wait 10 seconds and then retry to pick upstream and finally proxy will giveup and forward request to random server if no apropriate was found.

Consumer configuration

Tracks extrinsic inclusion and optionally tries to resubmit them, configuration:

SUB_REDIS_CONSUMER_NAME, SUB_REDIS_STREAM_NAME, SUB_REDIS_ADDR, SUB_REDIS_CONSUMER_GROUP, SUB_HOSTS, SUB_METRICS_LISTEN, SUB_UPSTREAM_MAXGAP, SUB_UPSTREAM_MINPEERS - same parameters as for proxy.

Worth mentioning:

• SUB_RETRY_DELAY - how often consumer should check for extrinsic inclusion
• SUB_TRY_RESUBMIT - resubmit extrinsic if it not appeared on-chain so far

Similar projects

• dshackle
• subway

Features

• Extendable: code written with middleware pattern, new blockchain can be onboarded implementing healthcheck function
• Rate limiting (set SUB_THROTTLE_LIMIT to nodes max-rpc-connections*number of upstream)
• Decode storage with any http client

  curl -s -d '{"id":1, "jsonrpc":"extensions/get-storage/1.0","method":"system.events"}' -v http://127.0.0.1:9944 | jq
  [
    {
      "type": {
        "module_id": "System",
        "event_id": "ExtrinsicSuccess"
      },
      "extrinsic_idx": 0,
      "params": [...]
    },
    {
      "type": {
        "module_id": "ParaInclusion",
        "event_id": "CandidateIncluded"
      }
    ...
  

Known issues

• Headers X-Real-IP, True-Client-IP, X-Forwarded-For used to identify clients IP can be crafted by maliciuos clients. Please prefer using substrate-rpc-proxy in trusted, dev environments or setup firewall to accept connections only from trusted addresses

History

Project started in Jan 2023 in P2P.ORG Polkadot SRE Team. Main goal of the project is to provide stable RPC service.

References

scale.go - Go implementation of scale codec