feat(gateway): Block and CAR response formats

[lidel]

Summary

This PR aims to add support for requesting alternative response format via:

• ?format= URL paramerer
• Accept: application/vnd.ipld.{format} HTTP header

This MVP supports two formats:

• raw – fetching single block
• car – fetching entire DAG behind a CID as a CARv1 stream

TLDR Demo

Downloading a Block

$ curl  -H 'Accept: application/vnd.ipld.raw' "http://127.0.0.1:8080/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN" --output block.bin
$ cat block.bin | ipfs block put 
$ ipfs cat QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
hello

Note: we return Content-Type: application/vnd.ipld.raw – see ipfs/in-web-browsers#191

Downloading a CAR

$ ipfs resolve -r  /ipns/webui.ipfs.io
$ curl  -H 'Accept: application/vnd.ipld.car' "http://127.0.0.1:8080/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu" --output webui.car
$ ipfs dag import webui.car

Note:

• We return Content-Type: application/vnd.ipld.car; version=1 – see [IANA #1228646] Request for media type application/vnd.ipld.car in-web-browsers#192
• For now only a full DAG, we will add custom selectors later – tracked in https://github.com/ipfs/go-ipfs/issues/8769

Rationale

Why we need both

• Verifiable HTTP Gateway Responses (Verifiable HTTP Gateway Responses in-web-browsers#128)
  • for mobile web browsers (content integrity without battery drain caused by full p2p)
  • for IoT devices and other thin clients (defaulting to HTTP, using p2p in LAN and as a fallback)
  • for hardened supply chain: we want to use CARs on CI (Github Actions), and as additional fallback logic for fetching fs-repo-migrations and similar updates
• unlock userland processing of IPLD data
  • users will be able to fetch arbitrary blocks, even ones in formats not supported by go-ipfs

References

• "Gateway support for /ipfs/{cid}?format=car|block|... " Gateway support for /ipfs/{cid}?format=car|raw|... #8234
• "Verifiable HTTP Gateway Responses" Verifiable HTTP Gateway Responses in-web-browsers#128
• "Make ipfs.dag.export built-in feature of HTTP gateways" Make ipfs.dag.export built-in feature of HTTP gateways in-web-browsers#170

TODO

• move file-specific logic to serveFile
  • Last-Modified only when Cache-Control is missing
  • Cache-Control for immutable content paths
  • add TODO about Last-Modified when UnixFS 1.5 is supported
  • run all tests on CI and fix any regressions
• implement router for ?as= (aka ?format=)
• implement serveBlock
  • content-length
  • content-disposition (as attachment + implicit filename of {cid}.ipfs.block)
  • cache-control: same as for a file
  • content-type: application/vnd.* – need decision in Specify mime type for car ipld/go-car#238 (comment)
  • etag
  • TBD: ability to test if block is present in local cache via HEAD /ipfs/{cid} (local test for block, no network fetch)
    • descoped, tracked in Gateway: fast check if CID is in local datastore cache (only-if-cached) #8783
  • tests - t0117-gateway-block.sh
• implement serveCar
  • TBD (is content-length deterministic? blocks may have different order, but returning full DAG should always return same total of bytes) too risky
  • content-disposition (as attachment + implicit filename of {cid}.ipfs.car)
  • content-disposition (as attachment + implicit filename of {cid}.ipfs.car)
  • etag
    • car streams for bigger dags won't be byte-for-byte identical, so went with "weak" Etag here
  • cache-control: no-cache, no-transform for now
    • chunked stream could be interrupted and browsers have handling due to trailers being garbage http.headers.Trailer - reading trailers not supported by Fetch API mdn/browser-compat-data#14703
    • interrupted stream includes X-Stream-Error so receiving end may be able to debug things, as long its not a browser
      W/"<etag_value>"` – it indicates that response can't be cached if byte-for-byte reproducibility is required.
  • TBD: support selector=multibase(cbor selector)
    • descoped from this PR, ?format=car&selector= is tracked in https://github.com/ipfs/go-ipfs/issues/8769
  • content-type: application/vnd.* – need decision in Specify mime type for car ipld/go-car#238 (comment) + always include version in response + handle when processing request
  • tests - t0118-gateway-car.sh
• create more useful metrics, deprecate unixfsGetMetric
  • descoped from this PR, improved metrics are tracked in Improved Gateway HTTP request metrics #8441

TODO (future PRs)

After this PR lands, we can add key IPLD formats (serveDagCbor, serveDagJson) – ipfs/in-web-browsers#182

[lidel]

Work in progress, dropping some notes so I don't forget.

[BigLep]

2022-03-03 discussion: transporting HTTP CARs around: currently sourcing how to handle errors.

[Jorropo]

Personal notes for a future feature request:

Get parameters are uselessly hard to deal in many languages. Even tho most of them have escape features, you sometime manually concatenate them with the first concatenation being ? and not &.

TL;DR: I want an alternative be Accept: application-x/car header.

That also allows to save up resending the header if you do multiple car requests in a row with HTTP2.

[Jorropo]

TBD (is content-length deterministic? blocks may have different order, but returning full DAG should always return same total of bytes)

FYI, Getting that info require a full root-traversal (you can actually skip raw-leaves but anyway that not on point).

First CAR intricates:

In most cases, the info we trivially have is the DagSize stored in the root.
However this is not enough to know the size of the final car.
Assuming equal DagSize a car made of lots of small blocks is bigger than one made of few big blocks. Because CARv1 have a fix varuint + CID overhead per block, more blocks => more overhead even at equal dag sizes.

If we had BlockCount + DagSize we can get a really good approximation.

I do say approximation because:

• Inlined blocks
• CID length
• Varuint size variability

Is variability you cannot account for without a traversal. Hopefully those are only small and doesn't change much. (Is it even ok to send wrong but close content lengths ?)

Secondly, duped blocks

If we want to be smart we don't send duped blocks multiple time, however you don't know how much of your dag size is duped that can have massive impacts on the real content size.

[lidel]

Get parameters are uselessly hard to deal in many languages.
I want an alternative be Accept: application/car header.

FWIW I've heard the same "uselessly hard" feedback about Accept header :)
Arguments being: not possible to create such request via browser's address bar, unable to create <a href= links, often being ignored by HTTP caches etc. etc.

This is to say, there is no silver bullet, too many use cases. We need to support both (query param, and Accept header). I've seen no consensus around mime-type for Block and CAR so skipped it in the initial PoC, waiting for ipld/go-car#238 to be resolved first, but I think you are right, we should support both from the start:

application/vnd.ipfs.block
application/vnd.ipfs.car
application/vnd.ipfs.car; version=1
application/vnd.ipfs.car; version=2

or to make things less ambiguous, reuse existing IPLD concepts:

application/vnd.ipld.raw
application/vnd.ipld.car
application/vnd.ipld.car; version=1
application/vnd.ipld.car; version=2

Is it even ok to send wrong but close content lengths?

I'd rather not risk this type of hackery – it would provide better UX via progress bars etc., but some overly smart HTTP clients may close the connection right after they receive the expected number of bytes, which would truncate the CAR stream.

[Jorropo]

Arguments being: not possible to create such request via browser's address bar, unable to create <a href= links, often being ignored by HTTP caches etc. etc.

I know, that why I think we should have both. :)

[Jorropo]

LGTM

CodeQL is unhappy but that is safe because in fact the user controlled input is passed in fmt's %q formater, which leaves us with fairly safe UTF-8.

[lidel]

Related histogram metrics are added in a separate PR: #8443