TS SDK (atproto/sync): correct way to handle "deleted blob" type errors?

[bleonard252]

I get errors like this one a lot:

Error: error in parsing and authenticating firehose event
    at Firehose.parseEvt (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:119:31)
    at eventLoopTick (ext:core/01_core.js:214:9)
    at async Firehose.processEvt (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:124:24)
    at async Firehose.start (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:85:21)
Caused by Error: block not found: bafyreihlwj5kicydksthycigmpheqliwmrungrztm3b63g323sblstfxmm, expected type: mst node
    at MemoryBlockstore.readObjAndBytes (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/storage/readable-blockstore.js:40:19)
    at eventLoopTick (ext:core/01_core.js:214:9)
    at async MemoryBlockstore.readObj (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/storage/readable-blockstore.js:45:21)
    at async MST.getEntries (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/mst/mst.js:152:26)
    at async MST.findGtOrEqualLeafIndex (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/mst/mst.js:517:25)
    at async MST.get (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/mst/mst.js:311:23)
    at async verifyProofs (file:///deno-dir/npm/registry.npmjs.org/@atproto/repo/0.5.3/dist/sync/consumer.js:123:23)
    at async parseCommitAuthenticated (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:157:25)
    at async Firehose.parseEvt (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:104:23)
    at async Firehose.processEvt (file:///deno-dir/npm/registry.npmjs.org/@atproto/sync/0.1.4/dist/firehose/index.js:124:24) {
  event: {
    ops: [
      {
        cid: null,
        path: "app.bsky.feed.post/3lbchi2szws2x",
        action: "delete"
      }
    ],
    rev: "3lbd22qltz32j",
    seq: 3973598396,
    prev: null,
    repo: "did:plc:4p4ahbjtsfv4huysyh6zm33s",
    time: "2024-11-19T18:16:34.490Z",
    blobs: [],
    since: "3lbczzlj5mx2t",
    blocks: <Buffer 3a a2 65 72 6f 6f 74 73 81 d8 2a 58 25 00 01 71 12 20 92 f3 8a 53 74 9f d2 b2 ae 6f 6b 36 de f3 d2 76 b2 9a 9c b5 0e 4b b8 ad da 08 4b a6 79 89 38 9a ... >,
    commit: CID(bafyreies6offg5e72kzk433lg3pphutwwknjzniojo4k3wqijothtcjyti),
    rebase: false,
    tooBig: false,
    "$type": "com.atproto.sync.subscribeRepos#commit"
  }
}

I'm not concerned with deletions right now, only new posts, and I also don't do anything with blobs. This clogs my logs and makes finding any actual errors more difficult. How can I hide this particular error? Is there a type check function I can do to suppress these errors (in the firehose's onError field, I expect)?

[bengotow]

I filtered these out in my implementation using something like this:

  onError: (err) => {
    if (
      err instanceof FirehoseParseError &&
      err.event.ops?.some((a) => a.action === "delete")
    ) {
      return;
    }
    console.error(err);
  },

[bleonard252]

I had to use this version to get types to work:

  onError: (err) => {
    if (
      err instanceof FirehoseParseError &&
      (err.event.ops as ComAtprotoSyncSubscribeRepos.RepoOp[]|undefined)?.some((a) => a.action === "delete")
    ) {
      return;
    }
    console.error(err);
  },

(where ComAtprotoSyncSubscribeRepos is imported from @atproto/api)

So far so good. Thank you so much, man. Hopefully this'll help someone else too.

[dholms]

This is actually a known issue that we're not emitting all necessary blocks for deletion proofs. We had a fix a couple weeks back but lost track of it in all the craziness.

Hoping to roll this out this week and hopefully stop the errors all together 😄

#3033

[dholms]

Update: rolled this out! My local testing shows that these errors have gone down to 0 (at least coming from our PDSs, there are still some self-hosters/alternate implementations in the network that may still be emitting faulty events)

[DavidBuchanan314]

as of v0.0.2, I believe millipds does this properly too. If anyone sees one of these errors in an event emitted from millipds >= v0.0.2 please ping me! (note, "v0.0.2.dev*" version strings are older than v0.0.2 proper - and you can find the version string on the index page or /xrpc/_health )

[callmephilip]

@bleonard252 did you manage to get firehose to work in Deno? (judging by exception traces)

[bleonard252]

@callmephilip It should work out of the box, but you do have to either allow every permission it asks for individually or use `--allow-all`. ---------- Blake Leonard https://blakes.dev Jan 8, 2025 14:21:48 Philip Nuzhnyi ***@***.***>:

…

@bleonard252[https://github.com/bleonard252] did you manage to get firehose to work in Deno? (judging by exception traces) — Reply to this email directly, view it on GitHub[#3044 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ADUJCOVNTVX5FR7RMU46JTT2JV3EVAVCNFSM6AAAAABSCYKTHWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZXG42TENA]. You are receiving this because you were mentioned. [Tracking image][https://github.com/notifications/beacon/ADUJCOSLU3FJTOSIWH7JXG32JV3EVA5CNFSM6AAAAABSCYKTHWWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAWO27I.gif]