feat: filter types #1198

kyscott18 · 2024-10-28T19:09:47Z

Tracking issue for all filter related features.

Filters should be deliberately designed, expressive, and modeled after the fundamentals of the Ethereum blockchain, not necessarily the Ethereum standard JSON-RPC. Raw blockchain data that is matched by a filter becomes an event, the arguments to an indexing function.

Proposed filter types

type LogFilter = {
  type: "log";
  chainId: number;
  address: Address | Address[] | Factory | undefined;
  topics: LogTopic[];
  fromBlock: number;
  toBlock: number | undefined;
  includeTransactionReceipts: boolean;
};

type BlockFilter = {
  type: "block";
  chainId: number;
  interval: number;
  offset: number;
  fromBlock: number;
  toBlock: number | undefined;
};

type TransferFilter = {   
  type: "transfer";   
  chainId: number;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  fromBlock: number;
  toBlock: number | undefined;   
  includeTransactionReceipt: boolean;
};

type TransactionFilter = {
  type: "transaction";
  chainId: number;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  functionSelectors: Hex | Hex[] | undefined;
  includeInner: boolean;
  includeFailed: boolean;
  callType: "call" | "staticcall" | "delegatecall" | "selfdestruct" | "create" | "create2" | "callcode" | undefined;
  fromBlock: number;
  toBlock: number | undefined;  
  includeTransactionReceipt: boolean;
};

Few things I'm not sure about:

is includeTransactionReceipt necessary? Does in need to include the logs property?
should transactions and traces be separate filters?

`ponder.config.ts`

The config should provide high level abstractions, such as contracts and accounts. These high-level sources also have the ability to post-process a filter result and add decoded args, etc.

`factory()`

I think it could be nice to have a helper function that can be used in any address field.

import { createConfig, factory } from "@ponder/core";

export default createConfig({
  contracts: {
    UniswapV3Pool: {
      network: "mainnet",
      abi: UniswapV3PoolAbi,
      address: factory({ 
        address: "0x1F98431c8aD98523631AE4a59f267346ea31F984",
        event: getAbiItem({ abi: UniswapV3FactoryAbi, name: "PoolCreated" }),
        parameter: "pool", 
      }),
      startBlock: 12369621,
    },
  },
});

The text was updated successfully, but these errors were encountered:

0xOlias · 2024-10-28T23:55:52Z

The address: factory(...) idea is excellent, much clearer than the union type we have today IMO.

I'm still not sure about includeTransactionReceipt. If we were being quite strict about modeling this after the EVM and not the standard RPC, IMO the receipt fields would simply always be included/joined with the transaction input. Obviously we know that's not how the standard RPC works, and it's actually very expensive to always fetch receipts.

Another idea (I know it's controversial 😉 ) would be to add a field selection option here, eg:

type LogFilter = {
  type: "log";
  chainId: number;
  address: Address | Address[] | Factory | undefined;
  topics: LogTopic[];
  fromBlock: number;
  toBlock: number | undefined;
  // includeTransactionReceipts: boolean;
  include: LogFilterField[] | undefined;
};

type LogFilterField = 
  "log.address" |
  "log.topics" |
  "log.data" |
  "block.number" |
  "block.timestamp" |
  "transaction.hash" |
  "transaction.input" |
  "transaction.gasUsed" // if included, the sync engine must fetch receipts
  // ...

For backwards compatibility (and perhaps as the forever default) the high-level contracts and accounts APIs could keep our existing includeTransactionReceipts option, and if set to true, add those fields to the include list.

kyscott18 · 2024-10-29T14:22:12Z

I definitely like the idea of just joining new properties into transaction rather than adding new top level fields.

0xOlias · 2024-10-29T18:59:12Z

I'm not quite convinced that traces and transactions should be combined into one filter type. From the EVM perspective, I can see why the EOA transaction (input + receipt) is the same logical "event" as the corresponding CALL trace. However, if we look at the fields available to us from the RPC responses, the EOA/"topCall" transaction has far more fields.

For mainnet transaction 0x50560570c4984213a573a28196f84669aaf179b0f1253b908b7621070b45f512

Transaction input & receipt (from eth_getBlockByNumber and eth_getBlockReceipts)

// input
{
  blockHash:"0x1197de9312833dc933878c1ab0134b2e02bf0218f808dc3179c83f4d185267d2",
  blockNumber:"0x1418caa",
  hash:"0x50560570c4984213a573a28196f84669aaf179b0f1253b908b7621070b45f512",
  yParity:"0x1",
  accessList:[],
  transactionIndex:"0x0",
  type:"0x2",
  nonce:"0x6",
  input:"0x0162e2d...",
  r:"0x4ac0fc4c3de38288d3b155905ed3ed9066e33381e6ddcac3094642f0eb47c42f",
  s:"0x32002d52e8fbfc6ae6a196c73595669ce8a663c4bb737f6dc466a6f82c12aefc",
  chainId:"0x1",
  v:"0x1",
  gas:"0x741a2",
  maxPriorityFeePerGas:"0x9b10b18400",
  from:"0xc6a7a6fc4a2ac1b8fc3ec1c0e343e2855aa35afc",
  to:"0x3328f7f4a1d1c57c35df56bbf0c9dcafca309c49",
  maxFeePerGas:"0x9f8956054d",
  value:"0x1bc16d674ec80000",
  gasPrice:"0x9e5f50134d"
}

// receipt
{
  transactionHash:"0x50560570c4984213a573a28196f84669aaf179b0f1253b908b7621070b45f512",
  blockHash:"0x1197de9312833dc933878c1ab0134b2e02bf0218f808dc3179c83f4d185267d2",
  blockNumber:"0x1418caa",
  logsBloom:"0x0024...",
  gasUsed:"0x2954f",
  contractAddress:null,
  cumulativeGasUsed:"0x2954f",
  transactionIndex:"0x0",
  from:"0xc6a7a6fc4a2ac1b8fc3ec1c0e343e2855aa35afc",
  to:"0x3328f7f4a1d1c57c35df56bbf0c9dcafca309c49",
  type:"0x2",
  effectiveGasPrice:"0x9e5f50134d",
  logs: [ ... ],
  status:"0x1"
}

Corresponding CALL trace (from debug_traceBlockByNumber)

{
  from:"0xc6a7a6fc4a2ac1b8fc3ec1c0e343e2855aa35afc",
  gas:"0x741a2",
  gasUsed:"0x2954f",
  to:"0x3328f7f4a1d1c57c35df56bbf0c9dcafca309c49",
  input:"0x0162e2d...",
  calls: [ ... ],
  value:"0x1bc16d674ec80000",
  type:"CALL"
}

The specific fields that seem useful for indexing in the EOA input/receipt are nonce, gasPrice, and logsBloom/logs.

With that said, it may also be useful to specify the return type for each filter and the available "joined" objects (which would form the arguments to the include option I proposed above).

LogFilter
  log
    block
    transaction (input and receipt)

BlockFilter
  block

TransferFilter
  transfer (trace?)
    block
    transaction (input and receipt)

TraceFilter
  trace
    block
    transaction (input and receipt)

TransactionFilter
  transaction
    block
    trace

kyscott18 · 2024-10-30T15:26:58Z

Updated filter types with corresponding raw events and field selection

type LogFilter = {
  type: "log";
  chainId: number;
  address: Address | Address[] | Factory | undefined;
  topics: LogTopic[];
  fromBlock: number;
  toBlock: number | undefined;
  include: (
    | "block.baseFeePerGas"
    | "block.difficulty"
    | "block.extraData"
    | "block.gasLimit"
    | "block.gasUsed"
    | "block.hash"
    | "block.logsBloom"
    | "block.miner"
    | "block.mixHash"
    | "block.nonce"
    | "block.number"
    | "block.parentHash"
    | "block.receiptsRoot"
    | "block.sha3Uncles"
    | "block.size"
    | "block.stateRoot"
    | "block.timestamp"
    | "block.totalDifficulty"
    | "block.transactionsRoot"
    | "transaction.blockHash"
    | "transaction.blockNumber"
    | "transaction.from"
    | "transaction.gas"
    | "transaction.hash"
    | "transaction.input"
    | "transaction.nonce"
    | "transaction.r"
    | "transaction.s"
    | "transaction.to"
    | "transaction.transactionIndex"
    | "transaction.v"
    | "transaction.value"
    | "log.address"
    | "log.blockHash"
    | "log.blockNumber"
    | "log.data"
    | "log.logIndex"
    | "log.removed"
    | "log.topics"
    | "log.transactionHash"
    | "log.transactionIndex"
    | "transactionReceipt.contractAddress"
    | "transactionReceipt.cumulativeGasUsed"
    | "transactionReceipt.effectiveGasPrice"
    | "transactionReceipt.gasUsed"
    | "transactionReceipt.logs"
  )[] | undefined;
};

type LogFilterRawEvent = {
  chainId: number;
  sourceIndex: number;
  checkpoint: string;
  log: Log;
  block: Block;
  transaction: Transaction;
  transactionReceipt: TransactionReceipt;
};

type BlockFilter = {
  type: "block";
  chainId: number;
  interval: number;
  offset: number;
  fromBlock: number;
  toBlock: number | undefined;
   include: (
    | "block.baseFeePerGas"
    | "block.difficulty"
    | "block.extraData"
    | "block.gasLimit"
    | "block.gasUsed"
    | "block.hash"
    | "block.logsBloom"
    | "block.miner"
    | "block.mixHash"
    | "block.nonce"
    | "block.number"
    | "block.parentHash"
    | "block.receiptsRoot"
    | "block.sha3Uncles"
    | "block.size"
    | "block.stateRoot"
    | "block.timestamp"
    | "block.totalDifficulty"
    | "block.transactionsRoot"
  )[] | undefined;
};

type BlockFilterRawEvent = {
  chainId: number;
  sourceIndex: number;
  checkpoint: string;
  block: Block;
};

type TransferFilter = {   
  type: "transfer";   
  chainId: number;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  fromBlock: number;
  toBlock: number | undefined;   
  include: (
    | "block.baseFeePerGas"
    | "block.difficulty"
    | "block.extraData"
    | "block.gasLimit"
    | "block.gasUsed"
    | "block.hash"
    | "block.logsBloom"
    | "block.miner"
    | "block.mixHash"
    | "block.nonce"
    | "block.number"
    | "block.parentHash"
    | "block.receiptsRoot"
    | "block.sha3Uncles"
    | "block.size"
    | "block.stateRoot"
    | "block.timestamp"
    | "block.totalDifficulty"
    | "block.transactionsRoot"
    | "transaction.blockHash"
    | "transaction.blockNumber"
    | "transaction.from"
    | "transaction.gas"
    | "transaction.hash"
    | "transaction.input"
    | "transaction.nonce"
    | "transaction.r"
    | "transaction.s"
    | "transaction.to"
    | "transaction.transactionIndex"
    | "transaction.v"
    | "transaction.value"
    | "transactionReceipt.contractAddress"
    | "transactionReceipt.cumulativeGasUsed"
    | "transactionReceipt.effectiveGasPrice"
    | "transactionReceipt.gasUsed"
    | "transactionReceipt.logs"
    | "trace.gas"
    | "trace.gasUsed"
    | "trace.from"
    | "trace.to"
    | "trace.input"
    | "trace.output"
    | "trace.value"
  )[] | undefined;
};

type TransferFilterRawEvent = {
  chainId: number;
  sourceIndex: number;
  checkpoint: string;
  block: Block;
  transaction: Transaction;
  transactionReceipt: TransactionReceipt;
  trace: Trace;
};

type TransactionFilter = {
  type: "transaction";
  chainId: number;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  functionSelectors: Hex | Hex[] | undefined;
  includeInner: boolean;
  includeFailed: boolean;
  callType: "call" | "staticcall" | "delegatecall" | "selfdestruct" | "create" | "create2" | "callcode" | undefined;
  fromBlock: number;
  toBlock: number | undefined;  
   include: (
    | "block.baseFeePerGas"
    | "block.difficulty"
    | "block.extraData"
    | "block.gasLimit"
    | "block.gasUsed"
    | "block.hash"
    | "block.logsBloom"
    | "block.miner"
    | "block.mixHash"
    | "block.nonce"
    | "block.number"
    | "block.parentHash"
    | "block.receiptsRoot"
    | "block.sha3Uncles"
    | "block.size"
    | "block.stateRoot"
    | "block.timestamp"
    | "block.totalDifficulty"
    | "block.transactionsRoot"
    | "transaction.blockHash"
    | "transaction.blockNumber"
    | "transaction.from"
    | "transaction.gas"
    | "transaction.hash"
    | "transaction.input"
    | "transaction.nonce"
    | "transaction.r"
    | "transaction.s"
    | "transaction.to"
    | "transaction.transactionIndex"
    | "transaction.v"
    | "transaction.value"
    | "transactionReceipt.contractAddress"
    | "transactionReceipt.cumulativeGasUsed"
    | "transactionReceipt.effectiveGasPrice"
    | "transactionReceipt.gasUsed"
    | "transactionReceipt.logs"
    | "trace.gas"
    | "trace.gasUsed"
    | "trace.from"
    | "trace.to"
    | "trace.input"
    | "trace.output"
    | "trace.value"
  )[] | undefined;
};

type TransactionFilterRawEvent = {
  chainId: number;
  sourceIndex: number;
  checkpoint: string;
  block: Block;
  transaction: Transaction;
  transactionReceipt: TransactionReceipt;
  trace: Trace;
};

Notes

I can see what you are getting at with some of the return data being confusing, (trace.input vs transaction.input). Not sure that we should do about this. Also not sure that separating traces and transaction would be very solve this, because the indexing function triggered by the trace may still want access to the transaction.

0xOlias · 2024-11-05T16:28:07Z

Edit: Decided against this (for now).

I think we should break out log topics and support passing a factory for those fields. This would unblock use cases like "filter for any ERC20 transfer sent to any smart account created by a specific factory".

type LogFilter = {
  type: "log";
  chainId: number;
  address: Address | Address[] | Factory | undefined;
  topic0: LogTopic | LogTopic[] | Factory | undefined;
  topic1: LogTopic | LogTopic[] | Factory | undefined;
  topic2: LogTopic | LogTopic[] | Factory | undefined;
  topic3: LogTopic | LogTopic[] | Factory | undefined;
  fromBlock: number;
  toBlock: number | undefined;
  include: ...
}

Note that there may be some complexity in the implementation, because addresses are 20 bytes but topics are 32.

0xOlias · 2024-11-05T21:22:09Z

Latest iteration. Pretty happy with it!

Notes:

Here "transaction" means input + receipt combined. No real distinction between inputs and receipts in this API.
In a normalized data model, you'd expect to implement this using four tables: blocks, transactions, logs, and traces. Transfers are backed by the traces table where trace.value > 0 && trace.reverted === false.
Note that the transaction filter supports including the trace associated with a transaction. This is really only useful for trace.output, which is the only trace field that's not already available on the transaction input + receipt.

type LogFilter = {
  chainId: number;
  fromBlock: number | undefined;
  toBlock: number | undefined;
  address: Address | Address[] | Factory | undefined;
  topic0: LogTopic | undefined;
  topic1: LogTopic | undefined;
  topic2: LogTopic | undefined; 
  topic3: LogTopic | undefined;
  // log + block + transaction fields
  include: string[] | undefined;
};

type BlockFilter = {
  chainId: number;
  fromBlock: number | undefined;
  toBlock: number | undefined;
  interval: number;
  offset: number;
  // block fields
  include: string[] | undefined;
};

type TransactionFilter = {
  chainId: number;
  fromBlock: number | undefined;
  toBlock: number | undefined;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  includeReverted: boolean;
  // transaction + block + trace fields
  include: string[] | undefined;
};

type TraceFilter = {
  chainId: number;
  fromBlock: number | undefined;
  toBlock: number | undefined;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  functionSelector: Hex | Hex[] | undefined;
  callType: "call" | "staticcall" | "delegatecall" | "selfdestruct" | "create" | "create2" | "callcode" | undefined;
  includeReverted: boolean;
  // trace + block + transaction fields
  include: string[] | undefined;
};

type TransferFilter = {   
  chainId: number;
  fromBlock: number | undefined;
  toBlock: number | undefined;
  fromAddress: Address | Address[] | Factory | undefined;
  toAddress: Address | Address[] | Factory | undefined;
  // transfer + block + transaction + trace fields
  include: string[] | undefined;
};

github-project-automation bot added this to Ponder Roadmap Oct 28, 2024

github-project-automation bot moved this to Todo in Ponder Roadmap Oct 28, 2024

kyscott18 mentioned this issue Oct 28, 2024

feature: Using eth_getBlockReceipts #1197

Open

kyscott18 mentioned this issue Oct 29, 2024

feat: use debug namespace for inner transactions #1201

Open

4 tasks

0xOlias mentioned this issue Nov 4, 2024

feat: combine all intervals into one table #1217

Merged

kyscott18 linked a pull request Nov 5, 2024 that will close this issue

accounts + native transfers #1210

Open

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: filter types #1198

feat: filter types #1198

kyscott18 commented Oct 28, 2024 •

edited

Loading

0xOlias commented Oct 28, 2024

kyscott18 commented Oct 29, 2024

0xOlias commented Oct 29, 2024 •

edited

Loading

kyscott18 commented Oct 30, 2024

0xOlias commented Nov 5, 2024 •

edited

Loading

0xOlias commented Nov 5, 2024 •

edited

Loading

feat: filter types #1198

feat: filter types #1198

Comments

kyscott18 commented Oct 28, 2024 • edited Loading

Proposed filter types

ponder.config.ts

factory()

0xOlias commented Oct 28, 2024

kyscott18 commented Oct 29, 2024

0xOlias commented Oct 29, 2024 • edited Loading

kyscott18 commented Oct 30, 2024

Notes

0xOlias commented Nov 5, 2024 • edited Loading

0xOlias commented Nov 5, 2024 • edited Loading

kyscott18 commented Oct 28, 2024 •

edited

Loading

`ponder.config.ts`

`factory()`

0xOlias commented Oct 29, 2024 •

edited

Loading

0xOlias commented Nov 5, 2024 •

edited

Loading

0xOlias commented Nov 5, 2024 •

edited

Loading