accounts + native transfers #1210

kyscott18 · 2024-11-01T19:44:22Z

Closes #1157 #1198 #1205

Tasks

`sync`

`sync-historical`

`sync-realtime`

fetchBlockEventData
handleBlock
factory support

API

AccountSource
ponder.config.ts
buildConfigAndIndexingFunctions

kyscott18 · 2024-11-04T02:08:18Z

packages/core/src/sync-historical/index.ts

+                      const blocks = Array.from(
+                        { length: interval[1] - interval[0] + 1 },
+                        (_, i) => interval[0] + i,
+                      );


I would feel a little better about this if we just used an array and a for loop

kyscott18 · 2024-11-04T02:21:16Z

packages/core/src/sync-historical/index.ts

+      if (block.transactions.find((t) => t.hash === hash) === undefined) {
        throw new Error(
-          `Detected inconsistent RPC responses. 'trace.blockHash' ${callTrace.blockHash} does not match 'block.hash' ${block.hash}`,
+          `Detected inconsistent RPC responses. 'trace.transactionHash' ${hash} not found in 'block.transactions' ${block.hash}`,
        );
      }


kyscott18 · 2024-11-04T02:22:35Z

packages/core/src/sync-historical/index.ts

+    let traces = await _debug_traceBlockByNumber(args.requestQueue, {
+      blockNumber: toHex(blockNumber),
+    });


When I was evaluating the debug rpc methods, I notice that traceBlockByHash was often way faster than traceBlockByNumber. It would require a different algorithm because block hash isn't immediately available, but something we should try before shipping

kyscott18 · 2024-11-04T02:47:21Z

packages/core/src/sync-historical/index.ts

+    const syncTraces = await _debug_traceBlockByNumber(args.requestQueue, {
+      blockNumber: toHex(blockNumber),
+    });


We should start to think about how we can avoid requesting the same block twice if there are multiple transfer and transaction filters

kyscott18 · 2024-11-05T03:28:24Z

packages/core/src/sync-realtime/index.ts

+    // });
+
+    // Remove traces that dont match transaction or transfer filter
+    traces = traces.filter((trace) => {


I think we need to do a depth first search over all the traces, either in this block or above where traces is declared.

It performs three functions:

processes child traces (each call can have several sub-calls)

orders traces for checkpoint.eventIndex

determine failed traces

See sync-historical

ponder/packages/core/src/sync-historical/index.ts

Lines 324 to 368 in 2117cec

const dfs = (

syncTrace: SyncTrace["result"][],

transactionHash: Hex,

parentTrace: SyncTrace["result"] | undefined,

) => {

for (const trace of syncTrace) {

if (filter.type === "transfer") {

if (

isTransferFilterMatched({

filter,

block: { number: toHex(blockNumber) },

trace,

})

) {

// TODO(kyle) handle factory

traces.push({ trace, transactionHash, position });

}

} else {

if (

isTransactionFilterMatched({

filter,

block: { number: toHex(blockNumber) },

trace,

})

) {

// TODO(kyle) handle factory

traces.push({ trace, transactionHash, position });

}

}

if (trace.error !== undefined) {

failedTraces.set(trace, trace.error);

} else if (parentTrace && failedTraces.has(parentTrace)) {

failedTraces.set(trace, failedTraces.get(parentTrace)!);

}

position++;

if (trace.calls) {

dfs(trace.calls, transactionHash, trace);

}

}

};

kyscott18 · 2024-11-05T03:29:53Z

packages/core/src/sync-realtime/index.ts


    return {
      block,
      logs,
      factoryLogs,
-      callTraces,
+      traces,


Should this type be the same as ??

ponder/packages/core/src/sync-historical/index.ts

Lines 313 to 317 in 2117cec

const traces: {

trace: SyncTrace["result"];

transactionHash: Hex;

position: number;

}[] = [];

kyscott18 · 2024-11-05T03:33:48Z

packages/core/src/sync-realtime/index.ts

    if (shouldRequestTraces) {
-      const traces = await _trace_block(args.requestQueue, {
+      traces = await _debug_traceBlockByNumber(args.requestQueue, {


Should use _debug_traceBlockByHash. It's more robust when reorgs occur because it unambiguously refers to the exact block

…or traces

…eBlock

packages/core/src/sync-realtime/index.ts

kyscott18 · 2024-11-18T02:54:26Z

packages/core/src/sync-realtime/index.ts

+      }
+      return isMatched;
+    });
+
    transactions = transactions.filter((t) => transactionHashes.has(t.hash));
    transactionReceipts = transactionReceipts.filter((t) =>


We should revisit this once we implement the includeTransactionReceipt feature

kyscott18 · 2024-11-18T15:54:02Z

packages/core/src/sync-realtime/filter.ts

+  if (
+    filter.includeReverted === false &&
+    (trace.revertReason !== undefined || trace.error !== undefined)
+  ) {
+    return false;
+  }


It's not super clear, this function returns true for the case "should this trace be saved in the database" and not "does this trace exactly match the filter such that it will cause an indexing function to run". The reason for the distinction is that we don't want the cache to be completely useless when the filter type slightly changes. For that reason this check should be removed.

In this example, if the user decides that they want to include reverted traces after they have already fully synced their app, every block will need to be resynced.

Maybe we could add some comments acknowledging the properties that are ignored.

packages/core/src/sync-realtime/filter.ts

kevinfaveri · 2024-11-19T11:11:31Z

Hello guys; I have a pressing timeline on getting something for native transfers and would like to explore using this here... Do you think it will be ready within next 1/2 weeks? Any timeline in mind?

Thank you in advance.

kyscott18 · 2024-11-19T21:19:06Z

Hello guys; I have a pressing timeline on getting something for native transfers and would like to explore using this here... Do you think it will be ready within next 1/2 weeks? Any timeline in mind?

We're hoping to get a prerelease out in the next few days, but don't want to make any promises on when we will release v0.8.

…ounts

kyscott18 · 2024-11-20T19:38:41Z

packages/core/src/sync-historical/index.ts

+          isTransactionFilterMatched({
+            filter: {
+              ...filter,
+              fromAddress: fromAddress,
+              toAddress: toAddress,
+            },
+            block,
+            transaction,
+          })


I like this, do you think we could get around overriding filter.fromAddress by adding an optional parameter fromChildAddresses: Set<Address>? Then we could reuse this in sync-realtime as well

kyscott18 · 2024-11-20T20:56:14Z

packages/core/src/sync-realtime/index.ts

@@ -173,6 +208,36 @@ export const createRealtimeSync = (
      }
    }

+    const isFactoryAddressMatched = ({


Could get rid of this if we update is[type]FilterMatched to handle factories

khaidarkairbek self-assigned this Nov 3, 2024

kyscott18 commented Nov 4, 2024

View reviewed changes

This was linked to issues Nov 5, 2024

[Feature] Native transfers #1157

Open

feat: filter types #1198

Open

feat: use debug namespace for inner transactions #1201

Open

kyscott18 commented Nov 5, 2024

View reviewed changes

Base automatically changed from v0.7 to main November 8, 2024 17:52

kyscott18 and others added 21 commits November 10, 2024 11:40

start new filter impl

2daf668

debug rpc types

35d5d00

withLog

3cfd157

add debug helper functions

f987047

syncTrace in sync-historical added with filter matching

cc5a3f6

syncTrace filtering modified, blockwise processing added

1a61820

sync-realtime: added tx/transfer factories

d15808d

sync: fragments and encoding for transaction/transfer filters

600304d

sync-realtime: transaction/trasfer filter added

b318ead

sync-realtime: handleBlock is modified for new filters

0ab76da

sync-realtime: fetchBlockEventData changed to debug and new filters f…

57bfc82

…or traces

update historical sync

3f3f82d

build checkpoint for traces

06b9060

record failed traces

3d944f0

implement insert traces db method

479be95

.

6681242

cleanup realtime

41412e6

corrections: minor changes

edbbea0

sync-historical: traceCache added

51e87df

sync-realtime: trace traversal added in fetchBlockEventData and handl…

a7052f3

…eBlock

sync: buildEvents and decodeEvents modfied

e7dcfbb

kyscott18 force-pushed the accounts branch from 8237db7 to e7dcfbb Compare November 10, 2024 16:41

fix debug type

5640a6a

kyscott18 commented Nov 18, 2024

View reviewed changes

packages/core/src/sync-realtime/index.ts Outdated Show resolved Hide resolved

kyscott18 commented Nov 18, 2024

View reviewed changes

kyscott18 and others added 2 commits November 17, 2024 21:55

cleanup realtime logs

9a60456

minor fix

f5ba161

kyscott18 commented Nov 18, 2024

View reviewed changes

kyscott18 added 2 commits November 18, 2024 16:02

build accounts

b4ca245

start historical sync tests

6f29077

update sync tests

a85ae63

khaidarkairbek and others added 7 commits November 19, 2024 16:59

sync: handle factory address

b12eacc

cleanup

f0fb96f

Merge branch 'accounts' of https://github.com/0xOlias/ponder into acc…

c9f4ed9

…ounts

fix transport tests

c9a40aa

realtime sync tests

3b4cd1f

fix typecheck

04c934e

account specific test

0ca8af0

kyscott18 marked this pull request as ready for review November 20, 2024 01:37

kyscott18 added 2 commits November 19, 2024 21:13

accounts example

0c97911

fix virtual types

7c2cfa4

kyscott18 commented Nov 20, 2024

View reviewed changes

kyscott18 and others added 8 commits November 20, 2024 16:00

cleanup sync

587e3de

note

ecb3482

transaction includeReverted

c9f5e1a

cleanup

aae6b18

fix: config

5bef6b0

sync: factory filter

ec42276

sync: factory filter cont.

017466c

set

597a65f

kyscott18 requested a review from 0xOlias November 23, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

accounts + native transfers #1210

accounts + native transfers #1210

kyscott18 commented Nov 1, 2024 •

edited

Loading

kyscott18 Nov 4, 2024

kyscott18 Nov 4, 2024

kyscott18 Nov 4, 2024

kyscott18 Nov 4, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 18, 2024

kyscott18 Nov 18, 2024

kevinfaveri commented Nov 19, 2024

kyscott18 commented Nov 19, 2024

kyscott18 Nov 20, 2024 •

edited

Loading

kyscott18 Nov 20, 2024

	const dfs = (
	syncTrace: SyncTrace["result"][],
	transactionHash: Hex,
	parentTrace: SyncTrace["result"] \| undefined,
	) => {
	for (const trace of syncTrace) {
	if (filter.type === "transfer") {
	if (
	isTransferFilterMatched({
	filter,
	block: { number: toHex(blockNumber) },
	trace,
	})
	) {
	// TODO(kyle) handle factory

	traces.push({ trace, transactionHash, position });
	}
	} else {
	if (
	isTransactionFilterMatched({
	filter,
	block: { number: toHex(blockNumber) },
	trace,
	})
	) {
	// TODO(kyle) handle factory

	traces.push({ trace, transactionHash, position });
	}
	}

	if (trace.error !== undefined) {
	failedTraces.set(trace, trace.error);
	} else if (parentTrace && failedTraces.has(parentTrace)) {
	failedTraces.set(trace, failedTraces.get(parentTrace)!);
	}

	position++;

	if (trace.calls) {
	dfs(trace.calls, transactionHash, trace);
	}
	}
	};

	const traces: {
	trace: SyncTrace["result"];
	transactionHash: Hex;
	position: number;
	}[] = [];

accounts + native transfers #1210

Are you sure you want to change the base?

accounts + native transfers #1210

Conversation

kyscott18 commented Nov 1, 2024 • edited Loading

Tasks

sync

sync-historical

sync-realtime

API

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinfaveri commented Nov 19, 2024

kyscott18 commented Nov 19, 2024

kyscott18 Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyscott18 commented Nov 1, 2024 •

edited

Loading

`sync`

`sync-historical`

`sync-realtime`

kyscott18 Nov 20, 2024 •

edited

Loading