fork-aware-tx-pool: add heavy load tests based on zombienet #7257

iulianbarbu · 2025-01-20T11:23:52Z

Description

Builds up towards addressing #5497 by creating some zombienet-sdk code infra that can be used to spin regular networks, as described in the fork aware transaction pool testing setup added here #7100. It will be used for developing tests against such networks, and to also spawn them on demand locally through tooling that will be developed in follow ups.

Integration

Node/runtime developers can run tests based on the zombienet-sdk infra that spins frequently used networks which can be used for analyzing behavior of various node related components, like fork aware transaction pool.

Review Notes

This is work in progress, and I'll follow up with some expected behavior which should be showcased, for some of the scenarios described in fatxpool: add heavy load testsuits #5497 .
The zombienet-sdk networks setup will be extracted in its own module/crate, and reused with a CLI tool that will be developed in a follow up (can be named zn-spawner), which would simplify the startup of networks that are used regularly for testing against fork-aware tx pool (at least), the logs management of such networks, and convenient integration with tools that observe certain behavior during tests, like tx-test-pool: https://github.com/michalkucharczyk/tx-test-tool/. The end result would look as if some is using zombienet-cli in a specific way (with given DSLs), and parts of these DSL files are also customizable. More and more stuff might end up being customizable, which will make this no different from zombienet-sdk, but having this zn-sdk infra glue code would still be useful because it builds standard ways of writing ZN based tests, and verifying behaviours of the networks.

Signed-off-by: Iulian Barbu <[email protected]>

lexnv · 2025-01-20T14:42:13Z

substrate/client/transaction-pool/tests/zombienet/small_network_yap.rs

+				"charlie".to_owned(),
+				vec![
+					"--force-authoring".into(),
+					("--pool-limit", "500000").into(),


dq: Would we want to configure this in the future?

One option would be to add a tiny builder wrapper around this struct, then we could image some scenarios like:

let small_net: SmallNetworkYap = SmallNetworkYapBuilder::default() .with_pool_limit(5k) .with_pool_kbytes(204) .build(relay_chain, para_chain); let small_net_2: SmallNetworkYap = SmallNetworkYapBuilder::default() .with_pool_limit(1k) .with_pool_kbytes(1k) .build(relay_chain, para_chain);

Could be a followup, feel free to ignore me :D

Very good point. Down the line we should enable the arg list to be passed according to the substrate-cli. I'll rethink this part a bit. We should consider that there is already a default arg list, and maybe we can use them in a dedicated constructor that would include them, but also provide a constructor version with no args, for custom initialization.

Signed-off-by: Iulian Barbu <[email protected]>

michalkucharczyk

So far looks good.

I was thinking about CLI. Maybe we don't need it, after all? Instead, we could use following:

$ cargo test --test stand_alone -- --exact run_single_collator_network

which would just:

spawn the network,
print out the location of the executed binaries (this one is important to me, to be absolutely sure that I don't run test on old binaries),
print out the location of logs file,
... or maybe just print zombienet summary - with ports, params, etc...
wait forever

Then one could use any tooling to just send transcations to this network.
We could start with this and see how it goes. We could do next iteration from here, and not over-complicate it from the beginning. In that way we would use the same config for manual testing, and for pre-defined test suits.

One more idea to control parameters would be using environment variables (not needed in the first step). Provides flexibility, less convenient to use comparing to CLI args, but much easier to implement.

export TXPOOLTESTS_POOL_LIMIT=1000
$ cargo test --test stand_alone -- --exact run_single_collator_network

Still we can have all the integration tests in different test module, reusing the same network configurations as those spawned in stand_alone mod, for example:

$ cargo test --test integration -- --exact single_collator_network__single_account_1M_txs

stand_alone tests would be excluded from cargo test command (as they never terminated on their own).

Any thoughts on this?

michalkucharczyk · 2025-01-21T08:31:06Z

substrate/client/transaction-pool/tests/zombienet/mod.rs

+
+/// Wrapper over a substrate node managed by zombienet.
+#[derive(Debug)]
+pub struct Node {


This should be named NodeConfig (or sth like this). From what I see it is used to provide params of single node.

michalkucharczyk · 2025-01-21T08:31:54Z

substrate/client/transaction-pool/tests/zombienet/limits_30.rs

+}
+
+#[async_trait::async_trait]
+impl Network for Limits30Network {


Methods seem to be copied into other files. Maybe it does not need to be trait? Or those methods could be provided in the trait. For sure copies should be avoided in the final shape.

michalkucharczyk · 2025-01-21T08:34:21Z

substrate/client/transaction-pool/tests/zombienet/limits_30.rs

+			"-lsub-libp2p=debug".into(),
+			"--pool-type=fork-aware".into(),
+			"--state-pruning=1024".into(),
+			"--rpc-max-subscriptions-per-connection=128000".into(),


This is also spreaded accross other networks. Maybe we should have common part that can be overwritten with particular settings?

michalkucharczyk · 2025-01-21T08:40:14Z

substrate/client/transaction-pool/tests/zombienet/yap.rs

+impl Network for YapNetwork {
+	fn ensure_bins_on_path(&self) -> bool {
+		// We need polkadot, polkadot-parachain, polkadot-execute-worker, polkadot-prepare-worker,
+		// (and ttxt? - maybe not for the network, but for the tests, definitely)


Would not say that ttxt would be needed here. Having purely-zombienet oriented abstractions sounds good. We could check ttxt in TxSpammer abstraction (which in future could just use lib for sending).

michalkucharczyk · 2025-01-21T08:54:08Z

substrate/client/transaction-pool/tests/zombienet/limits_30.rs

+		if !self.ensure_bins_on_path() {
+			return Err(anyhow!("Error: required bins weren't found on $PATH: polkadot"));
+		}
+		network_config.spawn_native().await.map_err(|err| anyhow!(format!("{}", err)))


Here I have one request:
Would be good to have some way to print the zombienet network params (just like it is done today with toml files). I would also like to see full paths to the binaries.

Enabling logs will be needed in stand_alone group of test (which just spawns networks and do no more, see my comment for details). For integrations test this may not be needed.

Printing the binaries path would be very helpful indeed. For cargo tests, we can reference the executable paths somewhere in the base dir of the network (if they aren't already mentioned there) - @pepoviola do you know?

But knowing the network basedir is another aspect with cargo test. We might be able to identify the basedir under tmp if we name it accordingly to the test in question, and add a timestamp suffix in the end for uniqueness. It would not be entirely perfect, but I find it usable. WDYT?

michalkucharczyk · 2025-01-21T09:11:19Z

substrate/client/transaction-pool/tests/zombienet/limits_30.rs

+
+// A zombienet network with two relaychain 'polkadot' validators and one parachain
+// validator based on yap-westend-live-2022 chain spec.
+pub struct Limits30Network {


One more thought:

We could have exactly the same struct for all the networks, and just have different values for different networks.
(This struct seems to be copied all over the files).

So maybe sth like this:

Network::new_limits_30_network() Network::new_single_collator() Network::new_old_pool_small() ...

Exactly! I have some TODOs already in zombienet/mod.rs after I realized this: https://github.com/iulianbarbu/polkadot-sdk/blob/21fda5fd98504cccfa9c0a087ddc466db8da2880/substrate/client/transaction-pool/tests/zombienet/mod.rs#L37C1-L43C68. I think they refer to the same thing.

pepoviola · 2025-01-21T09:30:51Z

substrate/client/transaction-pool/tests/zombienet/limits_30.rs

+
+#[async_trait::async_trait]
+impl Network for Limits30Network {
+	fn ensure_bins_on_path(&self) -> bool {


Hi, sorry to chime in. This is already checked by zombienet-sdk internally (for each cmd to execute and the workers).

Is zombienet-sdk capable of printing full executable paths? (I know, I am a bit paranoid on this 😅)

@pepoviola can you point me to where we're doing these checks in zombienet-sdk?

pepoviola · 2025-01-21T09:35:30Z

Hi @iulianbarbu / @michalkucharczyk, I'm working on an small cli to spawn from toml and we already can load tomls. Did you think that could be handy here?
Thanks!

michalkucharczyk · 2025-01-21T09:53:17Z

Hi @iulianbarbu / @michalkucharczyk, I'm working on an small cli to spawn from toml and we already can load tomls. Did you think that could be handy here? Thanks!

My 3 cents:
Our goal here is to have some abstraction that allows to run some testsuit against predefined network, and also run exactly the same predefined network to conduct manual tests.

We actually want to spawn network programatically. So I am not sure that cli will be helpful here. But having some API in zombienet that would accept toml and spawn the network could be potentially helpful. Especially when it comes to customization - instead of playing with CLI args or enviroment variables as I proposed in my previous comment we could just edit toml file.

On the other hand, it seems that using zn-sdk is not that difficult.

@iulianbarbu what is your opinion?

iulianbarbu · 2025-01-22T11:16:16Z

Responding to the last messages where @pepoviola chimed in:

having some API in zombienet that would accept toml and spawn the network could be potentially helpful. Especially when it comes to customization - instead of playing with CLI args or enviroment variables as I proposed in my previous comment we could just edit toml file.

+1 to this idea @michalkucharczyk . I personally prefer Rust and zn-sdk for the testsuite, while for manual runs, if we'd be able to import the tomls directly with zn-sdk, and have the option to also use them with a CLI, then we can have the best of both worlds. It would be just a preference for how we'd like to do the manual testing, because we can still run the testsuite locally, by changing things within the rust tests, but if we want just to run the network and then do other stuff against it, we'd have the CLI as well.

we already can load tomls.

yup, thanks @pepoviola for confirming this offline. For reference: https://docs.rs/zombienet-sdk/latest/zombienet_sdk/struct.NetworkConfig.html#method.load_from_toml.

I'm working on an small cli to spawn from toml

@pepoviola how different would be from the existing zombienet CLI and why do we need another one?

iulianbarbu added 5 commits January 16, 2025 18:10

wip: impl parachain network with zn-sdk

0283ab9

Signed-off-by: Iulian Barbu <[email protected]>

wip adding more stuff to the nodes to catch all network

67c2ee9

Signed-off-by: Iulian Barbu <[email protected]>

made network primitives more flexible

7afefe1

Signed-off-by: Iulian Barbu <[email protected]>

added todos for what's wip

6d10819

Signed-off-by: Iulian Barbu <[email protected]>

added network base_dir & the rest of the testing networks

0e084ab

Signed-off-by: Iulian Barbu <[email protected]>

iulianbarbu added the R0-silent Changes should not be mentioned in any release notes label Jan 20, 2025

iulianbarbu requested a review from michalkucharczyk January 20, 2025 11:23

iulianbarbu self-assigned this Jan 20, 2025

iulianbarbu changed the title ~~Ib zn test fatp~~ fork-aware-tx-pool: add heavy load tests based on zombienet Jan 20, 2025

iulianbarbu mentioned this pull request Jan 20, 2025

fatxpool: add heavy load testsuits #5497

Open

lexnv reviewed Jan 20, 2025

View reviewed changes

iulianbarbu added 3 commits January 20, 2025 22:22

renames and added builder support

daba086

Signed-off-by: Iulian Barbu <[email protected]>

added todos

f73874a

Signed-off-by: Iulian Barbu <[email protected]>

fixed derive_builder erros

21fda5f

Signed-off-by: Iulian Barbu <[email protected]>

michalkucharczyk reviewed Jan 21, 2025

View reviewed changes

pepoviola reviewed Jan 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fork-aware-tx-pool: add heavy load tests based on zombienet #7257

fork-aware-tx-pool: add heavy load tests based on zombienet #7257

iulianbarbu commented Jan 20, 2025 •

edited

Loading

lexnv Jan 20, 2025

iulianbarbu Jan 20, 2025

michalkucharczyk left a comment •

edited

Loading

michalkucharczyk Jan 21, 2025

michalkucharczyk Jan 21, 2025

michalkucharczyk Jan 21, 2025

michalkucharczyk Jan 21, 2025

michalkucharczyk Jan 21, 2025

iulianbarbu Jan 22, 2025

michalkucharczyk Jan 21, 2025

iulianbarbu Jan 22, 2025

pepoviola Jan 21, 2025

michalkucharczyk Jan 21, 2025

iulianbarbu Jan 22, 2025

pepoviola commented Jan 21, 2025

michalkucharczyk commented Jan 21, 2025

iulianbarbu commented Jan 22, 2025

fork-aware-tx-pool: add heavy load tests based on zombienet #7257

Are you sure you want to change the base?

fork-aware-tx-pool: add heavy load tests based on zombienet #7257

Conversation

iulianbarbu commented Jan 20, 2025 • edited Loading

Description

Integration

Review Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalkucharczyk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pepoviola commented Jan 21, 2025

michalkucharczyk commented Jan 21, 2025

iulianbarbu commented Jan 22, 2025

iulianbarbu commented Jan 20, 2025 •

edited

Loading

michalkucharczyk left a comment •

edited

Loading