This release contains the changes from polkadot-stable2409-2 to polkadot-stable2412.

ℹ️ Please note:
The tag corresponding to the current stable release polkadot-stable2412 and matching the old pattern will be available under polkadot-v1.17.0.

This release contains two additions comparing to the previous releases:

From this release, in addition to the Linux binaries, binaries compiled for MacOS are available for:

polkadot
polkadot-execute-worker
polkadot-prepare-worker
polkadot-parachain
polkadot-omni-node

gpg signature has changed slightly, all the binaries are signed with the new release team key: "ParityReleases [email protected]", which is part of the keyring as well as the previous key: "Parity Security Team [email protected]".

Changelog

Changelog for `Node Dev`

ℹ️ These changes are relevant to: Those who build around the client side code. Alternative client builders, SMOLDOT, those who consume RPCs. These are people who are oblivious to the runtime changes. They only care about the meta-protocol, not the protocol itself.

[#6729]: Fix order of resending messages after restart

At restart when dealing with a coalesced approval we might end up in a situation where we sent to approval-distribution the approval before all assignments covering it, in that case, the approval is ignored and never distribute, which will lead to no-shows.

[#6864]: Fix approval-voting canonicalize off by one

The approval-voting canonicalize was off by one, which lead to blocks being cleaned up every other 2 blocks. Normally, this is not an issue, but on restart we might end up sending NewBlocks to approval-distribution with finalized blocks.
This would be problematic in the case were finalization was already lagging before restart, so after restart approval-distribution will trigger aggression on the wrong already finalized block.

[#5038]: Plumb RPC listener up to caller

This PR allows the RPC server's socket address to be returned when initializing the server. This allows the library consumer to easily programmatically determine which port the RPC server is listening on.

[#6863]: Update merkleized-metadata to 0.2.0

0.1.2 was yanked as it was breaking semver.

[#5521]: Allow to call arbitrary runtime apis using RelayChainInterface

This PR adds a call_runtime_api method to RelayChainInterface trait, and a separate function also named call_runtime_api which allows the caller to specify the input and output types, as opposed to having to encode them.

[#6156]: Use bool::then instead of then_some with function calls

Fix misusage of bool::then_some.

[#5911]: Removed the possibility to start a shell parachain node

Removed the possibility to start a shell parachain node using the polkadot-parachain-lib or polkadot-parachain-bin.

[#5997]: Implement archive_unstable_storageDiff method

This PR implements the archive_unstable_storageDiff rpc-v2 method.
Developers can use this method to fetch the storage differences between two blocks. This is useful for oracles and archive nodes.
For more details see: https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/archive_unstable_storageDiff.md.

[#4837]: Add PVF execution priority

The new logic optimizes the distribution of execution jobs for disputes, approvals, and backings.
The main goal is to create back pressure for backing in the presence of disputes or numerous approval jobs.

[#5343]: Allow to disable gap creation during block import

New property BlockImportParams::create_gap allows to change whether to create block gap in case block has no parent (defaults to true keeping existing behavior), which is helpful for sync protocols that do not need to sync the gap after this happens. BlockImportOperation::create_gap() method was also introduced, though in most cases BlockImportParams::create_gap will be used.

[#6676]: Expose the unstable metadata v16

This PR exposes the unstable metadata V16. The metadata is exposed under the unstable u32::MAX number.
Developers can start experimenting with the new features of the metadata v16. Please note that this metadata is under development and expect breaking changes until stabilization.
The ExtrinsicMetadata trait receives a breaking change. Its associated type VERSION is rename to VERSIONS and now supports a constant static list of metadata versions.
The versions implemented for UncheckedExtrinsic are v4 (legacy version) and v5 (new version).
For metadata collection, it is assumed that all TransactionExtensions are under version 0.

[#5716]: Replace `lazy_static` with `LazyLock`

Replace all lazy_static usages with LazyLock from the Rust standard library. This will bring us less dependencies.

[#4639]: Added the fork-aware transaction pool implementation

Most important changes introduced by this PR:

The transaction pool references spread across codebase are now wrapper to a transaction pool trait object,
The fork-aware pool implementation was added.
The sc-transaction-pool refactored,
Trasnaction pool builder was introduced to allow to instantiation of either old or new transaction pool. Refer to PR description for
more details on how to enable fork-aware pool in the custom node.

[#5686]: sync: Remove checking of the extrinsics root

Remove checking the extrinsics root as part of the sync code.
With the introduction of system_version and the possibility to use the V1 layout for the trie when calculating the extrinsics root, it would require the sync code to fetch the runtime version first before knowing which layout to use when building the extrinsic root.
The extrinsics root is still checked when executing a block on chain.

[#6382]: gensis-config: patching default `RuntimeGenesisConfig` fixed

This PR fixes issue reported in #6306.
It changes the behavior of sc_chain_spec::json_patch::merge function which no longer removes any keys from the base JSON object.

[#5469]: Syncing strategy refactoring

Mostly internal changes to syncing strategies that is a step towards making them configurable/extensible in the future. It is unlikely that external developers will need to change their code.

[#5774]: Avoid unnecessary state reset of allowed_requests when no block requests are sent

Previously, the state of allowed_requests was always reset to the default even if there were no new block requests. This could cause an edge case because peer_block_request() will return early next time when there are no ongoing block requests.
This patch fixes it by checking whether block requests are empty before updating the state.

[#6588]: rpc server: fix subscription id_provider being reset to default one

The modification ensures that the id_provider variable is cloned instead of taken, which can help prevent issues related id provider being reset to the default.

[#5666]: Make syncing strategy an argument of the syncing engine

Syncing strategy is no longer implicitly created when building network, but needs to be instantiated explicitly.
Previously default implementation can be created with new function build_polkadot_syncing_strategy or custom syncing strategy could be implemented and used instead if desired, providing greater flexibility for chain
developers.

[#5883]: statement-distribution RFC103 implementation

Introduces checks for the new candidate descriptor fields: core_index and session_index.

[#5924]: Bump PoV request timeout

With asynchronous backing and PoV size 10MB, we can increase the PoV request timeout from 1.2s to 2s.

[#4846]: Make approval-voting runnable on a worker thread

Make approval-voting subsystem runnable on a separate worker thread without having to to always pass to it an orchestra context. It achieves that by refactoring existing functions to require only the minimal set of traits needed in the function instead of the general
Context

[#4849]: Introduce approval-voting-parallel subsystem

This introduces a new subsystem called approval-voting-parallel. It combines the tasks previously handled by the approval-voting and approval-distribution subsystems.

The new subsystem is enabled by default on all test networks. On production networks
like Polkadot and Kusama, the legacy system with two separate subsystems is still in use.
However, there is a CLI option --enable-approval-voting-parallel to gradually roll out the new subsystem on specific nodes. Once we are confident that it works as expected, it will be enabled by default on all networks.

The approval-voting-parallel subsystem coordinates two groups of workers:

Four approval-distribution workers that operate in parallel, each handling tasks based on the validator_index of the message originator.
One approval-voting worker that performs the tasks previously managed by the standalone approval-voting subsystem.

[#5787]: Move bitfield_distribution to blocking task pool and set capacity to 8192

This is moving bitfield_distribution to the blocking task pool because it does cpu intensive work and to make it snappier. Additionally, also increase the message capacity of the subsystem to make sure the queue does not get full if there is a burst of messages.

[#5847]: `candidate-validation`: RFC103 implementation

Introduces support for new v2 descriptor core_index and session_index fields.
The subsystem will check the values of the new fields only during backing validations.

[#5919]: substrate-offchain: upgrade hyper to v1

Bump depencency hyper of substrait-offchain for http from 0.14 to 1.
This changed APIs a bit;

sc_offchain::Offchainworker::new() now returns std::io::Result<Self> (Previously was Self)

[#6218]: Enable approval-voting-parallel by default on kusama

Enable approval-voting-parallel by default on kusama

[#6104]: LocalTransactionPool implemented for fork aware transaction pool

LocalTransactionPool trait is implemented for fork aware transaction pool.

[#5707]: Remove ValidateFromChainState

Removed the CandidateValidationMessage::ValidateFromChainState, which was previously used by backing, but is no longer relevant since initial async backing implementation

[#6696]: Make approval-distribution aggression a bit more robust and less spammy

The problem with the current implementation of approval-distribution aggression is that is too spammy, and can overload the nodes, so make it less spammy by moving back the moment we trigger L2 aggression and make resend enable only for the latest unfinalized block.

[#5768]: export NodeHealthProxyLayer

This PR export NodeHealthProxyLayer from sc-rpc-server.

[#6677]: chore: Update litep2p to v0.8.2

This includes a critical fix for debug release versions of litep2p (which are running in Kusama as validators).

While at it, have stopped the oncall pain of alerts around incoming_connections_total. We can rethink the metric expose of litep2p in Q1.

[#5635]: Fix edge case where state sync is not triggered

There is an edge case where the finalized block notification is received, but the conditions required to initiate the state sync are not fully met. In such cases, state sync would fail to start as expected and remain stalled.
This patch addresses it by storing the pending attempt and trying to start the state sync later when the conditions are satisfied.

[#5915]: Omni-Node renamings

This PR renames the polkadot-parachain-lib crate to polkadot-omni-node-lib and introduces a new polkadot-omni-node binary.

[#5880]: Fix prospective parachains test to use shuffled candidate list

Fix prospective parachains test to use shuffled candidate list.
Resolves #5617.

[#5908]: collation-generation: use v2 receipts

Implementation of RFC 103 for the collation-generation subsystem.
Also removes the usage of AsyncBackingParams.

[#6268]: Bump a timeout in zombienet coretime smoke test

polkadot/zombienet_tests/smoke/0004-coretime-smoke-test.zndsl still timeouts on CI from time to time. Bumping the timeout a bit more.

Related to #6226

[#6652]: rpc server: re-use server builder per rpc interface

This changes that the RPC server builder is re-used for each RPC interface which is more efficient than to build it for every connection.

[#6742]: Update litep2p backend to v0.8.3

This release includes two fixes for small memory leaks on edge-cases in the notification and request-response protocols.
While at it, have downgraded a log message from litep2p.

[#5830]: Remove jaeger from approval-voting and approval-distribution

Jaeger was remove from approval-voting and approval-distribution because it did not prove to improve the debugging and it wasted precious cpu cycles.

[#5737]: Make syncing service an argument of `build_network`

build_network is accompanied with lower-level build_network_advanced with simpler API that does not create syncing engine internally, but instead takes a handle to syncing service as an argument. In most cases typical syncing engine with polkadot syncing strategy and default block downloader can be created with newly introduced sc_service::build_default_syncing_engine() function, but lower-level build_default_block_downloader also exists for those needing more customization.

These changes allow developers higher than ever control over syncing implementation, but build_network is still available for easier high-level usage.

[#5762]: Fast return for invalid request of node health

Return directly when invalid request for node health api

[#5601]: Introduce `RpcParams` in sc-cli

Refactors and consolidates all RPC-related parameters in the run command into a dedicated RpcParams struct. This change allows downstream users to build custom run command without duplicating code.

[#6011]: collator protocol: validate descriptor version on the validator side

Implement checks needed for RFC 103 polkadot-fellows/RFCs#103 in the validator side of the collator protocol.

[#5789]: Prevents EthereumBlobExporter from consuming parameters when returning NotApplicable

When the EthereumBlobExporter returned a NotApplicable error, it consumed parameters universal_source, destination and message. As a result, subsequent exporters could not use these values. This PR corrects this incorrect behaviour. It also changes error type from Unroutable to NotApplicable when the global consensus system cannot be extracted from the universal_source, or when the source location cannot be converted to an agent ID. Lastly, it changes the error type from MissingArgument to NotApplicable when the parachain ID cannot be extracted from the location. These changes should have no effect - it is purely to correct behvaiour should multiple exporters be used.

[#5540]: Avoid unnecessary block gap updates

Previously, the block gap storage in database and state in BlockchainDb could be updated even if no changes occurred.
This commit refines the logic to ensure updates only occur when the block gap value actually changes, reducing unnecessary writes and enhancing overall efficiency.

[#4974]: Remove libp2p dependency from sc-network-sync

This PR removes libp2p::request_response::OutboundFailure from substrate/client/network/sync/src/engine.rs.

[#5811]: Improve `import_notification_stream` documentation

"Updates the doc comment on the import_notification_stream to make its behaviour clearer. Now it specifically states that this notification stream is fired on every import notification after the initial sync, and only when there are re-orgs in the initial sync."

[#5732]: Expose the unstable metadata v16

[#6690]: Fix Possible bug, Vote import failed after aggression is enabled

Fix the appearance of Possible bug: Vote import failed after aggression is enabled, the log itself is harmless because approval gets imported anyway and aggression is able to distribute it, nevertheless is something that can be easily be fixed by picking the highest required routing possible.

[#5592]: Introduce `BlockGap`

This is the first step towards #5406, refactoring the representation of block gap. This refactor converts the existing (NumberFor<Block>, NumberFor<Block>) into a dedicated BlockGap<NumberFor<Block>>
struct. This change is purely structural and does not alter existing logic, but lays the groundwork for the follow-up PR. The compatibility concern in the database caused by the new structure transition is addressed as well.

The BlockGap refactoring results in breaking changes in the Info structure returned in client.info().

[#5856]: Extend state tracking of chainHead to capture notification gaps

This PR extends the state tracking of the RPC-v2 chainHead methods.
ChainHead tracks the reported blocks to detect notification gaps.
This state tracking ensures we can detect NewBlock events for which we did not report previously the parent hash.

[#6015]: Rename QueueEvent::StartWork

When we send QueueEvent::StartWork, we have already completed the execution. Therefore, QueueEvent::FinishWork is a better match.

Changelog for `Runtime Dev`

ℹ️ These changes are relevant to: All of those who rely on the runtime. A parachain team that is using a pallet. A DApp that is using a pallet. These are people who care about the protocol (WASM, not the meta-protocol (client).)

[#6318]: Refactor pallet claims

Adds bounds on stored types for pallet claims.
Migrates benchmarking from v1 to v2 for pallet claims.

[#5892]: Treasury: add migration to clean up unapproved deprecated proposals

It is no longer possible to create Proposals storage item in pallet-treasury due to migration from governance v1 model but there are some Proposals whose bonds are still on hold with no way to release them.
The purpose of this migration is to clear Proposals which are stuck and return bonds to the proposers.

[#6299]: migrate pallet-recovery to benchmark V2 syntax

migrate pallet-recovery to benchmark V2 syntax

[#6073]: Refactor `pallet-grandpa` benchmarks to `v2`

Update benchmarks in GRANDPA pallet to use the second version of the frame_benchmarking runtime benchmarking framework.

[#6039]: Added Trusted Query API calls.

Added is_trusted_reserve and is_trusted_teleporter API calls to all the runtimes.
Given an asset and a location, they return if the chain trusts that location as a reserve or teleporter for that asset respectively.
You can implement them on your runtime by simply calling a helper function on pallet-xcm.

	impl xcm_runtime_apis::trusted_query::TrustedQueryApi<Block> for Runtime {
  fn is_trusted_reserve(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_reserve(asset, location)
  }
  fn is_trusted_teleporter(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_teleporter(asset, location)
  }
}

[#5939]: [pallet-revive] Bump PolkaVM and add static code validation

Statically validate basic block sizes and instructions.

[#5866]: [pallet-revive] Ethereum JSON-RPC integration

Changes Included:

A new pallet::call eth_transact.
A custom UncheckedExtrinsic struct to dispatch unsigned eth_transact calls from an Ethereum JSON-RPC proxy.
Generated types and traits to support implementing a JSON-RPC Ethereum proxy.

[#6778]: Added fallback_max_weight to Transact for sending messages to V4 chains

Removing the require_weight_at_most parameter in V5 Transact introduced a problem when converting a message from V5 to V4 to send to chains that didn't upgrade yet.
The local chain doesn't know how to decode calls for remote chains so it can't automatically populate require_weight_at_most required by V4 Transact.
To fix this, XCM v5 Transact now also takes a fallback_max_weight: Option<Weight> parameter.
This can be set to None if the instruction is not meant to be sent to chains running XCM versions lower than V5.
If set to Some(weight), a subsequent conversion to V4 will result in Transact { require_weight_at_most: weight, .. }.
The plan is to remove this workaround in V6 since there will be a good conversion path from V6 to V5.

[#6316]: Migrate pallet-election-provider-multi-phase benchmark to v2 and improve doc

Migrate pallet-election-provider-multi-phase benchmark to v2 and improve doc

[#5779]: [pallet-revive] last call return data API

This PR introduces 2 new syscall: return_data_size and return_data_copy, resembling the semantics of the EVM RETURNDATASIZE and RETURNDATACOPY opcodes.

The ownership of ExecReturnValue (the return data) has moved to the Frame.
This allows implementing the new contract API surface functionality in ext with no additional copies.
Returned data is passed via contract memory, memory is (will be) metered, hence the amount of returned data can not be statically known, so we should avoid storing copies of the returned data if we can.
By moving the ownership of the exectuables return value into the Frame struct we achieve this.

A zero-copy implementation of those APIs would be technically possible without that internal change by making the callsite in the runtime responsible for moving the returned data into the frame after any call.
However, resetting the stored output needs to be handled in ext, since plain transfers will not affect the stored return data (and we don't want to handle this special call case inside the runtime API).
This has drawbacks:

It can not be tested easily in the mock.
It introduces an inconsistency where resetting the stored output is handled in ext, but the runtime API is responsible to store it back correctly after any calls made.
Instead, with ownership of the data in Frame, both can be handled in a single place.
Handling both in fn run() is more natural and leaves less room for runtime API bugs.

The returned output is reset each time before running any executable in a nested stack.
This change should not incur any overhead to the overall memory usage as only the returned data from the last executed frame will be kept around at any time.

[#6171]: remove parachains_assigner

Remove the code of the parachains_assigner pallet, since coretime was released on all production networks.

[#6205]: pallet-message-queue: Fix max message size calculation

The max size of a message should not depend on the weight left in a given execution context. Instead the max message size depends on the service weights configured for the pallet. A message that may does not fit into on_idle is not automatically overweight, because it may can be executed successfully in on_initialize or in another block in on_idle when there is more weight left.

[#5845]: Fix compilation after renaming some of benchmarks in pallet_revive.

Changed the "instr" benchmark so that it should no longer return to little weight. It is still bogus but at least benchmarking should not work.

[#5861]: [pallet-revive] immutable data storage

This PR introduces the concept of immutable storage data, used for Solidity immutable variables.

This is a minimal implementation. Immutable data is attached to a contract; to ContractInfo fixed in size, we only store the length there, and store the immutable data in a dedicated storage map instead. Which comes at the cost of requiring an storage read (costly) for contracts using this feature.

We discussed more optimal solutions not requiring any additional storage accesses internally, but they turned out to be non-trivial to implement. Another optimization benefiting multiple calls to the same contract in a single call stack would be to cache the immutable data in Stack. However, this potential creates a DOS vulnerability (the attack vector is to call into as many contracts in a single stack as possible, where they all have maximum immutable data to fill the cache as efficiently as possible). So this either has to be guaranteed to be a non-issue by limits, or, more likely, to have some logic to bound the cache. Eventually, we should think about introducing the concept of warm and cold storage reads (akin to EVM). Since immutable variables are commonly used in contracts, this change is blocking our initial launch and we should only optimize it properly in follow-ups.

This PR also disables the set_code_hash API (which isn't usable for Solidity contracts without pre-compiles anyways). With immutable storage attached to contracts, we now want to run the constructor of the new code hash to collect the immutable data during set_code_hash. This will be implemented in a follow up PR.

[#6129]: Improved TrustedQueryAPI signatures.

Changed returned type of API methods from Result<bool, xcm_runtime_apis::trusted_query::Error> to a typed one:
type XcmTrustedQueryResult = Result<bool, xcm_runtime_apis::trusted_query::Error>

[#6169]: [Deprecation] deprecate treasury `spend_local` call and related items

Deprecates spend_local from the treasury pallet and items associated with it.

Migration

For users who were using only `spend_local` before

To replace spend_local functionality configure Paymaster pallet configuration to be PayFromAccount and configure AssetKind to be () and use spend call instead.
This way spend call will function as deprecated spend_local.

Example:

impl pallet_treasury::Config for Runtime {
  ..
  type AssetKind = ();
  type Paymaster = PayFromAccount<Self::Currency, TreasuryAccount>;
  // convert balance 1:1 ratio with native currency
  type BalanceConverter = UnityAssetBalanceConversion;
  ..
}

For users who were already using `spend` with all other assets, except the native asset

Use NativeOrWithId type for AssetKind and have a UnionOf for native and non-native assets, then use that with PayAssetFromAccount.

Example from kitchensink-runtime:

// Union of native currency and assets
pub type NativeAndAssets =
  UnionOf<Balances, Assets, NativeFromLeft, NativeOrWithId<u32>, AccountId>;

impl pallet_treasury::Config for Runtime {
  ..
  type AssetKind = NativeOrWithId<u32>;
  type Paymaster = PayAssetFromAccount<NativeAndAssets, TreasuryAccount>;
  type BalanceConverter = AssetRate;
  ..
}

// AssetRate pallet configuration
impl pallet_asset_rate::Config for Runtime {
  ..
  type Currency = Balances;
  type AssetKind = NativeOrWithId<u32>;
  ..
}

[#5526]: Fix enact_candidate weight generation

This PR works around an issue in multivariate linear regression of weight generation.

[#6496]: Backport #6418 to `stable2412`

This PR is a backport of #6418.

For context, TransactionExtension, introduced in #3685, is part of the stable2412 release, and this PR brings important fixes and quality of life improvements. Doing the backport now allows us to not break the interface later.

Opened against #6473 as the changes in the original PR were made on top of the changes in this backport.

[#6255]: [pallet-child-bounties] Index child bounties by parent bounty

Index child bounties by their parent bounty, ensuring that their indexes are independent of child bounties from other parent bounties. This will allow for predictable indexes and the ability to batch creation and approval calls together.

Migration for Runtime Pallet Instance

Use migration::v1::MigrateToV1Impl storage migration type to translate ids for the active child bounties and migrate the state to the new schema.

Migration for Clients

Use new ParentTotalChildBounties storage item to iterate over child bounties for a certain parent bounty;
Use new ChildBountyDescriptionsV1 storage item to get the bounty description instead of removed ChildBountyDescriptions;
Use V0ToV1ChildBountyIds storage item to look up the new child bounty id for a given old child bounty id;
Update the child bounty account id derivation from PalletId + "cb" + child_id to PalletId + "cb" + bounty_id + child_id.

Additional Notes

The ChildBountyCount storage item is deprecated and will be remove in May 2025.

[#5623]: Generic slashing side-effects

What?
Make it possible for other pallets to implement their own logic when a slash on a balance occurs.

How?
First we abstract the done_slash function of holds::Balanced to it's own trait that any pallet can implement.
Then we add a config type in pallet-balances that accepts a callback tuple of all the pallets that implement this trait.
Finally implement done_slash for pallet-balances such that it calls the config type.
Integration
The default implementation of done_slash is still an empty function, and the new config type of pallet-balances can be set to an empty tuple, so nothing changes by default.

[#6645]: xcm: fix local/remote exports when inner routers return `NotApplicable`

Resolved a bug in the local/remote exporters used for bridging. Previously, they consumed dest and msg without returning them when inner routers/exporters failed with NotApplicable. This PR ensures compliance with the SendXcm and ExportXcm traits.

[#5917]: XCM paid execution barrier supports more origin altering instructions

Updates the AllowTopLevelPaidExecutionFrom barrier to also support messages that
use DescendOrigin or AliasOrigin for altering the computed origin during execution.

[#5807]: [pallet-revive] last call return data API

This PR adds the EVM chain ID to Config as well as a corresponding runtime API so contracts can query it.

Related issue: paritytech/revive#44

[#5804]: Refactor get_account_id_from_seed / get_from_seed to one common place `get_account_id_from_seed / get_from_seed` were copied all over the place. This PR removes unnecessary code duplication.

Keyring::iter() provides the same functionality and is used instead.

[#5194]: FRAME: Support instantiable pallets in tasks.

In FRAME, tasks can now be used in instantiable pallet. Also some fix for expansion with conditional compilation in construct runtine.

[#6260]: [pallet-revive] code size API

This PR implements the contract API to query the code size of a given address.

[#5640]: [pallet-revive] Move event's topics

Move event's topics inside body

[#6278]: [pallet-revive] rpc server add docker file

Add a docker for pallet-revive eth-rpc

Tested with

sudo docker build . -t eth-rpc -f substrate/frame/revive/rpc/Dockerfile
sudo docker run --network="host" -e RUST_LOG="info,eth-rpc=debug" eth-rpc

[#6505]: [pallet-broker] Fix auto renew benchmarks

Fix the broker pallet auto-renew benchmarks which have been broken since #4424, yielding Weightless due to some prices being set too low, as reported in #6474.

Upon further investigation it turned out that the auto-renew contribution to rotate_sale was always failing but the error was mapped. This is also fixed at the cost of a bit of setup overhead.

[#5726]: revive: Limit the amount of static memory

Limit the amount of static memory a contract can declare.

[#5608]: [pallet-revive] update runtime types

Refactor the Ext trait to use U256 instead of BalanceOf or MomentOf

[#5664]: Calling an address without associated code is a balance transfer

This makes pallet_revive behave like EVM where a balance transfer is just a call to a plain wallet.

[#6261]: Add missing events to identity pallet

Extrinsics from pallet_identity that were missing an event emission on success now emit one.

[#6317]: eth-rpc fixes

Various fixes for the release of eth-rpc & ah-westend-runtime:

Bump asset-hub westend spec version
Fix the status of the Receipt to properly report failed transactions
Fix value conversion between native and eth decimal representation

[#6536]: Bridges testing improvements

This PR includes:

Refactored integrity tests to support standalone deployment of pallet-bridge-messages.
Refactored the open_and_close_bridge_works test case to support multiple scenarios, such as:
1. A local chain opening a bridge.
2. Sibling parachains opening a bridge.
3. The relay chain opening a bridge.
Previously, we added instance support for pallet-bridge-relayer but overlooked updating the DeliveryConfirmationPaymentsAdapter.

[#5971]: XCMv5 InitiateTransfer can preserve original origin across chains.

Beware: This only works if the following two rules are followed by the chain receiving such a message.

Alias to interior locations is valid (the exact same behaviour as DescendOrigin)
AssetHub can alias everything (most importantly sibling accounts and ethereum).
These can be set with the Aliasers configuration item, with the following adapters:
AliasChildLocation
AliasOriginRootUsingFilter with AssetHub and Everything
An example of the first one can be seen in asset-hub-westend and of the second one in penpal-runtime.

[#6148]: Fix migrations for pallet-xcm

pallet-xcm stores some operational data that uses Versioned* XCM types. When we add a new XCM version (XV), we deprecate XV-2 and remove XV-3.
This PR extends the existing MigrateToLatestXcmVersion to include migration for the Queries, LockedFungibles, and RemoteLockedFungibles storage types.
Additionally, more checks were added to try_state for these types.

[#6246]: [pallet-revive] implement the block hash API

Bound T::Hash to H256
Implement the block hash API

[#5888]: parachain-system: send core selector ump signal

Send the core selector ump signal in cumulus. Guarded by a compile time feature called experimental-ump-signals until nodes are upgraded to a version that includes #5423 for gracefully handling ump signals.

[#5743]: [pallet-revive] write sandbox output according to the provided output buffer length

Instead of error out if the provided output buffer is smaller than what we want to write, we can just write what fits into the output buffer instead.
We already write back the actual bytes written to the in-out pointer, so contracts can check it anyways.

This in turn introduces the benefit of allowing contracts to implicitly request only a portion of the returned data from calls and incantations.
Which is especially beneficial for YUL as the call family opcodes have a return data size argument and this change removes the need to work around it in contract code.

[#6781]: Bridges - revert-back congestion mechanism

With permissionless lanes PR#4949, the congestion mechanism based on sending Transact(report_bridge_status(is_congested)) from pallet-xcm-bridge-hub to pallet-xcm-bridge-hub-router was replaced with a congestion mechanism that relied on monitoring XCMP queues. However, this approach could cause issues, such as suspending the entire XCMP queue instead of isolating the affected bridge. This PR reverts back to using report_bridge_status as before.

[#6187]: [pallet-revive] rework balance transfers

This PR removes the transfer syscall and changes balance transfers to make the existential deposit (ED) fully transparent for contracts.

The transfer API is removed since there is no corresponding EVM opcode and transferring via a call introduces barely any overhead.

We make the ED transparent to contracts by transferring the ED from the call origin to nonexistent accounts. Without this change, transfers to nonexistant accounts will transfer the supplied value minus the ED from the contracts viewpoint, and consequentially fail if the supplied value lies below the ED. Changing this behavior removes the need for contract code to handle this rather annoying corner case and aligns better with the EVM. The EVM charges a similar deposit from the gas meter, so transferring the ED from the call origin is practically the same as the call origin pays for gas.

[#5665]: [pallet-contracts] remove riscv support

RISC-V support is now being built inside the new fork pallet-revive

[#6147]: [pallet-revive] Ethereum JSON-RPC

Add a new Ethereum JSON-RPC server that can be used a substrate chain configured with pallet-revive

[#5311]: No-op Impl Polling Trait

Provide a NoOp implementation of the Polling trait for unit where the trait is defined and skiping benchmarks that necessitate it's definition.

[#5502]: [pallet-revive] Add pallet to AH westend

Add pallet-revive to Westend runtime, and configure the runtime to accept Ethereum signed transaction

[#6141]: Improve `CheckMetadataHash` transaction extension weight and logic

The compilation now panics if the optional compile-time environment variable RUNTIME_METADATA_HASH contains an invalid value.
The weight for the CheckMetadataHash transaction extension is more accurate as it is almost compile-time.

[#6025]: Refactor staking pallet benchmarks to `v2`

Update benchmarks in staking pallet to the second version of the frame_benchmarking runtime benchmarking framework.

[#6384]: Relax requirements on `assign_core`.

Relax requirements for assign_core so that it accepts updates for the last scheduled entry.
This will allow the coretime chain to split up assignments into multiple messages, which allows for interlacing down to single block granularity.

Fixes: #6102

[#5813]: build_struct_json_patch macro added

This PR adds a macro that allows to construct a RuntimeGenesisConfig preset containing only provided fields, while performing the validation of the entire struct.

Related issue: #5700

[#5585]: Added SetAssetClaimer instruction to XCM v5.

Added SetAssetClaimer implementation to XCM v5. With asset_claimer set users can retrieve their trapped assets at any point in time without the need to go through OpenGov reclaim process.

[#5765]: Added foreign locations to local accounts converter to all the parachains.

Added foreign locations to local accounts converter to all the parachains.
i.e. added HashedDescription<AccountId, DescribeFamily> to LocationToAccountId

[#5274]: Enrich metadata IR with associated types of config traits

This feature is part of the upcoming metadata V16. The associated types of the Config trait that require the TypeInfo or Parameter bounds are included in the metadata of the pallet. The metadata is not yet exposed to the end-user, however the metadata intermediate representation (IR) contains these types.

Developers can opt out of metadata collection of the associated types by specifying without_metadata optional attribute to the #[pallet::config].

Furthermore, the without_metadata argument can be used in combination with the newly added #[pallet::include_metadata]
attribute to selectively include only certain associated types in the metadata collection.

[#6032]: Fix `feeless_if` in pallet section

Fix compilation with pallet::feeless_if in a pallet section: a local binding unexpectely resolved to a macro definition.

[#5941]: `SolochainDefaultConfig`: Use correct `AccountData`

SolochainDefaultConfig by default was setting AccountData to AccountInfo.
Thus, the actual account data was recursively nested the same type. By default it should be set (), because this is the only reasonable AccountData.

If you have used SolochainDefaultConfig before and did not overwrite, AccountData
you should now overwrite it to AccountInfo or you will need to write a migration to change the data.

[#6365]: pallet-revive: Use `RUSTUP_TOOLCHAIN` if set

We were not passing through the RUSTUP_TOOLCHAIN variable to the build.rs script of our fixtures. This means that setting the toolchain like cargo +1.81 build had no effect on the fixture build. It would always fall back to the default toolchain.

[#6221]: snowbridge: allow account conversion for Ethereum accounts

Replaced GlobalConsensusEthereumConvertsFor with EthereumLocationsConverterFor that allows Location to AccountId conversion for the Ethereum network root as before, but also for Ethereum contracts and accounts.

[#6809]: XCMv5 - SetHints instruction

Implementation of fellowship RFC 107.
The new SetHints instruction is a repackaging of SetAssetClaimer that also allows future "hints" which alter the default behaviour of the executor.
The AllowTopLevelPaidExecutionFrom barrier allows this instruction between WithdrawAsset and BuyExecution/PayFees to configure things before the actual meat of the program.

[#4251]: MBM `try-runtime` support

MBM try-runtime support

This MR adds support to the try-runtime trait such that the try-runtime-CLI will be able to support MBM testing here.
It mainly adds two feature-gated hooks to the SteppedMigration hook to facilitate testing. These hooks are named pre_upgrade and post_upgrade and have the same signature and implications as for single-block migrations.

Integration

To make use of this in your Multi-Block-Migration, just implement the two new hooks and test pre- and post-conditions in them:

#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, frame_support::sp_runtime::TryRuntimeError>
{
  // ...
}

#[cfg(feature = "try-runtime")]
fn post_upgrade(prev: Vec<u8>) -> Result<(), frame_support::sp_runtime::TryRuntimeError> {
    // ...
}

You may return an error or panic in these functions to indicate failure.
This will then show up in the try-runtime-CLI and can be used in CI for testing.

Changes:

Adds try-runtime gated methods pre_upgrade and post_upgrade
on SteppedMigration
Adds try-runtime gated methods nth_pre_upgrade
and nth_post_upgrade on SteppedMigrations
Modifies pallet_migrations
implementation to run pre_upgrade and post_upgrade steps at the appropriate times, and panic in the event of migration failure.

[#5693]: Remove `sp_runtime::RuntimeString` and replace with `Cow<'static, str>` or `String` depending on use case

Deprecate RuntimeString, replace with String or Cow<'static, str> where appropriate.

For downstream projects the upgrade will primarily consist of following two changes:

#[sp_version::runtime_version]
pub const VERSION: RuntimeVersion = RuntimeVersion {
-	spec_name: create_runtime_str!("statemine"),
-	impl_name: create_runtime_str!("statemine"),
+	spec_name: alloc::borrow::Cow::Borrowed("statemine"),
+	impl_name: alloc::borrow::Cow::Borrowed("statemine"),

		fn dispatch_benchmark(
			config: frame_benchmarking::BenchmarkConfig
-		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, sp_runtime::RuntimeString> {
+		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, alloc::string::String> {

SCALE encoding/decoding remains the same as before, but serde encoding in runtime has changed from bytes to string (it was like this in std environment already).

[#6360]: [eth-rpc] proxy /health

make the eth-rpc proxy /health and /health/readiness from the proxied substrate chain see #4802

[#5756]: Improve APIs for Tries in Runtime

This PR introduces a trait ProvingTrie which has all the function you need to use tries in the runtime.
This trait includes the ability to create, query, and prove data in a trie. Another trait ProofToHashes allows developers to express the computational complexity of proof verification using the proof data.

[#6849]: Add aliasers to westend chains

InitiateTransfer, the new instruction introduced in XCMv5, allows preserving the origin after a cross-chain transfer via the usage of the AliasOrigin instruction. The receiving chain needs to be configured to allow such this instruction to have its intended effect and not just throw an error.

In this PR, I add the alias rules specified in the RFC for origin preservation to westend chains so we can test these scenarios in the testnet.

The new scenarios include:

Sending a cross-chain transfer from one system chain to another and doing a Transact on the same message (1 hop)
Sending a reserve asset transfer from one chain to another going through asset hub and doing Transact on the same message (2 hops)

The updated chains are:

Relay: added AliasChildLocation
Collectives: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>
People: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>
Coretime: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>

AssetHub already has AliasChildLocation and doesn't need the other config item.
BridgeHub is not intended to be used by end users so I didn't add any config item.
Only added AliasChildOrigin to the relay since we intend for it to be used less.

[#6305]: Remove `riscv` feature flag

Since #6266 we no longer require a custom toolchain to build the pallet-revive-fixtures. Hence we no longer have to guard the build behind a feature flag.

[#3881]: Introduce a Generic Proving Trie

This PR introduces a Proving Trie object which can be used inside the runtime. This can allow for things like airdrops where a single hash is stored on chain representing the whole airdrop and individuals present a proof of their inclusion in the airdrop.

[#5745]: Implement `try_append` for `StorageNMap`

This PR introduces the try_append api which is available on other storage map types, but missing on StorageNMap.

[#5838]: enable wasm builder diagnostics propagation

substrate-wasm-builder is used as a build dependency by crates that implement FRAME runtimes.
Errors that occur in these crates can not be detected by IDEs that use rust-analyzer as a language server because rust-analyzer needs the errors to be reported as diagnostic message in json format to be able to publish them to language server clients. This PR adds WASM_BUILD_CARGO_ARGS environment variable, which can hold a space separated list of args that will be parsed and passed to the cargo command that it is used for building against wasm target. It can be used for the stated initial case, but it is also flexible enough to allow passing other arguments or formatting the messages using another available type.

[#5555]: Make salt optional

Remove address_len and salt_len from uapi as both are now fixed size

[#5548]: Use H160 when interfacing with contracts

When interfacing with a contract we now use the native ethereum address type and map it to AccountId32 when interfacing with the rest of substrate.

[#5630]: Introduce and Implement the `VestedTransfer` Trait

This PR introduces a new trait VestedTransfer which is implemented by pallet_vesting. With this, other pallets can easily introduce vested transfers into their logic.

[#6314]: Migrate pallet-elections-phragmen benchmark to v2 and improve doc

Part of:

#6202.

[#5556]: Make salt optional

By making salt optional we allow clients to use CREATE1 semantics when deploying a new contract.

[#4982]: Add useful error logs in pallet-xcm

This PR adds error logs to assist in debugging pallet-xcm.
Additionally, it replaces the usage of log with tracing.

[#6027]: Remove pallet::getter from pallet-offences

This PR removes pallet::getter from pallet-offences from type Reports. It also adds a test to verify that retrieval of Reports still works with storage::getter.

[#3970]: Update Treasury to Support Block Number Provider

The goal of this PR is to have the treasury pallet work on a parachain which does not produce blocks on a regular schedule, thus can use the relay chain as a block provider. Because blocks are not produced regularly, we cannot make the assumption that block number increases monotonically, and thus have new logic to handle multiple spend periods passing between blocks. To migrate existing treasury implementations, simply add type BlockNumberProvider = System to have the same behavior as before.

[#6263]: [pallet-revive] Update typeInfo

Update typeinfo impl to make it transparent for subxt

see paritytech/subxt#1845

[#4012]: `impl_runtime_apis!`: replace the use of `Self` with `Runtime`

Currently, if there is a type alias similar to type HeaderFor<T> in the scope, it makes sense to expect that HeaderFor<Runtime> and HeaderFor<Self> are equivalent. However, this is not the case. It currently leads to a compilation error that Self is not in scope, which is confusing. This PR introduces a visitor, similar to CheckTraitDecl in decl_runtime_apis!, ReplaceSelfImpl. It identifies usage of Self as a type argument in impl_runtime_apis! and replaces Self with an explicit Runtime type.

For example, the following example code will be transformed before expansion:

impl apis::Core<Block> for Runtime {
    fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
        let _: HeaderFor<Self> = header.clone();
        RuntimeExecutive::initialize_block(header)
    }
}

Instead, it will be passed to macro as:

impl apis::Core<Block> for Runtime {
    fn initialize_block(header: &HeaderFor<Runtime>) -> ExtrinsicInclusionMode {
        let _: HeaderFor<Runtime> = header.clone();
        RuntimeExecutive::initialize_block(header)
    }
}

[#5682]: Introduces `VerifyExistenceProof` trait

Introduces VerifyExistenceProof trait for verifying proofs in the runtime.
An implementation of the trait for binary and 16 patricia merkle tree is provided.

[#5995]: Use frame umbrella crate in pallet-proxy and pallet-multisig

Extends the FRAME umbrella crate and uses it in pallet-proxy and pallet-multisig.
Migrates benchmarking from v1 to v2 for pallet-proxy and pallet-multisig.
Allows CI to pick the umbrella crate weights template to run benchmarks.

[#6315]: Migrate pallet-election-provider-support-benchmarking benchmark to v2

Migrate pallet-election-provider-support-benchmarking benchmark to v2

[#6088]: [pallet-revive] EXTCODEHASH to match EIP-1052

Update ext_code_hash to match EIP-1052 specs.

[#6296]: Migrate pallet-glutton benchmark to v2

Update pallet-glutton to benchmarks v2.

[#5886]: Bump some dependencies

This bumps ethbloom, ethereum-types, primitive-types and rlp to their latest version.

Fixes: #5870

[#6291]: migrate pallet-remarks to v2 bench syntax

Part of:

#6202

[#5701]: [pallet-revive] uapi: allow create1 equivalent calls

The salt argument should be optional to allow create1 equivalent calls.

[#6264]: pallet-revive: Trade code size for call stack depth

This will reduce the call stack depth in order to raise the allowed code size. Should allow around 100KB of instructions. This is necessary to stay within the memory envelope. More code size is more appropriate for testing right now. We will re-evaluate parameters once we have 64bit support.

[#5675]: [pallet-revive] Add balance_of syscyall for fetching foreign balances

This adds an API method balance_of, corresponding to the BALANCE EVM opcode.

[#6023]: Fix storage in pallet section

Fix compilation of pallet::storage in a pallet section: a local binding definition was not correctly referenced due to macro hygiene.

[#5891]: Add benchmark overhead command to frame-omni-bencher

This adds the benchmark overhead command to the frame-omni-bencher library. This allows para- and relay chain teams to generate extrinsic and block base weights.

[#6418]: Follow up work on TransactionExtension - fix weights and clean up UncheckedExtrinsic

This PR removes the redundant extension version byte from the signed v4 extrinsic, previously unused and defaulted to 0. The extension version byte is also made to be part of the inherited implication handed to extensions in General transactions. Also, some system extensions benchmarks were adjusted through whitelisting to not count the reads for frequently read storage keys.

[#6295]: Migrate pallet-im-online benchmark to v2

Part of:

#6202.

[#5872]: [omni-bencher] Make all runtimes work

Changes:

Add --exclude-pallets to exclude some pallets from runtimes where we dont have genesis presets yet
Make --genesis-builder-policy=none work with --runtime
CI: Run the frame-omni-bencher for all runtimes

[#5876]: (XCMv5) implement RFC#100, add new InitiateTransfer instruction

There's a new XCM instruction in v5: InitiateTransfer.
It's meant as a general instruction that will do everything (and more) currently done by InitiateTeleport, InitiateReserveWithdraw and DepositReserveAsset.
Its main feature is the ability to do cross-chains transfers mixing teleported and reserve transferred assets.

/// Specify which type of asset transfer is required for a particular `(asset, dest)` combination.
pub enum AssetTransferFilter {
	/// teleport assets matching `AssetFilter` to `dest`
	Teleport(AssetFilter),
	/// reserve-transfer assets matching `AssetFilter` to `dest`, using the local chain as reserve
	ReserveDeposit(AssetFilter),
	/// reserve-transfer assets matching `AssetFilter` to `dest`, using `dest` as reserve
	ReserveWithdraw(AssetFilter),
}
/// Cross-chain transfer matching `assets` in the holding register as follows:
///
/// Assets in the holding register are matched using the given list of `AssetTransferFilter`s,
/// they are then transferred based on their specified transfer type:
///
/// - teleport: burn local assets and append a `ReceiveTeleportedAsset` XCM instruction to
///   the XCM program to be sent onward to the `dest` location,
///
/// - reserve deposit: place assets under the ownership of `dest` within this consensus system
///   (i.e. its sovereign account), and append a `ReserveAssetDeposited` XCM instruction
///   to the XCM program to be sent onward to the `dest` location,
///
/// - reserve withdraw: burn local assets and append a `WithdrawAsset` XCM instruction
///   to the XCM program to be sent onward to the `dest` location,
///
/// The onward XCM is then appended a `ClearOrigin` to allow safe execution of any following
/// custom XCM instructions provided in `remote_xcm`.
///
/// The onward XCM also potentially contains a `BuyExecution` instruction based on the presence
/// of the `remote_fees` parameter (see below).
///
/// If a transfer requires going through multiple hops, an XCM program can compose this instruction
/// to be used at every chain along the path, describing that specific leg of the transfer.
///
/// Parameters:
/// - `dest`: The location of the transfer next hop.
/// - `remote_fees`: If set to `Some(asset_xfer_filter)`, the single asset matching
///   `asset_xfer_filter` in the holding register will be transferred first in the remote XCM
///   program, followed by a `BuyExecution(fee)`, then rest of transfers follow.
///   This guarantees `remote_xcm` will successfully pass a `AllowTopLevelPaidExecutionFrom` barrier.
/// - `remote_xcm`: Custom instructions that will be executed on the `dest` chain. Note that
///   these instructions will be executed after a `ClearOrigin` so their origin will be `None`.
///
/// Safety: No concerns.
///
/// Kind: *Command*.
///
InitiateTransfer {
	destination: Location,
	remote_fees: Option<AssetTransferFilter>,
	assets: Vec<AssetTransferFilter>,
	remote_xcm: Xcm<()>,
}

[#6812]: xcm-executor: take delivery fee from transferred assets if necessary

In asset transfers, as a last resort, XCM delivery fees are taken from transferred assets rather than failing the transfer.

[#6288]: [pallet-revive] Add metrics to eth-rpc

Add metrics for eth-rpc

[#6087]: Expose private structs in pallet_nfts and pallet_uniques.

PR changes certain structs in pallet_nfts and pallet_uniques into public. It also changes 2 storages (collection & asset metadata) into public in pallet_uniques.

[#4257]: Rename `state_version` in `RuntimeVersion` to `system_version`.

This PR renames state_version in RuntimeVersion to system_version. system_version=2 signifies that extrinsic root derivation uses StateVersion::V1.

[#6212]: Added Trusted Query API calls for Westend and Rococo chains

Added is_trusted_reserve and is_trusted_teleporter API calls to relay chains.
Given an asset and a location, they return if the chain trusts that location as a reserve or teleporter for that asset respectively.
You can implement them on your runtime by simply calling a helper function on pallet-xcm.

	impl xcm_runtime_apis::trusted_query::TrustedQueryApi<Block> for Runtime {
  fn is_trusted_reserve(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_reserve(asset, location)
  }
  fn is_trusted_teleporter(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_teleporter(asset, location)
  }
}

[#6257]: fix claim queue size when validator groups count is smaller

Fixes a bug introduced in #5461, where the claim queue would contain entries even if the validator groups storage is empty (which happens during the first session). This PR sets the claim queue core count to be the minimum between the num_cores param and the number of validator groups.

[#6473]: add `TransactionSource` to `TransactionExtension::validate`

Add a the source of the extrinsic as an argument in TransactionExtension::validate.
The transaction source can be useful for transactions that should only be valid if it comes from the node. For example from offchain worker.
To update the current code. The transaction source can simply be ignored: _source: TransactionSource

[#5684]: [pallet-revive]

Update xcm runtime api, and fix pallet-revive xcm tests

[#6192]: [pallet-revive] fix hardcoded gas in tests

Fix hardcoded gas limits in tests

[#6885]: Omni-node: Detect pending code in storage and send go ahead signal in dev-mode.

When using the polkadot-omni-node with manual seal (--dev-block-time), it is now possible to perform runtime upgrades. The node will detect the pending validation code and send a go-ahead signal to the parachain.

[#5201]: Snowbridge free consensus updates

Allow free consensus updates to the Snowbridge Ethereum client if the headers are more than a certain number of headers apart. Relayers providing valid consensus updates are refunded for updates. Bridge users are not affected.

[#6174]: [pallet-revive] fix fixture build path

Fix fixture build path

Changelog for `Node Operator`

ℹ️ These changes are relevant to: Those who don't write any code and only run code.

[#5741]: make RPC endpoint `chainHead_v1_storage` faster

The RPC endpoint chainHead_v1_storage now relies solely on backpressure to determine how quickly to serve back values instead of handing back a fixed number of entries and then expecting the client to ask for more. This should improve the throughput for bigger storage queries significantly.

Benchmarks using subxt on localhost:

Iterate over 10 accounts on westend-dev -> ~2-3x faster
Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
Fetch 1024 descendant values -> ~500x faster

[#4639]: Added the fork-aware transaction pool implementation

New command line option was added, allowing to select implementation of transaction pool:
- --pool-type=fork-aware - new fork aware transaction pool,
- --pool-type=single-state - old transaction pool implementation which is still default,

[#4889]: Add CLI options for parachain chain specifications + fix bug for swallowing custom fields

Parachain ID and relay chain can be specified via the CLI arguments for when creating a chain spec.
A bug that also swallowed custom fields outside of the default config has also been fixed.

[#6454]: rpc server: fix ipv6 host filter for localhost

This PR fixes that ipv6 connections to localhost was faulty rejected by the host filter because only [::1] was allowed

[#6058]: backpressure `chainhead_v1_follow`

The RPC endpoint chainHead_v1_follow now relies on backpressure to determine whether or not the subscription should be closed instead of continuing to send more events to a consumer which can't keep up.
This should significantly improve memory consumption as substrate will be keeping less messages in memory.

[#5606]: Fix PVF precompilation for Kusama

Tweaks the PVF precompilation on Kusama to allow prepare PVFs when the node is an authority but not a validator.

Changelog for `Runtime User`

ℹ️ These changes are relevant to: Anyone using the runtime. This can be a token holder or a dev writing a front end for a chain.

[#6337]: Don't expose metadata for Runtime APIs that haven't been implemented

Prior to this PR, the metadata for runtime APIs would contain all methods for the latest version of each API, regardless of which version a runtime implements. This PR fixes that, so that the runtime API metadata reflects what is actually implemented.

[#6039]: Added Trusted Query API calls.

There's a new runtime API to check if a chain trust a Location as a reserve or teleporter for a given Asset.
It's implemented in all the relays and system parachains in Westend and Rococo.

[#6080]: Assets in pool with native can be used in query_weight_to_asset_fee in Asset Hubs

query_weight_to_asset_fee now works with assets in a pool with the native asset in both Westend and Rococo asset hubs.
This means all the information you get from query_acceptable_payment_assets can be used directly in query_weight_to_asset_fee to get the correct fees that need to be paid.

[#6373]: `chain-spec-builder`: info about patch/full files added

There was no good example of what is patch and full genesis config file. Some explanation and example were added to the chain-spec-builder doc.

[#6022]: [Coretime chain] Add high assignment count mitigation to testnets

We can handle a maximum of 28 assignments inside one XCM, while it's possible to have 80 (if a region is interlaced 79 times). This can be chunked on the coretime chain side but currently the relay does not support this. This PR truncates the additional assignments on Rococo and Westend to mitigate this until the relay is fixed. The first 27 assignments are taken, the final 28th is used to pad with idle to complete the mask. Any other assignments are dropped.

[#5971]: XCMv5 InitiateTransfer can preserve original origin across chains.

The new InitiateTransfer instruction can preserve the original origin across chains by setting preserve_origin: true in the instruction itself.
When it's set to true, it will append after the inner XCM, an AliasOrigin instruction instead of the usual ClearOrigin.
This instruction will try to alias to the original origin, thus preserving it.
This only works if the chain receiving the transfer supports the aliasing operation.
If not, preserve_origin: false works as before and will never fail because of this.

[#5687]: Westend/Rococo Asset Hub: auto incremented asset id for trust backed assets

Setup auto incremented asset id to 50_000_000 for trust backed assets.

Migration

This change does not break the API but introduces a new constraint. It implements an auto-incremented ID strategy for Trust-Backed Assets (50 pallet instance indexes on both networks), starting at ID 50,000,000. Each new asset must be created with an ID that is one
greater than the last asset created. The next ID can be fetched from the NextAssetId storage item of the assets pallet. An empty NextAssetId storage item indicates no constraint on the next asset ID and can serve as a feature flag for this release.

[#5765]: Added foreign locations to local accounts converter to all the parachains.

Now any user account can have a sovereign account on another chain controlled by the original account.

[#5999]: Westend: Constant yearly emission

Integrating the new inflation approach from polkadot-fellows/runtimes#471 into Westend first to check that it is working.

[#5435]: Support registering assets on Asset Hubs over bridge

Allows one Asset Hub on one side, to register assets on the other Asset Hub over the bridge.
Rococo <> Ethereum test bridge will be dropped in favor of Westend <> Ethereum test bridge.
This PR also changes emulated tests to simulate double bridging from Ethereum<>Westend<>Rococo.

[#5984]: Add page information to staking::PayoutStarted event

Adds page index that is claimed, and optional next page that can be claimed. If next is none, then the page is the last one.

[#4257]: Rename `state_version` in `RuntimeVersion` to `system_version`.

RuntimeVersion's state_version is renamed to system_version. Applications using that type and its field must update their code to reflect the changes. For easier migration serde serialization produces both new systemVersion and old stateVersion fields and deserialization supports stateVersion as an alias as too.

[#6212]: Added Trusted Query API calls for Westend and Rococo chains

There's a new runtime API to check if a chain trust a Location as a reserve or teleporter for a given Asset.
It's implemented in all the relays and system parachains in Westend and Rococo.

[#6357]: New runtime api that returns the associated pool accounts with a nomination pool.

Each nomination pool has two associated pot accounts: the bonded account, where funds are pooled for staking, and the reward account. This update introduces a runtime api that clients can query to retrieve these accounts.

Rust compiler versions

This release was built and tested against the following versions of rustc.
Other versions may work.

Rust Stable: "1.81.0"

Runtimes

The information about the runtimes included in this release can be found below.
The runtimes have been built using srtool v0.17.0 and rustc 1.81.0 (eeb90cda1 2024-09-04).

Westend

🏋️ Runtime size:             2.037 MB (2,135,690 bytes)
🗜 Compressed:               Yes, 73.61%
✨ Reserved meta:            OK - [6D, 65, 74, 61]
🎁 Metadata version:         V14
🔥 Core version:             westend-1017001 (parity-westend-0.tx27.au2)
🗳️ system.setCode hash:      0x66982e5b4340ac74a19a5a51ae7774ac05bcd5ef502f6b19a5f18d28afcce211
🗳️ authorizeUpgrade hash:    0xfcb5c327345609ecaa45bd679959f6d15a3c621c5ba6822af31c373c7ab5140c
🗳️ Blake2-256 hash:          0xfd9e364b5ee6fc709af5a746557ae9e5f8d5357c7cfcc50058f3b3bbd40303d5
📦 IPFS:                     QmY1mCdSL2CFFGALgPGbkzQxz6qH7GSqVJp31ye1EZc4RQ

Westend Assethub

🏋️ Runtime Size:          1.60 MB (1674969 bytes)
🔥 Core Version:          westmint-1017003 (westmint-0.tx16.au1)
🗜 Compressed:            Yes: 76.65%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0xd522fe21a324cf368516f02ba9f9d95c90881a0c44559a8ddb44cc13c7a05732
🗳️ authorizeUpgrade hash: 0x2150f7b02db7513f50305a16bc2a6723201f89c9c4d99967979a6da7b9848e1a
🗳️ Blake2-256 hash:       0x29bac726f658b40db916045e89ba2a5c6f404cfb266d4a24539740290bcfcb79
📦 IPFS:                  QmXkckKSkkxxdzdPemTngGNNThfhUCCXU7rZJhyupqyDws

Westend Bridgehub

🏋️ Runtime Size:          1.38 MB (1450346 bytes)
🔥 Core Version:          bridge-hub-westend-1017001 (bridge-hub-westend-0.tx6.au1)
🗜 Compressed:            Yes: 73.72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x52715185f7f28290fd05172e5b0f016ea0dac4808265323e6a23ba6368d80b0c
🗳️ authorizeUpgrade hash: 0xba8101845222ce0715fd4c063c9972e85b173c48c317a025bc1b969e1cbfe03d
🗳️ Blake2-256 hash:       0xcdc83f4c85dad25c3dfbfbce1ea57068b68105d4f95b29a677287bc527183aaf
📦 IPFS:                  QmfJn3gHYWdKsdnRUtZTB3MdFgSN7ihH1wQS4duNDStWVF

Westend Collectives

🏋️ Runtime Size:          1.21 MB (1267199 bytes)
🔥 Core Version:          collectives-westend-1017001 (collectives-westend-0.tx6.au1)
🗜 Compressed:            Yes: 75.58%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0xb064f17227559bf5fb5010c8a8ade578458da1ff1a228d66cf315fc10108b7bf
🗳️ authorizeUpgrade hash: 0x4bd35a76c3e53e93025ed4a3269deff9e01a60b1936414445f78b93f2f90183d
🗳️ Blake2-256 hash:       0xb96fee4c33e0f9a5bcb10700a971b542abe94a9f9e9e4a7e25ba5b8442c51313
📦 IPFS:                  QmVeSPx4KDMkt1yxfLKXozYaNDw3Qi5zZnJ5NmW6Dkh95w

Westend Coretime

🏋️ Runtime Size:          1.03 MB (1080704 bytes)
🔥 Core Version:          coretime-westend-1017001 (coretime-westend-0.tx2.au1)
🗜 Compressed:            Yes: 74.72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x9fad76f381887a24ef60bc7097c7ae109a5f31c3e5509132d274c40e61304e93
🗳️ authorizeUpgrade hash: 0xc5e10106007862b04a990da6f94288c46e9ccbeaa73e118365b93e0f35320df9
🗳️ Blake2-256 hash:       0x76973fbaf5b3c5da03b2f517b7385dafd51596f3d351e243bbbf5bf06443fdeb
📦 IPFS:                  QmT97ek69zN6nBBNUi1YD7o37L3RTEe8q1Bf3MipJaiKU6

Westend Glutton

🏋️ Runtime Size:          586.78 kB (600867 bytes)
🔥 Core Version:          glutton-westend-1017001 (glutton-westend-0.tx1.au1)
🗜 Compressed:            Yes: 72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x9a7fc73bd2e6858eeead2f237e5bde6f3e75e942fa1465c54d7dbf1204e7145b
🗳️ authorizeUpgrade hash: 0x3a974c855ecdd0068e0a3c7b0e3f80a7b1488d7051af67b2c6c76334557a812d
🗳️ Blake2-256 hash:       0x60849ff400d4d0a7e8dd47fff1350b2799e97abb3d64957f0e8f5a838ca5a457
📦 IPFS:                  QmeAst2t3foRqmuwX5GGMkxErRet56eeczvK3bvkgdYPdy

Westend People

🏋️ Runtime Size:          1.03 MB (1077104 bytes)
🔥 Core Version:          people-westend-1017001 (people-westend-0.tx2.au1)
🗜 Compressed:            Yes: 74.78%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x851eac6520032e3d957ecdff6b79989045117a0c59684243e4127a97ba203e3a
🗳️ authorizeUpgrade hash: 0x8bed1b714ce09aaa06f9948f0f893d5cccc97d838d8420fc3c0dc657c3242fa3
🗳️ Blake2-256 hash:       0x0352e12aece34ea64134a4cf8caa165961c8600aee2d5e69f2e511467a1e8009
📦 IPFS:                  QmSvKyCUCBsgVtXKXQWcsZfvqnenXCYBcFwPLZ91hPc8xf

Docker images

The docker images for the polkadot node binary and the polkadot-parachain binary can be found at Docker hub (will be available a few minutes after the release has been published):

You may also pull it with:

docker pull parity/polkadot:stable2412

docker pull parity/polkadot-parachain:stable2412

Polkadot stable2412

Changelog

Changelog for Node Dev

[#6729]: Fix order of resending messages after restart

[#6864]: Fix approval-voting canonicalize off by one

[#5038]: Plumb RPC listener up to caller

[#6863]: Update merkleized-metadata to 0.2.0

[#5521]: Allow to call arbitrary runtime apis using RelayChainInterface

[#6156]: Use bool::then instead of then_some with function calls

[#5911]: Removed the possibility to start a shell parachain node

[#5997]: Implement archive_unstable_storageDiff method

[#4837]: Add PVF execution priority

[#5343]: Allow to disable gap creation during block import

[#6676]: Expose the unstable metadata v16

[#5716]: Replace lazy_static with LazyLock

[#4639]: Added the fork-aware transaction pool implementation

[#5686]: sync: Remove checking of the extrinsics root

[#6382]: gensis-config: patching default RuntimeGenesisConfig fixed

[#5469]: Syncing strategy refactoring

[#5774]: Avoid unnecessary state reset of allowed_requests when no block requests are sent

[#6588]: rpc server: fix subscription id_provider being reset to default one

[#5666]: Make syncing strategy an argument of the syncing engine

[#5883]: statement-distribution RFC103 implementation

[#5924]: Bump PoV request timeout

[#4846]: Make approval-voting runnable on a worker thread

[#4849]: Introduce approval-voting-parallel subsystem

[#5787]: Move bitfield_distribution to blocking task pool and set capacity to 8192

[#5847]: candidate-validation: RFC103 implementation

[#5919]: substrate-offchain: upgrade hyper to v1

[#6218]: Enable approval-voting-parallel by default on kusama

[#6104]: LocalTransactionPool implemented for fork aware transaction pool

[#5707]: Remove ValidateFromChainState

[#6696]: Make approval-distribution aggression a bit more robust and less spammy

[#5768]: export NodeHealthProxyLayer

[#6677]: chore: Update litep2p to v0.8.2

[#5635]: Fix edge case where state sync is not triggered

[#5915]: Omni-Node renamings

[#5880]: Fix prospective parachains test to use shuffled candidate list

[#5908]: collation-generation: use v2 receipts

[#6268]: Bump a timeout in zombienet coretime smoke test

[#6652]: rpc server: re-use server builder per rpc interface

[#6742]: Update litep2p backend to v0.8.3

[#5830]: Remove jaeger from approval-voting and approval-distribution

[#5737]: Make syncing service an argument of build_network

[#5762]: Fast return for invalid request of node health

[#5601]: Introduce RpcParams in sc-cli

[#6011]: collator protocol: validate descriptor version on the validator side

[#5789]: Prevents EthereumBlobExporter from consuming parameters when returning NotApplicable

[#5540]: Avoid unnecessary block gap updates

[#4974]: Remove libp2p dependency from sc-network-sync

[#5811]: Improve import_notification_stream documentation

[#5732]: Expose the unstable metadata v16

[#6690]: Fix Possible bug, Vote import failed after aggression is enabled

[#5592]: Introduce BlockGap

[#5856]: Extend state tracking of chainHead to capture notification gaps

[#6015]: Rename QueueEvent::StartWork

Changelog for Runtime Dev

[#6318]: Refactor pallet claims

[#5892]: Treasury: add migration to clean up unapproved deprecated proposals

[#6299]: migrate pallet-recovery to benchmark V2 syntax

[#6073]: Refactor pallet-grandpa benchmarks to v2

[#6039]: Added Trusted Query API calls.

[#5939]: [pallet-revive] Bump PolkaVM and add static code validation

[#5866]: [pallet-revive] Ethereum JSON-RPC integration

[#6778]: Added fallback_max_weight to Transact for sending messages to V4 chains

[#6316]: Migrate pallet-election-provider-multi-phase benchmark to v2 and improve doc

[#5779]: [pallet-revive] last call return data API

[#6171]: remove parachains_assigner

[#6205]: pallet-message-queue: Fix max message size calculation

[#5845]: Fix compilation after renaming some of benchmarks in pallet_revive.

[#5861]: [pallet-revive] immutable data storage

[#6129]: Improved TrustedQueryAPI signatures.

[#6169]: [Deprecation] deprecate treasury spend_local call and related items

Migration

For users who were using only spend_local before

For users who were already using spend with all other assets, except the native asset

[#5526]: Fix enact_candidate weight generation

[#6496]: Backport #6418 to stable2412

[#6255]: [pallet-child-bounties] Index child bounties by parent bounty

Migration for Runtime Pallet Instance

Changelog for `Node Dev`

[#5716]: Replace `lazy_static` with `LazyLock`

[#6382]: gensis-config: patching default `RuntimeGenesisConfig` fixed

[#5847]: `candidate-validation`: RFC103 implementation

[#5737]: Make syncing service an argument of `build_network`

[#5601]: Introduce `RpcParams` in sc-cli

[#5811]: Improve `import_notification_stream` documentation

[#5592]: Introduce `BlockGap`

Changelog for `Runtime Dev`

[#6073]: Refactor `pallet-grandpa` benchmarks to `v2`

[#6169]: [Deprecation] deprecate treasury `spend_local` call and related items

For users who were using only `spend_local` before

For users who were already using `spend` with all other assets, except the native asset

[#6496]: Backport #6418 to `stable2412`

[#6645]: xcm: fix local/remote exports when inner routers return `NotApplicable`

[#5804]: Refactor get_account_id_from_seed / get_from_seed to one common place `get_account_id_from_seed / get_from_seed` were copied all over the place. This PR removes unnecessary code duplication.

[#6141]: Improve `CheckMetadataHash` transaction extension weight and logic

[#6025]: Refactor staking pallet benchmarks to `v2`

[#6384]: Relax requirements on `assign_core`.

[#6032]: Fix `feeless_if` in pallet section

[#5941]: `SolochainDefaultConfig`: Use correct `AccountData`

[#6365]: pallet-revive: Use `RUSTUP_TOOLCHAIN` if set

[#4251]: MBM `try-runtime` support

[#5693]: Remove `sp_runtime::RuntimeString` and replace with `Cow<'static, str>` or `String` depending on use case

[#6305]: Remove `riscv` feature flag

[#5745]: Implement `try_append` for `StorageNMap`

[#5630]: Introduce and Implement the `VestedTransfer` Trait

[#4012]: `impl_runtime_apis!`: replace the use of `Self` with `Runtime`

[#5682]: Introduces `VerifyExistenceProof` trait