Gas Profiling

[Fiddlekins]

Component

Forge

Describe the feature you would like

Hey everyone, first off thank you for the work on foundry so far, I've been enjoying working with it.

I'd like to ask about gas metering though - what dev workflows do people use/recommend for profiling functions at the moment, and what's the current general plans for future work on this aspect.

Here's the methods I'm presently aware of:

1. There's forge snapshot which gives you gas costs for test functions. This suffers from not giving per function breakdown, and whilst you can structure your tests to minimise the code running that isn't the function you're trying to profile that's pretty messy.

2. There's forge test --gas-report which summarises gas costs for function calls as well. At a glance this looks just like what I'm after, but re-running it a few times reveals that the min,avg,max values can differ by significant margins from run to run. It seems that it only tallies the gas cost for a function once per test function, even if the test is fuzzed? In doing so it would appear that it's perhaps inadvertently often using threshold fuzzed input that makes the function behave more drastically different, perhaps. I think I read in passing that only profiling a single run from a fuzzed test was a deliberate decision to keep execution times short, but I think it would benefit from a cli flag that enabled the slower but more representative profiling every run.

3. Then there's the vm cheatcodes to pause and resume gas consumption. In theory you can use these to ignore the gas cost for the code that sets up test scenarios, only metering the gas consumed by the function call that the test is testing. Unfortunately, judging by several other issues (GasMetering generates different results compared to gasleft() diff #4523, Gas meter shows huge number when using vm.pauseGasMetering() along with delete X #4370, etc) this doesn't always work as expected.

4. There's also the gasleft() approach where end users can manually keep track of gas consumed between two arbitrary points in a test function. This is perhaps the most reliable approach I know of right now, but even that isn't flawless unfortunately. Warm/cold access, memory expansion, and probably some other EVM details mean that even if you're ignoring the gas cost of the test setup code, it's still impacting the execution cost of the function you're trying to profile.

In order to address the shortcomings with approach 4 it seems to me that it requires fairly extensive work: some sort of cheatcode that can be used to tell forge to run the subsequent code as if it were a new transaction, giving it fresh state to work with. I'm not remotely familiar with forge internal working so no idea how practical attempting that even is, alas.

So to recap:

• what approach do people find works best currently?
• what are the general plans for how forge should do gas profiling in the future?
• is forge ever likely to support more accurate function profiling, or would it be recommended to use other tools (eg. hardhat) if that extra accuracy is important?

Additional context

No response

[nzhl]

Any progress on this

[cloud1992]

Any Answer for this... it is important for developers...

[zerosnacks]

We've since added the option to run tests in isolation: https://book.getfoundry.sh/reference/cli/forge/test?highlight=isolate#forge-test, this greatly improves the accuracy of tracking gas.

--gas-report now also uses this flag.

Other improvements like those developed in https://github.com/marktoda/forge-gas-snapshot/ can potentially be upstreamed

We also support the vm.lastCallGas cheatcode: https://github.com/foundry-rs/forge-std/blob/8948d45d3d9022c508b83eb5d26fd3a7a93f2f32/src/Vm.sol#L453-L454 to be able to get gas information about calls directly inside of your tests.

Converting this ticket into a discussion as there are no direct actionable items.

[zerosnacks]

We've recently added new cheatcodes to capture gas usage over arbitrary sections: #8952. This also includes internal gas tracking and it most similar to gasleft().

You can read more how to use them here: https://book.getfoundry.sh/forge/gas-section-snapshots

To determine the trade-offs between the different ways of capturing snapshots see: https://book.getfoundry.sh/forge/gas-tracking