--rerun-fails sometimes exits with an exit code of 0, when there are failures even in the reruns
#423
Labels
bug
Something isn't working
Comments
|
I've checked all the way back to 0.6.0 and the problem exists in all of them |
|
Thank you for the bug report! The exit code from Could you share the output of one of these runs with Also, which |
ImJeremyHe
added a commit
to EspressoSystems/nitro-espresso-integration
that referenced
this issue
Nov 7, 2024
This flag in gotestsum causes some bugs which multiple the test time. gotestyourself/gotestsum#417 gotestyourself/gotestsum#423
|
We've experienced the same issue. You can find an example here: And the workaround does not seem to work: (look at "Run functional test") Sufficient to say, this has eroded the trust in gotestsum in our team :-/ |
|
Could you share the output with The Most likely the bug is that the json output is not correctly including all the failed tests. |
|
I'm seeing the exit code sometimes correct, and sometimes wrong: correct non-zero exit codehttps://github.com/temporalio/temporal/actions/runs/11826484802/job/32952697612?pr=6809 at the bottom prints 
incorrect zero exit codehttps://github.com/temporalio/temporal/actions/runs/11826484802/job/32952697830?pr=6809 at the bottom prints 
I also ran it with https://github.com/temporalio/temporal/actions/runs/11825559427/job/32951811664?pr=6809 |
stephanos
added a commit
to temporalio/temporal
that referenced
this issue
Nov 14, 2024
## What changed? Disable `gotestsum` re-runs. ## Why? `gotestsum` has a bug that reports test failures with exit code 0, making the GHA job succeed. See gotestyourself/gotestsum#423 ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> Test failure results in GHA job failure. <img width="611" alt="Screenshot 2024-11-13 at 11 19 27 AM" src="https://github.com/user-attachments/assets/e7c8bd00-845c-4001-9f3c-c960d15097c4"> ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
bergundy
added a commit
to temporalio/temporal
that referenced
this issue
Nov 20, 2024
## What changed? Added a test-runner command for rerunning tests and modified the Makefile to use the new command. ## Why? gotestsum is supposed to do this but it is buggy: gotestyourself/gotestsum#423. If this runner works well we can stop using gotestsum if we choose but I would probably recommend using it since it claims to do exactly what we need. An alternative would be to either generate test reports using JSON output or using https://github.com/jstemmer/go-junit-report to create junit XML reports. ## How did you test it? - Ran functional tests multiple times locally - Added unit tests to the runner
Sneh1999
added a commit
to EspressoSystems/nitro-espresso-integration
that referenced
this issue
Nov 20, 2024
* add some mock merkle logic * use mock data * more mock validation code * Increase the timeout limit for the e2e validation check * Add SwitchSequencer * Fix the switch function * Create SwitchSequencer * Update the espresso-sequencer-go * Use light client as the trigger * Add the switch test * Make the linter happy * Add the switch test to the e2e ci * Increase the limit * Clean the log with the log helper * more mock code * Fix the e2e test * serialize proof variables correctly * update espresso-go version * wasm compatibility * remove stale file, lint fix * update to recent espresso go version * refactor block merkle verification * work on E2E * progress on merkle tree verification * add test vectors, fetch proofs at the correct indicies * cleanup, remove outfiles, fetch merkle proof at the correct justified height * add logic to compare circuit field * check that circuit representation equals u256 representation in the test * send byte slice to wasm instead of byte array which lives on the stack * cleanup, remove debug statements, add documentation * remove error logs * fix build * lint fix * progress on e2e tests, comment out vid stuff for now because of wasm io issues * update espresso-sequencer-go tag, lint ci fix * cleanup, add issues for unresolved problems * remove block that spams transactions * comment out unused txns * add test for advz equivalence and use a smaller srs * Allow dead code in validation * Specify foundry version * Add random_get stub in wasi_snapshot_preview1 * Add stubs required in wasi_snapshot_preview1 * Fix `make lint` * Enable prover service to talk to L1 on host * Add builder timeout This allows the sequencer to make progress past block 1. These env vars have been changed in espresso-sequencer. * Add missing env var, more logging in e2e test * Revert "Fix `make lint`" This reverts commit 8be940c. * Fix go linting * Parse espresso messages * Add timeout limit * Fix the e2e test * Add timeout limit * Run the e2e test and switch test in 1 thread * Update the e2e workflow * Increase time limit * Specify the foundryup version in Dockerfile * Add sock_accept stub * Increase timeout limit * Build successfully * move ark-srs to dev dependencies * remove debug statements * remove out.txt * Fix the machine test * Fix CI * Fix switch test * Fix CI * Increase CI time limie * update lockfile * remove commented out replaces * newline * Use the latest foundry * RESET ME: debug * RESET ME: not run the CI in a container * RESET ME: remove the parallel * Update script to run tests locally - Run the same commands as on CI. - Add ability to run `cargo +nightly` to dev shell. - make clean: remove brotli build files. * set -parallel=2 * Fix e2e test * Skip some tests in CI * Skip tests in stylus * Remove parallelism restriction * Impl escape hatch on validator side * Fix tests * Generate the osp of hotshot liveness * Fix the fraud proof (#124) * Update contracts submodule * Upgrade go client * Fix the staker challenge tests --------- Co-authored-by: sveitser <[email protected]> * Use espresso dev node * attempt to fix dockerfile * newline * WIP * WIP * download latest espresso module root * incorporate vid common data * Fix the validation pipeline * update go modules, e2e build step * add cbrotli wasm * ci fix attempt * dont build all test deps for e2e test * lint fixes * Use the self-hosted runner and install cbindgen in CI * Add time limit * Remove parallel in building * Fix the switch test * Decrease the waiting hotshot blocks * Use the ubuntu * Specify the foundry version * Enable -j * update foundry version * module root update * Fix the staker test * increase validation timeout for switch test * pin espresso-sequencer-go to new release v0.0.20 * Commit go.sum from running go mod tidy * sequencer info logs * Add escape hatch test in e2e test * Check the reactivate hotshot * Update validator to handle preconditions and panics before entering the espresso STF logic. * Improve readability of conditions in `handleEspressoPreConditions` and update the functions comments. * Further improve readability * Fix lint issue * Update the nitro testnode * Fix tests * Update contracts * Fix CI * Changes to `handleEspressoPreConditions` in `espresso_validation.go` to fix incorrect logic, and give more granular control to where and how the generated panic is handled in the validators STF via returned closures. * return nil when we need a no-op panic handler * Fix lint errors * Update contracts * Fix the validation * Remove the switch test * Fix CI * Remove postgres in espresso e2e test * Codeowners: add Alysia, Zach, Sneh (#154) For #146 * Remove Dockerfile.espresso (#156) * Remove Dockerfile.espresso * address review comments * add new line * Release 20240723 with 20240723-consensus This should include all our changes for nitro v3.1. - Check that the module root matches when downloading releases from github. * Check all "not found" errors Currently only the leveldb not found errors are handled. If a pebble not found error is returned by the backend it is propagated, causing the batchposter to fail to send transactions. With this change all DB not found errors are handled the same way. * Submit L2 Message to espresso network (#173) * Submit L2 Message to espresso network * add context timeout * check for hotshot liveness and add a polling interval to see if transactions were included in an espresso block * cleanup * lint * Draft the sovereign sequencer test * Add espresso transaction queue * Draft the espresso submission * Set a new l2 message type for the sovereign sequencer transactions * Attempt to write messages again * Fix typos and add some comments * Fix typo * add header to jst and submit transaction to espresso * Update go client for development * Confirm the transaction inclusion in hotshot * Fix the test * Unblock the building blocks * Update go client * Fix the sovereign test * add logs * cleanup * lint * Updage go client * not to fetch merkle proof if hotshot height is 0 * Cleanup * Fix CI * Add CI skip tests * remove unwanted variable * skipping some tests to make ci pass * skipping some tests to make ci pass --------- Co-authored-by: ImJeremyHe <[email protected]> * Persistent storage for sovereign builder (#180) * Persistence storage for sovereign builder * add unit test * add unit test to ci and add mutex * Optional header in jst * initialize pending transactions if not present * remove unwanted test * resolve bugs * remove nil pointer panic * cleanup * revert to strings.contains * fix isErrNotFound --------- Co-authored-by: ImJeremyHe <[email protected]> * Save digest artifacts into own directory per run (#182) * Save digest artifacts into own directory per run Currently it seems that we have too many digests in the folder and I think this breaks the assembly of the multiarch docker image. With this change only digests for the same run are saved in the same directory. * Fail docker image merge on wrong number of digests * Clean digests dir * Add dependabot rule to update nitro-testnode (#189) * Fix build (#191) * Add missing dependency * Use renamed variable, fix typo * dependabot: update contracts submodule (#193) * Pin the golint version in CI * Fix pebble panic (#192) * Fix pebble panic * lint * Update some outdated help on flags (#203) * Improve help texts for flags * Update e2e tests to use Sovereign sequencer (#204) * Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]> * Remove the global state for HotShot height (#210) * Remove the global state * Update the submodule * CI: use buildjet instead of self-hosted runner (#212) * CI: use buildjet instead of self-hosted runner * CI: use public runner / buildjet for docker build This removes usage of our self-hosted github runners. * CI: more disk space on public runner * Check disk space at end of runs * Try running CI tests from different directory This may avoid failures due to IPC paths being too long * Make workspace dir a symlink to /tmp/gha-work * Fix redis URL * Move redis tests to own workflow * CI: fix rust and go cache (#217) * rust-cache: add more target dirs * Don't set custom TMPDIR This seems to be the culprit for the extremely long path names that are too long for IPC paths. * modify arbosstate.go to allow for upgrade to arbOS version 35 (#216) * modify arbosstate.go to allow for upgrade to arbOS version 35 * Fix lint * Modify the chain config during the ArbOS upgrade --------- Co-authored-by: sveitser <[email protected]> * CI: make more space (#218) * Store transaction pos in batch poster (#219) * Move submit transaction pos from tx_streamer to batch poster * Remove espresso sequencer (#214) * Remove espresso sequencer * lint * remove enable espresso * commit go-ethereum * fix solgen/gen.go * fix build * revert .gitmodules * revert go.mod * add enable espresso to validation check * re-add enable espresso to tests --------- Co-authored-by: Mathis <[email protected]> * Update espresso-sequencer-go with new LC ABI (#230) * Update espresso-sequencer-go with new LC ABI Instead of using the block height from the first snapshot after the height in the justification it uses the latest height in the LC contract. Currently not 100% convinced about correctness but also believe it should work because any newer tree still contains the header of interest. * Undo LC ABI changes We're not comfortable with the previous change because making a proof w.r.t to the newest finalized state may cause issues if the batch poster is lagging behind. This is currently not easy to test so we decided to revert back to the old flow. * Use tag of espresso-sequencer-go * Stop setting the chain config during ArbOS upgrade in favor of setting it manually during migration process. (#234) * Use the tag before marketplace update (#238) * Sign payload for sovereign builder (#181) * Sign payload for sovereign builder * save signed payload to db * change parsing logic to include signature * add some comments * skip tests * address comments * add signature payload test * test working * fix logging and comments * move placement of log * fix ci * fix ci * only submit transaction if jst nil * add log to debug ci * use the latest instead of main for espresso-dev-node * pin to legacy lc version * pin to before marketplace version * remove log * Full node with espresso finality (#237) * wip * wip * Add espresso finality node test * fix ci * fix ci * pin to before marketplace version * fix tests * add comments * address comments * address comments * address comments * cleanup * fix test * fmt * rerun ci * ci skip tests * Update to sequencer.go to remove `EnableEspressoSovereign` flag, and instead sequence based on hotshot liveness at l1 block height. * Update node.go to remove reference to `EnableEspressoSovereign` * update Espresso tests to reflect removed config variables, and add escape hatch test to e2e test. * Add wait for hotshot liveness in sovereign sequencer test to ensure sovereign sequencer is enabled. * go fmt * Add comment to explain changes needed in replay binary when panics are no longer an issue. * Fix lint issues * Modify construction of lightClientReader in sequencer.go to allow for nil `l1Reader` * Update test configs to align with changes. * Update comments according to feedback. * Bump nitro-testnode from `03095e0` to `884a63c` (#240) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `03095e0` to `884a63c`. - [Commits](EspressoSystems/nitro-testnode@03095e0...884a63c) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mathis <[email protected]> * Revert removal of `EnableEspressoSovereign` * Add additional check to ensure that if EnableEspressoSovereign is on we can't construct a sequencer with a nil lightClientReader * Add `EnableEspressoSovereign` config flag to test configs. * Bump contracts from `d226e9d` to `2ae1bbd` (#244) Bumps [contracts](https://github.com/EspressoSystems/nitro-contracts) from `d226e9d` to `2ae1bbd`. - [Release notes](https://github.com/EspressoSystems/nitro-contracts/releases) - [Commits](EspressoSystems/nitro-contracts@d226e9d...2ae1bbd) --- updated-dependencies: - dependency-name: contracts dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pin rust to 1.81.0 See #257 * Set cargo config for espresso_crypto wasm build This was recently added for the other wasm compilation steps. * Pin rust in E2E test * CI: remove deprecated rust toolchain action * Fix if statement * Regenerate data-poster external-signer test certs (cherry picked from commit 8f961b0) * Use ArbOS chain config as the cannonial chain config. (#252) * Add checks to sequencer.go and transaction_streamer.go to utilize the chain config switch * Update interface.go to export interface for `GetArbOSConfigAtHeight()` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionEngine` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionNode` * Update `sequencer.go` and `transaction_streamer.go` to read from the `ArbOS` `chainConfig` as the cannonical `chainConfig` rather than the `chainConfig` held by the `Blockchain` struct. * Add ArbOS chain config check in batch poster to stop adding espresso messages to the transaction streamers DB if EnableEspresso is false. * Add missing error check in sequencer.go and nil check on arbos config in sequencer.go and batch_poster.go. * Remove ineffectual assignment in sequencer.go * Add test for new GetArbOSConfigAtHeight() method in ExecutionNode This commit adds an integration test for the GetArbOSConfigAtHeight() method on the Execution node that grabs the initial config, and validates it, then updates the config and ensures that the new method will fetch the updated config. * restore nitro-testnode to proper commit * Fix trie node node found error during recording I'm not exactly clear why we need to do this or if it's indeed the right approach to solving the issue however it does seem to make it possible to validate again. Without it we get errors like INFO [10-30|12:24:29.127] DelayedSequencer: Sequenced msgnum=1 startpos=1 INFO [10-30|12:24:30.143] Reading message result remotely. count=3 WARN [10-30|12:24:32.143] prepareblocks failed to get state for block hdrNum=0 err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" ERROR[10-30|12:24:32.143] Error while recording err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" status=2 * Fix the OSP test - Enable 4844 blobs to avoid contract deployment error - Fix address of OSP entry * Disable batch poster in Espresso finality node We don't need this type of node to have a batch poster. Enabling it now causes the test to fail because there's a check to ensure that redis is used for the batch poster on the second node. * Re-allow setting L1 stack config for tests After the sync the code would always use the default values. Don't think this is very clean but I couldn't think of something better yet. * CI: increase parallelism * Fix linting - Run `golang-lint run --fix` - Check error after `waitForConfigUpdate` * Move finality node test to espresso tests The race test detected a data race in https://github.com/EspressoSystems/nitro-espresso-integration/actions/runs/11705919237/job/32601803846?pr=254 * Update contracts submodule * Fix typo * Add wabt and cmake in wasm shell * Skip tests * Add tags in ci * Add ci skip tests * Add ci skip tests * Remove rerun-fails This flag in gotestsum causes some bugs which multiple the test time. gotestyourself/gotestsum#417 gotestyourself/gotestsum#423 * Revert "Remove rerun-fails" This reverts commit 6bef1c3. * CI: simplify - Add scripts to debug more easily locally. - Skip tests by substring match instead of exact match. - Don't run coverage on CI, we never look at it. - Move all Espresso tests to Espresso CI job. * Fix build with nix on macos (#296) * CI: install latest gotestsum This enables the --hide-summary argument * Update submodule forks with merge commits * Bump nitro-testnode from `07a25f1` to `77d1447` (#297) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `07a25f1` to `77d1447`. - [Commits](EspressoSystems/nitro-testnode@07a25f1...77d1447) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add FFI bindings for Espresso validation (#294) * Example call to espresso-crypto function stub in batch poster. This commit includes a log line and example call to the espresso-crypto function stubs in batch_poster.go * Result of `make fmt` Run `make fmt` to clean up formatting not apparent in editor * Make current lib.rs c abi compatible * result of make fmt * Add C ABI ffi between rust crypto lib and espressocrypto go module. This commit adds FFI support between exsisting Rust crypto helper code and the exsisting espressocrypto go module in native.go These functions will now serve as utility functions to use in the integration that will handle ffi to the Rust code. This commit also moves the espresso-crypto-helper code to the espressocrypto directory as we no longer need it for the arbitrator. * Build native.go successfully * Update the espresso-crypto and add test for merkle proof verification * update flake.nix to new go version and fix errors * Current state of work The current state of my work revolves around editing the Makefile to include the new espressocrypto ffi functions in the build process. Additionally, this commit restores some espressocrypto code to the arbitrator in an attempt to get a docker image built via the test node. I felt as though this would be easier for this PR as opposed to ripping out all of the code from the arbitrator. * Update the flake.lock * Add the namespace proof test * Current state of build system changes This commit represents the current state of the build system after testing today. Currently things compile locally, but not when building an image from the docker file. There are errors related to duplicate function definitions from what seem to be the FFI lib that stylus exposes. * Build nitro docker image sucessfully - Modify makefile to have a file target. Since the build artifact of our static library is a file we should use this as the make target (instead of only having PHONY targets). - Use rust 1.81 to build all shared rust libraries in docker. Our crate requires a minimum version of 1.81. However the other libraries like libstylus.a are currently compiled with 1.75 and this seems to cause issues with duplicate symbol definitions during linking. For example: 8.718 go build -ldflags="-extldflags '-ldl' -X github.com/offchainlabs/nitro/cmd/util/confighelpers.version= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.datetime= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.modified=" -o target/bin/nitro "/workspace/cmd/nitro" 42.74 # github.com/offchainlabs/nitro/cmd/nitro 42.74 /usr/local/go/pkg/tool/linux_amd64/link: running clang failed: exit status 1 42.74 /usr/bin/ld: /workspace/espressocrypto/lib/espresso-crypto-helper/target/release/libespresso_crypto_helper.a(std-1c4b19562077c20d.std.1f2242ed6435445e-cgu.0.rcgu.o): in function `__rdl_alloc': 42.74 /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/alloc.rs:397: multiple definition of `__rdl_alloc'; /workspace/execution/gethexec/../../target/lib/libstylus.a(std-90f6ddbf82de36ec.std.409886f6357001f0-cgu.0.rcgu.o):/rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/alloc.rs:389: first defined here Bump the rust version of the prover-header-builder to 1.81 and compile the espresso crypto library in the same container. * Fix LDFLAGS, remove unnecessary CFLAGS * Fix some merge errors - We need rust 1.81 to compile the espresso crypto lib - Re-add makefile rules * Fix merge errors * Remove FFI call in batch poster to expidite merge. * Update bf submodule in arbitrator to match upstream version (This was set to a different commit during my erroneous merge) * Remove extra change in espressocrypto_wasm.go * Fix error with temporary espressocryptowasm directory preventing CI from running * fix error of missing function definitions in espressocryptowasm directory to allow CI to run * Re-add nosec comment about gosec error that will be removed soon * result of make fmt --------- Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: ImJeremyHe <[email protected]> * Update the go client (#256) Still using the legacy dev node * ci: fix `git ls-remote` error in docker build (#304) * Use path context instead of git context With the default git context the repository is re-cloned at the start of the docker build and the .dockerignore file is ignored. > Be careful because any file mutation in the steps that precede the > build step will be ignored, including processing of the .dockerignore > file since the context is based on the Git reference. https://github.com/docker/build-push-action?tab=readme-ov-file#git-context This leads to some `.git` worktree files showing up inside the dockerbuild with triggers a bug in yarn. See yarnpkg/yarn#7537 (comment) for more details about the yarn bug. * Use the latest espresso dev node (#303) * Use the latest espresso dev node - Fix an issue that may cause the validation reorg - Waiting for L1 to be ready can avoid some potential bugs in dev node * Add proof validation to batch poster (#301) * Add espresso proof validation to batch poster Adds calls to the espresso validation code to the batch poster to validate the namespace proof and merkle proof in a TEE * result of make fmt * Fix lint errors * Add better logging * Fix batch poster after merge of upstream/integration * Remove espresso replay binary modifications (#307) * Remove espresso validation in the replay binary * Remove espresso-crypto related dependencies in arbitrator * remove espressocryptowasm * Update Makefile and Dockerfile to remove references to unused espresso crypto wasm lib * remove batch_poster.go.orig file * Fixes from make fmt * Remove espresso_ops_test * Remove additional code related to espresso validation in the staker * Remove changes on stateless block validator * Undo the stopwaiter Previous modifications on this was because we had to stop and start the seuquencers to function the escape hatch. Right now we don't need that anymore. These changes are from the upstream * Rename test flag --------- Co-authored-by: ImJeremyHe <[email protected]> * Add attestation quote in batch poster (#306) * Add attestation quote in batch poster * write file * update batch poster * use read file * update to userDataHash * reduce file permissions * nit * Submit multiple messages in one transaction (#308) * Submit multiple messages in one transaction * Remove the espresso_transaction_payload_signature_test.go * Update go client * Remove signature and data signer * Remove arbitrator/espresso-crypto-helper * Add size limit to hotshot transaction --------- Co-authored-by: Sneh Koul <[email protected]> * Use upstream contracts submodule (#310) We no longer need the customizations in nitro-contracts (except potentially some deployment tools) so this commit resets it to the before the celestia contracts were added celestiaorg/nitro-contracts@4f8c65c and makes the nitro code work with that version by removing the hotshot / light client address passed to the OSP. * Remove the legacy code and fix the batch poster (#318) * Remove the legacy code and fix the test * fix ci * format --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nomaxg <[email protected]> Co-authored-by: ImJeremyHe <[email protected]> Co-authored-by: sveitser <[email protected]> Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sneh1999
added a commit
to EspressoSystems/nitro-espresso-integration
that referenced
this issue
Nov 20, 2024
* Add SwitchSequencer * Fix the switch function * Create SwitchSequencer * Update the espresso-sequencer-go * Use light client as the trigger * Add the switch test * Make the linter happy * Add the switch test to the e2e ci * Increase the limit * Clean the log with the log helper * more mock code * Fix the e2e test * serialize proof variables correctly * update espresso-go version * wasm compatibility * remove stale file, lint fix * update to recent espresso go version * refactor block merkle verification * work on E2E * progress on merkle tree verification * add test vectors, fetch proofs at the correct indicies * cleanup, remove outfiles, fetch merkle proof at the correct justified height * add logic to compare circuit field * check that circuit representation equals u256 representation in the test * send byte slice to wasm instead of byte array which lives on the stack * cleanup, remove debug statements, add documentation * remove error logs * fix build * lint fix * progress on e2e tests, comment out vid stuff for now because of wasm io issues * update espresso-sequencer-go tag, lint ci fix * cleanup, add issues for unresolved problems * remove block that spams transactions * comment out unused txns * add test for advz equivalence and use a smaller srs * Allow dead code in validation * Specify foundry version * Add random_get stub in wasi_snapshot_preview1 * Add stubs required in wasi_snapshot_preview1 * Fix `make lint` * Enable prover service to talk to L1 on host * Add builder timeout This allows the sequencer to make progress past block 1. These env vars have been changed in espresso-sequencer. * Add missing env var, more logging in e2e test * Revert "Fix `make lint`" This reverts commit 8be940c. * Fix go linting * Parse espresso messages * Add timeout limit * Fix the e2e test * Add timeout limit * Run the e2e test and switch test in 1 thread * Update the e2e workflow * Increase time limit * Specify the foundryup version in Dockerfile * Add sock_accept stub * Increase timeout limit * Build successfully * move ark-srs to dev dependencies * remove debug statements * remove out.txt * Fix the machine test * Fix CI * Fix switch test * Fix CI * Increase CI time limie * update lockfile * remove commented out replaces * newline * Use the latest foundry * RESET ME: debug * RESET ME: not run the CI in a container * RESET ME: remove the parallel * Update script to run tests locally - Run the same commands as on CI. - Add ability to run `cargo +nightly` to dev shell. - make clean: remove brotli build files. * set -parallel=2 * Fix e2e test * Skip some tests in CI * Skip tests in stylus * Remove parallelism restriction * Impl escape hatch on validator side * Fix tests * Generate the osp of hotshot liveness * Fix the fraud proof (#124) * Update contracts submodule * Upgrade go client * Fix the staker challenge tests --------- Co-authored-by: sveitser <[email protected]> * Use espresso dev node * attempt to fix dockerfile * newline * WIP * WIP * download latest espresso module root * incorporate vid common data * Fix the validation pipeline * update go modules, e2e build step * add cbrotli wasm * ci fix attempt * dont build all test deps for e2e test * lint fixes * Use the self-hosted runner and install cbindgen in CI * Add time limit * Remove parallel in building * Fix the switch test * Decrease the waiting hotshot blocks * Use the ubuntu * Specify the foundry version * Enable -j * update foundry version * module root update * Fix the staker test * increase validation timeout for switch test * pin espresso-sequencer-go to new release v0.0.20 * Commit go.sum from running go mod tidy * sequencer info logs * Add escape hatch test in e2e test * Check the reactivate hotshot * Update validator to handle preconditions and panics before entering the espresso STF logic. * Improve readability of conditions in `handleEspressoPreConditions` and update the functions comments. * Further improve readability * Fix lint issue * Update the nitro testnode * Fix tests * Update contracts * Fix CI * Changes to `handleEspressoPreConditions` in `espresso_validation.go` to fix incorrect logic, and give more granular control to where and how the generated panic is handled in the validators STF via returned closures. * return nil when we need a no-op panic handler * Fix lint errors * Update contracts * Fix the validation * Remove the switch test * Fix CI * Remove postgres in espresso e2e test * Codeowners: add Alysia, Zach, Sneh (#154) For #146 * Remove Dockerfile.espresso (#156) * Remove Dockerfile.espresso * address review comments * add new line * Release 20240723 with 20240723-consensus This should include all our changes for nitro v3.1. - Check that the module root matches when downloading releases from github. * Check all "not found" errors Currently only the leveldb not found errors are handled. If a pebble not found error is returned by the backend it is propagated, causing the batchposter to fail to send transactions. With this change all DB not found errors are handled the same way. * Submit L2 Message to espresso network (#173) * Submit L2 Message to espresso network * add context timeout * check for hotshot liveness and add a polling interval to see if transactions were included in an espresso block * cleanup * lint * Draft the sovereign sequencer test * Add espresso transaction queue * Draft the espresso submission * Set a new l2 message type for the sovereign sequencer transactions * Attempt to write messages again * Fix typos and add some comments * Fix typo * add header to jst and submit transaction to espresso * Update go client for development * Confirm the transaction inclusion in hotshot * Fix the test * Unblock the building blocks * Update go client * Fix the sovereign test * add logs * cleanup * lint * Updage go client * not to fetch merkle proof if hotshot height is 0 * Cleanup * Fix CI * Add CI skip tests * remove unwanted variable * skipping some tests to make ci pass * skipping some tests to make ci pass --------- Co-authored-by: ImJeremyHe <[email protected]> * Persistent storage for sovereign builder (#180) * Persistence storage for sovereign builder * add unit test * add unit test to ci and add mutex * Optional header in jst * initialize pending transactions if not present * remove unwanted test * resolve bugs * remove nil pointer panic * cleanup * revert to strings.contains * fix isErrNotFound --------- Co-authored-by: ImJeremyHe <[email protected]> * Save digest artifacts into own directory per run (#182) * Save digest artifacts into own directory per run Currently it seems that we have too many digests in the folder and I think this breaks the assembly of the multiarch docker image. With this change only digests for the same run are saved in the same directory. * Fail docker image merge on wrong number of digests * Clean digests dir * Add dependabot rule to update nitro-testnode (#189) * Fix build (#191) * Add missing dependency * Use renamed variable, fix typo * dependabot: update contracts submodule (#193) * Pin the golint version in CI * Fix pebble panic (#192) * Fix pebble panic * lint * Update some outdated help on flags (#203) * Improve help texts for flags * Update e2e tests to use Sovereign sequencer (#204) * Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]> * Remove the global state for HotShot height (#210) * Remove the global state * Update the submodule * CI: use buildjet instead of self-hosted runner (#212) * CI: use buildjet instead of self-hosted runner * CI: use public runner / buildjet for docker build This removes usage of our self-hosted github runners. * CI: more disk space on public runner * Check disk space at end of runs * Try running CI tests from different directory This may avoid failures due to IPC paths being too long * Make workspace dir a symlink to /tmp/gha-work * Fix redis URL * Move redis tests to own workflow * CI: fix rust and go cache (#217) * rust-cache: add more target dirs * Don't set custom TMPDIR This seems to be the culprit for the extremely long path names that are too long for IPC paths. * modify arbosstate.go to allow for upgrade to arbOS version 35 (#216) * modify arbosstate.go to allow for upgrade to arbOS version 35 * Fix lint * Modify the chain config during the ArbOS upgrade --------- Co-authored-by: sveitser <[email protected]> * CI: make more space (#218) * Store transaction pos in batch poster (#219) * Move submit transaction pos from tx_streamer to batch poster * Remove espresso sequencer (#214) * Remove espresso sequencer * lint * remove enable espresso * commit go-ethereum * fix solgen/gen.go * fix build * revert .gitmodules * revert go.mod * add enable espresso to validation check * re-add enable espresso to tests --------- Co-authored-by: Mathis <[email protected]> * Update espresso-sequencer-go with new LC ABI (#230) * Update espresso-sequencer-go with new LC ABI Instead of using the block height from the first snapshot after the height in the justification it uses the latest height in the LC contract. Currently not 100% convinced about correctness but also believe it should work because any newer tree still contains the header of interest. * Undo LC ABI changes We're not comfortable with the previous change because making a proof w.r.t to the newest finalized state may cause issues if the batch poster is lagging behind. This is currently not easy to test so we decided to revert back to the old flow. * Use tag of espresso-sequencer-go * Stop setting the chain config during ArbOS upgrade in favor of setting it manually during migration process. (#234) * Use the tag before marketplace update (#238) * Sign payload for sovereign builder (#181) * Sign payload for sovereign builder * save signed payload to db * change parsing logic to include signature * add some comments * skip tests * address comments * add signature payload test * test working * fix logging and comments * move placement of log * fix ci * fix ci * only submit transaction if jst nil * add log to debug ci * use the latest instead of main for espresso-dev-node * pin to legacy lc version * pin to before marketplace version * remove log * Full node with espresso finality (#237) * wip * wip * Add espresso finality node test * fix ci * fix ci * pin to before marketplace version * fix tests * add comments * address comments * address comments * address comments * cleanup * fix test * fmt * rerun ci * ci skip tests * Update to sequencer.go to remove `EnableEspressoSovereign` flag, and instead sequence based on hotshot liveness at l1 block height. * Update node.go to remove reference to `EnableEspressoSovereign` * update Espresso tests to reflect removed config variables, and add escape hatch test to e2e test. * Add wait for hotshot liveness in sovereign sequencer test to ensure sovereign sequencer is enabled. * go fmt * Add comment to explain changes needed in replay binary when panics are no longer an issue. * Fix lint issues * Modify construction of lightClientReader in sequencer.go to allow for nil `l1Reader` * Update test configs to align with changes. * Update comments according to feedback. * Bump nitro-testnode from `03095e0` to `884a63c` (#240) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `03095e0` to `884a63c`. - [Commits](EspressoSystems/nitro-testnode@03095e0...884a63c) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mathis <[email protected]> * Revert removal of `EnableEspressoSovereign` * Add additional check to ensure that if EnableEspressoSovereign is on we can't construct a sequencer with a nil lightClientReader * Add `EnableEspressoSovereign` config flag to test configs. * Bump contracts from `d226e9d` to `2ae1bbd` (#244) Bumps [contracts](https://github.com/EspressoSystems/nitro-contracts) from `d226e9d` to `2ae1bbd`. - [Release notes](https://github.com/EspressoSystems/nitro-contracts/releases) - [Commits](EspressoSystems/nitro-contracts@d226e9d...2ae1bbd) --- updated-dependencies: - dependency-name: contracts dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pin rust to 1.81.0 See #257 * Set cargo config for espresso_crypto wasm build This was recently added for the other wasm compilation steps. * Pin rust in E2E test * CI: remove deprecated rust toolchain action * Fix if statement * Regenerate data-poster external-signer test certs (cherry picked from commit 8f961b0) * Use ArbOS chain config as the cannonial chain config. (#252) * Add checks to sequencer.go and transaction_streamer.go to utilize the chain config switch * Update interface.go to export interface for `GetArbOSConfigAtHeight()` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionEngine` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionNode` * Update `sequencer.go` and `transaction_streamer.go` to read from the `ArbOS` `chainConfig` as the cannonical `chainConfig` rather than the `chainConfig` held by the `Blockchain` struct. * Add ArbOS chain config check in batch poster to stop adding espresso messages to the transaction streamers DB if EnableEspresso is false. * Add missing error check in sequencer.go and nil check on arbos config in sequencer.go and batch_poster.go. * Remove ineffectual assignment in sequencer.go * Add test for new GetArbOSConfigAtHeight() method in ExecutionNode This commit adds an integration test for the GetArbOSConfigAtHeight() method on the Execution node that grabs the initial config, and validates it, then updates the config and ensures that the new method will fetch the updated config. * restore nitro-testnode to proper commit * Fix trie node node found error during recording I'm not exactly clear why we need to do this or if it's indeed the right approach to solving the issue however it does seem to make it possible to validate again. Without it we get errors like INFO [10-30|12:24:29.127] DelayedSequencer: Sequenced msgnum=1 startpos=1 INFO [10-30|12:24:30.143] Reading message result remotely. count=3 WARN [10-30|12:24:32.143] prepareblocks failed to get state for block hdrNum=0 err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" ERROR[10-30|12:24:32.143] Error while recording err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" status=2 * Fix the OSP test - Enable 4844 blobs to avoid contract deployment error - Fix address of OSP entry * Disable batch poster in Espresso finality node We don't need this type of node to have a batch poster. Enabling it now causes the test to fail because there's a check to ensure that redis is used for the batch poster on the second node. * Re-allow setting L1 stack config for tests After the sync the code would always use the default values. Don't think this is very clean but I couldn't think of something better yet. * CI: increase parallelism * Fix linting - Run `golang-lint run --fix` - Check error after `waitForConfigUpdate` * Move finality node test to espresso tests The race test detected a data race in https://github.com/EspressoSystems/nitro-espresso-integration/actions/runs/11705919237/job/32601803846?pr=254 * Update contracts submodule * Fix typo * Add wabt and cmake in wasm shell * Skip tests * Add tags in ci * Add ci skip tests * Add ci skip tests * Remove rerun-fails This flag in gotestsum causes some bugs which multiple the test time. gotestyourself/gotestsum#417 gotestyourself/gotestsum#423 * Revert "Remove rerun-fails" This reverts commit 6bef1c3. * CI: simplify - Add scripts to debug more easily locally. - Skip tests by substring match instead of exact match. - Don't run coverage on CI, we never look at it. - Move all Espresso tests to Espresso CI job. * Fix build with nix on macos (#296) * CI: install latest gotestsum This enables the --hide-summary argument * Update submodule forks with merge commits * Bump nitro-testnode from `07a25f1` to `77d1447` (#297) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `07a25f1` to `77d1447`. - [Commits](EspressoSystems/nitro-testnode@07a25f1...77d1447) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add FFI bindings for Espresso validation (#294) * Example call to espresso-crypto function stub in batch poster. This commit includes a log line and example call to the espresso-crypto function stubs in batch_poster.go * Result of `make fmt` Run `make fmt` to clean up formatting not apparent in editor * Make current lib.rs c abi compatible * result of make fmt * Add C ABI ffi between rust crypto lib and espressocrypto go module. This commit adds FFI support between exsisting Rust crypto helper code and the exsisting espressocrypto go module in native.go These functions will now serve as utility functions to use in the integration that will handle ffi to the Rust code. This commit also moves the espresso-crypto-helper code to the espressocrypto directory as we no longer need it for the arbitrator. * Build native.go successfully * Update the espresso-crypto and add test for merkle proof verification * update flake.nix to new go version and fix errors * Current state of work The current state of my work revolves around editing the Makefile to include the new espressocrypto ffi functions in the build process. Additionally, this commit restores some espressocrypto code to the arbitrator in an attempt to get a docker image built via the test node. I felt as though this would be easier for this PR as opposed to ripping out all of the code from the arbitrator. * Update the flake.lock * Add the namespace proof test * Current state of build system changes This commit represents the current state of the build system after testing today. Currently things compile locally, but not when building an image from the docker file. There are errors related to duplicate function definitions from what seem to be the FFI lib that stylus exposes. * Build nitro docker image sucessfully - Modify makefile to have a file target. Since the build artifact of our static library is a file we should use this as the make target (instead of only having PHONY targets). - Use rust 1.81 to build all shared rust libraries in docker. Our crate requires a minimum version of 1.81. However the other libraries like libstylus.a are currently compiled with 1.75 and this seems to cause issues with duplicate symbol definitions during linking. For example: 8.718 go build -ldflags="-extldflags '-ldl' -X github.com/offchainlabs/nitro/cmd/util/confighelpers.version= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.datetime= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.modified=" -o target/bin/nitro "/workspace/cmd/nitro" 42.74 # github.com/offchainlabs/nitro/cmd/nitro 42.74 /usr/local/go/pkg/tool/linux_amd64/link: running clang failed: exit status 1 42.74 /usr/bin/ld: /workspace/espressocrypto/lib/espresso-crypto-helper/target/release/libespresso_crypto_helper.a(std-1c4b19562077c20d.std.1f2242ed6435445e-cgu.0.rcgu.o): in function `__rdl_alloc': 42.74 /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/alloc.rs:397: multiple definition of `__rdl_alloc'; /workspace/execution/gethexec/../../target/lib/libstylus.a(std-90f6ddbf82de36ec.std.409886f6357001f0-cgu.0.rcgu.o):/rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/alloc.rs:389: first defined here Bump the rust version of the prover-header-builder to 1.81 and compile the espresso crypto library in the same container. * Fix LDFLAGS, remove unnecessary CFLAGS * Fix some merge errors - We need rust 1.81 to compile the espresso crypto lib - Re-add makefile rules * Fix merge errors * Remove FFI call in batch poster to expidite merge. * Update bf submodule in arbitrator to match upstream version (This was set to a different commit during my erroneous merge) * Remove extra change in espressocrypto_wasm.go * Fix error with temporary espressocryptowasm directory preventing CI from running * fix error of missing function definitions in espressocryptowasm directory to allow CI to run * Re-add nosec comment about gosec error that will be removed soon * result of make fmt --------- Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: ImJeremyHe <[email protected]> * Update the go client (#256) Still using the legacy dev node * ci: fix `git ls-remote` error in docker build (#304) * Use path context instead of git context With the default git context the repository is re-cloned at the start of the docker build and the .dockerignore file is ignored. > Be careful because any file mutation in the steps that precede the > build step will be ignored, including processing of the .dockerignore > file since the context is based on the Git reference. https://github.com/docker/build-push-action?tab=readme-ov-file#git-context This leads to some `.git` worktree files showing up inside the dockerbuild with triggers a bug in yarn. See yarnpkg/yarn#7537 (comment) for more details about the yarn bug. * Use the latest espresso dev node (#303) * Use the latest espresso dev node - Fix an issue that may cause the validation reorg - Waiting for L1 to be ready can avoid some potential bugs in dev node * Add proof validation to batch poster (#301) * Add espresso proof validation to batch poster Adds calls to the espresso validation code to the batch poster to validate the namespace proof and merkle proof in a TEE * result of make fmt * Fix lint errors * Add better logging * Fix batch poster after merge of upstream/integration * Remove espresso replay binary modifications (#307) * Remove espresso validation in the replay binary * Remove espresso-crypto related dependencies in arbitrator * remove espressocryptowasm * Update Makefile and Dockerfile to remove references to unused espresso crypto wasm lib * remove batch_poster.go.orig file * Fixes from make fmt * Remove espresso_ops_test * Remove additional code related to espresso validation in the staker * Remove changes on stateless block validator * Undo the stopwaiter Previous modifications on this was because we had to stop and start the seuquencers to function the escape hatch. Right now we don't need that anymore. These changes are from the upstream * Rename test flag --------- Co-authored-by: ImJeremyHe <[email protected]> * Add attestation quote in batch poster (#306) * Add attestation quote in batch poster * write file * update batch poster * use read file * update to userDataHash * reduce file permissions * nit * Submit multiple messages in one transaction (#308) * Submit multiple messages in one transaction * Remove the espresso_transaction_payload_signature_test.go * Update go client * Remove signature and data signer * Remove arbitrator/espresso-crypto-helper * Add size limit to hotshot transaction --------- Co-authored-by: Sneh Koul <[email protected]> * Use espresso-types instead of hard coded files * WIP: locally failed * Use upstream contracts submodule (#310) We no longer need the customizations in nitro-contracts (except potentially some deployment tools) so this commit resets it to the before the celestia contracts were added celestiaorg/nitro-contracts@4f8c65c and makes the nitro code work with that version by removing the hotshot / light client address passed to the OSP. * Remove the legacy code and fix the batch poster (#318) * Remove the legacy code and fix the test * fix nix build on linux - flake: add openssl, pkg-config - rust: update espresso-types to latest to not require testing features * nix: fix build on darwin - Update nixpkgs - Set NIX_LDFLAGS to find missing symbols - Leave note about updating flake since the apple SDKs in nixpkgs have been changed substantially. * Transmute nstable * add celestia-integration branch to the ci --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nomaxg <[email protected]> Co-authored-by: sveitser <[email protected]> Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: Sneh Koul <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sneh Koul <[email protected]>
ImJeremyHe
added a commit
to EspressoSystems/nitro-espresso-integration
that referenced
this issue
Dec 4, 2024
* Add SwitchSequencer * Fix the switch function * Create SwitchSequencer * Update the espresso-sequencer-go * Use light client as the trigger * Add the switch test * Make the linter happy * Add the switch test to the e2e ci * Increase the limit * Clean the log with the log helper * more mock code * Fix the e2e test * serialize proof variables correctly * update espresso-go version * wasm compatibility * remove stale file, lint fix * update to recent espresso go version * refactor block merkle verification * work on E2E * progress on merkle tree verification * add test vectors, fetch proofs at the correct indicies * cleanup, remove outfiles, fetch merkle proof at the correct justified height * add logic to compare circuit field * check that circuit representation equals u256 representation in the test * send byte slice to wasm instead of byte array which lives on the stack * cleanup, remove debug statements, add documentation * remove error logs * fix build * lint fix * progress on e2e tests, comment out vid stuff for now because of wasm io issues * update espresso-sequencer-go tag, lint ci fix * cleanup, add issues for unresolved problems * remove block that spams transactions * comment out unused txns * add test for advz equivalence and use a smaller srs * Allow dead code in validation * Specify foundry version * Add random_get stub in wasi_snapshot_preview1 * Add stubs required in wasi_snapshot_preview1 * Fix `make lint` * Enable prover service to talk to L1 on host * Add builder timeout This allows the sequencer to make progress past block 1. These env vars have been changed in espresso-sequencer. * Add missing env var, more logging in e2e test * Revert "Fix `make lint`" This reverts commit 8be940c. * Fix go linting * Parse espresso messages * Add timeout limit * Fix the e2e test * Add timeout limit * Run the e2e test and switch test in 1 thread * Update the e2e workflow * Increase time limit * Specify the foundryup version in Dockerfile * Add sock_accept stub * Increase timeout limit * Build successfully * move ark-srs to dev dependencies * remove debug statements * remove out.txt * Fix the machine test * Fix CI * Fix switch test * Fix CI * Increase CI time limie * update lockfile * remove commented out replaces * newline * Use the latest foundry * RESET ME: debug * RESET ME: not run the CI in a container * RESET ME: remove the parallel * Update script to run tests locally - Run the same commands as on CI. - Add ability to run `cargo +nightly` to dev shell. - make clean: remove brotli build files. * set -parallel=2 * Fix e2e test * Skip some tests in CI * Skip tests in stylus * Remove parallelism restriction * Impl escape hatch on validator side * Fix tests * Generate the osp of hotshot liveness * Fix the fraud proof (#124) * Update contracts submodule * Upgrade go client * Fix the staker challenge tests --------- Co-authored-by: sveitser <[email protected]> * Use espresso dev node * attempt to fix dockerfile * newline * WIP * WIP * download latest espresso module root * incorporate vid common data * Fix the validation pipeline * update go modules, e2e build step * add cbrotli wasm * ci fix attempt * dont build all test deps for e2e test * lint fixes * Use the self-hosted runner and install cbindgen in CI * Add time limit * Remove parallel in building * Fix the switch test * Decrease the waiting hotshot blocks * Use the ubuntu * Specify the foundry version * Enable -j * update foundry version * module root update * Fix the staker test * increase validation timeout for switch test * pin espresso-sequencer-go to new release v0.0.20 * Commit go.sum from running go mod tidy * sequencer info logs * Add escape hatch test in e2e test * Check the reactivate hotshot * Update validator to handle preconditions and panics before entering the espresso STF logic. * Improve readability of conditions in `handleEspressoPreConditions` and update the functions comments. * Further improve readability * Fix lint issue * Update the nitro testnode * Fix tests * Update contracts * Fix CI * Changes to `handleEspressoPreConditions` in `espresso_validation.go` to fix incorrect logic, and give more granular control to where and how the generated panic is handled in the validators STF via returned closures. * return nil when we need a no-op panic handler * Fix lint errors * Update contracts * Fix the validation * Remove the switch test * Fix CI * Remove postgres in espresso e2e test * Codeowners: add Alysia, Zach, Sneh (#154) For #146 * Remove Dockerfile.espresso (#156) * Remove Dockerfile.espresso * address review comments * add new line * Release 20240723 with 20240723-consensus This should include all our changes for nitro v3.1. - Check that the module root matches when downloading releases from github. * Check all "not found" errors Currently only the leveldb not found errors are handled. If a pebble not found error is returned by the backend it is propagated, causing the batchposter to fail to send transactions. With this change all DB not found errors are handled the same way. * Submit L2 Message to espresso network (#173) * Submit L2 Message to espresso network * add context timeout * check for hotshot liveness and add a polling interval to see if transactions were included in an espresso block * cleanup * lint * Draft the sovereign sequencer test * Add espresso transaction queue * Draft the espresso submission * Set a new l2 message type for the sovereign sequencer transactions * Attempt to write messages again * Fix typos and add some comments * Fix typo * add header to jst and submit transaction to espresso * Update go client for development * Confirm the transaction inclusion in hotshot * Fix the test * Unblock the building blocks * Update go client * Fix the sovereign test * add logs * cleanup * lint * Updage go client * not to fetch merkle proof if hotshot height is 0 * Cleanup * Fix CI * Add CI skip tests * remove unwanted variable * skipping some tests to make ci pass * skipping some tests to make ci pass --------- Co-authored-by: ImJeremyHe <[email protected]> * Persistent storage for sovereign builder (#180) * Persistence storage for sovereign builder * add unit test * add unit test to ci and add mutex * Optional header in jst * initialize pending transactions if not present * remove unwanted test * resolve bugs * remove nil pointer panic * cleanup * revert to strings.contains * fix isErrNotFound --------- Co-authored-by: ImJeremyHe <[email protected]> * Save digest artifacts into own directory per run (#182) * Save digest artifacts into own directory per run Currently it seems that we have too many digests in the folder and I think this breaks the assembly of the multiarch docker image. With this change only digests for the same run are saved in the same directory. * Fail docker image merge on wrong number of digests * Clean digests dir * Add dependabot rule to update nitro-testnode (#189) * Fix build (#191) * Add missing dependency * Use renamed variable, fix typo * dependabot: update contracts submodule (#193) * Pin the golint version in CI * Fix pebble panic (#192) * Fix pebble panic * lint * Update some outdated help on flags (#203) * Improve help texts for flags * Update e2e tests to use Sovereign sequencer (#204) * Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]> * Remove the global state for HotShot height (#210) * Remove the global state * Update the submodule * CI: use buildjet instead of self-hosted runner (#212) * CI: use buildjet instead of self-hosted runner * CI: use public runner / buildjet for docker build This removes usage of our self-hosted github runners. * CI: more disk space on public runner * Check disk space at end of runs * Try running CI tests from different directory This may avoid failures due to IPC paths being too long * Make workspace dir a symlink to /tmp/gha-work * Fix redis URL * Move redis tests to own workflow * CI: fix rust and go cache (#217) * rust-cache: add more target dirs * Don't set custom TMPDIR This seems to be the culprit for the extremely long path names that are too long for IPC paths. * modify arbosstate.go to allow for upgrade to arbOS version 35 (#216) * modify arbosstate.go to allow for upgrade to arbOS version 35 * Fix lint * Modify the chain config during the ArbOS upgrade --------- Co-authored-by: sveitser <[email protected]> * CI: make more space (#218) * Store transaction pos in batch poster (#219) * Move submit transaction pos from tx_streamer to batch poster * Remove espresso sequencer (#214) * Remove espresso sequencer * lint * remove enable espresso * commit go-ethereum * fix solgen/gen.go * fix build * revert .gitmodules * revert go.mod * add enable espresso to validation check * re-add enable espresso to tests --------- Co-authored-by: Mathis <[email protected]> * Update espresso-sequencer-go with new LC ABI (#230) * Update espresso-sequencer-go with new LC ABI Instead of using the block height from the first snapshot after the height in the justification it uses the latest height in the LC contract. Currently not 100% convinced about correctness but also believe it should work because any newer tree still contains the header of interest. * Undo LC ABI changes We're not comfortable with the previous change because making a proof w.r.t to the newest finalized state may cause issues if the batch poster is lagging behind. This is currently not easy to test so we decided to revert back to the old flow. * Use tag of espresso-sequencer-go * Stop setting the chain config during ArbOS upgrade in favor of setting it manually during migration process. (#234) * Use the tag before marketplace update (#238) * Sign payload for sovereign builder (#181) * Sign payload for sovereign builder * save signed payload to db * change parsing logic to include signature * add some comments * skip tests * address comments * add signature payload test * test working * fix logging and comments * move placement of log * fix ci * fix ci * only submit transaction if jst nil * add log to debug ci * use the latest instead of main for espresso-dev-node * pin to legacy lc version * pin to before marketplace version * remove log * Full node with espresso finality (#237) * wip * wip * Add espresso finality node test * fix ci * fix ci * pin to before marketplace version * fix tests * add comments * address comments * address comments * address comments * cleanup * fix test * fmt * rerun ci * ci skip tests * Update to sequencer.go to remove `EnableEspressoSovereign` flag, and instead sequence based on hotshot liveness at l1 block height. * Update node.go to remove reference to `EnableEspressoSovereign` * update Espresso tests to reflect removed config variables, and add escape hatch test to e2e test. * Add wait for hotshot liveness in sovereign sequencer test to ensure sovereign sequencer is enabled. * go fmt * Add comment to explain changes needed in replay binary when panics are no longer an issue. * Fix lint issues * Modify construction of lightClientReader in sequencer.go to allow for nil `l1Reader` * Update test configs to align with changes. * Update comments according to feedback. * Bump nitro-testnode from `03095e0` to `884a63c` (#240) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `03095e0` to `884a63c`. - [Commits](EspressoSystems/nitro-testnode@03095e0...884a63c) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mathis <[email protected]> * Revert removal of `EnableEspressoSovereign` * Add additional check to ensure that if EnableEspressoSovereign is on we can't construct a sequencer with a nil lightClientReader * Add `EnableEspressoSovereign` config flag to test configs. * Bump contracts from `d226e9d` to `2ae1bbd` (#244) Bumps [contracts](https://github.com/EspressoSystems/nitro-contracts) from `d226e9d` to `2ae1bbd`. - [Release notes](https://github.com/EspressoSystems/nitro-contracts/releases) - [Commits](EspressoSystems/nitro-contracts@d226e9d...2ae1bbd) --- updated-dependencies: - dependency-name: contracts dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pin rust to 1.81.0 See #257 * Set cargo config for espresso_crypto wasm build This was recently added for the other wasm compilation steps. * Pin rust in E2E test * CI: remove deprecated rust toolchain action * Fix if statement * Regenerate data-poster external-signer test certs (cherry picked from commit 8f961b0) * Use ArbOS chain config as the cannonial chain config. (#252) * Add checks to sequencer.go and transaction_streamer.go to utilize the chain config switch * Update interface.go to export interface for `GetArbOSConfigAtHeight()` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionEngine` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionNode` * Update `sequencer.go` and `transaction_streamer.go` to read from the `ArbOS` `chainConfig` as the cannonical `chainConfig` rather than the `chainConfig` held by the `Blockchain` struct. * Add ArbOS chain config check in batch poster to stop adding espresso messages to the transaction streamers DB if EnableEspresso is false. * Add missing error check in sequencer.go and nil check on arbos config in sequencer.go and batch_poster.go. * Remove ineffectual assignment in sequencer.go * Add test for new GetArbOSConfigAtHeight() method in ExecutionNode This commit adds an integration test for the GetArbOSConfigAtHeight() method on the Execution node that grabs the initial config, and validates it, then updates the config and ensures that the new method will fetch the updated config. * restore nitro-testnode to proper commit * Fix trie node node found error during recording I'm not exactly clear why we need to do this or if it's indeed the right approach to solving the issue however it does seem to make it possible to validate again. Without it we get errors like INFO [10-30|12:24:29.127] DelayedSequencer: Sequenced msgnum=1 startpos=1 INFO [10-30|12:24:30.143] Reading message result remotely. count=3 WARN [10-30|12:24:32.143] prepareblocks failed to get state for block hdrNum=0 err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" ERROR[10-30|12:24:32.143] Error while recording err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" status=2 * Fix the OSP test - Enable 4844 blobs to avoid contract deployment error - Fix address of OSP entry * Disable batch poster in Espresso finality node We don't need this type of node to have a batch poster. Enabling it now causes the test to fail because there's a check to ensure that redis is used for the batch poster on the second node. * Re-allow setting L1 stack config for tests After the sync the code would always use the default values. Don't think this is very clean but I couldn't think of something better yet. * CI: increase parallelism * Fix linting - Run `golang-lint run --fix` - Check error after `waitForConfigUpdate` * Move finality node test to espresso tests The race test detected a data race in https://github.com/EspressoSystems/nitro-espresso-integration/actions/runs/11705919237/job/32601803846?pr=254 * Update contracts submodule * Fix typo * Add wabt and cmake in wasm shell * Skip tests * Add tags in ci * Add ci skip tests * Add ci skip tests * Remove rerun-fails This flag in gotestsum causes some bugs which multiple the test time. gotestyourself/gotestsum#417 gotestyourself/gotestsum#423 * Revert "Remove rerun-fails" This reverts commit 6bef1c3. * CI: simplify - Add scripts to debug more easily locally. - Skip tests by substring match instead of exact match. - Don't run coverage on CI, we never look at it. - Move all Espresso tests to Espresso CI job. * Fix build with nix on macos (#296) * CI: install latest gotestsum This enables the --hide-summary argument * Update submodule forks with merge commits * Bump nitro-testnode from `07a25f1` to `77d1447` (#297) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `07a25f1` to `77d1447`. - [Commits](EspressoSystems/nitro-testnode@07a25f1...77d1447) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add FFI bindings for Espresso validation (#294) * Example call to espresso-crypto function stub in batch poster. This commit includes a log line and example call to the espresso-crypto function stubs in batch_poster.go * Result of `make fmt` Run `make fmt` to clean up formatting not apparent in editor * Make current lib.rs c abi compatible * result of make fmt * Add C ABI ffi between rust crypto lib and espressocrypto go module. This commit adds FFI support between exsisting Rust crypto helper code and the exsisting espressocrypto go module in native.go These functions will now serve as utility functions to use in the integration that will handle ffi to the Rust code. This commit also moves the espresso-crypto-helper code to the espressocrypto directory as we no longer need it for the arbitrator. * Build native.go successfully * Update the espresso-crypto and add test for merkle proof verification * update flake.nix to new go version and fix errors * Current state of work The current state of my work revolves around editing the Makefile to include the new espressocrypto ffi functions in the build process. Additionally, this commit restores some espressocrypto code to the arbitrator in an attempt to get a docker image built via the test node. I felt as though this would be easier for this PR as opposed to ripping out all of the code from the arbitrator. * Update the flake.lock * Add the namespace proof test * Current state of build system changes This commit represents the current state of the build system after testing today. Currently things compile locally, but not when building an image from the docker file. There are errors related to duplicate function definitions from what seem to be the FFI lib that stylus exposes. * Build nitro docker image sucessfully - Modify makefile to have a file target. Since the build artifact of our static library is a file we should use this as the make target (instead of only having PHONY targets). - Use rust 1.81 to build all shared rust libraries in docker. Our crate requires a minimum version of 1.81. However the other libraries like libstylus.a are currently compiled with 1.75 and this seems to cause issues with duplicate symbol definitions during linking. For example: 8.718 go build -ldflags="-extldflags '-ldl' -X github.com/offchainlabs/nitro/cmd/util/confighelpers.version= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.datetime= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.modified=" -o target/bin/nitro "/workspace/cmd/nitro" 42.74 # github.com/offchainlabs/nitro/cmd/nitro 42.74 /usr/local/go/pkg/tool/linux_amd64/link: running clang failed: exit status 1 42.74 /usr/bin/ld: /workspace/espressocrypto/lib/espresso-crypto-helper/target/release/libespresso_crypto_helper.a(std-1c4b19562077c20d.std.1f2242ed6435445e-cgu.0.rcgu.o): in function `__rdl_alloc': 42.74 /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/alloc.rs:397: multiple definition of `__rdl_alloc'; /workspace/execution/gethexec/../../target/lib/libstylus.a(std-90f6ddbf82de36ec.std.409886f6357001f0-cgu.0.rcgu.o):/rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/alloc.rs:389: first defined here Bump the rust version of the prover-header-builder to 1.81 and compile the espresso crypto library in the same container. * Fix LDFLAGS, remove unnecessary CFLAGS * Fix some merge errors - We need rust 1.81 to compile the espresso crypto lib - Re-add makefile rules * Fix merge errors * Remove FFI call in batch poster to expidite merge. * Update bf submodule in arbitrator to match upstream version (This was set to a different commit during my erroneous merge) * Remove extra change in espressocrypto_wasm.go * Fix error with temporary espressocryptowasm directory preventing CI from running * fix error of missing function definitions in espressocryptowasm directory to allow CI to run * Re-add nosec comment about gosec error that will be removed soon * result of make fmt --------- Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: ImJeremyHe <[email protected]> * Update the go client (#256) Still using the legacy dev node * ci: fix `git ls-remote` error in docker build (#304) * Use path context instead of git context With the default git context the repository is re-cloned at the start of the docker build and the .dockerignore file is ignored. > Be careful because any file mutation in the steps that precede the > build step will be ignored, including processing of the .dockerignore > file since the context is based on the Git reference. https://github.com/docker/build-push-action?tab=readme-ov-file#git-context This leads to some `.git` worktree files showing up inside the dockerbuild with triggers a bug in yarn. See yarnpkg/yarn#7537 (comment) for more details about the yarn bug. * Use the latest espresso dev node (#303) * Use the latest espresso dev node - Fix an issue that may cause the validation reorg - Waiting for L1 to be ready can avoid some potential bugs in dev node * Add proof validation to batch poster (#301) * Add espresso proof validation to batch poster Adds calls to the espresso validation code to the batch poster to validate the namespace proof and merkle proof in a TEE * result of make fmt * Fix lint errors * Add better logging * Fix batch poster after merge of upstream/integration * Remove espresso replay binary modifications (#307) * Remove espresso validation in the replay binary * Remove espresso-crypto related dependencies in arbitrator * remove espressocryptowasm * Update Makefile and Dockerfile to remove references to unused espresso crypto wasm lib * remove batch_poster.go.orig file * Fixes from make fmt * Remove espresso_ops_test * Remove additional code related to espresso validation in the staker * Remove changes on stateless block validator * Undo the stopwaiter Previous modifications on this was because we had to stop and start the seuquencers to function the escape hatch. Right now we don't need that anymore. These changes are from the upstream * Rename test flag --------- Co-authored-by: ImJeremyHe <[email protected]> * Add attestation quote in batch poster (#306) * Add attestation quote in batch poster * write file * update batch poster * use read file * update to userDataHash * reduce file permissions * nit * Submit multiple messages in one transaction (#308) * Submit multiple messages in one transaction * Remove the espresso_transaction_payload_signature_test.go * Update go client * Remove signature and data signer * Remove arbitrator/espresso-crypto-helper * Add size limit to hotshot transaction --------- Co-authored-by: Sneh Koul <[email protected]> * Use espresso-types instead of hard coded files * WIP: locally failed * Use upstream contracts submodule (#310) We no longer need the customizations in nitro-contracts (except potentially some deployment tools) so this commit resets it to the before the celestia contracts were added celestiaorg/nitro-contracts@4f8c65c and makes the nitro code work with that version by removing the hotshot / light client address passed to the OSP. * Remove the legacy code and fix the batch poster (#318) * Remove the legacy code and fix the test * fix nix build on linux - flake: add openssl, pkg-config - rust: update espresso-types to latest to not require testing features * nix: fix build on darwin - Update nixpkgs - Set NIX_LDFLAGS to find missing symbols - Leave note about updating flake since the apple SDKs in nixpkgs have been changed substantially. * Transmute nstable * add celestia-integration branch to the ci --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nomaxg <[email protected]> Co-authored-by: sveitser <[email protected]> Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: Sneh Koul <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sneh Koul <[email protected]>
Sneh1999
added a commit
to EspressoSystems/nitro-espresso-integration
that referenced
this issue
Dec 9, 2024
* Clean the espresso crypto lib (#317) * Add SwitchSequencer * Fix the switch function * Create SwitchSequencer * Update the espresso-sequencer-go * Use light client as the trigger * Add the switch test * Make the linter happy * Add the switch test to the e2e ci * Increase the limit * Clean the log with the log helper * more mock code * Fix the e2e test * serialize proof variables correctly * update espresso-go version * wasm compatibility * remove stale file, lint fix * update to recent espresso go version * refactor block merkle verification * work on E2E * progress on merkle tree verification * add test vectors, fetch proofs at the correct indicies * cleanup, remove outfiles, fetch merkle proof at the correct justified height * add logic to compare circuit field * check that circuit representation equals u256 representation in the test * send byte slice to wasm instead of byte array which lives on the stack * cleanup, remove debug statements, add documentation * remove error logs * fix build * lint fix * progress on e2e tests, comment out vid stuff for now because of wasm io issues * update espresso-sequencer-go tag, lint ci fix * cleanup, add issues for unresolved problems * remove block that spams transactions * comment out unused txns * add test for advz equivalence and use a smaller srs * Allow dead code in validation * Specify foundry version * Add random_get stub in wasi_snapshot_preview1 * Add stubs required in wasi_snapshot_preview1 * Fix `make lint` * Enable prover service to talk to L1 on host * Add builder timeout This allows the sequencer to make progress past block 1. These env vars have been changed in espresso-sequencer. * Add missing env var, more logging in e2e test * Revert "Fix `make lint`" This reverts commit 8be940c. * Fix go linting * Parse espresso messages * Add timeout limit * Fix the e2e test * Add timeout limit * Run the e2e test and switch test in 1 thread * Update the e2e workflow * Increase time limit * Specify the foundryup version in Dockerfile * Add sock_accept stub * Increase timeout limit * Build successfully * move ark-srs to dev dependencies * remove debug statements * remove out.txt * Fix the machine test * Fix CI * Fix switch test * Fix CI * Increase CI time limie * update lockfile * remove commented out replaces * newline * Use the latest foundry * RESET ME: debug * RESET ME: not run the CI in a container * RESET ME: remove the parallel * Update script to run tests locally - Run the same commands as on CI. - Add ability to run `cargo +nightly` to dev shell. - make clean: remove brotli build files. * set -parallel=2 * Fix e2e test * Skip some tests in CI * Skip tests in stylus * Remove parallelism restriction * Impl escape hatch on validator side * Fix tests * Generate the osp of hotshot liveness * Fix the fraud proof (#124) * Update contracts submodule * Upgrade go client * Fix the staker challenge tests --------- Co-authored-by: sveitser <[email protected]> * Use espresso dev node * attempt to fix dockerfile * newline * WIP * WIP * download latest espresso module root * incorporate vid common data * Fix the validation pipeline * update go modules, e2e build step * add cbrotli wasm * ci fix attempt * dont build all test deps for e2e test * lint fixes * Use the self-hosted runner and install cbindgen in CI * Add time limit * Remove parallel in building * Fix the switch test * Decrease the waiting hotshot blocks * Use the ubuntu * Specify the foundry version * Enable -j * update foundry version * module root update * Fix the staker test * increase validation timeout for switch test * pin espresso-sequencer-go to new release v0.0.20 * Commit go.sum from running go mod tidy * sequencer info logs * Add escape hatch test in e2e test * Check the reactivate hotshot * Update validator to handle preconditions and panics before entering the espresso STF logic. * Improve readability of conditions in `handleEspressoPreConditions` and update the functions comments. * Further improve readability * Fix lint issue * Update the nitro testnode * Fix tests * Update contracts * Fix CI * Changes to `handleEspressoPreConditions` in `espresso_validation.go` to fix incorrect logic, and give more granular control to where and how the generated panic is handled in the validators STF via returned closures. * return nil when we need a no-op panic handler * Fix lint errors * Update contracts * Fix the validation * Remove the switch test * Fix CI * Remove postgres in espresso e2e test * Codeowners: add Alysia, Zach, Sneh (#154) For #146 * Remove Dockerfile.espresso (#156) * Remove Dockerfile.espresso * address review comments * add new line * Release 20240723 with 20240723-consensus This should include all our changes for nitro v3.1. - Check that the module root matches when downloading releases from github. * Check all "not found" errors Currently only the leveldb not found errors are handled. If a pebble not found error is returned by the backend it is propagated, causing the batchposter to fail to send transactions. With this change all DB not found errors are handled the same way. * Submit L2 Message to espresso network (#173) * Submit L2 Message to espresso network * add context timeout * check for hotshot liveness and add a polling interval to see if transactions were included in an espresso block * cleanup * lint * Draft the sovereign sequencer test * Add espresso transaction queue * Draft the espresso submission * Set a new l2 message type for the sovereign sequencer transactions * Attempt to write messages again * Fix typos and add some comments * Fix typo * add header to jst and submit transaction to espresso * Update go client for development * Confirm the transaction inclusion in hotshot * Fix the test * Unblock the building blocks * Update go client * Fix the sovereign test * add logs * cleanup * lint * Updage go client * not to fetch merkle proof if hotshot height is 0 * Cleanup * Fix CI * Add CI skip tests * remove unwanted variable * skipping some tests to make ci pass * skipping some tests to make ci pass --------- Co-authored-by: ImJeremyHe <[email protected]> * Persistent storage for sovereign builder (#180) * Persistence storage for sovereign builder * add unit test * add unit test to ci and add mutex * Optional header in jst * initialize pending transactions if not present * remove unwanted test * resolve bugs * remove nil pointer panic * cleanup * revert to strings.contains * fix isErrNotFound --------- Co-authored-by: ImJeremyHe <[email protected]> * Save digest artifacts into own directory per run (#182) * Save digest artifacts into own directory per run Currently it seems that we have too many digests in the folder and I think this breaks the assembly of the multiarch docker image. With this change only digests for the same run are saved in the same directory. * Fail docker image merge on wrong number of digests * Clean digests dir * Add dependabot rule to update nitro-testnode (#189) * Fix build (#191) * Add missing dependency * Use renamed variable, fix typo * dependabot: update contracts submodule (#193) * Pin the golint version in CI * Fix pebble panic (#192) * Fix pebble panic * lint * Update some outdated help on flags (#203) * Improve help texts for flags * Update e2e tests to use Sovereign sequencer (#204) * Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]> * Remove the global state for HotShot height (#210) * Remove the global state * Update the submodule * CI: use buildjet instead of self-hosted runner (#212) * CI: use buildjet instead of self-hosted runner * CI: use public runner / buildjet for docker build This removes usage of our self-hosted github runners. * CI: more disk space on public runner * Check disk space at end of runs * Try running CI tests from different directory This may avoid failures due to IPC paths being too long * Make workspace dir a symlink to /tmp/gha-work * Fix redis URL * Move redis tests to own workflow * CI: fix rust and go cache (#217) * rust-cache: add more target dirs * Don't set custom TMPDIR This seems to be the culprit for the extremely long path names that are too long for IPC paths. * modify arbosstate.go to allow for upgrade to arbOS version 35 (#216) * modify arbosstate.go to allow for upgrade to arbOS version 35 * Fix lint * Modify the chain config during the ArbOS upgrade --------- Co-authored-by: sveitser <[email protected]> * CI: make more space (#218) * Store transaction pos in batch poster (#219) * Move submit transaction pos from tx_streamer to batch poster * Remove espresso sequencer (#214) * Remove espresso sequencer * lint * remove enable espresso * commit go-ethereum * fix solgen/gen.go * fix build * revert .gitmodules * revert go.mod * add enable espresso to validation check * re-add enable espresso to tests --------- Co-authored-by: Mathis <[email protected]> * Update espresso-sequencer-go with new LC ABI (#230) * Update espresso-sequencer-go with new LC ABI Instead of using the block height from the first snapshot after the height in the justification it uses the latest height in the LC contract. Currently not 100% convinced about correctness but also believe it should work because any newer tree still contains the header of interest. * Undo LC ABI changes We're not comfortable with the previous change because making a proof w.r.t to the newest finalized state may cause issues if the batch poster is lagging behind. This is currently not easy to test so we decided to revert back to the old flow. * Use tag of espresso-sequencer-go * Stop setting the chain config during ArbOS upgrade in favor of setting it manually during migration process. (#234) * Use the tag before marketplace update (#238) * Sign payload for sovereign builder (#181) * Sign payload for sovereign builder * save signed payload to db * change parsing logic to include signature * add some comments * skip tests * address comments * add signature payload test * test working * fix logging and comments * move placement of log * fix ci * fix ci * only submit transaction if jst nil * add log to debug ci * use the latest instead of main for espresso-dev-node * pin to legacy lc version * pin to before marketplace version * remove log * Full node with espresso finality (#237) * wip * wip * Add espresso finality node test * fix ci * fix ci * pin to before marketplace version * fix tests * add comments * address comments * address comments * address comments * cleanup * fix test * fmt * rerun ci * ci skip tests * Update to sequencer.go to remove `EnableEspressoSovereign` flag, and instead sequence based on hotshot liveness at l1 block height. * Update node.go to remove reference to `EnableEspressoSovereign` * update Espresso tests to reflect removed config variables, and add escape hatch test to e2e test. * Add wait for hotshot liveness in sovereign sequencer test to ensure sovereign sequencer is enabled. * go fmt * Add comment to explain changes needed in replay binary when panics are no longer an issue. * Fix lint issues * Modify construction of lightClientReader in sequencer.go to allow for nil `l1Reader` * Update test configs to align with changes. * Update comments according to feedback. * Bump nitro-testnode from `03095e0` to `884a63c` (#240) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `03095e0` to `884a63c`. - [Commits](EspressoSystems/nitro-testnode@03095e0...884a63c) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mathis <[email protected]> * Revert removal of `EnableEspressoSovereign` * Add additional check to ensure that if EnableEspressoSovereign is on we can't construct a sequencer with a nil lightClientReader * Add `EnableEspressoSovereign` config flag to test configs. * Bump contracts from `d226e9d` to `2ae1bbd` (#244) Bumps [contracts](https://github.com/EspressoSystems/nitro-contracts) from `d226e9d` to `2ae1bbd`. - [Release notes](https://github.com/EspressoSystems/nitro-contracts/releases) - [Commits](EspressoSystems/nitro-contracts@d226e9d...2ae1bbd) --- updated-dependencies: - dependency-name: contracts dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pin rust to 1.81.0 See #257 * Set cargo config for espresso_crypto wasm build This was recently added for the other wasm compilation steps. * Pin rust in E2E test * CI: remove deprecated rust toolchain action * Fix if statement * Regenerate data-poster external-signer test certs (cherry picked from commit 8f961b0) * Use ArbOS chain config as the cannonial chain config. (#252) * Add checks to sequencer.go and transaction_streamer.go to utilize the chain config switch * Update interface.go to export interface for `GetArbOSConfigAtHeight()` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionEngine` * Add implementation of `GetArbOSConfigAtHeight()` for `ExecutionNode` * Update `sequencer.go` and `transaction_streamer.go` to read from the `ArbOS` `chainConfig` as the cannonical `chainConfig` rather than the `chainConfig` held by the `Blockchain` struct. * Add ArbOS chain config check in batch poster to stop adding espresso messages to the transaction streamers DB if EnableEspresso is false. * Add missing error check in sequencer.go and nil check on arbos config in sequencer.go and batch_poster.go. * Remove ineffectual assignment in sequencer.go * Add test for new GetArbOSConfigAtHeight() method in ExecutionNode This commit adds an integration test for the GetArbOSConfigAtHeight() method on the Execution node that grabs the initial config, and validates it, then updates the config and ensures that the new method will fetch the updated config. * restore nitro-testnode to proper commit * Fix trie node node found error during recording I'm not exactly clear why we need to do this or if it's indeed the right approach to solving the issue however it does seem to make it possible to validate again. Without it we get errors like INFO [10-30|12:24:29.127] DelayedSequencer: Sequenced msgnum=1 startpos=1 INFO [10-30|12:24:30.143] Reading message result remotely. count=3 WARN [10-30|12:24:32.143] prepareblocks failed to get state for block hdrNum=0 err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" ERROR[10-30|12:24:32.143] Error while recording err="moved beyond genesis looking for state 0, genesis 0: missing trie node 147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 (path ) state 0x147a5a93c0202b07e7d7e3c936a36a8d97913056045be3af23b95b0aa0b356c7 is not available, not found" status=2 * Fix the OSP test - Enable 4844 blobs to avoid contract deployment error - Fix address of OSP entry * Disable batch poster in Espresso finality node We don't need this type of node to have a batch poster. Enabling it now causes the test to fail because there's a check to ensure that redis is used for the batch poster on the second node. * Re-allow setting L1 stack config for tests After the sync the code would always use the default values. Don't think this is very clean but I couldn't think of something better yet. * CI: increase parallelism * Fix linting - Run `golang-lint run --fix` - Check error after `waitForConfigUpdate` * Move finality node test to espresso tests The race test detected a data race in https://github.com/EspressoSystems/nitro-espresso-integration/actions/runs/11705919237/job/32601803846?pr=254 * Update contracts submodule * Fix typo * Add wabt and cmake in wasm shell * Skip tests * Add tags in ci * Add ci skip tests * Add ci skip tests * Remove rerun-fails This flag in gotestsum causes some bugs which multiple the test time. gotestyourself/gotestsum#417 gotestyourself/gotestsum#423 * Revert "Remove rerun-fails" This reverts commit 6bef1c3. * CI: simplify - Add scripts to debug more easily locally. - Skip tests by substring match instead of exact match. - Don't run coverage on CI, we never look at it. - Move all Espresso tests to Espresso CI job. * Fix build with nix on macos (#296) * CI: install latest gotestsum This enables the --hide-summary argument * Update submodule forks with merge commits * Bump nitro-testnode from `07a25f1` to `77d1447` (#297) Bumps [nitro-testnode](https://github.com/EspressoSystems/nitro-testnode) from `07a25f1` to `77d1447`. - [Commits](EspressoSystems/nitro-testnode@07a25f1...77d1447) --- updated-dependencies: - dependency-name: nitro-testnode dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add FFI bindings for Espresso validation (#294) * Example call to espresso-crypto function stub in batch poster. This commit includes a log line and example call to the espresso-crypto function stubs in batch_poster.go * Result of `make fmt` Run `make fmt` to clean up formatting not apparent in editor * Make current lib.rs c abi compatible * result of make fmt * Add C ABI ffi between rust crypto lib and espressocrypto go module. This commit adds FFI support between exsisting Rust crypto helper code and the exsisting espressocrypto go module in native.go These functions will now serve as utility functions to use in the integration that will handle ffi to the Rust code. This commit also moves the espresso-crypto-helper code to the espressocrypto directory as we no longer need it for the arbitrator. * Build native.go successfully * Update the espresso-crypto and add test for merkle proof verification * update flake.nix to new go version and fix errors * Current state of work The current state of my work revolves around editing the Makefile to include the new espressocrypto ffi functions in the build process. Additionally, this commit restores some espressocrypto code to the arbitrator in an attempt to get a docker image built via the test node. I felt as though this would be easier for this PR as opposed to ripping out all of the code from the arbitrator. * Update the flake.lock * Add the namespace proof test * Current state of build system changes This commit represents the current state of the build system after testing today. Currently things compile locally, but not when building an image from the docker file. There are errors related to duplicate function definitions from what seem to be the FFI lib that stylus exposes. * Build nitro docker image sucessfully - Modify makefile to have a file target. Since the build artifact of our static library is a file we should use this as the make target (instead of only having PHONY targets). - Use rust 1.81 to build all shared rust libraries in docker. Our crate requires a minimum version of 1.81. However the other libraries like libstylus.a are currently compiled with 1.75 and this seems to cause issues with duplicate symbol definitions during linking. For example: 8.718 go build -ldflags="-extldflags '-ldl' -X github.com/offchainlabs/nitro/cmd/util/confighelpers.version= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.datetime= -X github.com/offchainlabs/nitro/cmd/util/confighelpers.modified=" -o target/bin/nitro "/workspace/cmd/nitro" 42.74 # github.com/offchainlabs/nitro/cmd/nitro 42.74 /usr/local/go/pkg/tool/linux_amd64/link: running clang failed: exit status 1 42.74 /usr/bin/ld: /workspace/espressocrypto/lib/espresso-crypto-helper/target/release/libespresso_crypto_helper.a(std-1c4b19562077c20d.std.1f2242ed6435445e-cgu.0.rcgu.o): in function `__rdl_alloc': 42.74 /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/alloc.rs:397: multiple definition of `__rdl_alloc'; /workspace/execution/gethexec/../../target/lib/libstylus.a(std-90f6ddbf82de36ec.std.409886f6357001f0-cgu.0.rcgu.o):/rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/alloc.rs:389: first defined here Bump the rust version of the prover-header-builder to 1.81 and compile the espresso crypto library in the same container. * Fix LDFLAGS, remove unnecessary CFLAGS * Fix some merge errors - We need rust 1.81 to compile the espresso crypto lib - Re-add makefile rules * Fix merge errors * Remove FFI call in batch poster to expidite merge. * Update bf submodule in arbitrator to match upstream version (This was set to a different commit during my erroneous merge) * Remove extra change in espressocrypto_wasm.go * Fix error with temporary espressocryptowasm directory preventing CI from running * fix error of missing function definitions in espressocryptowasm directory to allow CI to run * Re-add nosec comment about gosec error that will be removed soon * result of make fmt --------- Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: ImJeremyHe <[email protected]> * Update the go client (#256) Still using the legacy dev node * ci: fix `git ls-remote` error in docker build (#304) * Use path context instead of git context With the default git context the repository is re-cloned at the start of the docker build and the .dockerignore file is ignored. > Be careful because any file mutation in the steps that precede the > build step will be ignored, including processing of the .dockerignore > file since the context is based on the Git reference. https://github.com/docker/build-push-action?tab=readme-ov-file#git-context This leads to some `.git` worktree files showing up inside the dockerbuild with triggers a bug in yarn. See yarnpkg/yarn#7537 (comment) for more details about the yarn bug. * Use the latest espresso dev node (#303) * Use the latest espresso dev node - Fix an issue that may cause the validation reorg - Waiting for L1 to be ready can avoid some potential bugs in dev node * Add proof validation to batch poster (#301) * Add espresso proof validation to batch poster Adds calls to the espresso validation code to the batch poster to validate the namespace proof and merkle proof in a TEE * result of make fmt * Fix lint errors * Add better logging * Fix batch poster after merge of upstream/integration * Remove espresso replay binary modifications (#307) * Remove espresso validation in the replay binary * Remove espresso-crypto related dependencies in arbitrator * remove espressocryptowasm * Update Makefile and Dockerfile to remove references to unused espresso crypto wasm lib * remove batch_poster.go.orig file * Fixes from make fmt * Remove espresso_ops_test * Remove additional code related to espresso validation in the staker * Remove changes on stateless block validator * Undo the stopwaiter Previous modifications on this was because we had to stop and start the seuquencers to function the escape hatch. Right now we don't need that anymore. These changes are from the upstream * Rename test flag --------- Co-authored-by: ImJeremyHe <[email protected]> * Add attestation quote in batch poster (#306) * Add attestation quote in batch poster * write file * update batch poster * use read file * update to userDataHash * reduce file permissions * nit * Submit multiple messages in one transaction (#308) * Submit multiple messages in one transaction * Remove the espresso_transaction_payload_signature_test.go * Update go client * Remove signature and data signer * Remove arbitrator/espresso-crypto-helper * Add size limit to hotshot transaction --------- Co-authored-by: Sneh Koul <[email protected]> * Use espresso-types instead of hard coded files * WIP: locally failed * Use upstream contracts submodule (#310) We no longer need the customizations in nitro-contracts (except potentially some deployment tools) so this commit resets it to the before the celestia contracts were added celestiaorg/nitro-contracts@4f8c65c and makes the nitro code work with that version by removing the hotshot / light client address passed to the OSP. * Remove the legacy code and fix the batch poster (#318) * Remove the legacy code and fix the test * fix nix build on linux - flake: add openssl, pkg-config - rust: update espresso-types to latest to not require testing features * nix: fix build on darwin - Update nixpkgs - Set NIX_LDFLAGS to find missing symbols - Leave note about updating flake since the apple SDKs in nixpkgs have been changed substantially. * Transmute nstable * add celestia-integration branch to the ci --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nomaxg <[email protected]> Co-authored-by: sveitser <[email protected]> Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: Sneh Koul <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sneh Koul <[email protected]> * Remove the oudated WASM binary (#329) * Add escape hatch (#330) * Clean flags (#335) * Checkout submodules recursively * Remove unnecessary flags in tx-streamer and sequencer * Remove redundant code (#340) * fix docker build (#338) * docker: install openssl, pkg-config Needed to compile openssl-sys crate. * ci: run amd64 docker build on PRs * Revert "ci: run amd64 docker build on PRs" This reverts commit 0974e29. * Avoid ARM build on PRs GitHub actions makes this really complicated. * ci: exclude: needs a list * symbol -> string * Fix conditional * Fix PR trigger * Update contracts submodule * Run yarn and forge install * Run yarn and forge install for espresso e2e as well * Run yarn and forge install for espresso e2e as well * fix e2e * fix tests * cleanup * comment out multiplatform build * fix tests * ci: skip multiplatform build on PRs * ci: fix indentation * ci: skip wavm extraction on PRs We don't push the image so we can't extract the wavm root. * ci: remove wavm machine release We are not planning to have an STF that differs from upstream. So this shouldn't be needed anymore. --------- Co-authored-by: Sneh Koul <[email protected]> * Use verifier address (#341) * WIP: use geth submodule * replace EnableEspresso flag into verifier address * Update geth submodule * Nit * Increase the poll interval * Fix arbos test * DelayThreshold * Stop processing if no verifier address in chain config * chores * Lint * Fix tests * Fix * Clean up (#351) * Clean up * Remove changes to parse_l2.go * Clean up * remove attestation log --------- Co-authored-by: Sneh Koul <[email protected]> * Fix logs for when espresso should not be enabled (#357) * Fix logs for when espresso should not be enabled * fix transacion_streamer * Add dockerfiles and other files needed to decode attestation quote (#358) * Update contract submodule * Update espresso dev node docker file * Remove unnecessary directory * Remove unnecessary changes * skip failing test --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nomaxg <[email protected]> Co-authored-by: sveitser <[email protected]> Co-authored-by: Zach Showalter <[email protected]> Co-authored-by: Sneh Koul <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sneh Koul <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version: from main
Unfortunately this is happening on a private repository, so I can't share a minimal example. If you do require one and the issue isn't obvious, I can work to make one.
Here's the command:
I've noticed that if I add
--hide-summary outputthis problem doesn't happen. It might be something to do with how the test results are being outputted (potentially a race condition between different processes there?)The text was updated successfully, but these errors were encountered: