Some problem when testing the forceBatch

[Federico2014]

System information

zkEVM Node version: v0.6.0
OS & Version: OSX
Network: private testnet

I find some timestamp and proof verficatoin problems when testing the forceBatch. Here I will describe the process as explicitly as possible.
Here, batches 4-5 are ground into a sequence batch, where only batch 6 is forced batch.

After successful sequencing by the on-chain contract, the problems emerged when the synchronizer executed the batch 5:

2024-06-13T18:07:39.398+0800    WARN    etrog/processor_l1_sequence_batches.go:373      Batch: 5. Different field StateRoot. 
       Virtual: 0x43d607833c18fe91a85ad6b0cbdd782148f18cbbdcef481010d6c17ef471134b, 
       Trusted: 0x6acef6cc2f3820f308578296c7afc16e2225066386590f404696fc4a6cdb5155

Then I debugged the problem, I found the executor causes it when the synchronizer executes batch 5:

processBatchResponse, err := s.executorClient.ProcessBatchV2(ctx, processBatchRequest)

The processBatchResponse has the Rom error as follows, which seems to be caused by the timestamp.

Then I analyzed the timestamp for batches 4-6 in the last three rows.

And the forced batch timestamp is :

I find the timestamp used when the synchronizer processes batch 4 and 5, is always the forced batch timestamp: 2024-06-13 09:59:36+00, which is caused by the sequenceSender:

zkevm-node/sequencesender/sequencesender.go

Line 244 in a39edda

to, data, err := s.etherman.BuildSequenceBatchesTxData(s.cfg.SenderAddress, sequences, uint64(lastSequence.LastL2BLockTimestamp), firstSequence.BatchNumber-1, s.cfg.L2Coinbase)

lastSequence.LastL2BLockTimestamp is the forceBatch timestamp because:

zkevm-node/sequencesender/sequencesender.go

Line 331 in a39edda

seq.LastL2BLockTimestamp = seq.ForcedBatchTimestamp

The batch 4 timestamp 2024-06-13 09:41:27 is smaller than the forced batch timestamp 2024-06-13 09:59:36, so it can be synchronized successfully. But for batch 5, the timestamp is 2024-06-13 10:04:56, which is greater than the forced batch timestamp 2024-06-13 09:59:36, causing the executor romError when synchronizing.
Then I adjusted the timestamp by modifying the following code from

zkevm-node/synchronizer/actions/elderberry/processor_l1_sequence_batches.go

Line 73 in a39edda

err = g.previousProcessor.ProcessSequenceBatches(ctx, l1Block.SequencedBatches[order.Pos], l1Block.BlockNumber, time.Unix(int64(sbatch.SequencedBatchElderberryData.MaxSequenceTimestamp), 0), dbTx)

to

err = g.previousProcessor.ProcessSequenceBatches(ctx, l1Block.SequencedBatches[order.Pos], l1Block.BlockNumber, time.Unix(l1Block.ReceivedAt.Unix(), 0), dbTx)

then batch 5 is synchronized successfully. But for the forced batch 6, the synchronizer shows an error again, it seems the executor mistakes the force batch as normal batch, I find there is a bug here:
https://github.com/0xPolygonHermez/zkevm-node/blob/a39eddac16fc9623860eddce8b1f457342f3a3ba/synchronizer/actions/etrog/processor_l1_sequence_batches.go#L157

The forcedBlockHashL1 value is not assigned yet. So I added the following code in line 151:

			var fBHL1 common.Hash = sbatch.PolygonRollupBaseEtrogBatchData.ForcedBlockHashL1
			forcedBlockHashL1 = &fBHL1

Then batch 6 is successfully synchronized.

After batches 4-6 are successfully synchronized as virtual batches, the aggregator can generate the proof for batches 4-6. But when submitting the proof to the on-chain verifier contract, it failed the verification as invalid proof.
https://github.com/0xPolygonHermez/zkevm-contracts/blob/1ad7089d04910c319a257ff4f3674ffd6fc6e64e/contracts/v2/PolygonRollupManager.sol#L1050

I don't know the reasons why the proof failed. I guess we'd better not group the normal batch and force batch together, which will avoid timestamp and proof verification failure proof. so I add the code snippet in sequenceSender module:

I redeployed the new testnet and tested the forced batch, made the forced batch sequenced and proved alone, and then the whole process worked successfully.

Furthermore, after the sequencer processes the forced batch, the sequencer will show the following error and halt when processing the next batch.

I don't know the reasons. After I restarted the sequencer, it will work normally. But after processing the next force batch, the error shows again.

[gimersettle]

Hi!

I have been working on a local implementation of the ZKEVM project, and I need to test the forceBatches mechanism. I'm trying to sign and serialize a transaction using ethers.js v6. The transaction was processed and accepted by the contract, but after a few minutes, I got an error because the transaction was not properly deserialized. Do you have any suggestions? This is the script:

const tx = {
to: "0x1234567890123456789012345678901234567890",
nonce: 8,
gasPrice: ethers.parseUnits('200', 'gwei'),
gasLimit: 600000,
value: ethers.parseUnits('10', 'ether'),
chainId: 1001
}

const signedTx = await signer.signTransaction(tx);

const sequencesArray = [signedTx];
const txRaw = ethers.concat(sequencesArray);

const forceTx = await plasmaFree.forceBatch(txRaw, batchFee);
await forceTx.wait();

This is the error that I got from the Sequencer:

2024-08-12 14:49:09 {"level":"error","ts":1723495749.8932047,"caller":"sequencer/forcedbatch.go:47","msg":"error when processing forced batch 1, error: failed to process/execute forced batch 1, error: invalid RLP%!(EXTRA string=\n/home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/home/runner/work/cdk-validium-node/cdk-validium-node/sequencer/forcedbatch.go:47 github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).processForcedBatches()\n/home/runner/work/cdk-validium-node/cdk-validium-node/sequencer/batch.go:220

Am I serializing the Tx properly? Do you have any suggestions?

[Federico2014]

Hi!

I have been working on a local implementation of the ZKEVM project, and I need to test the forceBatches mechanism. I'm trying to sign and serialize a transaction using ethers.js v6. The transaction was processed and accepted by the contract, but after a few minutes, I got an error because the transaction was not properly deserialized. Do you have any suggestions? This is the script:

const tx = { to: "0x1234567890123456789012345678901234567890", nonce: 8, gasPrice: ethers.parseUnits('200', 'gwei'), gasLimit: 600000, value: ethers.parseUnits('10', 'ether'), chainId: 1001 }

const signedTx = await signer.signTransaction(tx);

const sequencesArray = [signedTx];
const txRaw = ethers.concat(sequencesArray);

const forceTx = await plasmaFree.forceBatch(txRaw, batchFee);
await forceTx.wait();

This is the error that I got from the Sequencer:

2024-08-12 14:49:09 {"level":"error","ts":1723495749.8932047,"caller":"sequencer/forcedbatch.go:47","msg":"error when processing forced batch 1, error: failed to process/execute forced batch 1, error: invalid RLP%!(EXTRA string=\n/home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/home/runner/work/cdk-validium-node/cdk-validium-node/sequencer/forcedbatch.go:47 github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).processForcedBatches()\n/home/runner/work/cdk-validium-node/cdk-validium-node/sequencer/batch.go:220

Am I serializing the Tx properly? Do you have any suggestions?

You can refer the script: initialClaim to submit the forceBatch.

[gimersettle]

You can refer the script: initialClaim to submit the forceBatch.

Thank you! We solved the problem by using the javascript utils in this repo, the function rawTxToCustomRawTx(...) formats the transaction to the Polygon's format.

[gimersettle]

Hi @Federico2014, did you manage to solve this issue? I have the same problem: forced transactions are not processed, and the synchronizer fails to verify them. Do you have any thoughts?

[Federico2014]

Hi @Federico2014, did you manage to solve this issue? I have the same problem: forced transactions are not processed, and the synchronizer fails to verify them. Do you have any thoughts?

I have solved the problem described above. But I have no idea about yours, they seem not alike.

[gimersettle]

Hi @Federico2014, The problem was similar. The aggregator fails when a sequence has both regular and forced batches. I implemented your suggestion, and it worked! One more question, do you know the parameters that we need to change in order to speed up the process of sequencing forced batches?

[Federico2014]

Hi @Federico2014, The problem was similar. The aggregator fails when a sequence has both regular and forced batches. I implemented your suggestion, and it worked! One more question, do you know the parameters that we need to change in order to speed up the process of sequencing forced batches?

@gimersettle It is determined by the ForcedBatchesTimeout parameter.

func (f *finalizer) setNextForcedBatchDeadline() {
	f.nextForcedBatchDeadline = now().Unix() + int64(f.cfg.ForcedBatchesTimeout.Duration.Seconds())
}