transaction can't be replaced, mining has already started

[MatthiasLohr]

Hi,

ganache (v7.0.3) is telling me to open an issue here:

Error: transaction can't be replaced, mining has already started. (please open an issue with reproduction steps: https://github.com/trufflesuite/ganache/issues/new)\n    at shouldReplace (/app/dist/node/1.js:2:129092)

Any idea?

Best regards
Matthias

[davidmurdoch]

Can you provide reproduction steps?

[MatthiasLohr]

Actually, not really. I'm running a script of mine (written in Python, using web3py library) which is a simulation of something over and over again and in (roughly estimated) 1 of 250 executions I'm seeing this error.

I have running two processes interacting with Ganache, in alternating roles, sending transactions to Ganache respectively the other process waiting to see that transaction.

I'm running this kind of simulation within a minikube Kubernetes cluster, with each Pod having its own two client processes and Ganache instance.

What can I provide to help to locate this issue?

[davidmurdoch]

This issue happens when Ganache receives a transaction with a nonce that matches the nonce of a transaction that is already in the process of being mined.

If you aren't filling in the nonce yourself, and you are letting Ganache generate it, then this could be due to this issue: #2489.

If you are filling in the nonce yourself then the bug is likely on your end, if you aren't, then it would seem that this is a Ganache bug.

[MatthiasLohr]

I did some digging and found the following: I'm filling nonce myself and transaction is sent in a retry loop using eth_sendRawTransaction. The error above occurs when the retry loop tries to send the transaction the second time. However, for the first try, I'm getting an error such as "tx is not in the chain after 120 seconds" (where 120 seconds is default timeout by web3py) but ganache does not give any reason or hint why the transaction is neither failing not accepted... any idea?

[davidmurdoch]

Before running the retry can you try checking if the transaction is in the transaction pool by calling the RPC method txpool_content?

[MatthiasLohr]

Similar to this:

{
	'pending': {
		'0x598ec01a78be6945e5cb4c0451c5cf185211e96d': {
			'2': {
				'type': '0x2',
				'hash': '0x40308dd3a4f9510cd333b8a77f30783a2f4172c0d228e89cfe991d3761d98dc7',
				'chainId': '0x539',
				'nonce': '0x2',
				'blockHash': None,
				'blockNumber': None,
				'transactionIndex': None,
				'from': '0x598ec01a78be6945e5cb4c0451c5cf185211e96d',
				'to': '0xda88fe680c2a8fb454817fba3d0ea2549939001a',
				'value': '0x0',
				'maxPriorityFeePerGas': '0x3b9aca00',
				'maxFeePerGas': '0x3b9b6180',
				'gasPrice': '0x3b9b0419',
				'gas': '0xc165',
				'input': '0x4ed4283cde8963f17b437df08278d6b7cd11063b8ef5641e7a5fca72e5e2bd27e46f1aae000000000000000000000000598ec01a78be6945e5cb4c0451c5cf185211e96d000000000000000000000000598ec01a78be6945e5cb4c0451c5cf185211e96d000000000000000000000000000000000000000000000000000000007735940000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000004115b9e1ce726eb09dde3e8ed6fed1ac804df1230f2e007ad7b5a96aa8923e79a666491698be212c9d3e240271fed4485cdff1697ba5e777116fe0c573bc5408e31b00000000000000000000000000000000000000000000
000000000000000000',
				'accessList': [],
				'v': '0x1',
				'r': '0xbf5c6f097a17a4ed4632d263bad0f4870a1de1b3fa3473fa4b095cf1d9518e4e',
				's': '0x6daf49d178fb4daa5f36a3624c386c1e38fbcaa84922e0791eabb1b2d5227b13'
			}
		}
	},
	'queued': {}
}

[MatthiasLohr]

Another one:

{
	'pending': {
		'0x65a3a53e899601d0c49c668bb1a7bc06a5b23f35': {
			'9': {
				'type': '0x2',
				'hash': '0x1a1c9219b16d69b8ba2cd4e6106615db59c6bc780875d2782bd903af18ddbde0',
				'chainId': '0x539',
				'nonce': '0x9',
				'blockHash': None,
				'blockNumber': None,
				'transactionIndex': None,
				'from': '0x65a3a53e899601d0c49c668bb1a7bc06a5b23f35',
				'to': '0xda88fe680c2a8fb454817fba3d0ea2549939001a',
				'value': '0x0',
				'maxPriorityFeePerGas': '0x3b9aca00',
				'maxFeePerGas': '0x3b9dbeee',
				'gasPrice': '0x3b9bec43',
				'gas': '0xc165',
				'input': '0x4ed4283ccd70955b00f5f72689ebd8976d5cee78a3a2a6ea0b366bbb5e08e75a8511c30a00000000000000000000000065a3a53e899601d0c49c668bb1a7bc06a5b23f350000000000000000000000
0065a3a53e899601d0c49c668bb1a7bc06a5b23f35000000000000000000000000000000000000000000000000000000003b9aca0000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000
000000000000000000000000000041602ff4ab7d5872f79d69316fd617b5c127632b8c4c4be1cea785d8ea1a5c029a72082dd6fe911bcca2e03d74445eff7c9365cd58611422924a6339dc982b0e541b00000000000000000000000000000000000000000000
000000000000000000',
				'accessList': [],
				'v': '0x0',
				'r': '0x5232b9fa002ea4289d6e351236f6fcc4e638ff36e12475c370dc4c56ceb4dcb4',
				's': '0x2053b1c2620f1eb080b76efcfaaa93a68340724505da093ef69361774e0e5bac'
			}
		}
	},
	'queued': {}
}

[davidmurdoch]

Are these two transactions the transactions that are being retried?

[MatthiasLohr]

Yep. These two have not been "mined" or "sealed" after 60 seconds (despite having --miner.blockTime=1.

These logs are written before the retry was sent. This is 60 seconds after submitting it the first time.

[davidmurdoch]

Can you also output eth_getTransactionCount for the from account for the timed-out transaction?

[MatthiasLohr]

Which block identifier should I use? latest or pending?

[davidmurdoch]

latest

[MatthiasLohr]

Ok, I updated the code (sorry, this takes a while and then I have to run it again until the error occurs again).

The result from eth_getTransactionCount equals to the key in the dict an the nonce (at least in the first two errors occured).

[davidmurdoch]

Hm, I was hoping it would be as simple as the nonce of the transaction being too high. Is it possible to run this with the ganache flags set to --miner.blockTime 0 --miner.instamine=eager --chain.vmErrorsOnRPCResponse. These settings will result in an Error being returned for calls to eth_sendRawTransaction for transactions that fail.

[MatthiasLohr]

Sorry for the delay, but for now, with these options, it looks like the error is not showing up... will try again...

[davidmurdoch]

It must be the blocktime option that is causing the problem. We're investigating a race condition issues right now and this additional information will help. Thanks!

Possibly related: #2489

[teknoxic]

Any update on this?
Im still having the issue i made here:
Transactions can't be replaced, mining has already started #2680

Which we think is the same as this issue.
It causes problems for applications that use ganache-cli when a single user makes multiple transactions in quick succession. (which is quite common)

[MicaiahReid]

@teknoxic I have a PR that I should be able to pick back up before too long, #3498. I think this PR will likely fix the issue. If you are able to provide specific reproduction steps, I can confirm that this PR will fix it.

I don't have a specific timeline currently, but I am aware that this is a problem and will try to get it done ASAP.

[davidmurdoch]

@teknoxic you might be able to work around this for now by setting --chain.asyncRequestProcessing=true when starting ganache.