bug(v1.1.x): whole x/bank store lost after block commit

[beer-1]

Hey we have a chain using cosmos@v0.50 and iavl@v1.1.4, but we have encountered wired situation.

Pruning: default
App chain: https://github.com/initia-labs/minievm/tree/v0.6.6-5
State snapshot:

• https://storage.cloud.google.com/minitia-temporary/bfb-snapshot-data.tar.gz
• gs://minitia-temporary/bfb-snapshot-data.tar.gz

All data under "s/k:bank/" have been removed after height 4876147 commit.
You can find the 4876147 block here. It only contains normal ibc transfer message. The tx is registering a new denom metadata pairs on x/bank and suspecting this write makes some problems on our iavl tree.

To prevent other doubt, we don't have params delete or metadata delete functions like normal cosmos chain.

Query Result

Height: 4876146

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/params -H 'x-cosmos-block-height: 4876146' | jq

{
  "params": {
    "send_enabled": [],
    "default_send_enabled": true
  }
}

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/denoms_metadata -H 'x-cosmos-block-height: 4876146' | jq

{
  "metadatas": [
    {
      "description": "OPinit token of uinit",
      "denom_units": [
        {
          "denom": "uinit",
          "exponent": 0,
          "aliases": []
        }
      ],
      "base": "l2/83db4af0de8965c851672915eaa857f0911385f7d71f9da1decb2504d3d3ffdd",
      "display": "uinit",
      "name": "uinit OPinit token",
      "symbol": "uinit",
      "uri": "",
      "uri_hash": ""
    }
  ],
  "pagination": {
    "next_key": null,
    "total": "1"
  }
}

Height: 4876147

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/params -H 'x-cosmos-block-height: 4876147' | jq

{
  "params": {
    "send_enabled": [],
    "default_send_enabled": false
  }
}

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/denoms_metadata -H 'x-cosmos-block-height: 4876147' | jq

{
  "metadatas": [],
  "pagination": {
    "next_key": null,
    "total": "0"
  }
}

At the above heights, app chain was not broken, but broken at height 4876606 and around that height our nodes return denom metadata not found but other node denom metadata exist. new denom metadata addition happened on our node block, but not on broken node.

not sure why removed params now accessible after new denom metadata addition.

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/params --height=4876605
{
  "params": {
    "send_enabled": [],
    "default_send_enabled": false
  }
}

$ curl https://rest.culinaris-2.initia.tech/cosmos/bank/v1beta1/params --height=4876606
{
  "params": {
    "send_enabled": [],
    "default_send_enabled": true
  }
}

iaviewer result

Height: 4876146

$ iaviewer data ./application.db "s/k:bank/" 4876146

Got version: 4878188
Printing all keys with hashed values (to detect diff)
  016C322F38336462346166306465383936356338353136373239313565616138353766303931313338356637643731663964613164656362323530346433643366666464
    8974521987DEC91845EFC3418CE57FDEB800F1544E3201FB1F0BD1A99CEDA6AE
  05
    27C24FCB8474773E2AF799D0848495FF053272D33C432DC26277993DF45C9276
Hash: 50AF2AC87FB93C4D364242018B284CBA9F2B746C6E199FC1B2761C08789B51A1
Size: 2

Height: 4876147

$ iaviewer data ./application.db "s/k:bank/" 4876147

Got version: 4878188
Printing all keys with hashed values (to detect diff)
Hash: E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
Size: 0

Height: latest block

• DenomsMeta are re-stored by app chain logics (ibc transfer)
• Params only exists in latest height

$ iaviewer data ./application.db "s/k:bank/" 4878187

Got version: 4878188
Printing all keys with hashed values (to detect diff)
  016962632F46303832423635433838453442364435454631444232343343444131443333314430303237353945393338413046354344334646444335443533423345333439
    06E1CA4F35A12BB9D9ED1D0E1E7DE52601B289C7F6DD8C42D0586DC0D3B82ECF
  016C322F38336462346166306465383936356338353136373239313565616138353766303931313338356637643731663964613164656362323530346433643366666464
    8974521987DEC91845EFC3418CE57FDEB800F1544E3201FB1F0BD1A99CEDA6AE
  05
    27C24FCB8474773E2AF799D0848495FF053272D33C432DC26277993DF45C9276
Hash: 42EF3E298658B787D2C4C27F4554C98C509EA2D9C4DDF8F551CDDF0F10EDA6CB
Size: 2

Height: latest block - 1

Still lost params

$ iaviewer data ./application.db "s/k:bank/" 4878187

Got version: 4878188
Printing all keys with hashed values (to detect diff)
  016962632F46303832423635433838453442364435454631444232343343444131443333314430303237353945393338413046354344334646444335443533423345333439
    06E1CA4F35A12BB9D9ED1D0E1E7DE52601B289C7F6DD8C42D0586DC0D3B82ECF
  016C322F38336462346166306465383936356338353136373239313565616138353766303931313338356637643731663964613164656362323530346433643366666464
    8974521987DEC91845EFC3418CE57FDEB800F1544E3201FB1F0BD1A99CEDA6AE
Hash: 42EF3E298658B787D2C4C27F4554C98C509EA2D9C4DDF8F551CDDF0F10EDA6CB
Size: 2

[beer-1]

seems this PR is related and not back ported to release/v1.1.x

we saw the logs

1:05PM INF receive packet callback succeeded channel-id=channel-2415 module=server port-id=transfer result=RESPONSE_RESULT_TYPE_SUCCESS
1:05PM ERR iavl set error error="Value missing for key [0 0 0 0 0 0 0 1 0 0 0 1] corresponding to nodeKey 73000000000000000100000001" module=server