[Bug]: avalanche blockchain addValidator is failed for a PoS L1 deployed to Fuji

[0xbarchitect]

Describe the bug
Our team at derachain are trying to leverage ACP-77 to create our PoS L1 blockchain. We have successfully created the blockchain using avalanche-cli, then deployed it to Fuji testnet. Once we tried to add the new validators to L1 on Fuji, it seems does not work as expected.

To Reproduce

1. Create the blockchain using CLI, selecting PoS as the chain type.
2. Deploy L1 to Fuji Testnet successfully.
3. Add new validators to L1 on Fuji, the logs is printed out as the transaction is succeeded.
4. Verify the validator set using CLI, it turns out as empty.

Expected behavior
The validator set is added successfully.

Screenshots

1. Blockchain config
   

• SubnetID 2L6CmVuASKTG99FKhkqbRpitvgP4p4aXh52bFPwhc1FQFNnF6t
• BlockchainID HLySu3YcULhhhW19VSeeLwKkrbujGBJ2tzmiNB7y5mRcQRKiV

2. Deployment to Fuji Testnet is succeeded and it is tracked on the Subnet-networks site https://subnets-test.avax.network/subnets/2L6CmVuASKTG99FKhkqbRpitvgP4p4aXh52bFPwhc1FQFNnF6t

3. The validator set is added using CLI command avalanche blockchain addValidator , we have added totally 5 validators, with weight equal 20 for each node. All the nodes is configured to track and fully bootstrapped the L1.

NODE1
Validator weight: 20
ValidationID: p3msdN2oiSPeiTYii8Mjv1DzkoVp798uHjFnmvc91G3z155i
RegisterL1ValidatorTx fee: 0.000099502 AVAX
RegisterL1ValidatorTx ID: 21eoE6tHNaMma5J7S39ayihVW4QU5pHo5zjMkEVeGvFH6Hqw3d

NODE2
Validator weight: 20
ValidationID: wpLhcSPJMTDBFtyb8PiFUE8CHcVnwD3M6gu1eVjk5aBvWfK9Y
RegisterL1ValidatorTx fee: 0.000099502 AVAX
RegisterL1ValidatorTx ID: 2HxztTo2xP85fadfM1JPmgyc9xXY8WGUbwHdvUuLKjjqDaoqAc

NODE3
Validator weight: 20
ValidationID: uE21rPFMqFbZy1TnbnWaGoMuPiUzt87iYrKXFp2WuTWav4HVu
RegisterL1ValidatorTx fee: 0.000093580 AVAX
RegisterL1ValidatorTx ID: 8KJzUpjSWakC7BbpjP8N86yyiFjhwFbuxMZPSTdmc4YktnBd7

NODE4
Validator weight: 20
ValidationID: 2aJu2G5Gbtdgu5s2Hh3Vtqeq9aD86miDC5CjhxqiW2c9uTpr44
RegisterL1ValidatorTx fee: 0.000093620 AVAX
RegisterL1ValidatorTx ID: TVtGp69Za3tmpmE9bwcUavwRhy3YGfNe5ZPhWTYs3SN188Thn

NODE5
Validator weight: 20
ValidationID: 67gYBXqmVTqeXweYzTnaAX9MWwq1hfMTYLyo3hWvSxQA2tyPz
RegisterL1ValidatorTx fee: 0.000093660 AVAX
RegisterL1ValidatorTx ID: jcmzk2UoRaHxF3RuB4NBew1rFDqhvgiJxdAwPi42GdQsQa6FL

4. Verify the validator set using CLI command avalanche blockchain validators returns empty
   

Logs
If applicable, please include the relevant logs that indicate a problem.
HLySu3YcULhhhW19VSeeLwKkrbujGBJ2tzmiNB7y5mRcQRKiV.zip

Operating System
Ubuntu 22.04

Additional context
Add any other context about the problem here.

[felipemadero]

Thanks for the issue. Please use the command avalanche validator list and check the commands under that command group avalanche validator . We will deprecated or adapt avalanche blockchain validators pretty soon.

[0xbarchitect]

Hi @felipemadero
Using the avalanche validator list command as you suggested, I retrieved the list as in the following screenshots.

I have recognized the 5 latter nodes in the list is exactly the 5 validators that I had added beforehand. But I got confused with the first node, the NodeID-111111111111111111116DBWJs, which has weight of 100 and zero remaining-balance. What is this node and how is it added to our L1 validator set?

Because there is an error that shown in the screenshots:

could not get balance for node NodeID-111111111111111111116DBWJs due to failed to decode client response: fetching L1 validator "11111111111111111111111111111111LpoYY" failed: not found

I worry this error will lead to a more significant problem for L1 operation.

Moreover, when I check the logs of the local bootstrap validator node, which is created during deployment of the L1 to Fuji testnet and resides in the folder ~/.avalanche-cli/local/deratest250102-local-node-fuji/node1 , I got this error:

[01-04|16:16:49.501] WARN health/worker.go:252 check started failing {"name": "health", "name": "HLySu3YcULhhhW19VSeeLwKkrbujGBJ2tzmiNB7y5mRcQRKiV", "tags": ["2L6CmVuASKTG99FKhkqbRpitvgP4p4aXh52bFPwhc1FQFNnF6t"], "error": "not connected to enough stake: connected to 50.000000%; required at least 100.000000%"}

Screenshots:

How to fix the error?