Proposal: Add opcode for verifying Ledger signatures with 512-bit dataHash

[aleksandr-hramcov]

Problem

Ledger devices (due to internal security policies) do not allow signing arbitrary raw data directly. To mitigate this, the Ledger app prepends 0xFFFFFFFF (4 bytes) to the data before signing.

Relevant code:
https://github.com/broxus/ledger-app-everscale/blob/master/src/handler/sign.c#L10

static const char SIGN_MAGIC[] = {0xFF, 0xFF, 0xFF, 0xFF};
// ...
memcpy(context->to_sign, SIGN_MAGIC, SIGN_MAGIC_LENGTH);
memcpy(context->to_sign + SIGN_MAGIC_LENGTH, cdata->ptr + cdata->offset, TO_SIGN_LENGTH);

As a result, instead of signing a pure 256-bit hash (uint256), the device signs a 512-bit payload.

However, in the Tycho VM (as well as the original TON TVM), signature verification using CHKSIGNU only supports 256-bit data inputs:

From TVM Solidity docs (https://github.com/broxus/TVM-Solidity-Compiler/blob/master/API.md#tvmchecksign):

tvm.checkSign(uint256 dataHash, uint256 signHi, uint256 signLo, uint256 pubkey) returns (bool);
tvm.checkSign(uint256 dataHash, TvmSlice signature, uint256 pubkey) returns (bool);
tvm.checkSign(TvmSlice data, TvmSlice signature, uint256 pubkey) returns (bool);

This makes it impossible to verify signatures made on Ledger inside a smart contract.

Moreover, externalIn messages signed with Ledger are also rejected by the network because the signature does not match the message hash (uint256 vs uint512 mismatch).

Proposed solution

Introduce a new TVM opcode (or modify the existing CHKSIGNU) to allow signature verification over a 512-bit hash.

Two options:

1. Add a dedicated opcode, e.g., CHKSIGNU512, to verify signatures from a 512-bit data hash.
2. Modify CHKSIGNU to support both uint256 and uint512 inputs, and internally try verifying with and without the 0xFFFFFFFF prefix.

[rualekseev]

Done.
сommit: cd2dd8fd00de9911ec83327ea2e270180b7e4093

[rualekseev]

Reopened.
The opcodes:

• ED25519_CHKSIGNU
• ED25519_CHKSIGN_EXT
  were removed due to conflicts with new opcodes in TON VM.
  Commit: 76f7250b8ffddacc1bcbb795dd1c8a52f13b411e