SSL_ERROR_SIGN_HASHES_FAILURE with TLS 1.2 on Win10

[schiermi]

I try to use p11capi_w64.dll with Windows 10 64 bit and Firefox 67.0.4.
When the server requests the certificate the selection dialog pops up as configured. After selecting the users only available certificate firefox displays an SSL error page with the error code: SSL_ERROR_SIGN_HASHES_FAILURE.

Firefox selected cipher from servers available cipher list: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

[schiermi]

debug output:
00045753 57.82608032 [4892] S9: registered sesson id
00045754 57.82609940 [4892] C_OpenSession: 0
00045755 57.82611465 [4892] C_SignInit: enter
00045756 57.82613373 [4892] S9: found and locked session
00045757 57.82637405 [4892] copying container name 67ecd2b999046a689e7515264b0caa68_d766644d-7784-49a4-a71b-29cfc50bfb65
00045758 57.82640457 [4892] S9: unlocked session
00045759 57.82642365 [4892] C_SignInit: 0
00045760 57.82643509 [4892] C_GetAttributeValue: enter
00045761 57.82645416 [4892] S1: found and locked session
00045762 57.82647324 [4892] S1: unlocked session
00045763 57.82648468 [4892] C_GetAttributeValue: 0
00045764 57.82649612 [4892] C_Sign: enter
00045765 57.82651520 [4892] S9: found and locked session
00045766 57.82733154 [4892] Container name 67ecd2b999046a689e7515264b0caa68_d766644d-7784-49a4-a71b-29cfc50bfb65; provider name Microsoft Enhanced Cryptographic Provider v1.0
00045767 57.82738876 [4892] parse_rsa_pkcs_der_hash at ../src/p11-capi-rsa.c, line 88
00045768 57.82740402 [4892] n_input 51
00045769 57.82742691 [4892] input: 3031300D060960864801650304020105000420BB0B480D9A9F333B001E371149F4DF223EF0AED74357303635D9D27E016627F8
00045770 57.82744598 [4892] oid_str: 2.16.840.1.101.3.4.2.1
00045771 57.82884598 [4892] response form CertOIDToAlgId: 0xffffffff
00045772 57.82885742 [4892] forcing algorithm to CALG_SHA_256 (0x800c)
00045773 57.82950592 [4892] checkpoint at ../src/p11-capi-rsa.c, line 282
00045774 57.82951736 [4892] algorithm 0x800C
00045775 57.82954025 [4892] capifail 80090008 at ../src/p11-capi-rsa.c, line 327
00045776 57.82955933 [4892] checkpoint (ret = 32) at ../src/p11-capi-session.c, line 886
00045777 57.82957077 [4892] S9: unlocked session
00045778 57.82958984 [4892] C_Sign: 32
00045779 57.82960129 [4892] C_CloseSession: enter
00045780 57.82962036 [4892] S9: found and locked session
00045781 57.82963181 [4892] S9: removed session from list
00045782 57.82964325 [4892] S9: unlocked session
00045783 57.82968140 [4892] S9: destroyed

[schiermi]

It seems to me that PROV_RSA_FULL in Windows 10 is finally not capable/willing to handle CALG_SHA_256 due to error 80090008.

I found some C# code where the key associated to a PROV_RSA_FULL provider is re-used with PROV_RSA_AES:
https://referencesource.microsoft.com/#System.IdentityModel/System/IdentityModel/Tokens/X509AsymmetricSecurityKey.cs,434

Unfortunately I'm not savvy enough to translate this into C.