RedTrust KSP with TLS 1.2 Client Certificate

The EVOLIUM REDTRUST CSP is not capable of signing a non-hash. In other words, it’s not possible to do TLS 1.2 with client certificates with the RedTrust KSP.

This is a common problem in CSP/KSP implementations.  They assume that everything to be signed is a hash, such as SHA1, SHA256, SHA384, SHA512, and require the hash algorithm to be specified (even though the CSP is not hashing the data, but is simply receiving the already-hashed bytes to be signed), and they require the data length to be the size of the hash (20 bytes for SHA1, 32 bytes for SHA256, etc.).

The problem is that in the TLS 1.2 Protocol, the data to be signed for the CertificateVerify handshake message is NOT a hash.   It is a small bit of DER encoded ASN.1 that contains the hash.  For example:

In addition, the Microsoft NCrypt function to sign is named NCryptSignHash (see https://learn.microsoft.com/en-us/windows/win32/api/ncrypt/nf-ncrypt-ncryptsignhash ) and it implies only hashes will ever be signed.

But the little chunk of ASN.1 to be signed for the TLS 1.2 CertificateVerify is NOT a hash.  In fact, the above example is 35 bytes in length.

The NCryptSignHash’s pPaddingInfo.pszAlgId member  to can be set to 0 to satisfy some CSP/KSP implementations, which then allows for non-hashes to be signed.

But with RedTrust KSP, you’ll see the following error in the RedTrust KSP log:

Enter RtKSPSignHash, hProvider -> 452025600, hKey -> 454191712, cbHashValue -> 35, cbSignaturee -> 0, dwFlags -> 2
RtKSPSignHash: container: rt-TU3499N817XT7TJ-786D8686E774E7F9E99EC19EE8929BA695069DC1_524
RtKSPSignHash BCRYPT_PKCS1_PADDING_INFO
ERR RtKSPSignHash: bCRYPT_PKCS1_PADDING_INFO->pszAlgId is null
RtKSPSignHash: hash in base64 to sign -> MCEwCQYFKw4DAhoFAAQU7KfMfMzVFt8ZQWrwiQ2ZRqA3FyY=
RtKSPSignHash: Call API NCryptSignHash. Ask for lenght.

ERR RtKSPSignHash: Call API NCryptSignHash return fail.

Solution:

This is only a problem with TLS 1.2.  The content and format of the CertificateVerify message changed from TLS 1.1 to TLS 1.2 to TLS 1.3.   In both TLS 1.1 and TLS 1.3, it is truly a hash that is signed, and therefore this problem does not exist.  If the server is TLS 1.3 capable, then it will be the version automatically agreed upon between Chilkat and the server (the TLS client indicates what capabilities it has, and the server chooses).