cryptography: PKCS#7 EnvelopedData decryption exposes a Bleichenbacher oracle through distinguishable errors and timing

Summary

pkcs7_decrypt_der, pkcs7_decrypt_pem, and pkcs7_decrypt_smime reported the
outcome of decrypting a RecipientInfo's encryptedKey in several
distinguishable ways, one of which disclosed the exact length recovered from the
RSA operation. The same distinction was also observable by timing. An
application that decrypts attacker-supplied EnvelopedData and reflects the
outcome gives the attacker a Bleichenbacher oracle against the
content-encryption key.

Introduced in 44.0.0. Fixed in 50.0.0.

Details

Decryption ran as: RSA PKCS#1 v1.5 decrypt of encryptedKey → build an AES
cipher from the result → AES-CBC decrypt and PKCS#7 unpad. Each stage failed
differently, with no RFC 3218 mitigation:

1. invalid RSA padding → Decryption failed
2. valid padding, bad key length → Invalid key size (N) for AES., disclosing N
3. correct length, wrong key → Invalid padding bytes.
4. the real key → plaintext

Case 1 is reachable only where the linked library lacks implicit rejection:
OpenSSL 3.0 and 3.1, LibreSSL, and BoringSSL. On OpenSSL 3.2+, used in our wheels,
invalid padding instead returns a synthetic plaintext of
pseudorandom length, so the error channel does not distinguish conforming
ciphertexts.

Exploitation requires a service that auto-decrypts untrusted EnvelopedData
matching the victim certificate and answers adaptively at high volume, such as
an S/MIME gateway or mail filter.

Fix

Per RFC 3218, the content-encryption algorithm is now resolved before the
private key is used, so the expected key length is known in advance. If the RSA
decryption fails or recovers a key of the wrong length, a random key of the
expected length is substituted and decryption continues down an identical path.
All failures now report identically and perform the same work.

Not addressed by this fix

EnvelopedData does not authenticate its content. Tampering with
encryptedContent alone yields a CBC padding oracle that recovers plaintext at
roughly 256 queries per byte, without recovering any key, on every backend. This
is a property of PKCS#7 rather than of this implementation, cannot be fixed in
the library, and is now documented.

Credit

Reported by @X1AOxiang.