I think EVP_PKEY_bits(pubkey) would be more appropriate here (it's the actual bits of the RSA key). The security bits is the estimate of security strength equivalent and is not linear to key size.

This has to be in bits, not in bytes (regardless if we are requiring a security strength (the current code is doing) or a minimum size of key); using bytes is very confusing for readers as key sizes are always measured in bits.

The current code is using a security strength bits, as demanded in NIST SP800-57 (see reasoning in the comment below). Conceptually it's saying "This key is as strong as X bits symmetric cipher" regardless of the actual key size.

The proposed change would assume that the underlying key is a RSA key and requiring a minimal key size of 2048 bits (represented as 256 bytes), according to table 2 (page 54), the corresponding security strength is 112 bits, so it's equivalent to the current code.

I don't really have a strong opinion on which approach would be better here, but personally I prefer using the current one (use security bits instead of key length) because a) it's closer to the standard text (112 bits of strength acceptable up to 2030), and b) we don't have to change the code to support other public key encryption methods, because they would be mapped to the same strength bits table.

(see comment above; compare with bits not bytes. Personally I'd prefer using the security bits as we do today).

This should be EVP_PKEY_bits(pubkey), or EVP_PKEY_size(pubkey) * 8.

This is incredibly helpful/enlightening context; I'll digest this and mold the current code into what it needs to be.

I'll be sure to add references to the other NIST documents for further reading.

ERR_load_crypto_strings is deprecated as of OpenSSL 1.1; see D40353

Good catch!