decryptcore: update for OpenSSL 1.1 API

[emaste]

Summary

ERR_load_crypto_strings is deprecated in OpenSSL 1.1, and OpenSSL 1.1
generally does not require explicit initialization.  However, we do need
to ensure that initialization is done before entering capability mode so
call OPENSSL_init_crypto instead.  Also include header needed for
ERR_error_string.

Sponsored by:   The FreeBSD Foundation

[khorben_defora.org]

Are you sure that it is still necessary to call OPENSSL_init_crypto() before entering capability mode?
OpenSSL 1.1 seems to use getentropy(3) on FreeBSD >= 12, or arc4rand(9) otherwise (through sysctl).
Anyway, better safe than sorry.

[def]

sbin/decryptcore/decryptcore.c

185

It would be good to check a return value here.

185

Shouldn't it use OPENSSL_INIT_LOAD_CONFIG so that a configuration file is loaded before entering the sandbox?

[emaste]

sbin/decryptcore/decryptcore.c

185

Documentation says "As of OpenSSL 1.1.1 this is a default option for libssl". Do you think explicit OPENSSL_INIT_LOAD_CONFIG would make things clearer? (And if so, what about other options?)

[khorben_defora.org]

The documentation also says "This is not a default option for libcrypto." and this call is for libcrypto (OPENSSL_init_crypto() and not OPENSSL_init_ssl()) so @def is probably right.

[def]

sbin/decryptcore/decryptcore.c

185

My interpretation of the documentation was that it's used by default in libssl but in case you want to initialize crypto manually you have to do it yourself.

Also, decryptcore is linked with libcrypto and, if I understand correctly, this doesn't apply to our case here. However, I couldn't find where OPENSSL_init_crypto() gets called in libcrypto.

[khorben_defora.org]

In the case of dumpon (similar situation), I found that some initialisation code is in .init in libcrypto.so. I think it comes from ossl_init_base() which in turns calls OPENSSL_cpuid_setup() (caught in gdb below). If I understand it correctly, OPENSSL_init_crypto() makes sure ossl_init_base() has been called or calls it explicitly. But then OPENSSL_init_crypto() also does more than that, and I do not see it being called at all here.

# gdb dumpon
GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from dumpon...
Reading symbols from /usr/lib/debug//sbin/dumpon.debug...
(gdb) break OPENSSL_cpuid_setup
Function "OPENSSL_cpuid_setup" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (OPENSSL_cpuid_setup) pending.
(gdb) break OPENSSL_init_crypto
Function "OPENSSL_init_crypto" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (OPENSSL_init_crypto) pending.
(gdb) run -k test /dev/null
Starting program: /sbin/dumpon -k test /dev/null

Breakpoint 1, OPENSSL_cpuid_setup () at /usr/home/khorben/Projects/FreeBSD/src/crypto/openssl/crypto/cryptlib.c:104
warning: Source file is more recent than executable.
104         if (trigger)
(gdb) continue
Continuing.
[Detaching after fork from child process 45228]
dumpon: Unable to read data from test: error:0909006C:PEM routines:get_name:no start line
dumpon: genkey pipe read
[Inferior 1 (process 44661) exited with code 01]

[khorben_defora.org]

The documentation from OpenSSL 3 is more clear about the behaviour of OPENSSL_init_crypto() with regard to the OPENSSL_INIT_LOAD_CONFIG option:

With this option an OpenSSL configuration file will be automatically loaded and used by
calling OPENSSL_config(). This is a default option. Note that in OpenSSL 1.1.1 this was the
default for libssl but not for libcrypto (see OPENSSL_init_ssl(3) for further details about
libssl initialisation). In OpenSSL 1.1.0 this was a nondefault option for both libssl and
libcrypto. See the description of OPENSSL_INIT_new(), below.