security/ca_root_nss: Add a ca-merge utility to permit including private CAs

[feld]

Summary

This is a Proof of Concept which provides a ca-merge utliity in the
ca_root_nss package. An update to ftp/curl is included. Explanation
below:

This is roughly inspired by the RHEL utility update-ca-trust(8)
https://www.unix.com/man-page/centos/8/update-ca-trust/

Open PRs: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160387 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229329

TL;DR: Put your CAs in %%PREFIX%%/etc/ssl/ca-trust/source/anchors and run ca-merge; test your apps

The overall design is as follows:

1. /usr/local/share/certs/ca-root-nss.crt is no longer the canonical

root. This is merely the roots that FreeBSD ships via this package. The true
CA root will change to /usr/local/etc/ssl/cert.pem. All symlinks that
pointed to ca-root-nss.crt will now point to this cert.pem instead. This
is intentional. I do not believe ca-root-nss.crt as shipped by this
package should ever be modified.

This also means every port in the tree that we have set the CA path to
ca-root-nss.crt also needs a modification and a PORTREVISION bump.

2. Include ca-merge script which finds files in PEM/DER format in a

designated location. This script is run automatically each time the
package is installed and can be run on demand by users as well. This
guarantees updates will not break CAs. This script deserves thorough
review, but it should have reasonable level of error handling at this
time. At no point should a system be left with a broken or missing CA
root. The script's failure mode is to simply install the ca-root-nss.crt
as the root CA. It is currently working for me as expected.

3. No longer execute the lang/mono cert-sync utility at pkg install

time. This is automatically handled by the ca-merge utility and will
guarantee that OpenSSL-consumers, Mono, and Java apps will all get the
same CA roots trusted.

I have intentionally chosen to mimic the directory structure of the RHEL
update-ca-trust utility as I imagine this could grow further
capabilities as time goes on. The update-ca-trust utliity also makes it
easy to blacklist CAs, for example.

I also feel like this is going to require some documentation and/or a man page.

Test Plan

Have users try this. Get feedback. Shame me for shell scripting issues.

curl with the attached patch should work. Otherwise use the
curl --cacert /usr/local/etc/ssl/cert.pem option for now. Additionally
you can test with openssl directly via openssl s_client functionality to
verify. Python should work out of the box as I believe it looks for
/etc/ssl/cert.pem.

[feld]

Use @sample for cert.pem. We will install the ca-root-nss.crt here, but the cert.pem will
get changed by ca-merge. This is cleaner and leaves us with no strange packaging issues.

[michaelo]

This looks quite nice, albeit I need to test this at work, I am trying to image how _merge_jks would work since this would require to have Java installed even for those do not have/use Java at all. Maybe a "-x keytool" would suffice to build the store?! What do you propose as path for cacerts? /usr/local/etc/ssl/cacerts?

ftp/curl/Makefile

76

Isn't the ca-root-nss.crt an implentation detail now since we don't depend on it directly?

security/ca_root_nss/files/ca-merge.sh.in

46

I dislike that the temp file is still ca-root-nss especially because your commit message says that never modify this file

56

This should go to stderr.

[michaelo]

Have you checked your script with: https://www.shellcheck.net/?

[romain]

Looks great!

Besides the in-line comments, maybe it's work considering setting $PATH to some well-known value / use full path of commands to be less sensitive to the user's environment?

security/ca_root_nss/files/ca-merge.sh.in

16

Output diagnostic to stderr? >&2

41

It looks like you properly exit each time it's appropriate, and _clean never return any value, so $? should always be 0.

I't rather exit 0 at the end of the script instead of this end of this function if needed.

98

DEBUG (and FAILED) are not initialized on startup, so some variable may already exist in the users' environment, causing inconsistencies here.

Initializing them to 0 (FAILED=0) at the beginning of the script and comparing values to integer ([ $DEBUG -eq 1 ]) may help avoid this.

[feld]

security/ca_root_nss/files/ca-merge.sh.in

116

This should be removed or _merge will run twice. Leftover from previous variations of the script.

[feld]

In D16352#347099, @romain wrote:

Looks great!

Besides the in-line comments, maybe it's work considering setting $PATH to some well-known value / use full path of commands to be less sensitive to the user's environment?

Indeed, setting PATH would be better because full path of commands might be unreliable if people are expecting to use this in appliances or FreeBSD derivatives. I'd like it to "just work" for all consumers.

[feld]

ftp/curl/Makefile

76

That's simply a test to prove that the security/ca_root_nss package is correctly installed. It is still a valid test even with these changes.

security/ca_root_nss/files/ca-merge.sh.in

46

I can agree that it would be less confusing if we operate on a "cert.pem" in ${TMPDIR} instead.

[feld]

security/ca_root_nss/files/ca-merge.sh.in

73

and -> an

[feld]

Update ca-merge to address identified concerns

[feld]

These issues have now been addressed. I have two remaining tasks:

1. do not install cert.pem unless it has actually changed.
2. Add java support based on the keytool commands found here: https://gist.github.com/jimblom/8ca3d775a7dcc67ef13130b104f17fa2

[feld]

Now include Java support.

Note, you will have to specify the trustStore until java ports are updated:

java -Djavax.net.ssl.trustStore=/usr/local/etc/ssl/cacerts

[feld]

Fix error output to stderr

[feld]

Fix inconsistency with update-ca-trust: files should be in a subdir named "anchors"

[matthew]

Adding a 'dummy-run' mode -- so it will report on what would be changed without modifying anything on disk -- would be useful.
Bonus points if you run that as a one of the daily periodic jobs.

You should definitely log the actions of this script to syslog.

Given you're going to split up the monolithic ca-root-nss.crt file into individual certs for the java keystore processing anyhow, using the c_rehash(1)
mechanism for managing the cert store looks more attractive. Not sure how that ties in with the mono cert-sync thing though.

security/ca_root_nss/files/ca-merge.sh.in

9

Use the ':' trick to allow reading CAPATH from the environment?

: ${CAPATH=%%PREFIX%%/etc/ssl/ca-trust/source/anchors}

10

No test for whether mktemp(1) succeeded?

You're overloading the meaning of TMPDIR here. That's probably fine though.
However I would add:

trap _clean EXIT INT KILL

so the temporary directory gets cleaned up if the process is killed.

42

If you add the trap as suggested above, no need to call _clean explicitly

71

Comment does not match code: you're not actually calculating a checksum here, but directly comparing the files.

79

So, if I already have a /etc/ssl/cert.pem with my locally added custom certs this overwrites them with the generic ca-root-nss.crt which will not make me a happy bunny. Wouldn't the safer strategy here be to just exit leaving the existing trusted cert store alone but loudly flagging up problems.

101

csplit(1) ?

[michaelo]

security/ca_root_nss/files/ca-merge.sh.in

79

How do you intend to detect that?

[michaelo]

So I ran the snippet which perfoms openssl verify against company-internal CAs and I think that this might not really work. Here is our certificate listing. I wrote a simple Java tool which traverses the HTML code, downloads the certs in the physical order and writes them into a combined PEM file.

$ openssl verify siemens-certs.pem
siemens-certs.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZB7, OU = Siemens Trust Center, CN = Siemens Issuing CA Intranet Server 2017
error 20 at 0 depth lookup:unable to get local issuer certificate
$ echo $?
2

Which is the first one in the file. One needs to re-arrange the file by issuer first. Lets try separate files now (siemens-cert-%02d.pem):

$ openssl verify *
siemens-cert-01.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZB7, OU = Siemens Trust Center, CN = Siemens Issuing CA Intranet Server 2017
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-02.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZB9, OU = Siemens Trust Center, CN = Siemens Issuing CA Internet Server 2017
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-03.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA1, OU = Siemens Trust Center, CN = Siemens Root CA V3.0 2016
error 18 at 0 depth lookup:self signed certificate
OK
siemens-cert-04.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA2, OU = Siemens Trust Center, CN = Siemens Issuing CA EE Auth 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-05.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZAD, OU = Siemens Trust Center, CN = Siemens Issuing CA EE Network Smartcard Auth 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-06.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA3, OU = Siemens Trust Center, CN = Siemens Issuing CA EE Enc 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-07.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA4, OU = Siemens Trust Center, CN = Siemens Issuing CA Intranet Code Signing 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-08.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA5, OU = Siemens Trust Center, CN = Siemens Issuing CA Multi Purpose 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-09.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA6, OU = Siemens Trust Center, CN = Siemens Issuing CA Medium Strength Authentication 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-10.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZAB, OU = Siemens Trust Center, CN = Siemens Issuing CA MSA Impersonalized Entities 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-11.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA7, OU = Siemens Trust Center, CN = Siemens Issuing CA Intranet Server 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-12.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA8, OU = Siemens Trust Center, CN = Siemens Issuing CA Internet Code Signing 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-13.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA9, OU = Siemens Trust Center, CN = Siemens Issuing CA Internet Server 2016
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-14.pem: OK
siemens-cert-15.pem: OK
siemens-cert-16.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY8, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA Internet Code Signing 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-17.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY9, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA Class Internet Server 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-18.pem: OK
siemens-cert-19.pem: C = DE, O = Siemens, serialNumber = ZZZZZZVS, OU = Copyright (C) Siemens 2007 All Rights Reserved, CN = Siemens Root-CA for Special Purposes 2007
error 18 at 0 depth lookup:self signed certificate
OK
siemens-cert-20.pem: OK
siemens-cert-21.pem: OK
siemens-cert-22.pem: OK
siemens-cert-23.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY3, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA EE Enc 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-24.pem: OK
siemens-cert-25.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY5, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA Multipurpose 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-26.pem: OK
siemens-cert-27.pem: OK
siemens-cert-28.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY8, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA Internet Code Signing 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-29.pem: C = DE, O = Siemens, serialNumber = ZZZZZZY9, OU = Copyright (C) Siemens AG 2013 All Rights Reserved, CN = Siemens Issuing CA Class Internet Server 2013
error 20 at 0 depth lookup:unable to get local issuer certificate
siemens-cert-30.pem: OK
siemens-cert-31.pem: OK

Doesn't really work either.
The only option I see is that to sort the list in Java by cert.getSubjectX500Principal().equals(cert.getIssuerX500Principal()) to get the roots and the write then intermediate ones.

If this is something which cannot be done by this tool, it must document this properly for the admin somehow.

[michaelo]

I have restructured my application by collecting roots and intermediates and then writing them and do now get:

$ openssl verify siemens-certs.pem
siemens-certs.pem: C = DE, ST = Bayern, L = Muenchen, O = Siemens, serialNumber = ZZZZZZA1, OU = Siemens Trust Center, CN = Siemens Root CA V3.0 2016
error 18 at 0 depth lookup:self signed certificate
OK
$ echo $?
0

This might not be possible/scale for others.

[feld]

Address all raised concerns except the use of csplit(1) which needs further testing

[feld]

most concerns addressed except csplit which I need to do some more testing with.

[feld]

Use sed and split

csplit does not play well here. This seems to be the cleanest way to extract certs.

[feld]

csplit(1) does not work well here. I'd like someone to prove me wrong if I'm misunderstanding how to use it.

[feld]

In D16352#350399, @1983-01-06_gmx.net wrote:

So I ran the snippet which perfoms openssl verify against company-internal CAs and I think that this might not really work. Here is our certificate listing. I wrote a simple Java tool which traverses the HTML code, downloads the certs in the physical order and writes them into a combined PEM file.

This does seem like a very unusual scenario. Siemens appears to have a very complicated internal CA root. I expect most people who have to get their servers to trust an internal CA are simply adding a single root CA certificate to the bundle; maybe two if they're dealing with two companies merging. I'm not sure how to approach this other than adding additional documentation to tell users how to structure their roots so it works nicely with ca-merge.

[michaelo]

In D16352#352606, @feld wrote:

In D16352#350399, @1983-01-06_gmx.net wrote:

So I ran the snippet which perfoms openssl verify against company-internal CAs and I think that this might not really work. Here is our certificate listing. I wrote a simple Java tool which traverses the HTML code, downloads the certs in the physical order and writes them into a combined PEM file.

This does seem like a very unusual scenario. Siemens appears to have a very complicated internal CA root. I expect most people who have to get their servers to trust an internal CA are simply adding a single root CA certificate to the bundle; maybe two if they're dealing with two companies merging. I'm not sure how to approach this other than adding additional documentation to tell users how to structure their roots so it works nicely with ca-merge.

Personally, I too do not see a better way than to sort by root/non-root. The best *you* can do is to provide proper documentation. In a complex case like mine, it is best to assemble a bundle on your own into a single PEM file. Everything else will not work.

[feld]

Are there any objections to this in its current state?

[matthew]

I can't see any obvious problems, apart from the trivially repeated PORTREVISION thing, but I haven't had a chance to test properly just yet.

ftp/curl/Makefile

7

Deja vu? PORTREVISION appears twice

[michaelo]

Let me run the script again and share the findings by Friday.

[michaelo]

After deinstallation I see this:

You may need to manually remove /usr/local/etc/ssl/cert.pem if it is no longer needed.
[1/1] Deleting files for ca_root_nss-3.39_1: 100%
2112 # tree /usr/local/etc/ssl
/usr/local/etc/ssl
├── ca-trust
│   └── source
│       └── anchors
│           └── siemens-certs.pem
├── cacerts
└── cert.pem

cert.pem is retained because it has been modified after installation. cacerts is retained, but without that warning. Maybe it would make sense to install the cacerts from the original cert.pem, mark as sample and then add the certs?

security/ca_root_nss/files/ca-merge.sh.in

55

"CA" > "CA(s)" since there couple be multiple ones on one file.

56

This one does not work if the input file contains more than one cert. Intermediate CAs require the Root CA to be present in the same file before otherwise 'verify' will fail upfront.

The line shall read "openssl crl2pkcs7 -nocrl -certfile ${i} | openssl pkcs7 -print_certs -text >> ${TMPDIR}/cert.pem".

[michaelo]

Folks, I took another step on this on a new machine with FreeBSD 12-STABLE by using OpenSSL from base, even with my changes it does not work. There is a misconception of openssl verify. Here it is used to verify the syntax of the input, but verify does verify the input syntactially and semantically against the root store on the system compiled into the OpenSSL objects. Our Siemens root is deemed to fail because it is selfsigned. This creates a chicken-and-egg situation. There is no switch to tell OpenSSL to syntactically verify the file, one has to apply a trick: openssl verify -no-CAfile -trusted ${i} ${i}.

I have also added a @rmtry on the generated cacerts file. Everything will be in a clean state as long as no custom CAs will be added.

See the modified patch:

enterprise-cas.patch8 KBDownload

It works for OpenSSL 1.1.1, version 1.0.2 does not provide these commands. Will provide another patch for that.

[michaelo]

Here is a modified version for OpenSSL 1.0.2. These need to merged by checking the version of OpenSSL.

enterprise-cas-1.0.2.patch8 KBDownload

[michaelo]

@feld , please close this review. It has been overcome by events.

[feld]

overcome by events