Add support for &#34;preferred-chain&#34; Issuer CA #503

@Eihrister commented on GitHub (Nov 11, 2020):

We would also appreciate this, as we're having many users relying on this for the time being.. which would give them time to manage this as well as possible.

@Eihrister commented on GitHub (Nov 11, 2020): We would also appreciate this, as we're having many users relying on this for the time being.. which would give them time to manage this as well as possible.

adam commented

@Eihrister commented on GitHub (Nov 13, 2020):

I just checked. Upon the final step, in this line (line 1138 in my local 0.6.6 copy):

result="$(signed_request "${finalize}" '{"csr": "'"${csr64}"'"}' | jsonsh)"

The result will contain a certificate key with a URL as value. The URL is then fetched in the end, with these lines (lines 1157 + 1158 in my local 0.6.6 copy):

certificate="$(echo "${result}" | get_json_string_value certificate)"
crt="$(signed_request "${certificate}" "")"

However, then the problem starts. This URL returns Link:-headers that are ignored by dehydrated. The first was unimportant as we obtained the certificate anyway. But now a new, second Link:- header has been added, with rel=alternate as flag, and the URL for the alternative chain. Until the change in january, this links to the new chain. Starting with the change in january, it will point to the old chain that we want.

This looks like this:

< link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< link: <https://acme-v02.api.letsencrypt.org/acme/cert/042481de76237659d8492e9c12d013faa7ad/1>;rel="alternate"

It's not super trivial to add support for the new header, but, in case someone doesn't get around to it, the ugly hotfix/workaround could be to just manually edit your local copy and append /0 to the certificate-URL ... it's only for half a year maximum anyway (although proper support is always preferred, as this might happen more often in the future with other ACME servers):

crt="$(signed_request "${certificate}/0" "")"

Omitting a specific chain (/0 or /1) will give you the default chain (currently the old root, in january this will be the new root).
Appending /0 will give you the old root. Now, and I assume with the change in january as well.
Appending /1 will give you the new root. Now, and I assume with the change in january as well.

@Eihrister commented on GitHub (Nov 13, 2020): I just checked. Upon the final step, in this line (line 1138 in my local 0.6.6 copy): ``` result="$(signed_request "${finalize}" '{"csr": "'"${csr64}"'"}' | jsonsh)" ``` The result will contain a `certificate` key with a URL as value. The URL is then fetched in the end, with these lines (lines 1157 + 1158 in my local 0.6.6 copy): ``` certificate="$(echo "${result}" | get_json_string_value certificate)" crt="$(signed_request "${certificate}" "")" ``` However, then the problem starts. This URL returns `Link:`-headers that are ignored by `dehydrated`. The first was unimportant as we obtained the certificate anyway. But now a new, second `Link:`- header has been added, with `rel=alternate` as flag, and the URL for the alternative chain. Until the change in january, this links to the *new* chain. Starting with the change in january, it will point to the *old* chain that we want. This looks like this: ``` < link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index" < link: <https://acme-v02.api.letsencrypt.org/acme/cert/042481de76237659d8492e9c12d013faa7ad/1>;rel="alternate" ``` It's not super trivial to add support for the new header, but, in case someone doesn't get around to it, the ugly hotfix/workaround could be to just manually edit your local copy and append `/0` to the certificate-URL ... it's only for half a year maximum anyway (although proper support is always preferred, as this might happen more often in the future with other ACME servers): ``` crt="$(signed_request "${certificate}/0" "")" ``` - Omitting a specific chain (`/0` or `/1`) will give you the default chain (currently the old root, in january this will be the new root). - Appending `/0` will give you the old root. Now, and I assume with the change in january as well. - Appending `/1` will give you the new root. Now, and I assume with the change in january as well.

adam commented

@rossnick commented on GitHub (Nov 13, 2020):

Great, I just tried it and was able to get a certificate signed by the new root. I confirmed in chrome and with openssl s_client the certificate chain. However, in firefox, I still see the old root. I get the same behaviour with valid-isrgrootx1.letsencrypt.org. That's strange...

So in the end, the modification doesn't seem to be that hard to implement...

@rossnick commented on GitHub (Nov 13, 2020): Great, I just tried it and was able to get a certificate signed by the new root. I confirmed in chrome and with openssl s_client the certificate chain. However, in firefox, I still see the old root. I get the same behaviour with valid-isrgrootx1.letsencrypt.org. That's strange... So in the end, the modification doesn't seem to be that hard to implement...

adam commented

@Eihrister commented on GitHub (Nov 13, 2020):

So in the end, the modification doesn't seem to be that hard to implement...

Could even make it a numerical option in the config file, and append whatever it's set to to the final certificate URL as above. 0 for the old chain, 1 for the new chain, empty for LE's default as it is currently.

@Eihrister commented on GitHub (Nov 13, 2020): > So in the end, the modification doesn't seem to be that hard to implement... Could even make it a numerical option in the config file, and append whatever it's set to to the final certificate URL as above. 0 for the old chain, 1 for the new chain, empty for LE's default as it is currently.

adam commented

2025-12-29 01:26:20 +01:00

@lukas2511 commented on GitHub (Nov 13, 2020):

Pushed an implementation for this in 7dfde364a3. Not really happy with the code quality but not sure on how to make it much nicer.

@lukas2511 commented on GitHub (Nov 13, 2020): Pushed an implementation for this in 7dfde364a3c960e4ec4b3c32026929ba2a39b040. Not really happy with the code quality but not sure on how to make it much nicer.

adam commented

@rossnick commented on GitHub (Nov 13, 2020):

crt="$(signed_request "${certificate}/0" "")"

* Omitting a specific chain (`/0` or `/1`) will give you the default chain (currently the old root, in january this will be the new root).

* Appending `/0` will give you the old root. Now, and I assume with the change in january as well.

* Appending `/1` will give you the new root. Now, and I assume with the change in january as well.

One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose.

@rossnick commented on GitHub (Nov 13, 2020): > ``` > crt="$(signed_request "${certificate}/0" "")" > ``` > > * Omitting a specific chain (`/0` or `/1`) will give you the default chain (currently the old root, in january this will be the new root). > > * Appending `/0` will give you the old root. Now, and I assume with the change in january as well. > > * Appending `/1` will give you the new root. Now, and I assume with the change in january as well. One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose.

adam commented

2025-12-29 01:26:21 +01:00

@lukas2511 commented on GitHub (Nov 13, 2020):

crt="$(signed_request "${certificate}/0" "")"
* Omitting a specific chain (`/0` or `/1`) will give you the default chain (currently the old root, in january this will be the new root).

* Appending `/0` will give you the old root. Now, and I assume with the change in january as well.

* Appending `/1` will give you the new root. Now, and I assume with the change in january as well.
One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose.

Because there was no reference in the specs on how alternate chains are ordered and if the paths are consistent i took the (assumingly) same route as certbot and use matching on the chain roots CN instead of anything else.

@lukas2511 commented on GitHub (Nov 13, 2020): > > ``` > > crt="$(signed_request "${certificate}/0" "")" > > ``` > > > > > > ``` > > * Omitting a specific chain (`/0` or `/1`) will give you the default chain (currently the old root, in january this will be the new root). > > > > * Appending `/0` will give you the old root. Now, and I assume with the change in january as well. > > > > * Appending `/1` will give you the new root. Now, and I assume with the change in january as well. > > ``` > > One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose. Because there was no reference in the specs on how alternate chains are ordered and if the paths are consistent i took the (assumingly) same route as certbot and use matching on the chain roots CN instead of anything else.

adam commented

2025-12-29 01:26:21 +01:00

@Eihrister commented on GitHub (Nov 13, 2020):

Pushed an implementation for this in 7dfde36. Not really happy with the code quality but not sure on how to make it much nicer.

Thanks a lot!

One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose.

There's 2 intermediaries with the same CN, issued by different roots. In fact, the DST intermediary expires in March next year already (so not sure where the "half a year" came from).... and the ISRG intermediary expires on October next year, so they'll get changed either way. They'll have to be.

Because there was no reference in the specs on how alternate chains are ordered and if the paths are consistent i took the (assumingly) same route as certbot and use matching on the chain roots CN instead of anything else.

Yeah, this is a lot better. Just a shame they don't really seem to give you a way of requesting a list of chain paths first, and then appending the ID for the one you're looking for, instead of having to iterate over alternate links. But hey, this works and is what we need, so again, thanks!

@Eihrister commented on GitHub (Nov 13, 2020): > Pushed an implementation for this in [7dfde36](https://github.com/dehydrated-io/dehydrated/commit/7dfde364a3c960e4ec4b3c32026929ba2a39b040). Not really happy with the code quality but not sure on how to make it much nicer. Thanks a lot! > One thing I'm worried about is that 1 (alternate chain) would meed the old root after january . We (by that I mean dehydrated as a client of LE) don't have any control over this I suppose. There's 2 intermediaries with the same CN, issued by different roots. In fact, the DST intermediary expires in March next year already (so not sure where the "half a year" came from).... and the ISRG intermediary expires on October next year, so they'll get changed either way. They'll have to be. > Because there was no reference in the specs on how alternate chains are ordered and if the paths are consistent i took the (assumingly) same route as certbot and use matching on the chain roots CN instead of anything else. Yeah, this is a lot better. Just a shame they don't really seem to give you a way of requesting a list of chain paths first, and then appending the ID for the one you're looking for, instead of having to iterate over alternate links. But hey, this works and is what we need, so again, thanks!

adam commented

2025-12-29 01:26:21 +01:00

@Pavel077 commented on GitHub (Nov 21, 2020):

I'm trying to test how preferred-chain options works and I got stuck.
What should I set PREFERRED_CHAIN variable in config to get a cert with new ISRG ROOT X1?
Or is this not possible until Jan,11 2021?

I've tried setting both PREFERRED_CHAIN="DST Root CA X3"
and
PREFERRED_CHAIN="Let's Encrypt Authority X3"
but always get

ERROR: Alternative chain with CN = DST Root CA X3 not found respectfully
or
ERROR: Alternative chain with CN = Let's Encrypt Authority X3 not found

@Pavel077 commented on GitHub (Nov 21, 2020): I'm trying to test how preferred-chain options works and I got stuck. What should I set PREFERRED_CHAIN variable in config to get a cert with new ISRG ROOT X1? Or is this not possible until Jan,11 2021? I've tried setting both **PREFERRED_CHAIN="DST Root CA X3"** and **PREFERRED_CHAIN="Let's Encrypt Authority X3"** but always get **ERROR: Alternative chain with CN = DST Root CA X3 not found** respectfully or **ERROR: Alternative chain with CN = Let's Encrypt Authority X3 not found**

adam commented

2025-12-29 01:26:22 +01:00

@lukas2511 commented on GitHub (Nov 21, 2020):

On Let's Encrypt production the following CAs should currently be available: DST Root CA X3, ISRG Root X1

On testing: Fake LE Intermediate X1

I've also pushed a commit that shows available options if the set one isn't found.

@lukas2511 commented on GitHub (Nov 21, 2020): On Let's Encrypt production the following CAs should currently be available: `DST Root CA X3`, `ISRG Root X1` On testing: `Fake LE Intermediate X1` I've also pushed a commit that shows available options if the set one isn't found.

adam commented

2025-12-29 01:26:23 +01:00

@mritzmann commented on GitHub (Nov 26, 2020):

I have seen that this has already been implemented. But the last release is from 2019. Will there be a new release on time in December or should we work directly with the master branch? :-)

@mritzmann commented on GitHub (Nov 26, 2020): I have seen that this has already been implemented. But the last release [is from 2019](https://github.com/dehydrated-io/dehydrated/releases). Will there be a new release on time in December or should we work directly with the master branch? :-)

adam commented

2025-12-29 01:26:23 +01:00

@rossnick commented on GitHub (Dec 7, 2020):

I tried to use the PREFERRED_CHAIN config variable (or command-line), and it yields this :
+ Requesting certificate...
usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-no_alt_chains] [-attime timestamp] [-engine e] cert1 cert2 ...
recognized usages:
sslclient SSL client
sslserver SSL server
nssslserver Netscape SSL server
smimesign S/MIME signing
smimeencrypt S/MIME encryption
crlsign CRL signing
any Any Purpose
ocsphelper OCSP helper
timestampsign Time Stamp signing
ERROR: Error while fetching CN from certificate chain

@rossnick commented on GitHub (Dec 7, 2020): I tried to use the PREFERRED_CHAIN config variable (or command-line), and it yields this : ` + Requesting certificate...` `usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-no_alt_chains] [-attime timestamp] [-engine e] cert1 cert2 ...` `recognized usages:` ` sslclient SSL client` ` sslserver SSL server` ` nssslserver Netscape SSL server` ` smimesign S/MIME signing` ` smimeencrypt S/MIME encryption` ` crlsign CRL signing` ` any Any Purpose` ` ocsphelper OCSP helper` ` timestampsign Time Stamp signing` `ERROR: Error while fetching CN from certificate chain`

adam commented

@lukas2511 commented on GitHub (Dec 7, 2020):

I tried to use the PREFERRED_CHAIN config variable (or command-line), and it yields this :
+ Requesting certificate...
usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-no_alt_chains] [-attime timestamp] [-engine e] cert1 cert2 ...
recognized usages:
sslclient SSL client
sslserver SSL server
nssslserver Netscape SSL server
smimesign S/MIME signing
smimeencrypt S/MIME encryption
crlsign CRL signing
any Any Purpose
ocsphelper OCSP helper
timestampsign Time Stamp signing
ERROR: Error while fetching CN from certificate chain

What openssl version are you using?

@lukas2511 commented on GitHub (Dec 7, 2020): > I tried to use the PREFERRED_CHAIN config variable (or command-line), and it yields this : > ` + Requesting certificate...` > `usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-no_alt_chains] [-attime timestamp] [-engine e] cert1 cert2 ...` > `recognized usages:` > ` sslclient SSL client` > ` sslserver SSL server` > ` nssslserver Netscape SSL server` > ` smimesign S/MIME signing` > ` smimeencrypt S/MIME encryption` > ` crlsign CRL signing` > ` any Any Purpose` > ` ocsphelper OCSP helper` > ` timestampsign Time Stamp signing` > `ERROR: Error while fetching CN from certificate chain` What openssl version are you using?

adam commented

@rossnick commented on GitHub (Dec 7, 2020):

OpenSSL 1.0.2k-fips - openssl-1.0.2k-19.el7.x86_64 from Centos 7

@rossnick commented on GitHub (Dec 7, 2020): OpenSSL 1.0.2k-fips - openssl-1.0.2k-19.el7.x86_64 from Centos 7

adam commented

@Pavel077 commented on GitHub (Dec 7, 2020):

I've hit the same problem on openssl 1.0.2r, looks like it does not know show_chain option. Had to compile openssl 1.1.1 in a custom location for function get_last_cn() to work.
I also had to add -r option to sed there: _sed -r -e 's/.* CN ?= ?([^/,]*).*/\1/'

@Pavel077 commented on GitHub (Dec 7, 2020): I've hit the same problem on openssl **1.0.2r**, looks like it does not know ```show_chain``` option. Had to compile openssl **1.1.1** in a custom location for function ```get_last_cn()``` to work. I also had to add **-r** option to sed there: ```_sed -r -e 's/.* CN ?= ?([^/,]*).*/\1/'```

adam commented

@rossnick commented on GitHub (Dec 8, 2020):

Ok, with taht in mind, I installed openssl11 from epel repo on a centos vm. I was not able to change the executable used by dehydrated via env variable or config, had to modify the base script, but that's a different issue.
It seems to be working a little more, but it didn't find the alternate chain with CN = ISRG Root X1. Error seems to be different, not sure it work entierly :

Requesting certificate...
C = US, O = Let's Encrypt, CN = R3
error 2 at 1 depth lookup: unable to get issuer certificate
C = US, O = Let's Encrypt, CN = R3
error 2 at 1 depth lookup: unable to get issuer certificate
error /dev/fd/62: verification failed
ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: error /dev/fd/62: verification failed, err or /dev/fd/62: verification failed

@rossnick commented on GitHub (Dec 8, 2020): Ok, with taht in mind, I installed openssl11 from epel repo on a centos vm. I was not able to change the executable used by dehydrated via env variable or config, had to modify the base script, but that's a different issue. It seems to be working a little more, but it didn't find the alternate chain with CN = ISRG Root X1. Error seems to be different, not sure it work entierly : + Requesting certificate... C = US, O = Let's Encrypt, CN = R3 error 2 at 1 depth lookup: unable to get issuer certificate C = US, O = Let's Encrypt, CN = R3 error 2 at 1 depth lookup: unable to get issuer certificate error /dev/fd/62: verification failed ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: error /dev/fd/62: verification failed, err or /dev/fd/62: verification failed

adam commented

@Pavel077 commented on GitHub (Dec 8, 2020):

Had this issue too, but i thought it was due to my EOL'd centos 6.
I had to modify get_last_cn function so that it stores temporary cert into file, and the I run openssl checks againt it.
Since I'm not a programmer I had a hard time saving cert into file from dehydrated script, hopefully Lukas will show a correct way to use printf for this, but through trial and error way I found this ugly, but working solution:

get_last_cn() {
printf "abc${1}" >/tmp/11111.pem
sed -e s/abc//g -i /tmp/11111.pem
  #cn="$("${OPENSSL}" verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem | tail -n 1 | _sed -r -e 's/.* CN ?= ?([^/,]*).*/\1/')"
  cn="$("${OPENSSL}" verify  -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem 2>&1 | tail -n 1 | awk -F '=' {'print $NF'} | xargs)"
  if [ -z "${cn}" ]; then
    _exiterr "Error while fetching CN from certificate chain"
  fi
  printf "${cn}"
}

Please note that I use awk and xargs, not sed to extract issuer CN. I've googled about sed's -r option later and did not try it to avoid getting banned by LetsEncrypt for excessive number of requests. But at least on centos 6 running sed from command line without "-r" gave me error as described here: https://stackoverflow.com/questions/16637799/sed-error-invalid-reference-1-on-s-commands-rhs

P.S. my OPENSSL var in config is set like this:
OPENSSL="/usr/local/ssl-1.1.1g/bin/openssl"

@Pavel077 commented on GitHub (Dec 8, 2020): Had this issue too, but i thought it was due to my EOL'd centos 6. I had to modify ```get_last_cn``` function so that it stores temporary cert into file, and the I run openssl checks againt it. Since I'm not a programmer I had a hard time saving cert into file from dehydrated script, hopefully Lukas will show a correct way to use ```printf``` for this, but through trial and error way I found this ugly, but working solution: ``` get_last_cn() { printf "abc${1}" >/tmp/11111.pem sed -e s/abc//g -i /tmp/11111.pem #cn="$("${OPENSSL}" verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem | tail -n 1 | _sed -r -e 's/.* CN ?= ?([^/,]*).*/\1/')" cn="$("${OPENSSL}" verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem 2>&1 | tail -n 1 | awk -F '=' {'print $NF'} | xargs)" if [ -z "${cn}" ]; then _exiterr "Error while fetching CN from certificate chain" fi printf "${cn}" } ``` Please note that I use *awk* and *xargs*, not *sed* to extract issuer CN. I've googled about sed's *-r* option later and did not try it to avoid getting banned by LetsEncrypt for excessive number of requests. But at least on centos 6 running sed from command line without "-r" gave me error as described here: [https://stackoverflow.com/questions/16637799/sed-error-invalid-reference-1-on-s-commands-rhs](https://stackoverflow.com/questions/16637799/sed-error-invalid-reference-1-on-s-commands-rhs) P.S. my OPENSSL var in config is set like this: ```OPENSSL="/usr/local/ssl-1.1.1g/bin/openssl"```

adam commented

@rossnick commented on GitHub (Dec 9, 2020):

I see. I was able to advance a bit. I ended up hitting constantly the "JWS has no anti-replay nonce" error. Implemented a temp fix found in this thread : https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324/14

@Pavel077 with your get_last_cn fix, it still can't find the CN I asked (with prod "ISRG Root X1") and the command. While the temp file 1111.pem is still there, I issued the command :
openssl11 verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem
it failes with :
C = US, O = Let's Encrypt, CN = R3
error 2 at 1 depth lookup: unable to get issuer certificate
error /tmp/11111.pem: verification failed

I don't know what is supposed to be the content of the file, but I outputed some more debug, and found that get_last_cn is called 2 times with 2 concatenated certs. The first one is always the cert I am issued, I mean the same,. and the second in the Let's Encrypt R3, one time issued by DST Root and the other from ISRG. With debug added before the call to get_last_cn, it may by that the cert passed to the get_last_cn isn't right the second time.

I ended up busting my issuence quota. So I'll sleep on this until friday.

@rossnick commented on GitHub (Dec 9, 2020): I see. I was able to advance a bit. I ended up hitting constantly the "JWS has no anti-replay nonce" error. Implemented a temp fix found in this thread : https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324/14 @Pavel077 with your get_last_cn fix, it still can't find the CN I asked (with prod "ISRG Root X1") and the command. While the temp file 1111.pem is still there, I issued the command : openssl11 verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem it failes with : C = US, O = Let's Encrypt, CN = R3 error 2 at 1 depth lookup: unable to get issuer certificate error /tmp/11111.pem: verification failed I don't know what is supposed to be the content of the file, but I outputed some more debug, and found that get_last_cn is called 2 times with 2 concatenated certs. The first one is always the cert I am issued, I mean the same,. and the second in the Let's Encrypt R3, one time issued by DST Root and the other from ISRG. With debug added before the call to get_last_cn, it may by that the cert passed to the get_last_cn isn't right the second time. I ended up busting my issuence quota. So I'll sleep on this until friday.

adam commented

@Pavel077 commented on GitHub (Dec 9, 2020):

I guess I know what is going on.
Perhaps your new openssl version does not have a list of trusted CAs?
I did not know how to force it to use /etc/ssl/certs/ca-bundle.crt which came with centos, so I did it this way:

Download fresh cacert.pem from Mozilla CA certificate store: https://curl.haxx.se/ca/cacert.pem
Put it into /usr/local/ssl-1.1.1g/certs (my custom openssl home is /usr/local/ssl-1.1.1g)
Extract individual certificates from bundle cacert.pem: awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "cert." c ".pem"}' ./cacert.pem
Run /usr/local/ssl-1.1.1g/bin/c_rehash -v /usr/local/ssl-1.1.1g/certs
This adds symlinks in the same folder where you put .pem certs for CAs - /usr/local/ssl-1.1.1g/certs

I think that now your command openssl verify will say that your cert is ok, this is my example with cert hashes:

/usr/local/ssl-1.1.1g/bin/openssl  verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem
/tmp/11111.pem: OK
Chain:
depth=0: CN = my-example-domain.org (untrusted)
depth=1: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1

And this how it looks if i delete hashes:

/usr/local/ssl-1.1.1g/bin/openssl  verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
error 2 at 1 depth lookup: unable to get issuer certificate
error /tmp/11111.pem: verification failed

You can even add LE staging certificate to your trusted CA list this way by adding their files downloaded from https://letsencrypt.org/docs/staging-environment/ and running c_rehash again to continue testing while you are banned on LE production environment.

rehash manual

HTH

@Pavel077 commented on GitHub (Dec 9, 2020): I guess I know what is going on. Perhaps your new openssl version does not have a list of trusted CAs? I did not know how to force it to use **/etc/ssl/certs/ca-bundle.crt** which came with centos, so I did it this way: 1) Download fresh cacert.pem from Mozilla CA certificate store: [https://curl.haxx.se/ca/cacert.pem](https://curl.haxx.se/ca/cacert.pem) 2) Put it into **/usr/local/ssl-1.1.1g/certs** (my custom openssl home is **/usr/local/ssl-1.1.1g**) 3) Extract individual certificates from bundle cacert.pem: ```awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "cert." c ".pem"}' ./cacert.pem``` 4) Run ```/usr/local/ssl-1.1.1g/bin/c_rehash -v /usr/local/ssl-1.1.1g/certs``` This adds symlinks in the same folder where you put .pem certs for CAs - **/usr/local/ssl-1.1.1g/certs** I think that now your command ```openssl verify``` will say that your cert is ok, this is my example with cert hashes: ``` /usr/local/ssl-1.1.1g/bin/openssl verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem /tmp/11111.pem: OK Chain: depth=0: CN = my-example-domain.org (untrusted) depth=1: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1 ``` And this how it looks if i delete hashes: ``` /usr/local/ssl-1.1.1g/bin/openssl verify -CAfile /tmp/11111.pem -show_chain /tmp/11111.pem C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 error 2 at 1 depth lookup: unable to get issuer certificate error /tmp/11111.pem: verification failed ``` You can even add LE staging certificate to your trusted CA list this way by adding their files downloaded from ```https://letsencrypt.org/docs/staging-environment/``` and running ```c_rehash``` again to continue testing while you are banned on LE production environment. [rehash manual](https://www.openssl.org/docs/man1.1.1/man1/c_rehash.html) HTH

adam commented

@rossnick commented on GitHub (Dec 9, 2020):

Ok, I was able to have a verify with the expanded certificate bundle and adding CApath option to the verify. I'll play again with this later at the end of the week. But it seems that this solution won't be viable on a base RHEL/Centos 7 system without serious hacks.

@rossnick commented on GitHub (Dec 9, 2020): Ok, I was able to have a verify with the expanded certificate bundle and adding CApath option to the verify. I'll play again with this later at the end of the week. But it seems that this solution won't be viable on a base RHEL/Centos 7 system without serious hacks.

adam commented

@lukas2511 commented on GitHub (Dec 9, 2020):

I just pushed a modification with fbcaac89f9 that should be compatible with a lot more openssl versions.

@lukas2511 commented on GitHub (Dec 9, 2020): I just pushed a modification with fbcaac89f9545c4e32fee8c64870d5ae16393e28 that should be compatible with a lot more openssl versions.

adam commented

@lukas2511 commented on GitHub (Dec 9, 2020):

Closing this ticket as I'm seeing this feature as implemented and stable. Further discussion about weird issues is obviously still allowed.

@lukas2511 commented on GitHub (Dec 9, 2020): Closing this ticket as I'm seeing this feature as implemented and stable. Further discussion about weird issues is obviously still allowed.

adam commented

@rossnick commented on GitHub (Dec 11, 2020):

Thanks. For the stock openssl in centos 7, it now works fine on the part where it try to find the CN of the issuer. I get :
ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3, issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1

I'm not entierly sure that the sed command works as expected on my system when piped with the string :
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3
I am no sed or perl regex master, so I cannot try to begin to know why.

But whem I uses @Pavel077 awk command :
awk -F '=' {'print $NF'} | xargs
instead of the sed, it works super fine.

@rossnick commented on GitHub (Dec 11, 2020): Thanks. For the stock openssl in centos 7, it now works fine on the part where it try to find the CN of the issuer. I get : ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3, issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1 I'm not entierly sure that the sed command works as expected on my system when piped with the string : issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3 I am no sed or perl regex master, so I cannot try to begin to know why. But whem I uses @Pavel077 awk command : awk -F '=' {'print $NF'} | xargs instead of the sed, it works super fine.

adam commented

@lukas2511 commented on GitHub (Dec 11, 2020):

Thanks. For the stock openssl in centos 7, it now works fine on the part where it try to find the CN of the issuer. I get :
ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3, issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1

I'm not entierly sure that the sed command works as expected on my system when piped with the string :
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3
I am no sed or perl regex master, so I cannot try to begin to know why.

But whem I uses @Pavel077 awk command :
awk -F '=' {'print $NF'} | xargs
instead of the sed, it works super fine.

Can you check if 29b67962ac solves this for you?

@lukas2511 commented on GitHub (Dec 11, 2020): > Thanks. For the stock openssl in centos 7, it now works fine on the part where it try to find the CN of the issuer. I get : > ERROR: Alternative chain with CN = ISRG Root X1 not found, available options: issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3, issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1 > > I'm not entierly sure that the sed command works as expected on my system when piped with the string : > issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3 > I am no sed or perl regex master, so I cannot try to begin to know why. > > But whem I uses @Pavel077 awk command : > awk -F '=' {'print $NF'} | xargs > instead of the sed, it works super fine. Can you check if 29b67962ac07d5d2b24f69c863a9ef31e8ac53ee solves this for you?

adam commented

@rossnick commented on GitHub (Dec 12, 2020):

Yes, indeed ! The new default KEY_ALGO had bit me, but that's another story. Had to restart httpd instead of reload.

@rossnick commented on GitHub (Dec 12, 2020): Yes, indeed ! The new default KEY_ALGO had bit me, but that's another story. Had to restart httpd instead of reload.

adam commented