mirror of
https://github.com/dehydrated-io/dehydrated.git
synced 2026-01-11 22:30:44 +01:00
JWS has no anti-replay nonce #440
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @throw1008a on GitHub (Oct 8, 2019).
So I needed to renew a cert on Monday, October 7, 2019, and got an error about "JWS has no anti-replay nonce"; this was with dehydrated 0.6.2 (it has been running fine for over a year). Doing some testing, I moved up to dehydrated 0.6.3, and got the following on a cert that needed renewal:
Then in 0.6.4 and 0.6.5 I get:
The version of cURL:
curl 7.47.1 (x86_64-redhat-linux-gnu) libcurl/7.47.1 OpenSSL/1.0.2p zlib/1.2.7 libidn/1.28 libssh2/1.4.3 nghttp2/1.31.1After some playing around, I added an extra configuration file with the following contents:
CURL_OPTS="--http1.1"No longer seem to get the “JWS has no anti-replay nonce” error.
See also this thread:
@throw1008a commented on GitHub (Oct 8, 2019):
Full output of cURL version:
@ilude commented on GitHub (Oct 8, 2019):
Master branch seems to fix this issue. Any chance we can get a v0.6.6 release tag?
@hefty-pty commented on GitHub (Oct 8, 2019):
crossposting from https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324/14
dehydrated output from where it goes wrong: https://pastebin.com/2VWLpgth
it ultimately seems to be an error with the letsencrypt infrastructure, but maybe dehydrated could handle an tls-connection error while trying to retrieve the nonce more graceful?
@hefty-pty commented on GitHub (Oct 8, 2019):
also: shouldn't the curlret status in line 534 (master v0.6.5) be set before the touch command?
@lukas2511 commented on GitHub (Oct 9, 2019):
Master branch currently is exactly at the state of v0.6.5. All other changes I'm working on are currently only on my local clone.
I've found what's going on here, it seems that fetching a nonce fails and for whatever reason (didn't have the time to debug it yet) dehydrated doesn't exit at that point but just keeps running with an empty nonce.
I'll have to implement a loooot of retry logic in the (very) near future as the availability of Let's Encrypts CDN seems to be getting worse by the day... All of the missing nonces and "error 35" messages etc... meh :-/
@lukas2511 commented on GitHub (Oct 9, 2019):
Good find. That may actually be the logic error. I'll commit that to master branch. Would be great if somebody could test it.
@pR0Ps commented on GitHub (Oct 9, 2019):
@lukas2511 Just pulled master and updated my certs - everything worked perfectly.
Before pulling master I was on v0.6.2 and seeing the anti-replay error. Might just be me getting lucky though.
@hefty-pty commented on GitHub (Oct 9, 2019):
here a quickfix that implements while loops in http_request() for retry. patch is for v0.6.5, so it also fixes the curlret status issue already patched in master.
https://pastebin.com/igFHAp6D
@ilude commented on GitHub (Oct 9, 2019):
Understood. I was just looking at SHA's yesterday morning when the issue occurred and master was different because of the merge. I was on the v0.6.5 tag and was getting the error and when I updated to master it went away and I have not seen it since. So my best assumption at this point is that I was getting the error due to something on LetsEncrypt's side of things, as you noted, and it was all just a timing issue.
@lukas2511 commented on GitHub (Oct 9, 2019):
Just to be clear: The underlying issue hasn't been completely fixed yet. This error occurs due to connection issues with the CA and there currently is no retry-logic in dehydrated. The only thing the last commit fixed should be the type of the error message, instead of failing with "no nonce" dehydrated should (at least in theory...) now notice the broken HEAD request and fail early with a correct error message, but I think there is another issue which again prevents this from working correctly. I'll have to do some work on it, until then you can always just rerun the script, at least with ACME it at least kinda keeps the state and doesn't have to rerun everything again.
@bonidier commented on GitHub (Oct 11, 2019):
Hi,
FYI I've encountered this errors 2 days ago (curl error 35 + nonce) with a big SAN request using http-01, and it's OK today without any change (dehydrated 0.6.5).
This can also be due to requests limit from ACME servers.
Does playing with Curl options
--retry/--retry-delay/--retry-max-timeintoCURL_OPTSmay easily resolve retry logic (instead of reinventing the wheel) ?@hefty-pty commented on GitHub (Oct 11, 2019):
I looked at this, but you'll need at least curl 7.55.0 (which features --retry-connrefused) as earlier versions don't retry with connection errors.
@njh commented on GitHub (Nov 11, 2019):
I started experiencing the "JWS has no anti-replay nonce" problem on my Debian 9 machine (dehydrated 0.3.1 and curl 7.52.1). It was coming towards my cert expiry date and I was starting to panic!
Thankfully it was fixed easily by upgrading to Debian 10 (dehydrated 0.6.2 and curl 7.64.0).
@ukleinek commented on GitHub (Nov 12, 2019):
FTR: This is a duplicate of #559
@thnee commented on GitHub (Nov 14, 2019):
I started getting this error completely out of nowhere on my FreeBSD server, maybe one month ago or so. Hadn't touched anything on the machine for almost a year, and it just started happening.
Then I realized I was running an old version of dehydrated, 0.6.2, so I upgraded to 0.6.5 and the error stopped happening. Or maybe it was because one of the tools such as bash or curl was also upgraded at the same time, but that seems less likely. Either way, it certainly appears to be solved in the latest version.
This error was happening consistently every time I ran dehydrated 0.6.2 and immediately stopped happening after upgrading, which seems to indicate that this is not just because of some random connection error.
@davehayes commented on GitHub (Nov 18, 2019):
I've recently seen this error using curl 7.66.0 AND 7.65.0, both on 0.6.2 of dehydrated.
I am confirming that upgrading to 0.6.5 fixes the issue. One more datapoint for you.
@mleklund commented on GitHub (Nov 20, 2019):
0.6.5 fixes this for me as well.
@hmoffatt commented on GitHub (Dec 29, 2019):
I have 0.6.5 and this still occurs most times I run dehydrated. Sometimes connecting to the directory also fails with “curl returned 35”. My curl is 7.52.1.
@hmoffatt commented on GitHub (Jan 13, 2020):
It seems to work after upgrading curl to 7.64.0 (Debian buster).
@jose1711 commented on GitHub (Jan 28, 2020):
Was unable to solve constant "curl returned 35" so I tried with
certbotand that worked (Debian 10, recently upgraded from Squeeze)@bonidier commented on GitHub (Jan 28, 2020):
Hi @jose1711 , like mentioned by @lukas2511 CURL error 35 means ACME server does not handle all requests as expected and may fail randomly (maybe a load or connection limit from their side). It can works by retrying later, that's why a retry logic must be implemented.
The Curl manpages says :
35 SSL connect error. The SSL handshaking failed.@jose1711 commented on GitHub (Jan 29, 2020):
Actually I did try this at least 20 times before giving up.
@hmoffatt commented on GitHub (Jan 29, 2020):
@jose1711 it has been solid for me since upgrading cURL.
@GTAXL commented on GitHub (Aug 8, 2020):
Any update on this? I'm getting this on one of my servers, always, never have been able to issue a cert. Latest version and everything.
@KalaNag66 commented on GitHub (Nov 19, 2020):
Some curl versions show different behavior with HTTP/2 connections. Here I have curl/7.64.0 which creates a HTTP/2 connection and then sends "HEAD /acme/new-nonce HTTP/2". I also have a curl/7.47.1 which also creates a HTTP/2 connection but sends "HEAD /acme/new-nonce HTTP/1.1". At least the nginx at acme-staging-v02.api.letsencrypt.org handles this slighly different, omitting the whitespace from the headers in the latter case. Thus you won't get "replay-nonce: 0004VvdMDFA-aEtxcMiuPTiF0RdiXGPtxAwUsxxbuNt9vEI" but "replay-nonce:0004dM2_4WoTBObCXot8l7IyVx2ED79zeAmiXZbA89xet90". Since dehydrated parses this with
nonce="$(http_request head "${CA_NEW_NONCE}" | grep -i ^Replay-Nonce: | awk -F ': ' '{print $2}' | tr -d '\n\r')"
the awk does not see ": " and does not emit a nonce, resulting in the error observed. I don't know enough about HTTP/2 to tell wether curl or nginx is in error, but dehydrated should tolerate a missing whitespace.
This is why upgrading curl or falling back to --http1.1 remedies the situation.
@lukas2511 commented on GitHub (Dec 10, 2020):
The main issue is solved and every further fix depends on still-to-be-implemented retry-logic of dehydrated. Closing this issue.