no renewal with pebble #578

New Issue

adam · 2025-12-29T01:27:27+01:00

adam commented

2025-12-29 01:27:27 +01:00

Originally created by @ibukanov on GitHub (Sep 6, 2022).

Originally assigned to: @lukas2511 on GitHub.

It seems dehydrated does not with pebble, a test acme server that is useful for local testing of certificate renewals.

To reproduce, install pebble as instructed in the above link. Typically it is necessary to adjust test/config/pebble-config.json in pebble checkout to match local server HHTP port from 5002 to 8080 or whatever.

Then adjust /etc/localhost to contain:

127.0.0.1 www.example.com

Then configure HTTP server running locally to respond to www.example.com

It is also necessary to set CURL_CA_BUNDLE environment variable to path-pebble-checkout/test/certs/pebble.minica.pem.

Then run pebble and then dehydrated like in:

env CURL_CA_BUNDLE=/path-to-pebble-checkout/pebble.minica.pem dehydrated --acount --accept-terms --ca https://127.0.0.1:14000/dir
env CURL_CA_BUNDLE=/path-to-pebble-checkout/pebble.minica.pem dehydrated --cron --force --accept-terms --ca https://127.0.0.1:14000/dir --domain www.example.com

While the first command reports success, te seconds fails with the log:

# INFO: Using main config file /etc/lms-dehydrated/config
Processing www.example.com
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for www.example.com
 + 1 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for www.example.com authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
 + Requesting certificate...
  + ERROR: An error occurred while sending post-request to https://127.0.0.1:14000/finalize-order/lPlATXt7PCfm1I8lZeY4jMlzZ69NWKJjQoSJRHzzK0c (Status 400)

Details:
HTTP/1.1 400 Bad Request
Cache-Control: public, max-age=0, no-cache
Content-Type: application/problem+json; charset=utf-8
Link: <https://127.0.0.1:14000/dir>;rel="index"
Replay-Nonce: Ry56gB38RD-9RyDlCgJFmg
Date: Tue, 06 Sep 2022 19:32:00 GMT
Content-Length: 147

{
   "type": "urn:ietf:params:acme:error:badNonce",
   "detail": "JWS has an invalid anti-replay nonce: R809jiUv50iSokVBmHNwDA",
   "status": 400
}

EXPECTED value GOT EOF

Originally created by @ibukanov on GitHub (Sep 6, 2022). Originally assigned to: @lukas2511 on GitHub. It seems dehydrated does not with [pebble](https://github.com/letsencrypt/pebble), a test acme server that is useful for local testing of certificate renewals. To reproduce, install pebble as instructed in the above link. Typically it is necessary to adjust test/config/pebble-config.json in pebble checkout to match local server HHTP port from 5002 to 8080 or whatever. Then adjust /etc/localhost to contain: 127.0.0.1 www.example.com Then configure HTTP server running locally to respond to www.example.com It is also necessary to set CURL_CA_BUNDLE environment variable to path-pebble-checkout/test/certs/pebble.minica.pem. Then run pebble and then dehydrated like in: ``` env CURL_CA_BUNDLE=/path-to-pebble-checkout/pebble.minica.pem dehydrated --acount --accept-terms --ca https://127.0.0.1:14000/dir env CURL_CA_BUNDLE=/path-to-pebble-checkout/pebble.minica.pem dehydrated --cron --force --accept-terms --ca https://127.0.0.1:14000/dir --domain www.example.com ``` While the first command reports success, te seconds fails with the log: ``` # INFO: Using main config file /etc/lms-dehydrated/config Processing www.example.com + Signing domains... + Generating private key... + Generating signing request... + Requesting new certificate order from CA... + Received 1 authorizations URLs from the CA + Handling authorization for www.example.com + 1 pending challenge(s) + Deploying challenge tokens... + Responding to challenge for www.example.com authorization... + Challenge is valid! + Cleaning challenge tokens... + Requesting certificate... + ERROR: An error occurred while sending post-request to https://127.0.0.1:14000/finalize-order/lPlATXt7PCfm1I8lZeY4jMlzZ69NWKJjQoSJRHzzK0c (Status 400) Details: HTTP/1.1 400 Bad Request Cache-Control: public, max-age=0, no-cache Content-Type: application/problem+json; charset=utf-8 Link: <https://127.0.0.1:14000/dir>;rel="index" Replay-Nonce: Ry56gB38RD-9RyDlCgJFmg Date: Tue, 06 Sep 2022 19:32:00 GMT Content-Length: 147 { "type": "urn:ietf:params:acme:error:badNonce", "detail": "JWS has an invalid anti-replay nonce: R809jiUv50iSokVBmHNwDA", "status": 400 } EXPECTED value GOT EOF ```

adam closed this issue

2025-12-29 01:27:27 +01:00

adam commented

2025-12-29 01:27:28 +01:00

@ibukanov commented on GitHub (Sep 7, 2022):

The problem is that hydrated does not properly handle nonce. The code uses HEAD to get one, but ACME specs does not allow that. In particularly, the code must track nonce from the previous requests as required by ACME. pebble explicitly deviates from boulder in that respect to test spec conformance.

@ibukanov commented on GitHub (Sep 7, 2022): The problem is that hydrated does not properly handle nonce. The code uses HEAD to get one, but ACME specs does not allow that. In particularly, the code must track nonce from the previous requests as required by ACME. pebble explicitly deviates from boulder in that respect to test spec conformance.

adam commented

2025-12-29 01:27:28 +01:00

@lukas2511 commented on GitHub (Sep 7, 2022):

Mh, has there been an update to the spec I haven't seen yet? The rfc clearly states:

 To get a fresh nonce, the client sends a HEAD request to the newNonce
   resource on the server.  The server's response MUST include a Replay-
   Nonce header field containing a fresh nonce and SHOULD have status
   code 200 (OK).  The server MUST also respond to GET requests for this
   resource, returning an empty body (while still providing a Replay-
   Nonce header) with a status code of 204 (No Content).

Maybe it's intentionally a badNonce error to check for compliance with another part of the rfc?

An error response with the
   "badNonce" error type MUST include a Replay-Nonce header field with a
   fresh nonce that the server will accept in a retry of the original
   query (and possibly in other requests, according to the server's
   nonce scoping policy).  On receiving such a response, a client SHOULD
   retry the request using the new nonce.

But that's only a SHOULD, so except for some test scenarios it should still work in normal operation... I'll look into this... Last time I tried it worked against pebble without any issues.

@lukas2511 commented on GitHub (Sep 7, 2022): Mh, has there been an update to the spec I haven't seen yet? The rfc clearly states: ``` To get a fresh nonce, the client sends a HEAD request to the newNonce resource on the server. The server's response MUST include a Replay- Nonce header field containing a fresh nonce and SHOULD have status code 200 (OK). The server MUST also respond to GET requests for this resource, returning an empty body (while still providing a Replay- Nonce header) with a status code of 204 (No Content). ``` Maybe it's intentionally a `badNonce` error to check for compliance with another part of the rfc? ``` An error response with the "badNonce" error type MUST include a Replay-Nonce header field with a fresh nonce that the server will accept in a retry of the original query (and possibly in other requests, according to the server's nonce scoping policy). On receiving such a response, a client SHOULD retry the request using the new nonce. ``` But that's only a `SHOULD`, so except for some test scenarios it should still work in normal operation... I'll look into this... Last time I tried it worked against pebble without any issues.

adam commented

2025-12-29 01:27:28 +01:00

@lukas2511 commented on GitHub (Sep 7, 2022):

The rfc also states that clients are free to use nonces as they wish, even allowing for something like grabbing a pool of nonces before doing anything else. The only more strict definition is about the badNonce messages, which would require to use the nonce that came with the error response in a retry.

@lukas2511 commented on GitHub (Sep 7, 2022): The rfc also states that clients are free to use nonces as they wish, even allowing for something like grabbing a pool of nonces before doing anything else. The only more strict definition is about the `badNonce` messages, which would require to use the nonce that came with the error response in a retry.

adam commented

2025-12-29 01:27:29 +01:00

@lukas2511 commented on GitHub (Sep 7, 2022):

Okay the spec hasn't changed, this is actually a test-feature of pebble, forcing implementation of retries on badNonce errors... https://github.com/letsencrypt/pebble#invalid-anti-replay-nonce-errors

@lukas2511 commented on GitHub (Sep 7, 2022): Okay the spec hasn't changed, this is actually a test-feature of pebble, forcing implementation of retries on `badNonce` errors... https://github.com/letsencrypt/pebble#invalid-anti-replay-nonce-errors

adam commented

2025-12-29 01:27:29 +01:00

@lukas2511 commented on GitHub (Sep 7, 2022):

I've implemented a workaround for this in 6fb8eba56a. Not going to merge it into master yet, as I'm hoping to clean up some implementation details soon, hopefully making such retry logic a lot easier.

@lukas2511 commented on GitHub (Sep 7, 2022): I've implemented a workaround for this in 6fb8eba56a67af8e8b5528b669a5da923b5d2182. Not going to merge it into master yet, as I'm hoping to clean up some implementation details soon, hopefully making such retry logic a lot easier.

adam commented

2025-12-29 01:27:29 +01:00

@ibukanov commented on GitHub (Sep 7, 2022):

The checkout from 6fb8eba56a works, thanks for a quick fix and for the link about the pebble on purpose failing here.

I suspect that others have not reported the issue previously because by default pebble fails in 5% of requests. If one uses it just to get single certificate, then the issue is not that visible. But it is almost certain to fail when one uses a test a setup with 15 different host names...

@ibukanov commented on GitHub (Sep 7, 2022): The checkout from https://github.com/dehydrated-io/dehydrated/commit/6fb8eba56a67af8e8b5528b669a5da923b5d2182 works, thanks for a quick fix and for the link about the pebble on purpose failing here. I suspect that others have not reported the issue previously because by default pebble fails in 5% of requests. If one uses it just to get single certificate, then the issue is not that visible. But it is almost certain to fail when one uses a test a setup with 15 different host names...

adam commented

2025-12-29 01:27:30 +01:00

@lukas2511 commented on GitHub (Oct 31, 2022):

fix has now been released

@lukas2511 commented on GitHub (Oct 31, 2022): fix has now been released

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/dehydrated#578