Certs rate limiting #84

@yusing commented on GitHub (May 31, 2025):

You said the container kept restarting right? So there should be logs from previous restart unless you manually docker compose down.

Try again on 2025-06-01 17:24:08 UTC then.

@yusing commented on GitHub (May 31, 2025): You said the container kept restarting right? So there should be logs from previous restart unless you manually `docker compose down`. Try again on 2025-06-01 17:24:08 UTC then.

adam commented

@zachkont commented on GitHub (Jun 1, 2025):

Unfortunately I'm running it via portainer with gitops updates and I only noticed after updating and the original container has since been deleted. I believe it only started happening after updating to v0.13.5 too but I can't be sure. Anything else I can try?

@zachkont commented on GitHub (Jun 1, 2025): Unfortunately I'm running it via portainer with gitops updates and I only noticed after updating and the original container has since been deleted. I believe it only started happening after updating to v0.13.5 too but I can't be sure. Anything else I can try?

adam commented

@yusing commented on GitHub (Jun 2, 2025):

Update to the current latest version and try again?

@yusing commented on GitHub (Jun 2, 2025): Update to the current latest version and try again?

adam commented

@zachkont commented on GitHub (Jun 2, 2025):

Still the same :/

06-02 09:23 INF GoDoxy version v0.13.8

06-02 09:23 INF loaded route providers

                  • docker@local 8 routes

                

2025/06/02 09:23:14 [INFO] acme: Trying to resolve account by key

06-02 09:23 INF reused acme registration from private key

2025/06/02 09:23:14 [INFO] [*.lab.domain.me, lab.domain.me] acme: Obtaining bundled SAN certificate

06-02 09:23 FTL autocert setup error

                  • acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-06-03 03:03:47 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames

@zachkont commented on GitHub (Jun 2, 2025): Still the same :/ ``` 06-02 09:23 INF GoDoxy version v0.13.8 06-02 09:23 INF loaded route providers • docker@local 8 routes 2025/06/02 09:23:14 [INFO] acme: Trying to resolve account by key 06-02 09:23 INF reused acme registration from private key 2025/06/02 09:23:14 [INFO] [*.lab.domain.me, lab.domain.me] acme: Obtaining bundled SAN certificate 06-02 09:23 FTL autocert setup error • acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-06-03 03:03:47 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames ```

adam commented

@zachkont commented on GitHub (Jun 2, 2025):

If it helps, my server just had a filesystem corruption issue and I've had to manually run fsck on the affected volume, which is also where docker volumes reside. Could it be related to that and if so, how can I make it recover?

@zachkont commented on GitHub (Jun 2, 2025): If it helps, my server just had a filesystem corruption issue and I've had to manually run `fsck` on the affected volume, which is also where docker volumes reside. Could it be related to that and if so, how can I make it recover?

adam commented

@yusing commented on GitHub (Jun 2, 2025):

too many certificates (5)

Do you have other reverse proxy / certbot registering certs for the same domain? I don't think it's related to filesystem corruption.

@yusing commented on GitHub (Jun 2, 2025): > too many certificates (5) Do you have other reverse proxy / certbot registering certs for the same domain? I don't think it's related to filesystem corruption.

adam commented

@zachkont commented on GitHub (Jun 2, 2025):

No I don't, and godoxy has been working perfectly so far for me

@zachkont commented on GitHub (Jun 2, 2025): No I don't, and godoxy has been working perfectly so far for me

adam commented

@zachkont commented on GitHub (Jun 2, 2025):

In order to help with debugging as much as possible, here is a domain where this happens: https://crt.sh/?q=kontoulis.cloud

@zachkont commented on GitHub (Jun 2, 2025): In order to help with debugging as much as possible, here is a domain where this happens: https://crt.sh/?q=kontoulis.cloud

adam commented

@yusing commented on GitHub (Jun 2, 2025):

It's weird, I only see two certs obtained on 2nd June and 31th May but those errors showed 5.

Could you try running with bare docker compose (no portainer) so failed requests can be logged? Please also clean up certs directory.

@yusing commented on GitHub (Jun 2, 2025): It's weird, I only see two certs obtained on 2nd June and 31th May but those errors showed 5. Could you try running with bare docker compose (no portainer) so failed requests can be logged? Please also clean up `certs` directory.

adam commented

@zachkont commented on GitHub (Jun 2, 2025):

So I have already tried cleaning up the certs directory, got an error saying acme.key is not readable, added only that back in, got the error I mentioned above.

Now I just tried moving priv.key and cert.crt back into certs/ and I get an additional line

06-02 18:22 INF GoDoxy version v0.13.8

06-02 18:22 INF loaded route providers

                  • docker@local 8 routes

                

06-02 18:22 INF next renewal in -8 days and 14 minutes

06-02 18:22 INF certs expired, renewing

2025/06/02 18:22:02 [INFO] acme: Trying to resolve account by key

06-02 18:22 INF reused acme registration from private key

2025/06/02 18:22:02 [INFO] [*.lab.kontoulis.cloud, lab.kontoulis.cloud] acme: Obtaining bundled SAN certificate

06-02 18:22 FTL autocert setup error

                  • acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-06-03 03:04:03 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames

I will try running without portainer and post findings

@zachkont commented on GitHub (Jun 2, 2025): So I have already tried cleaning up the certs directory, got an error saying `acme.key` is not readable, added only that back in, got the error I mentioned above. Now I just tried moving `priv.key` and `cert.crt` back into `certs/` and I get an additional line ``` 06-02 18:22 INF GoDoxy version v0.13.8 06-02 18:22 INF loaded route providers • docker@local 8 routes 06-02 18:22 INF next renewal in -8 days and 14 minutes 06-02 18:22 INF certs expired, renewing 2025/06/02 18:22:02 [INFO] acme: Trying to resolve account by key 06-02 18:22 INF reused acme registration from private key 2025/06/02 18:22:02 [INFO] [*.lab.kontoulis.cloud, lab.kontoulis.cloud] acme: Obtaining bundled SAN certificate 06-02 18:22 FTL autocert setup error • acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-06-03 03:04:03 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames ``` I will try running without portainer and post findings

adam commented

@zachkont commented on GitHub (Jun 3, 2025):

However, the documentation states

Up to 5 certificates can be issued per exact same set of hostnames every 7 days.

so it's not a daily limit which explains the error right? Why does the app register so many new credentials for the same domain? Shouldn't it check if it already exists or something?

@zachkont commented on GitHub (Jun 3, 2025): However, the [documentation](https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames) states >Up to 5 certificates can be issued per exact same set of hostnames every 7 days. so it's not a daily limit which explains the error right? Why does the app register so many new credentials for the same domain? Shouldn't it check if it already exists or something?

adam commented

@yusing commented on GitHub (Jun 3, 2025):

Why does the app register so many new credentials for the same domain?

Because of the file system corruption?

Btw you do not move those domains those certs back in, in this case. Just clean the directory and obtain a new one after "unban".

@yusing commented on GitHub (Jun 3, 2025): > Why does the app register so many new credentials for the same domain? Because of the file system corruption? Btw you do not move those domains those certs back in, in this case. Just clean the directory and obtain a new one after "unban".

adam commented

@zachkont commented on GitHub (Jun 3, 2025):

Because of the file system corruption?

hmm so maybe because it could not read the existing cert file it thought there was none and tried to make a new one? Which looks like it started on 2025-05-28, before that it seemed more reasonable.

However, it does look like the app does not handle this error gracefully so maybe that can be improved if you'd like. At least avoid the container going in a crash loop. In any case, I'll report back when the "ban" is lifted and close this, thanks for being so responsive and helpful

@zachkont commented on GitHub (Jun 3, 2025): >Because of the file system corruption? hmm so maybe because it could not read the existing cert file it thought there was none and tried to make a new one? Which looks like it started on 2025-05-28, before that it seemed more reasonable. However, it does look like the app does not handle this error gracefully so maybe that can be improved if you'd like. At least avoid the container going in a crash loop. In any case, I'll report back when the "ban" is lifted and close this, thanks for being so responsive and helpful

adam commented

@zachkont commented on GitHub (Jun 10, 2025):

Hi again, I'm back with a different error:

06-10 15:45 INF GoDoxy version v0.14.2
06-10 15:45 INF failed to load ACME private key, generating a now one error="open certs/acme.key: no such file or directory"
06-10 15:45 WRN skipping conflicting route
• route with alias godoxy already exists
container godoxy
conflicting container godoxy-frontend

06-10 15:45 INF loaded route providers
• docker@local 9 routes

06-10 15:45 WRN errors in config
• config load error
• save ACME private key
• open certs/acme.key: permission denied

06-10 15:45 INF autocert not configured
06-10 15:45 INF http_routes: added home-assistant
06-10 15:45 INF http_routes: added godoxy
06-10 15:45 INF http_routes: added portainer
06-10 15:45 INF http_routes: added wallos
06-10 15:45 INF http_routes: added orb-sensor
06-10 15:45 INF http_routes: added everything-presence-mmwave-configurator
06-10 15:45 INF http_routes: added music-assistant
06-10 15:45 INF http_routes: added adguard
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xaedae0]

I deleted the contents of the /certs dir as you mentioned but it tries to find them instead of recreating. the folder permissions are
drwxr-xr-x 2 root docker 4.0K Jun 3 11:31 certs

@zachkont commented on GitHub (Jun 10, 2025): Hi again, I'm back with a different error: > 06-10 15:45 INF GoDoxy version v0.14.2 > 06-10 15:45 INF failed to load ACME private key, generating a now one error="open certs/acme.key: no such file or directory" > 06-10 15:45 WRN skipping conflicting route > • route with alias godoxy already exists > container godoxy > conflicting container godoxy-frontend > > > 06-10 15:45 INF loaded route providers > • docker@local 9 routes > > 06-10 15:45 WRN errors in config > • config load error > • save ACME private key > • open certs/acme.key: permission denied > > 06-10 15:45 INF autocert not configured > 06-10 15:45 INF http_routes: added home-assistant > 06-10 15:45 INF http_routes: added godoxy > 06-10 15:45 INF http_routes: added portainer > 06-10 15:45 INF http_routes: added wallos > 06-10 15:45 INF http_routes: added orb-sensor > 06-10 15:45 INF http_routes: added everything-presence-mmwave-configurator > 06-10 15:45 INF http_routes: added music-assistant > 06-10 15:45 INF http_routes: added adguard > panic: runtime error: invalid memory address or nil pointer dereference > [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xaedae0] I deleted the contents of the `/certs` dir as you mentioned but it tries to find them instead of recreating. the folder permissions are `drwxr-xr-x 2 root docker 4.0K Jun 3 11:31 certs`

adam commented

@yusing commented on GitHub (Jun 10, 2025):

route with alias godoxy already exists

rename godoxy container to something else like godoxy-proxy

open certs/acme.key: permission denied

check for UID and GID in your .env and chown to corresponding user and group ids

@yusing commented on GitHub (Jun 10, 2025): > route with alias godoxy already exists rename godoxy container to something else like `godoxy-proxy` > open certs/acme.key: permission denied check for `UID` and `GID` in your `.env` and `chown` to corresponding user and group ids

adam commented