Sharing with the ssl-cert group #534

@kousu commented on GitHub (Apr 24, 2021):

I'm going to solve this for now with a hook that does chgrp -R ssl-cert /var/lib/dehydrated && chmod -R g+rX /var/lib/dehydrated/ on every renewal, but I don't want to use too many hooks because the lack of #270 means it's easy for them to get out of hand or be accidentally broken when combining features using ansible or other config-management setups.

@kousu commented on GitHub (Apr 24, 2021): I'm going to solve this for now with a hook that does `chgrp -R ssl-cert /var/lib/dehydrated && chmod -R g+rX /var/lib/dehydrated/` on every renewal, but I don't want to use too many hooks because the lack of #270 means it's easy for them to get out of hand or be accidentally broken when combining features using ansible or other config-management setups.

adam commented

@kousu commented on GitHub (Apr 24, 2021):

Just to compare, I've solved this in the past with certbot on linux by doing:

sudo chmod -R g+s /etc/letsencrypt
sudo chgrp -R ssl-cert /etc/letsencrypt

And iirc on BSD systems, the group is just inherited by default so it's even easier there. But dehydrated would still need this patch:

-umask 077
+umask 007

to be compatible with that.

Indeed, this seems to solve my problem without deploying a hook:

# apt-get autoremove -y dehydrated # start fresh
# apt-get install -y dehydrated
# sed -i 's/umask 077/umask 007/' $(command -v dehydrated)  # patch dehydrated
# for daemon in netdata postfix; do usermod -a -G ssl-cert $daemon; done
# mkdir -p /var/lib/dehydrated/certs
# chgrp ssl-cert /var/lib/dehydrated/certs
# chmod -R g+s /var/lib/dehydrated/certs

which is a less invasive patch than adding a new config option.

@kousu commented on GitHub (Apr 24, 2021): Just to compare, I've solved this in the past with `certbot` on linux by doing: ``` sudo chmod -R g+s /etc/letsencrypt sudo chgrp -R ssl-cert /etc/letsencrypt ``` And iirc on BSD systems, the group is just inherited by default so it's even easier there. But dehydrated would still need this patch: ```diff -umask 077 +umask 007 ``` to be compatible with that. Indeed, this seems to solve my problem without deploying a hook: ``` # apt-get autoremove -y dehydrated # start fresh # apt-get install -y dehydrated # sed -i 's/umask 077/umask 007/' $(command -v dehydrated) # patch dehydrated # for daemon in netdata postfix; do usermod -a -G ssl-cert $daemon; done # mkdir -p /var/lib/dehydrated/certs # chgrp ssl-cert /var/lib/dehydrated/certs # chmod -R g+s /var/lib/dehydrated/certs ``` which is a less invasive patch than adding a new config option.

adam commented

@egberts commented on GitHub (Apr 25, 2021):

First of all, ssl-cert group is for admins who are tasked with creation and distribution of TLS/SSL, et. al., only.

To give ssl-cert group to any servers would be giving away the store to all things, PEM-wise.

Which is not a good thing.

The proper thing to do is to use your openssl after changing to this group, then copy/move the files into each server’s own /etc subdirectories using its server’s file ownership/permission.

@egberts commented on GitHub (Apr 25, 2021): First of all, `ssl-cert` group is for admins who are tasked with creation and distribution of TLS/SSL, et. al., only. To give `ssl-cert` group to any servers would be giving away the store to all things, PEM-wise. Which is not a good thing. The proper thing to do is to use your openssl after changing to this group, then copy/move the files into each server’s own /etc subdirectories using its server’s file ownership/permission.

adam commented

@kousu commented on GitHub (Apr 25, 2021):

Thanks for chiming in. That's an interesting approach. I can't tell if it's "proper" though. I get the sense there is no consensus around managing certs.

You're suggesting something like this?

cp /var/lib/dehydrated/example.com/privkey.pem /etc/postfix/privkey.pem
chown postfix:postfix /etc/postfix/privkey.pem
cp /var/lib/dehydrated/example.com/fullchain.pem /etc/postfix/cert.pem
chown postfix:postfix /etc/postfix/cert.pem


cp /var/lib/dehydrated/example.com/privkey.pem /etc/netdata/privkey.pem
chown netdata:netdata /etc/netdata/privkey.pem
cp /var/lib/dehydrated/example.com/fullchain.pem /etc/netdata/cert.pem
chown netdata:netdata /etc/postfix/cert.pem

...

On my systems /etc/ is root:root everywhere, so the custom permissions would stick out oddly.

Plus it would be hard to maintain since you'd have a hard time remembering what to copy where. I could do most of it with dehydrated's hook feature, but what's on disk and what's in the script could fall out of sync, especially if I removed certs.

Most linuxes come with /etc/ssl/certs and /etc/ssl/private, because that's what openssl expects, which would seem to be another option, but I've never seen /etc/ssl/private actually used. Debian comes with /etc/ssl/private/ssl-cert-snakeoil.key but I've never seen it used; its certbot uses /etc/letsencrypt and its dehydrated uses /var/lib/dehydrated/certs.

To give ssl-cert group to any servers would be giving away the store to all things, PEM-wise.

Which is not a good thing.

This is probably what I meant by "imperfect" above. But I don't think it's that big of a problem.

In the common case, I have a single server, with a single domain, with multiple daemons, and I want them all to share the same single cert. Then, it doesn't matter if the daemons can see each others certs, they're all the same cert. That's my current use-case.

In the vhosting case, with multiple sites on single daemons, unix permissions aren't enough to save you; if you have, say, nginx or postfix running multiple domains, they're running them all as the www-data or postfix user, so if one domain is compromised they all are. That's just the risk of vhosting, and part of why vhosting is cheaper than other plans.

The only imperfect case I see is if you have different daemons running different subdomains. In that case, I agree you can do better than the status quo. You can give postfix access to /var/lib/dehydrated/mail.example.org/ (or copy the contents elsewhere if you want) and nginx /var/lib/dehydrated/www.example.org and they don't need to cross paths. In fact, if you did have certs segrated to specific daemons, maybe you could even make it permanent by chgrp www-data /var/lib/dehydrated/www.example.org; chmod u+rXs /var/lib/dehydrated/www.example.org --- except that won't work with dehydrated because of the umask 077 above; it needs to become umask 027 or umask 007.

The most common status quo I know of -- nginx is an example -- is for daemons to start as root, load their certs, and then setuid() to become their real user. In that case, you can isolate the different certs, because after setuid() no certs are readable anymore by the different daemons. But also in that case there's no need to copy your certs to any special folders in the first place.

But other daemons, like netdata, disagree with using setuid(). They would rather start as their daemon user immediately. It's these daemons that I'm having trouble integrating with dehydrated, and these daemons that made me chgrp ssl-cert /etc/letsencrypt back in the day, and chgrp ssl-cert /var/lib/dehydrated/certs now.

I think, probably, the "right" answer is to make everything root:root and rely on daemons doing setuid(). But I can't control all the daemons. So personally, I don't think there really is a right answer, only best efforts.

Andddd to bring this back on topic, my best effort would be helped a lot with this patch:

-umask 077 # paranoid umask, we're creating private keys 
+umask 027 # paranoid umask, we're creating private keys

But maybe there's a better solution floating out there?

@kousu commented on GitHub (Apr 25, 2021): Thanks for chiming in. That's an interesting approach. I can't tell if it's "proper" though. I get the sense there is no consensus around managing certs. You're suggesting something like this? ``` cp /var/lib/dehydrated/example.com/privkey.pem /etc/postfix/privkey.pem chown postfix:postfix /etc/postfix/privkey.pem cp /var/lib/dehydrated/example.com/fullchain.pem /etc/postfix/cert.pem chown postfix:postfix /etc/postfix/cert.pem cp /var/lib/dehydrated/example.com/privkey.pem /etc/netdata/privkey.pem chown netdata:netdata /etc/netdata/privkey.pem cp /var/lib/dehydrated/example.com/fullchain.pem /etc/netdata/cert.pem chown netdata:netdata /etc/postfix/cert.pem ... ``` On my systems `/etc/` is `root:root` everywhere, so the custom permissions would stick out oddly. Plus it would be hard to maintain since you'd have a hard time remembering what to copy where. I could do most of it with dehydrated's hook feature, but what's on disk and what's in the script could fall out of sync, especially if I removed certs. Most linuxes come with /etc/ssl/certs and /etc/ssl/private, because that's what `openssl` expects, which would seem to be another option, but I've never seen `/etc/ssl/private` actually used. Debian comes with `/etc/ssl/private/ssl-cert-snakeoil.key` but I've never seen it used; its `certbot` uses `/etc/letsencrypt` and its `dehydrated` uses `/var/lib/dehydrated/certs`. > To give ssl-cert group to any servers would be giving away the store to all things, PEM-wise. > > Which is not a good thing. This is probably what I meant by "imperfect" above. But I don't think it's that big of a problem. In the common case, I have a single server, with a single domain, with multiple daemons, and I want them all to share the same single cert. Then, it doesn't matter if the daemons can see each others certs, they're all the same cert. That's my current use-case. In the vhosting case, with multiple sites on single daemons, unix permissions aren't enough to save you; if you have, say, nginx or postfix running multiple domains, they're running them all as the `www-data` or `postfix` user, so if one domain is compromised they all are. That's just the risk of vhosting, and part of why vhosting is cheaper than other plans. The only imperfect case I see is if you have different daemons running different subdomains. In that case, I agree you can do better than the status quo. You can give postfix access to `/var/lib/dehydrated/mail.example.org/` (or copy the contents elsewhere if you want) and nginx `/var/lib/dehydrated/www.example.org` and they don't need to cross paths. In fact, if you did have certs segrated to specific daemons, maybe you could even make it permanent by `chgrp www-data /var/lib/dehydrated/www.example.org; chmod u+rXs /var/lib/dehydrated/www.example.org` --- except that won't work with dehydrated because of the `umask 077` above; it needs to become `umask 027` or `umask 007`. The most common status quo I know of -- nginx is an example -- is for daemons to start as root, load their certs, and then `setuid()` to become their real user. In that case, you can isolate the different certs, because after `setuid()` no certs are readable anymore by the different daemons. But also in that case there's no need to copy your certs to any special folders in the first place. But other daemons, like `netdata`, [disagree](http://web.archive.org/web/20210425022659/https://learn.netdata.cloud/docs/agent/netdata-security#your-systems-are-safe-with-netdata) with using `setuid()`. They would rather start as their daemon user immediately. It's these daemons that I'm having trouble integrating with `dehydrated`, and these daemons that made me `chgrp ssl-cert /etc/letsencrypt` back in the day, and `chgrp ssl-cert /var/lib/dehydrated/certs` now. I think, probably, the "right" answer is to make everything root:root and rely on daemons doing `setuid()`. But I can't control all the daemons. So personally, I don't think there really is a right answer, only best efforts. Andddd to bring this back on topic, my best effort would be helped a lot with this patch: ```diff -umask 077 # paranoid umask, we're creating private keys +umask 027 # paranoid umask, we're creating private keys ``` But maybe there's a better solution floating out there?

adam commented

@txr13 commented on GitHub (Apr 25, 2021):

First off, dehydrated does use permissions 0600, but not necessarily root:root. Literally the first two options in the config file are the user and group you want it to run as. So if you want the group set to something custom like ssl-cert or whatever, set the DEHYDRATED_GROUP in your config file as that. Now dehydrated runs as root:$DEHYDRATED_GROUP and you can use the deploy_cert hook to set whatever permissions you want.

Alternately, if you call dehydrated from a non-root crontab, it will happily create files as whatever user it runs under. Doesn't have to be root. I use this for my own central certificate management server. Still creates the permissions as 0600, but that doesn't matter to me because of my export process.

Because I have a central certificate management server, I have a separate, non-dehydrated update script which compares the hash of the current certificate on disk against the recorded hash of the last update run. If the current certificate on disk has been updated, the script then checks for a certificate-specific export script or otherwise runs the default certificate export options I desire. The export script for each certificate will handle the necessary steps for whatever the target use is, including concatenating the files into the target format (eg. privkey/fullchain, or fullchain/privkey, or fullchain and privkey in separate files, or export to PKCS#8, or export to PKCS#12, etc.), sending the certificate to a different server (via file mounts for Windows or scp for *nix), and calling the daemon's update script on those servers.

The point being, dehydrated just does what it does--request and write out certificates. There's enough different options out there that it's far easier to simply write yourself an export / update script per certificate (for large installations) or do the necessary in your deploy_cert hook (for small installations). I mean, that's why the deploy_cert hook exists--to make any needed modifications and/or move the certificate to its final location and/or call whatever update process needs to be invoked. I much prefer dehydrated itself to simply write the certificates in a reasonably locked-down fashion, especially since you can use the config file to specify which user and group to run as.

@txr13 commented on GitHub (Apr 25, 2021): First off, dehydrated does use permissions `0600`, but not necessarily `root:root`. Literally the first two options in the config file are the user and group you want it to run as. So if you want the group set to something custom like `ssl-cert` or whatever, set the `DEHYDRATED_GROUP` in your config file as that. Now dehydrated runs as `root:$DEHYDRATED_GROUP` and you can use the `deploy_cert` hook to set whatever permissions you want. Alternately, if you call dehydrated from a non-root crontab, it will happily create files as whatever user it runs under. Doesn't have to be root. I use this for my own central certificate management server. Still creates the permissions as `0600`, but that doesn't matter to me because of my export process. Because I have a central certificate management server, I have a separate, non-dehydrated update script which compares the hash of the current certificate on disk against the recorded hash of the last update run. If the current certificate on disk has been updated, the script then checks for a certificate-specific export script or otherwise runs the default certificate export options I desire. The export script for each certificate will handle the necessary steps for whatever the target use is, including concatenating the files into the target format (eg. privkey/fullchain, or fullchain/privkey, or fullchain and privkey in separate files, or export to PKCS#8, or export to PKCS#12, etc.), sending the certificate to a different server (via file mounts for Windows or scp for *nix), and calling the daemon's update script on those servers. The point being, dehydrated just does what it does--request and write out certificates. There's enough different options out there that it's far easier to simply write yourself an export / update script per certificate (for large installations) or do the necessary in your `deploy_cert` hook (for small installations). I mean, that's why the `deploy_cert` hook exists--to make any needed modifications and/or move the certificate to its final location and/or call whatever update process needs to be invoked. I much prefer dehydrated itself to simply write the certificates in a reasonably locked-down fashion, especially since you can use the config file to specify which user and group to run as.

adam commented

@lukas2511 commented on GitHub (Apr 25, 2021):

Yea, like @txr13 mentioned it's not easy to implement a solution that will fit all use-cases, that's why dehydrated just tries to create the files with secure permissions and it is up to your hook scripts to change ownership/permissions, deploy the certificate to somewhere else, reload services, etc.

I will not add any more privilege-modifying or -seperating stuff to dehydrated in the near future. Tbh I now think it was a big mistake to add even the sudo stuff in the first place, but now it's here and I don't want to break compatibility.

@lukas2511 commented on GitHub (Apr 25, 2021): Yea, like @txr13 mentioned it's not easy to implement a solution that will fit all use-cases, that's why dehydrated just tries to create the files with secure permissions and it is up to your hook scripts to change ownership/permissions, deploy the certificate to somewhere else, reload services, etc. I will not add any more privilege-modifying or -seperating stuff to dehydrated in the near future. Tbh I now think it was a big mistake to add even the sudo stuff in the first place, but now it's here and I don't want to break compatibility.

adam commented

@stokito commented on GitHub (Jun 8, 2022):

@kousu you made very good points. I think that it should be the centralized location of certs and all ACME clients should place certs there with the same permissions. Please see my comment for Let's Encrypt https://github.com/certbot/certbot/issues/1425#issuecomment-1150116062

@stokito commented on GitHub (Jun 8, 2022): @kousu you made very good points. I think that it should be the centralized location of certs and all ACME clients should place certs there with the same permissions. Please see my comment for Let's Encrypt https://github.com/certbot/certbot/issues/1425#issuecomment-1150116062

adam commented

https://egbert.net/blog/articles/ca-certificates-rebuild-on-debian.html

@egberts commented on GitHub (Jun 8, 2022):

I am not saying that there is a right way but it is very clear that there is a severe divergence of repositories of PKI Root CAs.

I even have a problem with Debian PKI Root CA approach. Of which I’ve detailed here:

@egberts commented on GitHub (Jun 8, 2022): I am not saying that there is a right way but it is very clear that there is a severe divergence of repositories of PKI Root CAs. I even have a problem with Debian PKI Root CA approach. Of which I’ve detailed here: https://egbert.net/blog/articles/ca-certificates-rebuild-on-debian.html

adam commented

2025-12-29 01:26:57 +01:00

@egberts commented on GitHub (Jun 8, 2022):

Also, it does make it easier to managed multiple (but private) PKI Root CAs when having a centralized SSL directory as I do here for my some-23 odd CA certificate types.

https://github.com/egberts/tls-ca-manage

@egberts commented on GitHub (Jun 8, 2022): Also, it does make it easier to managed multiple (but private) PKI Root CAs when having a centralized SSL directory as I do here for my some-23 odd CA certificate types. https://github.com/egberts/tls-ca-manage

adam commented

@kousu commented on GitHub (Jun 8, 2022):

Those links are interesting, @egberts, but I'm not sure I understand why they're relevant. It sounds like tls-ca-manage does the same thing as dehydrated but with a private CA? I'm using dehydrated because I want to use a public CA that my users won't have trouble with.

Anyway the point is, this was wontfixed. The best I can do is write a deploy_hook that does chown -R :ssl-cert ... && chmod -R g+rX ....

That's not so bad. It at least doesn't require writing one hook per daemon. I means I can treat my ansible role that uses dehydrated as dependency of other ansible roles: I just need to make sure they deploy daemons to :ssl-cert with their TLS configs pointed at /etc/ssl/acme.

What I was trying to avoid was having to have each ansible role write hooks into dehydrated (which would break encapsulation IMO). "Just use hooks to solve it" is a reasonable answer for managing individual servers by hand, but it's a stretch when managing a fleet. I think anyone with a fleet to manage is going to have the same problem, no matter whether they're using ansible or puppet or terraform or whatever other configuration tool. I think this is the point @stokito is making to certbot, too, who apparently also haven't figured out how to handle giving SSL to more than one daemon on a server at a time?

@kousu commented on GitHub (Jun 8, 2022): Those links are interesting, @egberts, but I'm not sure I understand why they're relevant. It sounds like `tls-ca-manage` does the same thing as `dehydrated` but with a private CA? I'm using `dehydrated` because I want to use a public CA that my users won't have trouble with. Anyway the point is, this was wontfixed. The best I can do is write a `deploy_hook` that does `chown -R :ssl-cert ... && chmod -R g+rX ...`. That's not so bad. It at least doesn't require writing one hook per daemon. I means I can treat my ansible role that uses `dehydrated` as dependency of other ansible roles: I just need to make sure they deploy daemons to `:ssl-cert` with their TLS configs pointed at `/etc/ssl/acme`. What I was trying to avoid was having to have each ansible role write hooks into `dehydrated` (which would break encapsulation IMO). "Just use hooks to solve it" is a reasonable answer for managing individual servers by hand, but it's a stretch when managing a fleet. I think anyone with a fleet to manage is going to have the same problem, no matter whether they're using ansible or puppet or terraform or whatever other configuration tool. I think this is the point @stokito is making to certbot, too, who apparently also haven't figured out how to handle giving SSL to more than one daemon on a server at a time?

adam commented

@egberts commented on GitHub (Jun 8, 2022):

These given links are only examples of why no daemon should ever use the ssl-cert group ID/name.

No Daemons are privileged to examine keys of other servers’ PEM through ssl-cert supplementary group name/ID.

@egberts commented on GitHub (Jun 8, 2022): These given links are only examples of why no daemon should ever use the `ssl-cert` group ID/name. No Daemons are privileged to examine keys of other servers’ PEM through `ssl-cert` supplementary group name/ID.

adam commented

@stokito commented on GitHub (Jun 8, 2022):

Maybe as a default behavior that may be fine for most cases. Those who need more can adjust users manually. You know, today almost everything is dockerized so files are isolated.
Personally I working in direction of self hosting when non experienced users can install their own services on their RaspberryPi or even on a router with OpenWrt. And ideally they shouldn't even open a terminal to edit something.

@stokito commented on GitHub (Jun 8, 2022): Maybe as a default behavior that may be fine for most cases. Those who need more can adjust users manually. You know, today almost everything is dockerized so files are isolated. Personally I working in direction of self hosting when non experienced users can install their own services on their RaspberryPi or even on a router with OpenWrt. And ideally they shouldn't even open a terminal to edit something.

adam commented