Let's Encrypt retired and new issuer CAs
The major changes in the Let's Encrypt issuer CA lineup noted in my previous post:
https://list.sys4.de/hyperkitty/list/dane-users@list.sys4.de/message/ZTM3XQM...
are now largely completed. Of the ~46000 domains with working DANE-TA(2) TLSA records matching a Let's Encrypt intermediate issuer, just 62 are still based on R3, and none on X3, X4, R4, E1 or E2.
These last few R3 issued certificates will either be renewed or will expire by September 4th.
Therefore, if you haven't done so already, please read the fine advice in:
https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
and switch to R10..R14 or E5..E9 (or rarely both) as appropriate. If you prefer to instead pin the ISRG root CAs, you MUST ensure that your SMTP server's chain file also includes the ISRG X1 or ISRG X2 root (whichever happened to issue the intermediate CA cert), and then you can publish TLSA records matching these roots.
https://dane.sys4.de/common_mistakes#4 https://github.com/Mailu/Mailu/issues/2138 https://datatracker.ietf.org/doc/html/rfc7671#section-5.2.3
Note that some MTA operators have made the mistake of listing just R10 or R11 (similary just E5 or E6), whichever was the first new issuer they saw, without understanding that the issuer will randomly rotate between these, and may in an emergency be one of their backups.
DO NOT be tempted to skimp on the list of published TAs, if you're keen on using DANE-TA(2) with Let's Encrypt, publish the full set, and keep track of periodic Let's Encrypt service announcements.
An of course, DO NOT neglect monitoring, perhaps based on:
https://list.sys4.de/hyperkitty/list/dane-users@list.sys4.de/thread/NKDBQABS...
And of course, it may be simplest to stop playing Let's Encrypt TA whack-a-mole, and switch to "3 1 1" records:
https://mail.sys4.de/pipermail/dane-users/2018-February/000440.html
Perhaps with the aid of:
https://github.com/tlsaware/danebot
or similar/equivalent. Best of luck, but, if can you pay attention to detail, you should not need it.
hi,
If you prefer to instead pin the ISRG root CAs, you MUST ensure that your SMTP server's chain file also includes the ISRG X1 or ISRG X2 root (whichever happened to issue the intermediate CA cert), and then you can publish TLSA records matching these roots.
thx for the heads-up.
i'm using dane "3 1 2" records; all checks are good.
i note LE's current Chain of Trust currently are
roots: ISRG Root X1: RSA 4096, until 2030-06-04 (generated 2015-06-04) ISRG Root X2: ECDSA P-384, until 2035-09-04 (generated 2020-09-04) and
intermediates: Let’s Encrypt E5: ECDSA P-384, until 2027-03-12 Let’s Encrypt E6: ECDSA P-384, until 2027-03-12 Let’s Encrypt R10: RSA 2048, until 2027-03-12 Let’s Encrypt R11: RSA 2048, until 2027-03-12
to date, my cert generation has had
DEFAULT_PREFERRED_CHAIN='ISRG Root X1'
, i.e. RSA.
and, i publish both RSA and ECDSA DANE records.
reading @
https://letsencrypt.org/upcoming-features/ Completed Features ECDSA Root and Intermediates
Enabled: June 06, 2024
We are issuing certificates from our production ECDSA intermediates to ECDSA leaf certificates. See the Chains of Trust documentation for full details on our PKI hierarchy.
iiuc, that's full/official support for ECDSA issuance.
and a switch to ECDSA
DEFAULT_PREFERRED_CHAIN='ISRG Root X2'
should be (?) reasonably well tolerated.
i'm unclear on
is it DANE-safe? or DANE-recommended?
(re-)reading the OP, I _think_ we're ok ... but, best to double-check.
On Sun, Aug 25, 2024 at 12:07:34PM -0400, pgnd wrote:
I'm using dane "3 1 2" records; all checks are good.
Which domain(s)?
I note LE's current Chain of Trust currently are
roots: ISRG Root X1: RSA 4096, until 2030-06-04 (generated 2015-06-04)
See below, X1 is presently set to expire in 2035.
ISRG Root X2: ECDSA P-384, until 2035-09-04 (generated 2020-09-04)and
intermediates: Let’s Encrypt E5: ECDSA P-384, until 2027-03-12 Let’s Encrypt E6: ECDSA P-384, until 2027-03-12 Let’s Encrypt R10: RSA 2048, until 2027-03-12 Let’s Encrypt R11: RSA 2048, until 2027-03-12
(plus 3 backups for each pair)
to date, my cert generation has had
DEFAULT_PREFERRED_CHAIN='ISRG Root X1'
, i.e. RSA.
Yes, The root is RSA, but in many cases the underlying chain is ECDSA, There are two versions of E5..9 certs, with one signed by X1 and the other by X2.
_25._tcp.mail.censored.example. IN TLSA 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3 _25._tcp.mail.censored.example. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d mail.censored.example[192.0.2.1]: pass: TLSA match: depth = 2, name = mail.censored.example TLS = TLS13 with AES256GCM-SHA384,X25519,PubKeyALG_EC name = mail.censored.example depth = 0 Issuer CommonName = E5 Issuer Organization = Let's Encrypt notBefore = 2024-08-16T19:30:03Z notAfter = 2024-11-14T19:30:02Z Subject CommonName = mail.censored.example pkey sha256 [nomatch] <- 3 1 1 8fb34b1f452c2acba0e9c418defd705f09b9abf6006da7bfc6deafa5149bf7f2 depth = 1 Issuer CommonName = ISRG Root X1 Issuer Organization = Internet Security Research Group notBefore = 2024-03-13T00:00:00Z notAfter = 2027-03-12T23:59:59Z Subject CommonName = E5 Subject Organization = Let's Encrypt pkey sha256 [nomatch] <- 2 1 1 3586d4ecf070578cbd27aedce20b964e48bc149faeb9dad72f46b857869172b8 depth = 2 Issuer CommonName = ISRG Root X1 Issuer Organization = Internet Security Research Group notBefore = 2015-06-04T11:04:38Z notAfter = 2035-06-04T11:04:38Z Subject CommonName = ISRG Root X1 Subject Organization = Internet Security Research Group pkey sha256 [matched] <- 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3
Note that the 2nd TLSA RR in the above (lightly anonymised) example has a digest matching R3, rather than E5. It still works via X1, but it is fair to assume that the operator intended for both TLSA records to be effective. A more robust TLSA RRset based on LE's CAs would list both X1 and X2, and then E5..E9:
X1: IN TLSA 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3 X2: IN TLSA 2 1 1 762195c225586ee6c0237456e2107dc54f1efc21f61a792ebd515913cce68332 E5: IN TLSA 2 1 1 3586d4ecf070578cbd27aedce20b964e48bc149faeb9dad72f46b857869172b8 E6: IN TLSA 2 1 1 d016e1fe311948aca64f2de44ce86c9a51ca041df6103bb52a88eb3f761f57d7 E7: IN TLSA 2 1 1 cbbc559b44d524d6a132bdac672744da3407f12aae5d5f722c5f6c7913871c75 E8: IN TLSA 2 1 1 885bf0572252c6741dc9a52f5044487fef2a93b811cdedfad7624cc283b7cdd5 E9: IN TLSA 2 1 1 f1440a9b76e1e41e53a4cb461329bf6337b419726be513e42e19f1c691c5d4b2
and, I publish both RSA and ECDSA DANE records.
If you also have both certificate algorithms deployed live on your server keep in mind that you then need two sets of "3 1 1" records, one for each algorithm:
https://mail.sys4.de/pipermail/dane-users/2017-August/000416.html
a switch to ECDSA
DEFAULT_PREFERRED_CHAIN='ISRG Root X2'
should be (?) reasonably well tolerated.
i'm unclear on
is it DANE-safe? or DANE-recommended?
Yes, though it is in principle possible that there are DANE-enabled senders that support only RSA, and then might not be able to send you mail. At this point in time, that'd be the hypothetical sender's fault, they'd have little excuse to not support ECDSA. I don't know whether the problem is real or how prevalent it might be.
Simpler to stick with RSA for now, there's little reason to be worried about brute-force attacks on 2048-bit RSA keys, and email servers serve a lot fewer connections per second than HTTP servers, so the bandwidth and CPU differences are not significant.
I have no experience using Let's Encrypt (perhaps some ACME clients make this easier than others) with simultaneous RSA and ECDSA certs for the same hostname.
On Mon, Aug 26, 2024 at 11:20:52AM +1000, Viktor Dukhovni wrote:
and, I publish both RSA and ECDSA DANE records.
If you also have both certificate algorithms deployed live on your server keep in mind that you then need two sets of "3 1 1" records, one for each algorithm:
https://mail.sys4.de/pipermail/dane-users/2017-August/000416.html
I see that you've indeed published "3 1 2" records for both your RSA and your ECDSA certificate. This is in my view an "expert" configuration. Make sure you have monitoring in place on your end to catch any problems that might occur around future certificate renewals.
Note also that the "ISRG X1" or "ISRG X2" root CA cert (whichever is the issuer of your intermediate CA cert) is not included in your server certificate chain file, so the TLSA records for these won't work with at least the DANE TLSA code in Postfix and Exim and likely other MTAs.
_25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 025490860b498ab73c6a12f27a49ad5fe230fafe3ac8f6112c9b7d0aad46941d _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 2bbad93ab5c79279ec121507f272cbe0c6647a3aae52e22f388afab426b4adba _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 3586d4ecf070578cbd27aedce20b964e48bc149faeb9dad72f46b857869172b8 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 6ddac18698f7f1f7e1c69b9bce420d974ac6f94ca8b2c761701623f99c767dc7 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 762195c225586ee6c0237456e2107dc54f1efc21f61a792ebd515913cce68332 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 885bf0572252c6741dc9a52f5044487fef2a93b811cdedfad7624cc283b7cdd5 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 919c0df7a787b597ed056ace654b1de9c0387acf349f73734a4fd7b58cf612a4 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 cbbc559b44d524d6a132bdac672744da3407f12aae5d5f722c5f6c7913871c75 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 d016e1fe311948aca64f2de44ce86c9a51ca041df6103bb52a88eb3f761f57d7 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 f1440a9b76e1e41e53a4cb461329bf6337b419726be513e42e19f1c691c5d4b2 _25._tcp.mx2-edge.censored.net. IN TLSA 2 1 1 f1647a5ee3efac54c892e930584fe47979b7acd1c76c1271bca1c5076d869888 _25._tcp.mx2-edge.censored.net. IN TLSA 3 1 2 1b1436e45b1d56e4183f45d81f0fff48ea193aeda60d99b037945cc1d20fbfecd3da2b5d5de75fdfe0cf0420891d649957568c0cb8dc7cdae83ff4d21ac4e3fa _25._tcp.mx2-edge.censored.net. IN TLSA 3 1 2 eec904205869cafa231f037b958e3ce7cde10443464261b01f5b95d852dd50cdefee7cead8b79792dca7fb1ea4f138fc615d1a2018133fc2d94d1260e012bf5a
mx2-edge.censored.net[192.0.2.1]: pass: TLSA match: depth = 0, name = mx2-edge.censored.net TLS = TLS13 with CHACHA20POLY1305-SHA256,X25519,PubKeyALG_EC name = mx1-edge.censored.net name = mx2-edge.censored.net depth = 0 Issuer CommonName = E6 Issuer Organization = Let's Encrypt notBefore = 2024-08-25T15:51:37Z notAfter = 2024-11-23T15:51:36Z Subject CommonName = mx1-edge.censored.net pkey sha512 [matched] <- 3 1 2 eec904205869cafa231f037b958e3ce7cde10443464261b01f5b95d852dd50cdefee7cead8b79792dca7fb1ea4f138fc615d1a2018133fc2d94d1260e012bf5a depth = 1 Issuer CommonName = ISRG Root X2 Issuer Organization = Internet Security Research Group notBefore = 2024-03-13T00:00:00Z notAfter = 2027-03-12T23:59:59Z Subject CommonName = E6 Subject Organization = Let's Encrypt pkey sha256 [matched] <- 2 1 1 d016e1fe311948aca64f2de44ce86c9a51ca041df6103bb52a88eb3f761f57d7
hi,
a bit late, well early, to focus on all your comments, so will do that in a while ...
but, one quick question re:
Note also that the "ISRG X1" or "ISRG X2" root CA cert (whichever is the issuer of your intermediate CA cert) is not included in your server certificate chain file, so the TLSA records for these won't work with at least the DANE TLSA code in Postfix and Exim and likely other MTAs.
my Postfix config has
smtpd_tls_chain_files = /sec/ssl/le/deploy/deploy/${v_MY_DOM}/priv.ec.key, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/fullchain.ec.crt.pem, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/priv.rsa.key, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/fullchain.rsa.crt.pem
where LE's fullchain certs are the usual concat of my dom's cert.pem with their intermediate cert chain.
so, as said, NOT including the X1 and X1 root certs.
my understanding of the "3 1 2" records is that the inclusion of the X1/X2 root certs is _not_ required.
and that your warning is re: the "2 1 1" use case. correct?
atm, i cleaned up my includes -- removing the old/deprecated intermediated -- and have the current ones published. my intention is to verify i've got my "3 1 2" setup working after monkeying with it, and unpublish the "2 1 1" records.
thx for the reminder re: self-checking with danesmtp!
On Mon, Aug 26, 2024 at 02:01:04AM -0400, pgnd wrote:
Note also that the "ISRG X1" or "ISRG X2" root CA cert (whichever is the issuer of your intermediate CA cert) is not included in your server certificate chain file, so the TLSA records for these won't work with at least the DANE TLSA code in Postfix and Exim and likely other MTAs.
My Postfix config has
smtpd_tls_chain_files = /sec/ssl/le/deploy/deploy/${v_MY_DOM}/priv.ec.key, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/fullchain.ec.crt.pem, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/priv.rsa.key, /sec/ssl/le/deploy/deploy/${v_MY_DOM}/fullchain.rsa.crt.pem
where LE's fullchain certs are the usual concat of my dom's cert.pem with their intermediate cert chain. So, as said, NOT including the X1 and X2 root certs.
Right, the "fullchain" files are not as full as they could be.
My understanding of the "3 1 2" records is that the inclusion of the X1/X2 root certs is _not_ required.
Yes, matching TLSA "3 X X" records are sufficient, and don't require anything more in the chain, and are immune from certificate expiration and even matching SAN requirements. So they are the least fragile option provided your certificate rollover process keeps the keys stable, or prepublishes the next key well before rollover.
and that your warning is re: the "2 1 1" use case. correct?
Yes.
My intention is to verify i've got my "3 1 2" setup working after monkeying with it, and unpublish the "2 1 1" records.
That will improve security (avoids trust in weak DV cert issuance), and provided you have good monitoring, and a robust "3 1 1 + 3 1 1" rollover process (when changing keys), you should be all set.
thx for the reminder re: self-checking with danesmtp!
Monitoring is not optional. If you've deployed DANE TLSA records for your server, and are not monitoring them, you're not doing yourself or anyone else a favour, the records will at some point break, and both senders and your users will be unhappy.
If you make non-trivial improvements in the danesmtp() bash function, please share.
and that your warning is re: the "2 1 1" use case. correct?
Yes.
My intention is to verify i've got my "3 1 2" setup working after monkeying with it, and unpublish the "2 1 1" records.
That will improve security (avoids trust in weak DV cert issuance), and provided you have good monitoring, and a robust "3 1 1 + 3 1 1" rollover process (when changing keys), you should be all set.
thx for this, and yesterday's comments.
after simplifying to just the "3 1 2" certs, i see the one-algo-not-the-other 'good' results @ online checks,
https://stats.dnssec-tools.org/explore/ https://dane.sys4.de https://dnsviz.net/ https://www.huque.com/bin/danecheck
, as you'd warned.
i've switched out my own monitoring for danesmtp. once i remembered that running it from my residential lan was hitting ISP port 25 blocks (::facepalm::), it's easy enough for once a day scans, and notify on fail, for each of my certs+algos checks.
cheers!
On Mon, Aug 26, 2024 at 05:36:55PM -0400, pgnd wrote:
after simplifying to just the "3 1 2" certs, i see the one-algo-not-the-other 'good' results @ online checks,
https://stats.dnssec-tools.org/explore/ https://dane.sys4.de https://dnsviz.net/ https://www.huque.com/bin/danecheck
, as you'd warned.
i've switched out my own monitoring for danesmtp. once i remembered that running it from my residential lan was hitting ISP port 25 blocks (::facepalm::), it's easy enough for once a day scans, and notify on fail, for each of my certs+algos checks.
For your own servers, I'd recomment checking once an hour, if not more often. Some (legitimate) senders have fairly short queue lifetimes, and some are aggressive (silly) enough to bounce mail as soon as TLS authentication fails, without waiting for the issue to be resolved.
Of course the domain in question may not carry sufficiently "important" traffic to warrant prompt detection/notification, but as a default, I'd recommend checking hourly rather than daily.
Also set your TLSA RR TTLs to at most an hour.
For your own servers, I'd recomment checking once an hour, if not more often. Some (legitimate) senders have fairly short queue lifetimes, and some are aggressive (silly) enough to bounce mail as soon as TLS authentication fails, without waiting for the issue to be resolved.
Of course the domain in question may not carry sufficiently "important" traffic to warrant prompt detection/notification, but as a default, I'd recommend checking hourly rather than daily.
i hadn't expected _that_ frequent -- tho makes sense. i've historically run checks 1/day+ ...
easy enough to do.
Also set your TLSA RR TTLs to at most an hour.
+1
o/
I've tried to follow this thread.
I have one question...
Is there a site i can visit to tell me whether or not my TLSA and/or other cert DNS entries are OK with the new certs?
On 8/26/2024 10:46 PM, Viktor Dukhovni wrote:
On Mon, Aug 26, 2024 at 05:36:55PM -0400, pgnd wrote:
after simplifying to just the "3 1 2" certs, i see the one-algo-not-the-other 'good' results @ online checks,
https://stats.dnssec-tools.org/explore/ https://dane.sys4.de https://dnsviz.net/ https://www.huque.com/bin/danecheck
, as you'd warned.
i've switched out my own monitoring for danesmtp. once i remembered that running it from my residential lan was hitting ISP port 25 blocks (::facepalm::), it's easy enough for once a day scans, and notify on fail, for each of my certs+algos checks.
For your own servers, I'd recomment checking once an hour, if not more often. Some (legitimate) senders have fairly short queue lifetimes, and some are aggressive (silly) enough to bounce mail as soon as TLS authentication fails, without waiting for the issue to be resolved.
Of course the domain in question may not carry sufficiently "important" traffic to warrant prompt detection/notification, but as a default, I'd recommend checking hourly rather than daily.
Also set your TLSA RR TTLs to at most an hour.
On Tue, Aug 27, 2024 at 12:57:03AM -0400, Mike via dane-users wrote:
I've tried to follow this thread.
I have one question...
Is there a site i can visit to tell me whether or not my TLSA and/or other cert DNS entries are OK with the new certs?
The DANE survey (https://stats.dnssec-tools.org/explore) shows a detailed breakdown of the DNSSEC/DANE status of directly delegated (from a TLD or similar registry, not internal within an organisation) DNSSEC-signed domains. However, the data is not "real-time", domains are checked once a day, presently some time between 16:00 and 22:00 UTC. So if you make changes, the survey may not show you the current state.
For a real-time check you can perform yourself, use the "danesmtp" bash function, described at:
https://list.sys4.de/hyperkitty/list/dane-users@list.sys4.de/thread/NKDBQABS...
All you need is OpenSSL 1.1.1 or later and a bash-compatible shell.
Another real-time option is https://dane.sys4.de, but the results are noticeably more basic (simple) than from the survey. The results are cached, but you can request a refresh (every ~5 minutes, IIRC).
Checks are also possible via:
* https://www.huque.com/bin/danecheck
Not a domain check, you have to explicitly check a particular MX host, and specify port 25.
Don't forget to choose the "SMTP" radio button under "STARTTLS Application"
* https://internet.nl/test-mail/
But some of their criteria are too strict (pedantic).
There are others, measurement and analysis quality varies...
On Tue, Aug 27, 2024 at 03:18:42PM +1000, Viktor Dukhovni wrote:
Checks are also possible via:
* https://www.huque.com/bin/danecheck Not a domain check, you have to explicitly check a particular MX host, and specify port 25. Don't forget to choose the "SMTP" radio button under "STARTTLS Application"
I neglected to find and post Shumon's SMTP-specific test site, that does check all the MX hosts of a domain:
https://www.huque.com/bin/danecheck-smtp
FWIW, as with many other sites, this does not probe multi-certificate deployments, where often multiple connections are required with different offers of client supported TLS algorithms in order to test both RSA and ECDSA (or some day also Ed25519 if/when that becomes popular in EE certificates).
Automated regular tests should perform local validation, the test sites are for occasional ad hoc sanity checks.
On Mon, Aug 26, 2024 at 12:15:47AM +1000, Viktor Dukhovni wrote:
The major changes in the Let's Encrypt issuer CA lineup noted in my previous post:
https://list.sys4.de/hyperkitty/list/dane-users@list.sys4.de/message/ZTM3XQMI3XP7PWMWJTXBYDPVU4UENE24/are now largely completed. Of the ~46000 domains with working DANE-TA(2) TLSA records matching a Let's Encrypt intermediate issuer, just 62 are still based on R3, and none on X3, X4, R4, E1 or E2.
These last few R3 issued certificates will either be renewed or will expire by September 4th.
Therefore, if you haven't done so already, please read the fine advice in:
https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.htmland switch to R10..R14 or E5..E9 (or rarely both) as appropriate.
With all the R3, R4, E1 and E2 certifiates now expired, I've updated the text of the above webpage, and added MX hosts still listing R3, R4, E1 or E2 to the table:
https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html#stale
Please be sure to publish TLSA records for the FULL list of CAs in each group:
- R10–R14 if using any of these. - E5–E9 if using any of these. - ISRG X1 and ISRG X2 if using either of these.
It is disappointing to see some operators react to a survey notice of a problem by publishing a single TLSA RR matching e.g. just R10, only to have a problem ~30-60 days later when the new certificate is from R11.
They may then publish, just both R10 and R11, leaving out R12–R14, which might be used with little warning, if needed.
Such dogged failure to plan for the inevitable is not a positive character trait. :-(
participants (3)
-
Mike -
pgnd -
Viktor Dukhovni