TAKE NOTE: "2 1 1" TLSA records vs. apparent change of Let's Encrypt default certificate chain
The DANE/DNSSEC survey (https://stats.dnssec-tools.org) has seen a recent spike in the number of MX hosts whose "2 1 1" TLSA records no longer match their certificate chain. The records in question all shar the same digest value, for various TLSA base domains:
_25._tcp.mx1.example. IN TLSA 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3
I was initially puzzled as to why this might be happening, but then it occurred to me that the reason why is clear.
The above hash is the hash of the ISRG X1 root CA key, but it is also of course the key hash of its outdated **cross-certificate** issued by DST. That DST cross cert was needed for compatability with some old Android systems that did not get root CA updates (or updates of any kind).
It must be that Let's Encrypt finally stopped by default including that cross certificate in their chains. So instead of a chain that looks like:
- depth 0: EE (server) certificate - depth 1: Let's Encrypt R3/E1 issuer CA - depth 2: ISRG X1 cross cert issued by DT
with the certificate at depth 2 matching the TLSA record, they now generate just:
- depth 0: EE (server) certificate - depth 1: Let's Encrypt R3/E1 issuer CA
with the ISRG (now standalone) root CA not included in the chain!
Leaving out the root CA works fine for WebPKI, where clients need to have a locally trusted copy of the root, but not for certificate usage DANE-TA(2), which does not rely on any local CA store:
https://dane.sys4.de/common_mistakes#4 https://datatracker.ietf.org/doc/html/rfc7672#section-3.1.2
Bottom line, if you're relying on that "2 1 1" record matching the ISRG root to match your Let's Encrypt chain, you're about to be disappointed, if you aren't already. See:
http://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
for better alternatives, or switch to "3 1 1", perhaps with the aid of "danebot" (still hoping some early adopters will pitch in to further improve it, to support some additional workflows):
https://github.com/tlsaware/danebot
On Wed, Nov 15, 2023 at 12:13:15AM -0500, Viktor Dukhovni wrote:
It must be that Let's Encrypt finally stopped by default including that cross certificate in their chains.
As pointed out helpfully by Geert Hendrickx on the postfix-users list:
They plan to stop providing the cross-signed "long chain" by default in February 2024, and completely in June, as the cross-sign expires in September. Dropping it last week was unintended.
The ensuing conversation on the LE forum uncovered a second potential future incompatibility to plan for:
https://community.letsencrypt.org/t/short-chain-and-dane/208422/8?u=ietf-dan...
Let's Encrypt are apparently also planning to *randomise* the choice of intermediate issuer CA used with each renewal. Instead of consistently using say "R3", they'll randomly choose one of R3/R4/E1/E2.
Therefore, anyone who publishes TLSA records for just one of the 4 issuers, will eventually also be "disappointed".
If you're using Let's Encrypt as your CA and prefer to publish DANE-TA(2), rather than DANE-EE(3) TLSA records, please look over:
http://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
carefully, and publish all four of the **required** TLSA records, for each MX host:
_25._tcp.mx1.org.example. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d ; R3 _25._tcp.mx1.org.example. IN TLSA 2 1 1 e5545e211347241891c554a03934cde9b749664a59d26d615fe58f77990f2d03 ; R4 _25._tcp.mx1.org.example. IN TLSA 2 1 1 276fe8a8c4ec7611565bf9fce6dcace9be320c1b5bea27596b2204071ed04f10 ; E1 _25._tcp.mx1.org.example. IN TLSA 2 1 1 bd936e72b212ef6f773102c6b77d38f94297322efc25396bc3279422e0c89270 ; E2 ...
or if you prefer:
_25._tcp.mx1.org.example. IN CNAME _25._tlsa.org.example. _25._tcp.mx2.org.example. IN CNAME _25._tlsa.org.example. ... _25._tcp.mxN.org.example. IN CNAME _25._tlsa.org.example. ; _25._tlsa.org.example. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d ; R3 _25._tlsa.org.example. IN TLSA 2 1 1 e5545e211347241891c554a03934cde9b749664a59d26d615fe58f77990f2d03 ; R4 _25._tlsa.org.example. IN TLSA 2 1 1 276fe8a8c4ec7611565bf9fce6dcace9be320c1b5bea27596b2204071ed04f10 ; E1 _25._tlsa.org.example. IN TLSA 2 1 1 bd936e72b212ef6f773102c6b77d38f94297322efc25396bc3279422e0c89270 ; E2
My previous post on this topic noted that covered Let's Encrypt are planning to *randomise* the choice of intermediate issuer CA used with each renewal.
It now turns out that they will also be switching to new underlying intermediate CAs. So you'll a random choice of *new* issuers.
https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/L7XoAXt_s1c/...
- We will be generating 5 RSA and 5 ECDSA intermediates, instead of 2 each. We plan to automatically rotate issuance between multiple intermediates for improved redundancy.
- We will be shortening their validity period from 5 years to 3 years, to reflect our commitment to issue new intermediates every 2 years.
So anyone relying on DANE-TA(2) (certificate usage 2) needs to closely watch for upcoming announcements from LE, and be prepared to add TLSA records for the new intemediates soon. Or stop playing their game, and switch to a robust "3 1 1" + "3 1 1" model with a stable by default key during certificate renewals.
On Fri, Dec 08, 2023 at 01:59:33PM -0500, Viktor Dukhovni wrote:
It now turns out that they will also be switching to new underlying intermediate CAs. So you'll a random choice of *new* issuers.
https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/L7XoAXt_s1c/m/k_vdk9rQAwAJ - We will be generating 5 RSA and 5 ECDSA intermediates, instead of 2 each. We plan to automatically rotate issuance between multiple intermediates for improved redundancy. - We will be shortening their validity period from 5 years to 3 years, to reflect our commitment to issue new intermediates every 2 years.So anyone relying on DANE-TA(2) (certificate usage 2) needs to closely watch for upcoming announcements from LE, and be prepared to add TLSA records for the new intemediates soon. Or stop playing their game, and switch to a robust "3 1 1" + "3 1 1" model with a stable by default key during certificate renewals.
This has now (as of 2024-06-06) taken place, and I'm starting to see Let's Encrypt certificates from R10, R11, E5 and E6, and of course one's TLSA published TLSA RRset should always include the backup issuers.
However, it is possible to publish TLSA RRs that match just the "R*" CAs when you have RSA keys, or just the "E*" CAs for ECDSA keys. But don't forget to take appropriate action before switching algorithms or choosing to have keys/certs for both algorithms.
For more details:
- https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
It is sadly again the case that some MTA operators employ "fire and forget", with no active monitoring, and no keeping up with service announcements from Let's Encrypt. They then end up upon automated renewal with a certificate chain that does not match their TLSA records, and so a partial outage.
With a bit of care, one can instead publish TLSA records matching one of the "ISRG X1" or "ISRG X2" roots, but one then has to carefully ensure that the root CAs ore appended to the server's chain file (not the case with, e.g., certbot), so the ACME chain file may require post-processing before it is configured as the MTA's certificate chain.
Of course that's still not safe to then indefinitely ignore, even the roots are subject to occasional bitrot. Only "3 1 1" records are under your control and change only when *you* decide to switch to new keys.
Hence, my best advice is to stop playing Let's Encrypt whack-a-mole, and use "3 1 1" records with stable keys (not automatically replaced with every renewal). You should choose when to rekey, and prepublish matching TLSA records before you do so:
- https://mail.sys4.de/pipermail/dane-users/2018-February/000440.html - https://github.com/tlsaware/danebot
And of course, DO NOT forget monitoring. Perhaps via:
- https://list.sys4.de/hyperkitty/list/dane-users@list.sys4.de/thread/NKDBQABS...
• Viktor Dukhovni [2024-06-07 10:54]: [...]
This has now (as of 2024-06-06) taken place, and I'm starting to see Let's Encrypt certificates from R10, R11, E5 and E6, and of course one's TLSA published TLSA RRset should always include the backup issuers.
However, it is possible to publish TLSA RRs that match just the "R*" CAs when you have RSA keys, or just the "E*" CAs for ECDSA keys. But don't forget to take appropriate action before switching algorithms or choosing to have keys/certs for both algorithms.
For more details:
[...]
beware that publishing TLSA RRs for *all* LE keys (10+4 for now, and only 10 in 3 months' time) could cause trouble when exchange online tries to do delivery... see https://www.mail-archive.com/mailop@mailop.org/msg22141.html for more details.
On Mon, Jun 10, 2024 at 12:46:39PM +0200, Kirill Miazine wrote:
This has now (as of 2024-06-06) taken place, and I'm starting to see Let's Encrypt certificates from R10, R11, E5 and E6, and of course one's TLSA published TLSA RRset should always include the backup issuers.
However, it is possible to publish TLSA RRs that match just the "R*" CAs when you have RSA keys, or just the "E*" CAs for ECDSA keys. But don't forget to take appropriate action before switching algorithms or choosing to have keys/certs for both algorithms.
For more details:
beware that publishing TLSA RRs for *all* LE keys (10+4 for now, and only 10 in 3 months' time) could cause trouble when exchange online tries to do delivery... see https://www.mail-archive.com/mailop@mailop.org/msg22141.html for more details.
Are you sure the issue related to the TLSA DNS response size? Note also that an MX with just RSA keys or just ECDSA keys needs only half (5) of the new intermediate CA records and at most one of the legacy CA records (since the E2 and R4 backups will never be used at this point).
So 6 records rather than 14 should be enough, or 12 for both RSA and ECDSA. Or if one post-processes the chain file to append the root CA certs, just two TLSA records (matching ISRG X1 and X2) would suffice.
https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
Hi
• Viktor Dukhovni [2024-06-10 14:00]:
On Mon, Jun 10, 2024 at 12:46:39PM +0200, Kirill Miazine wrote:
This has now (as of 2024-06-06) taken place, and I'm starting to see Let's Encrypt certificates from R10, R11, E5 and E6, and of course one's TLSA published TLSA RRset should always include the backup issuers.
However, it is possible to publish TLSA RRs that match just the "R*" CAs when you have RSA keys, or just the "E*" CAs for ECDSA keys. But don't forget to take appropriate action before switching algorithms or choosing to have keys/certs for both algorithms.
For more details:
beware that publishing TLSA RRs for *all* LE keys (10+4 for now, and only 10 in 3 months' time) could cause trouble when exchange online tries to do delivery... see https://www.mail-archive.com/mailop@mailop.org/msg22141.html for more details.
Are you sure the issue related to the TLSA DNS response size? Note also that an MX with just RSA keys or just ECDSA keys needs only half (5) of the new intermediate CA records and at most one of the legacy CA records (since the E2 and R4 backups will never be used at this point).
I didn't say it was related to DNS response size, as I don't have any insight into what is going on at Microsoft -- it was just an observation that Microsoft seems to choke on DNSSEC validation when number of TLSA RRs is getting high with a rather misleading error message: "dnssec-invalid: Destination domain returned invalid DNSSEC records"
Testing with sending from Outlook to domains with increasing number of TLSA RRs gave me the number of TLSA RRs Microsoft was able to validate: 12. I don't have any insight in whatever Microsoft is doing, but I was notified of a similar issue by a local mail provider with some tens of thousands mailboxes, so I shared the observations with the fellow postmasters.
So 6 records rather than 14 should be enough, or 12 for both RSA and ECDSA. Or if one post-processes the chain file to append the root CA certs, just two TLSA records (matching ISRG X1 and X2) would suffice.
https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html
Yeah, once the issue is known, the solutions are multiple.
Notifying dane-users of my mailop post was solely due to the following sentence on the page above:
If your server has both RSA and ECDSA keys, you'll need to publish TLSA records matching both the "R*" and "E*" issuer CAs.
Counting records matching both the "R*" and "E*" issuer CAs in the table on that page gives me 14.
I used to have LE TLSAs referenced by a single name, it worked well since 2020, when I first started using DANE. IIRC E* certs appeared during the previous round of intermediate certs update. But the number was still manageable. Now I'll just have TLSA RRs in two names: one for RSA and one for ECDSA.
participants (1)
-
Kirill Miazine -
Viktor Dukhovni