Below a little text about TLS certificate agility and some pitfalls herein (thanks to Viktor and Patrick for input and review):
The Transport Layer Protocol (TLS) supports the introduction of new key algorithms for x509 certificates. For a long time, keys based on the RSA algorithm were the only available choice. Nowadays, TLS also permits keys based on elliptic curve algorithms such as ECDSA. Other, new algorithms might follow in the future.
Modern algorithms promise same or better security with less CPU usage and smaller key sizes. This makes the new algoritms interesting for operators of TLS secured services. However older software (on client or peer server side) might not support and understand the newer algorithms, so both the new and the old need be configured for some transition time. Ideally, both communicating endpoints will negotiate the best algorithm (in terms of security and performance) supported by both sides. The scheme of supporting multiple certificate key algorithms is called certificate agility.
Care must be taken when multiple key algorithms are used in a TLS setup together with DNS-based Authentication of Named-Entities (DANE) as documented in RFC 7672 ([https://tools.ietf.org/html/rfc7672.html]). With SMTP-DANE, data authenticating the TLS certificates (the hashes of TLS certificate, hashes of the public key, or the full certificates) are stored in DNS, secured by DNSSEC. Upon connection, a delivering mail-server (MTA) will recieve the certificate chain from the destination MTA and will try to validate this certificate chain via the data stored in DNS as one or more TLSA resource records. The TLSA records contain either information on the leaf-node certificates (called DANE-EE(3)) or information about the certification authority (DATE-TA(2)).
With DANE-EE(3) and TLS certificate agility in use, each distinct certificate that is configured on the server side does require it's own TLSA record containing either the hash of the certificate, hash of the public key of the certificate or the certificate itself. With DANE-TA(2), multiple TLSA-Records are only needed if the certificates are issued by different certification authorities (care must be taken to compare the CA keys in the certificate chain, as certain CA companies or "brands" are using different certificate authorities to sign x509 certificates).
The postfix mail server supports certificate agility, in the example below two certificates are configured, one based on RSA keys, and a 2nd certificate using ECDSA keys:
,---- | # RSA certificate configuration | smtpd_tls_key_file=/etc/postfix/tls/rsa-key.pem | smtpd_tls_cert_file=/etc/postfix/tls/rsa-cert.pem | # ECDSA certificate configuration | smtpd_tls_eckey_file=/etc/postfix/tls/ecdsa-key.pem | smtpd_tls_eccert_file=/etc/postfix/tls/ecdsa-cert.pem `----
This configuration requires two separate DANE-EE(3) TLSA records in DNS, one for RSA and one for the ECDSA certificate. If both certifcates are issued from the same CA, only one DANE-TA(2) TLSA record is required.
OpenSSL 1.0.x does not handle multiple chain files gracefully, and so both certificate chain files need to contain all the issuer CA certificates needed to validate either end-entity (leaf) certificate. This is fixed in OpenSSL 1.1.0.
The selection of which certificate algorithm to use is based on the TLS handshake. Older TLS software prefers RSA over ECDSA, while some more modern software will have a preference for ECDSA. Today most DANE commandline tools (ldns-dane, gnutls-cli ...), as well as online DANE services such as the DANE SMTP Validator ([https://dane.sys4.de]), are only testing one certificate (based on the preference or configuration of the underlying TLS library used to implement the tools). A missing TLSA record can go unnoticed but can lead to rejection of mail. The DANE SMTP Validator will be updated to check all possible TLS certificate key algorithms.
---
Carsten Strotmann