5 Mar 2020
Blogs
The Let's Encrypt CAA Code Bug – A Plain View
What happened
Let’s Encrypt recently found a bug in their CAA checking code and after remediating the bug [1] on 2020-02-29 UTC (the evening of Friday February 28, U.S. Eastern time), announced they would revoke approximately 2.6% of their active certificates that were potentially affected by the bug, totalling approximately 3 million certificates [2].
Let’s Encrypt company engineers provided a technical update [1]:
|
“ On 2020-02-29 UTC, Let’s Encrypt found a bug in our CAA code. Our CA software, Boulder, checks for CAA records at the same time it validates a subscriber’s control of a domain name. Most subscribers issue a certificate immediately after domain control validation, but we consider a validation good for 30 days. That means in some cases we need to check CAA records a second time, just before issuance. Specifically, we have to check CAA within 8 hours prior to issuance (per BRs §3.2.2.8), so any domain name that was validated more than 8 hours ago requires rechecking. The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt. We confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance at 03:10. We deployed a fix at 05:22 UTC and then re-enabled issuance. Our preliminary investigation suggests the bug was introduced on 2019-07-25. We will conduct a more detailed investigation and provide a postmortem when it is complete. “ |
Cert Revocation, Renewal and Replacement
Let’s Encrypt report they are aiming to “complete revocations before the deadline of 2020-03-05 03:00 UTC, we are planning to start revoking affected certificates at 2020-03-04 20:00 UTC (3:00pm US EST)”. Those affected should continue to renew and replace affected with new certificates. [3]
Impact
Whilst this might not be seen as critical on the surface, a certificate is fundamentally used to establish and maintain site trustworthiness between two parties. Essentially, certificates are used by browsers to ensure that the site we intended to visit is really the one we’ve arrived at.
Leaving a revoked certificate in-place could trigger errors in browsers and other applications, cause loss availability and/or trust, all potentially causing harm to the companies that rely on them.
Impacted Customer Communications
From Let’s Encrypt
Let’s Encrypted reported they “have sent notification emails to affected subscribers who have registered an email address”, although believe some customers “may not have received an email if they did not provide an email address while registering” their ACME account. [3]
In this latter scenario, Let’s Encrypt are directing customers with any need to re-subscribe to email notifications to https://letsencrypt.org/docs/expiration-emails/ . [3]
It is worth considering that email delivery issues or spam filtering may also be the cause of missing the email which ultimately advises affected customers to renew their certificates. [3]
If you are looking for the missing email you can search for the following subject line within your mailbox or email gateway logs:
| “ACTION REQUIRED: Renew these Let’s Encrypt certificates by March 4” |
If you are unsure whether your hostname is affected, use the checking tools described in this post.
Via AUSCERT
As a passionate not-for-profit CERT organisation, we routinely monitor industry updates, news and other intel feeds. Due to this practice, we were promptly aware of the public bug announcement from Let’s Encrypt and following a proactive course of action, we identified AUSCERT Members with affected certificates and are currently working with them.
Identifying an affected certificate
Let’s Encrypt have published a page hosting the list of affected serial numbers relating to the 2020.02.29 CAA Rechecking Incident [3]. That page details the downloadable file contains a list of all affected certs, sorted by account ID. [4]
Checking Tools/methods
There are several methods or tools providing a means to check for an affected certificate.
- Online
- Common Tools
- Curl
- OpenSSL
- Purpose built script
Online
If you want to double check whether a given hostname still needs its certificate replaced, you can use the tool seen in the screenshot below available at: https://checkhost.unboundtest.com/ .
Common Tools
Curl
The curl command on a linux system can be used in conjunction with online tool https://checkhost.unboundtest.com/ against a target website to show its current certificate serial number. The following two example indicate affected and non-affected certificate responses.
Response 1: Affected Certificate
|
$ curl -XPOST -d ‘fqdn=www.REDACTED.au’ https://checkhost.unboundtest.com/checkhost The certificate currently available on www.REDACTED.au needs renewal because it is affected by the Let’s Encrypt CAA rechecking problem. Its serial number is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. See your ACME client documentation for instructions on how to renew a certificate. |
Response 2: Non-Affected Certificate
|
$ curl -XPOST -d ‘fqdn=letsencrypt.org’ https://checkhost.unboundtest.com/checkhost The certificate currently available on letsencrypt.org is OK. It is not one of the certificates affected by the Let’s Encrypt CAA rechecking problem. Its serial number is 03a1c95bdaa36a8268327f2253cbd3ba2436 |
OpenSSL
As seen in the following examples, the openssl command (linux) can be used against a target website to show its current certificate serial number:
|
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial Number | tr -d : Response: Serial Number 0fd078dd48f1a2bd4d0f2ba96b6038fe |
|
openssl s_client -connect letsencrypt.org:443 -servername letsencrypt.org -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial Number | tr -d : Response: Serial Number 03a1c95bdaa36a8268327f2253cbd3ba2436 |
Purpose-Built Script
- Github – Let’s Encrypt CAA (lecaa) checking scripts [5]
A purpose-built script hosted on Github [5] and created by Hanno Böck [6] “…allows you to efficiently check affected hosts”. Hanno Böck advised on his github page that the script was created after “Let’s Encrypt announced a bug in their system’s CAA checks, which forced them to revoke 3 million certificates on very short notice”.
Let’s Encrypt credit the lecaa script as useful tool and refer customer to use it by advising “if you have a large list of domains you need to check, this tool will be more effective. [3]
Where certificates are found that are not affected, Let’s Encrypt said “even if you received an email, it’s possible that the affected certificates have been replaced by newer certs not affected by the bug. (Either due to being issued in the last few days since it was fixed, or simply by not meeting the specific timing criteria necessary for the bug to trigger.) In that case, it’s not necessary to renew them again”. [3]
Questions
Anyone who has questions should review the Q & A’s seen on Let’s Encrypt’s FAQ [2], then should questions remain after such review, they should contact Let’s Encrypt directly.
References
[1] 2020.02.29 CAA Rechecking Bug
https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591
[2] Revoking certain certificates on March 4
https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864
[3] Download affected certificate serials for 2020.02.29 CAA Rechecking Incident
https://letsencrypt.org/caaproblem/
[4] File containing serial number of the affected certificates
https://d4twhgtvn0ff5.cloudfront.net/caa-rechecking-incident-affected-serials.txt.gz
[5] Github – Purpose Built Checker (lecaa)
https://github.com/hannob/lecaa
[6] Hanno Böck
https://hboeck.de/
