Why Did Our HTTPS Calls Suddenly Fail? Solving the 2020 AddTrust Root Expiration
This article explains how an expired AddTrust External CA Root caused HTTPS requests to Sentry to fail, shows how to reproduce the error with curl and OpenSSL, and provides step‑by‑step fixes for Ubuntu, Docker‑Alpine, and macOS environments.
Problem
Our application’s HTTPS requests to the Sentry server started failing with SSL certificate verification errors, despite the server’s own certificate appearing valid.
Analysis
Log entries showed
Raven::Error - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired). Checking the domain’s certificate chain revealed that the AddTrust External CA Root expired on May 30 2020.
Using
curl https://sentry.xxx.comreproduced the error, while other hosts (CentOS, other domains) succeeded, indicating the problem was specific to the worker host.
Running
openssl s_client -showcerts -servername sentry.xxx.com -connect sentry.xxx.com:443confirmed the expired root certificate.
Solution
Remove the expired AddTrust root from the CA configuration and update the CA bundle.
Ubuntu host
<code>sed -i "/AddTrust_External_Root.crt/d" /etc/ca-certificates.conf</code>Then refresh the certificates:
<code>sudo update-ca-certificates --fresh</code>Restart the affected services.
Docker (Alpine) container
<code>sed -i "/AddTrust_External_Root.crt/d" /etc/ca-certificates.conf</code> <code>update-ca-certificates -f -v</code>Ideally add these commands to the Dockerfile before
CMDso the container always has an up‑to‑date CA bundle.
macOS
Backup the system certificate file and comment out the expired AddTrust block:
<code>sudo cp /etc/ssl/cert.pem ~/etc-ssl-cert.pem-20200601</code> <code>sudo sed -i "/^### AddTrust/,/^-.*END/ s/^/#/g" /etc/ssl/cert.pem</code>After updating the CA store, the
curl https://sentry.xxx.comcommand succeeds.
Additional notes
The issue only affects server‑side HTTPS calls that encounter the expired AddTrust root; browsers typically have updated trust stores, so end‑users are not impacted. Use configuration management tools (Ansible, SaltStack) for host‑wide fixes, and remember to restart services after updating the CA bundle.
Source: https://aliasmee.github.io/post/resolve-certificate-verify-failed-with-2020-may-30/
Efficient Ops
This public account is maintained by Xiaotianguo and friends, regularly publishing widely-read original technical articles. We focus on operations transformation and accompany you throughout your operations career, growing together happily.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.