The CA bundle you updated is probably not the one your service uses

The eMudhra mess has produced the usual reassuring sentence: “We updated the CA certificates on the server.” Which server? Which process? Which bundle? The browser on your laptop can be safe while a background worker in last month’s container image still trusts the old chain.

Start with the incident account. Eight intermediate keys were recoverable. Mozilla and Chrome blocked them, then moved against the issuing ECC G3 root. Your job is to establish which clients in your estate got that decision, and which clients are still reading a frozen trust file.

Trace the verifier, not the library name

Take one outbound connection your application actually makes. Find its HTTP/TLS library, then find where that library gets trust anchors. Possibilities include the operating-system store, a CA file mounted into the container, a bundle shipped with a language package, and a custom store set by environment variable or application config. Record the path and package version inside the running workload, not on the build host.

For a Python process using certifi, this tells you which bundle that environment has installed:

python -c 'import certifi; print(certifi.__version__, certifi.where())'

Run it with the same interpreter or virtual environment the service runs. Then check whether its bundle includes the relevant emSign root. The certifi maintainers’ discussion identifies an updated release, but a version on your shell’s Python proves nothing about a different virtual environment or a bundled installer.

For a remote server, inspect what it sends and whether this OpenSSL invocation accepts it:

openssl s_client -connect example.org:443 -servername example.org \
  -verify_hostname example.org -verify_return_error -showcerts </dev/null

Replace example.org with the real endpoint. -servername sends SNI; -verify_hostname checks the name; -verify_return_error makes a verification error fail the handshake rather than merely printing it. OpenSSL’s manual is explicit about that last gotcha. This command uses OpenSSL’s configured trust path unless you point it at the application’s exact CA file. It is an inspection tool, not a certificate for every TLS client on the box.

Prove the deployed path

Update the OS trust package, the relevant language bundles, and any pinned container base images. Rebuild and redeploy those images. Restart long-running processes so cached TLS contexts and connections are replaced. If you use a custom CA bundle for mTLS or an origin connection, review that separately; a system package update will not edit your private file.

Then make the application itself connect through its normal code path. Check a known-good endpoint still works and test rejection against the affected chain using the current browser/root-program guidance. Record the application version, bundle path, image digest, and result. A curl success from your laptop is a nice way to test curl on your laptop.

If you serve a certificate chaining to emSign ECC Root CA - G3, plan a replacement chain with your CA provider. The eight compromised intermediates reportedly issued no leaf certificates; other certificates still chain to that root and face distrust as stores update. Do not “fix” a validation error with -k, verify=False, or an equivalent switch. That changes a failed identity check into no identity check at all.