$RodHat_
Console Tips

Debug a TLS handshake by hand with openssl s_client

Published by

Debug a TLS handshake by hand with openssl s_client
Photo: AI-generated — no human photographer / RodHat AI Cover

curl says SSL certificate problem: unable to get local issuer certificate. The browser says the site is fine. Somebody concludes the problem is curl.

The problem is almost never curl. The browser is quietly compensating for a server misconfiguration, and the difference between those two behaviors is the bug.

The command

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

-servername is not optional. Without it, no SNI is sent, and a server hosting forty vhosts on one IP hands you whichever certificate it considers the default — usually the wrong one. Half the “wrong certificate returned” reports are somebody forgetting -servername. Newer OpenSSL sets it from -connect, older doesn’t; type it anyway.

</dev/null stops it sitting there waiting for you to type HTTP at it.

Reading the output

The part that matters is at the top:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

s: is subject, i: is issuer. The chain is correct when each certificate’s issuer is the next one’s subject, walking down, until you reach something in your trust store. Cert 0’s issuer is R11; cert 1’s subject is R11. Linked. Cert 1’s issuer is ISRG Root X1, which is in the trust store. Done.

Now the broken version:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
Verify return code: 20 (unable to get local issuer certificate)

One certificate. The server sent the leaf and nothing else. The intermediate R11 is missing, so there’s no path from the leaf to any root you trust.

And the browser works fine anyway. Browsers cache intermediates they’ve seen before, and many follow the Authority Information Access extension to fetch the missing one over HTTP. curl, Java, Go, Python and every other client either won’t or can’t. So it works on your laptop and fails in the service mesh, which is the exact shape of a bug that survives three sprints.

The fix is on the server: concatenate the intermediate into your certificate file. Every ACME client emits a fullchain.pem for this reason. Somebody configured cert.pem instead. That’s the whole bug, and it’s the single most common TLS misconfiguration there is.

The verify codes worth knowing

The last line is the verdict:

  • 0 (ok) — done.
  • 20 unable to get local issuer certificate — missing intermediate, as above. Or a genuinely untrusted CA (internal PKI, and your client doesn’t have the root).
  • 21 unable to verify the first certificate — same family; the chain doesn’t reach a trusted root.
  • 10 certificate has expired — check notAfter, and check the intermediate’s expiry too. An expired intermediate on a valid leaf is a fun morning.
  • 19 self signed certificate in certificate chain — internal CA not in the trust store, or somebody’s MITM proxy is intercepting you. Both look identical here. Look at the issuer name; it usually confesses.
  • 62 hostname mismatch — needs -verify_hostname, see below.

The flags that answer specific questions

# Does it work with the system trust store, explicitly?
openssl s_client -connect host:443 -servername host -CApath /etc/ssl/certs

# Does the cert actually cover this hostname? (s_client does NOT check by default)
openssl s_client -connect host:443 -servername host -verify_hostname host

# Is this server the reason we can't drop TLS 1.2?
openssl s_client -connect host:443 -servername host -tls1_3

# Client certificate auth
openssl s_client -connect host:443 -cert client.pem -key client.key

# STARTTLS — the protocol matters
openssl s_client -connect mail:587 -starttls smtp
openssl s_client -connect db:5432  -starttls postgres

That -verify_hostname deserves emphasis: s_client does not check that the certificate matches the hostname unless you ask it to. People get a Verify return code: 0 and conclude everything’s fine while serving a cert for the wrong name. Every real client checks. s_client doesn’t, by default. That inconsistency has burned a lot of people.

Inspect the certificate itself

Pipe it into x509:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -text

Go straight to Subject Alternative Name. The CN field has been deprecated for hostname matching for years — browsers ignore it entirely. If the name you’re connecting to isn’t in the SAN list, the certificate does not cover it, no matter what the CN says.

The quick version for expiry checks in a script:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Why do it by hand

Because the online SSL checkers tell you a grade and the error messages tell you a category, and neither shows you the actual bytes the server sent. The chain listing does. Once you can look at four lines of s:/i: pairs and see that link two is missing, TLS stops being a black box that occasionally curses at you and becomes a linked list you can read — and “certificate verify failed” turns into “somebody deployed cert.pem instead of fullchain.pem,” which takes one line in nginx to fix.