Symptom: Persistent TLS handshake failure observed in order-service logs when attempting to connect to payment-gateway. Logs show generic "TLS handshake failed" or "remote host closed connection" messages, without specific certificate or protocol errors.
Hypothesis List:
- Hypothesis 1: Certificate Chain Validation Failure
- Hypothesis 2: Server Name Indication (SNI) Mismatch
- Hypothesis 3: Incompatible Cipher Suites
Checks (with commands):
Check for Hypothesis 1: Verify payment-gateway's certificate chain and order-service's trust store. * Inspect payment-gateway's presented certificate chain: openssl s_client -connect payment-gateway.internal.svc.cluster.local:443 -showcerts -servername payment-gateway.internal.svc.cluster.local < /dev/null * *Expected output / interpretation*: The output should show the full certificate chain, including the root CA. Look for Verify return code: 0 (ok) and ensure all certificates are valid and not expired. Any error codes here indicate an issue with the certificate itself or the chain. * Check order-service's trust store for the issuing CA: sudo keytool -list -keystore /etc/ssl/certs/java/cacerts (for Java applications) or ls /etc/ssl/certs/ (for system-wide CAs) * *Expected output / interpretation*: Confirm the root CA or intermediate CA that signed payment-gateway's certificate is present in order-service's trust store.
Check for Hypothesis 2: Confirm SNI is being sent correctly by order-service and accepted by payment-gateway. * Simulate order-service's connection with explicit SNI: openssl s_client -connect payment-gateway.internal.svc.cluster.local:443 -servername payment-gateway.internal.svc.cluster.local * *Expected output / interpretation*: A successful connection and handshake details. If this works but order-service fails, it suggests order-service might not be sending the correct SNI. If it fails, the payment-gateway might not be configured for the SNI. * Capture traffic on payment-gateway to inspect SNI: sudo tcpdump -i any -s 0 -A 'port 443 and tcp[((tcp[12:1]&0xf0)>>2):4]=0x16030100' | grep 'Server Name' * *Expected output / interpretation*: Look for the Server Name field in the captured TLS Client Hello packets. It should match payment-gateway.internal.svc.cluster.local.
Check for Hypothesis 3: Identify common cipher suites between order-service and payment-gateway. * List payment-gateway's supported cipher suites: nmap --script ssl-enum-ciphers -p 443 payment-gateway.internal.svc.cluster.local * *Expected output / interpretation*: A list of cipher suites supported by the server. Note preferred ones. * Inspect order-service's configured cipher suites (application-dependent, example for Java): grep -r "jdk.tls.client.cipherSuites" /path/to/java/config * *Expected output / interpretation*: Configuration files should list the cipher suites order-service is configured to offer. Compare this list with payment-gateway's supported ciphers to find common ground.
Likely Fixes:
Fix for Hypothesis 1: * Missing Root/Intermediate CA: Import the necessary CA certificate into order-service's trust store. * Expired Certificate: Renew payment-gateway's certificate. * Incorrect Chain Order: Ensure payment-gateway is serving the full chain in the correct order (leaf, intermediate, root).
Fix for Hypothesis 2: * `order-service` not sending SNI: Configure order-service to send the correct SNI (payment-gateway.internal.svc.cluster.local). This is often a client-side library configuration. * `payment-gateway` misconfigured for SNI: Adjust payment-gateway's server configuration to correctly handle the expected SNI, ensuring the corresponding certificate is served.
Fix for Hypothesis 3: * No Common Cipher Suite: Configure order-service or payment-gateway to support at least one common, secure cipher suite. Prioritize modern, strong ciphers.
Verification: * Restart order-service and payment-gateway (if configuration changes require it). * Monitor order-service logs for successful connection messages to payment-gateway. * Execute a test transaction or API call from order-service to payment-gateway to confirm end-to-end functionality. * Review payment-gateway access logs for successful TLS handshakes from order-service.