TLS Handshake: Encryption, Certificates & MITM Prevention

Hard

Interview Question: "Walk through the TLS handshake from TCP establishment to encrypted data transmission. Why does TLS use hybrid encryption, how does TLS 1.3 achieve a 1-RTT handshake compared to TLS 1.2's 2-RTT, why was RSA key exchange banned in TLS 1.3, and how does OCSP stapling prevent MITM attacks?"

Before a browser and a server can start exchanging encrypted data, they need to do two things: prove the server is who it says it is, and agree on a secret key that nobody else listening on the network can guess. That whole negotiation is the TLS handshake. Once it's done, you get three guarantees:

1. Confidentiality — encrypted data looks like gibberish to anyone sniffing the network.
2. Integrity — if anyone tampers with a packet in transit, the receiving end can tell.
3. Authentication — a certificate proves you're actually talking to the real server, not an imposter.

---

Why TLS uses two different kinds of encryption

A common interview trap: "Why not just use RSA or elliptic-curve encryption for everything?"

Because there's a tradeoff between agreeing on a secret with a stranger and encrypting data fast, and no single algorithm family is good at both:

Type	Algorithms	Good at	Bad at	Used for
Asymmetric (public/private key)	RSA, ECDSA, X25519	Letting two strangers agree on a secret over an open network, with no prior setup.	Slow. The math (modular exponentiation) is heavy — encrypting gigabytes of video with it would melt a server's CPU.	Only the handshake: proving identity and agreeing on a session key.
Symmetric (one shared key)	AES-GCM, ChaCha20-Poly1305	Fast. Hardware acceleration (Intel AES-NI) pushes over 4 GB/s with barely any CPU cost.	You need a way to get both sides the same secret key without anyone else seeing it — which is exactly what symmetric crypto can't do on its own.	All the actual application data, once the handshake has produced a shared key.

So TLS uses asymmetric crypto for the hard part — proving identity and agreeing on a secret — and then switches to symmetric crypto, which is cheap, for the bulk of the traffic.

---

TLS 1.2 vs. TLS 1.3: cutting the handshake in half

TLS 1.3 (RFC 8446) cut the time to first encrypted byte roughly in half compared to TLS 1.2.

TLS 1.2 — 2 round trips

1️⃣ClientHello

client proposes ciphers

2️⃣ServerHello + Cert + KeyExchange

3️⃣ClientKeyExchange + Finished

4️⃣Server Finished → data flows

TLS 1.3 — 1 round trip

1️⃣ClientHello + Key Share (guessed)

sends its key share immediately

2️⃣ServerHello + Key Share + Cert + Finished

data flows right after

What actually happens in TLS 1.3:

1. ClientHello: instead of waiting to agree on an algorithm first, the client just guesses the common choice (usually the curve X25519) and sends its half of the key exchange in the very first packet.
2. ServerHello + cert + Finished: the server picks a matching cipher, computes the shared secret using the client's key share, sends back its own key share plus its certificate — all in one response.
3. Data flows: the client computes the same shared secret on its end and starts sending encrypted HTTP data in its very next round trip.

Returning visitors can go even faster with 0-RTT, sending encrypted data in the very first packet using a key saved from a previous session. That's only safe for requests that don't change anything (like GET), because a 0-RTT packet can be captured and replayed by an attacker.

---

Why RSA key exchange got banned in TLS 1.3

In TLS 1.2, a server could use static RSA key exchange: the client picks a secret, encrypts it with the server's public RSA key, and sends it over. That sounds fine — until you think about what happens if the server's private key ever leaks.

The problem: if an attacker (or a government) records years of encrypted traffic and later steals the server's long-term private RSA key — through a breach, a subpoena, whatever — they can go back and decrypt every single one of those old recorded sessions. The key never changes, so one leak unlocks everything it ever protected.

The fix — forward secrecy: TLS 1.3 bans static RSA key exchange outright. Instead, every session generates a brand-new, throwaway key pair using Diffie-Hellman (specifically ECDHE). Both sides derive the same shared secret through math, without ever sending that secret over the wire. Once the session ends, the throwaway keys are wiped from memory. Steal the server's permanent certificate key a year later, and it still won't decrypt last year's traffic — the keys that mattered are already gone.

---

Certificates: how TLS stops a man-in-the-middle

Encryption alone doesn't stop an attacker on public Wi-Fi from just pretending to be the server:

Attacker intercepts the connection → claims to be "bank.com" → sends its own public key.
Client encrypts its secrets to the attacker → attacker reads every password in plaintext.

Certificates close this gap:

1. Certificate Authorities (CAs) like Let's Encrypt or DigiCert have root certificates baked directly into your OS and browser as trusted.
2. A CA verifies a company actually owns bank.com, then signs a certificate containing that domain and its public key, using the CA's own private key.
3. When the real server presents its certificate, your browser checks that signature against the CA's public key, which it already trusts — following the chain Root CA → Intermediate CA → bank.com.
4. An attacker can't forge that signature. Any attempt to swap in a fake key trips a browser warning (NET::ERR_CERT_AUTHORITY_INVALID).

OCSP stapling: browsers used to ask the CA directly, "is this certificate still valid?" — which leaked which sites you were visiting and added latency. Now the server periodically asks the CA itself, gets back a signed, timestamped "still valid" proof, and attaches that proof directly to the handshake. The browser just checks the signature — no separate query needed.

---

Say this in the interview

"TLS uses hybrid encryption: asymmetric crypto (ECDHE) handles the handshake — authenticating the server and agreeing on a secret — and then symmetric crypto (AES-GCM) handles all the actual data, because it's orders of magnitude faster. TLS 1.3 cut the handshake from 2 round trips to 1 by having the client guess its key share and send it in the first packet. It also banned static RSA key exchange and made forward secrecy mandatory — every session gets a throwaway key, so a leaked server key later can't decrypt old traffic. Certificates, signed by a trusted CA, prove you're talking to the real server and not an attacker; OCSP stapling lets the server prove its certificate hasn't been revoked without the browser having to ask the CA directly."

---

Python Verification: Hybrid Encryption Performance Benchmark

The following executable Python script benchmarks the massive performance difference between Asymmetric RSA operations and Symmetric AES-GCM bulk encryption, demonstrating why TLS mandates a hybrid architecture:

"""
TLS Hybrid Encryption Benchmark: Asymmetric (RSA) vs Symmetric (AES)
Demonstrates:
  1. High computational cost of asymmetric cryptography (handshake only)
  2. Nanosecond throughput of symmetric AES encryption (bulk data)
"""

import time
import os
import hashlib
from typing import Tuple

try:
    from cryptography.hazmat.primitives.asymmetric import rsa, padding
    from cryptography.hazmat.primitives import hashes
    from cryptography.hazmat.primitives.ciphers.aead import AESGCM
    CRYPTO_AVAILABLE = True
except ImportError:
    CRYPTO_AVAILABLE = False


def benchmark_asymmetric_rsa(payload: bytes) -> Tuple[float, float]:
    """Generates an RSA-2048 keypair and encrypts/decrypts a small secret."""
    private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
    public_key = private_key.public_key()

    start = time.perf_counter()
    ciphertext = public_key.encrypt(
        payload,
        padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None)
    )
    encrypt_time = time.perf_counter() - start

    start_dec = time.perf_counter()
    plaintext = private_key.decrypt(
        ciphertext,
        padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None)
    )
    decrypt_time = time.perf_counter() - start_dec
    assert plaintext == payload
    return encrypt_time, decrypt_time


def benchmark_symmetric_aes(payload: bytes, iterations: int = 500) -> float:
    """Encrypts bulk data using 256-bit AES-GCM."""
    key = AESGCM.generate_key(bit_length=256)
    aesgcm = AESGCM(key)
    nonce = os.urandom(12)

    start = time.perf_counter()
    for _ in range(iterations):
        ct = aesgcm.encrypt(nonce, payload, None)
        _ = aesgcm.decrypt(nonce, ct, None)
    total_time = time.perf_counter() - start
    return total_time / iterations


def main():
    print("=== TLS Hybrid Encryption Performance Benchmark ===\n")
    if not CRYPTO_AVAILABLE:
        print("Note: 'cryptography' library not installed. Simulating mathematical scaling comparison:")
        print("  • RSA-2048 Modular Exponentiation: ~1.5 ms per operation (Heavy CPU math)")
        print("  • AES-256-GCM Hardware Acceleration: ~0.002 ms per 64KB (2,000x faster!)")
        print("Conclusion: TLS uses RSA/ECDHE solely to exchange the AES symmetric key.")
        return

    # Secret pre-master key (32 bytes)
    SECRET_KEY_PAYLOAD = os.urandom(32)
    # Bulk application payload (64 KB)
    BULK_DATA = os.urandom(64 * 1024)

    print("1. Benchmarking Asymmetric RSA-2048 (Used only during Handshake):")
    rsa_enc, rsa_dec = benchmark_asymmetric_rsa(SECRET_KEY_PAYLOAD)
    print(f"  RSA Encryption (Public Key):  {rsa_enc * 1000:6.3f} ms")
    print(f"  RSA Decryption (Private Key): {rsa_dec * 1000:6.3f} ms")

    print("\n2. Benchmarking Symmetric AES-256-GCM (Used for all Application Data):")
    aes_per_op = benchmark_symmetric_aes(BULK_DATA, iterations=200)
    print(f"  AES-GCM 64KB Encrypt + Decrypt: {aes_per_op * 1000:6.4f} ms")

    speedup = (rsa_dec) / aes_per_op if aes_per_op > 0 else 1.0
    print(f"\nResult: Symmetric AES-GCM processed 64KB of data ~{speedup:.0f}x faster than RSA decrypted 32 bytes!")
    print("Direct cause: Validates why TLS restricts asymmetric math strictly to key exchange.")

if __name__ == "__main__":
    main()