Firewall TLS inspection and post-quantum key exchange

Firewall TLS inspection with post-quantum key exchange is about how firewalls, proxies and other middleboxes handle TLS 1.3 handshakes that use hybrid key agreement. The usual example is X25519MLKEM768, which pairs classical X25519 with ML-KEM from FIPS 203. These handshakes have bigger messages and new group codepoints, and that can lead to dropped connections, failed decryption or a quiet fall back to classical key exchange.

For organisations running OT and critical infrastructure, this is a practical problem, not a theoretical one. The same firewall often sits on site-to-site links, cloud uplinks and remote access paths. One misbehaving inspection engine can break post-quantum protection on all of them. This article explains what changes in the handshake, why middleboxes struggle with it, and how to decide where inspection should and shouldn’t happen.

How large is a TLS ClientHello with ML-KEM-768?

A hybrid X25519MLKEM768 key share alone adds 1,216 bytes to the ClientHello. That pushes the full message past a single standard TCP segment. Classical X25519 needs only 32 bytes.

The numbers come straight from draft-ietf-tls-ecdhe-mlkem, the IETF TLS working group document that defines the hybrid groups for RFC 8446 (TLS 1.3):

Source: IETF, draft-ietf-tls-ecdhe-mlkem-00, 2025-03-23.

The size of the client share is 1216 bytes (1184 bytes for the ML-KEM part and 32 bytes for X25519). The size of the server share is 1120 bytes (1088 bytes for the ML-KEM part and 32 bytes for X25519).

The total ClientHello size depends on everything else the client sends: SNI, ALPN, extra key shares, padding and so on. At a 2025 NIST workshop on KEM guidance, one presentation showed an ML-KEM-768 ClientHello of about 3.5 KB, split across three TCP segments of 1,460, 1,460 and about 580 bytes. ML-KEM large public key/ ciphertext sizes can cause fragmented TLS Client Hello packets. Whatever the exact figure, a single packet no longer holds the whole handshake opener.

Why does my firewall block or drop post-quantum TLS ClientHello messages?

Most drops happen because the inspection engine expects the whole ClientHello in the first TCP segment, or because it rejects a named group it doesn’t recognise. The cryptography itself is rarely the cause.

ClientHello split across several segments

Application identification, SNI-based URL filtering and IPS signatures often read the first data packet of a flow. When a hybrid ClientHello spans two or three segments, an engine that doesn’t reassemble the TCP stream sees a truncated TLS record. Depending on how it is configured, it then drops the flow, resets it, or classifies it as unknown and blocks it under a default-deny rule. TLS handshakes failing on Client Hello due to enterprise middle boxes incompatibility was reported to the NIST workshop as one of the first problems seen when hybrid support was switched on by default.

Unknown group codepoints

X25519MLKEM768 is registered as Value: 4588 (0x11EC). Strict protocol validators, and older TLS parsers built before this codepoint existed, may treat an unknown group in supported_groups or key_share as a protocol anomaly.

Path MTU and tunnel overhead

On OT networks, traffic often already runs inside IPsec, GRE or vendor tunnels with reduced MTU. When the ClientHello gets bigger, broken path MTU discovery or missing MSS clamping becomes visible. You see handshakes that stall rather than fail cleanly.

This isn’t new. When TLS 1.3 itself was rolled out, two browser measurements put the extra error rates at roughly 1.5% and 3.4%. Eric Rescorla, editor of RFC 8446, told the IETF TLS working group that negotiating TLS 1.3 seems to cause increased failures with a variety of middleboxes. Post-quantum key exchange is testing the same weak spots again.

Can a firewall decrypt and inspect hybrid ML-KEM TLS traffic?

Only through active interception, where the firewall terminates TLS and opens a new session. To keep post-quantum protection, it must support the hybrid group on both legs.

TLS 1.3 under RFC 8446 already ruled out passive decryption with a server’s private key, because every key exchange is ephemeral. Hybrid ML-KEM doesn’t change that. An out-of-band sensor that only sees mirrored traffic can’t decrypt X25519MLKEM768 sessions, just as it couldn’t decrypt classical TLS 1.3 sessions.

An inline inspection proxy works differently. It acts as the server towards the client and as the client towards the real server. On each leg it has to generate or process ML-KEM keys itself. That means its TLS stack needs a FIPS 203 implementation and support for the draft-ietf-tls-ecdhe-mlkem codepoints. It also has to cope with the extra per-handshake computation and memory at the connection rates your network produces.

For hybrid groups, NIST describes ML-KEM as presently believed to be secure, even against adversaries in possession of a large-scale fault-tolerant quantum computer. That protection only covers the legs where ML-KEM is actually negotiated.

Does SSL inspection downgrade post-quantum connections back to classical key exchange?

Often, yes. If the inspection proxy doesn’t support hybrid groups, it negotiates classical key exchange on one or both legs. Neither the client nor the server will report this as an error.

This isn’t a downgrade attack in the RFC 8446 sense, because the proxy is a legitimate TLS endpoint that the client trusts through an enterprise CA. The result is still classical-only protection. Under RFC 8446, Section 9.3 requires that such proxies remove any extensions they do not understand, and the proxy is free to choose any group it and the peer share.

The HelloRetryRequest case is worth watching. Suppose a client sends only an X25519MLKEM768 key share but also lists X25519 in supported_groups. A proxy without hybrid support responds with a HelloRetryRequest asking for X25519. The handshake succeeds, takes an extra round trip, and leaves the session classical. Clients that send only a post-quantum key share, with no classical group listed, fail outright against such a proxy. That is louder, but safer to diagnose.

How do I test whether middleboxes break post-quantum handshakes before rollout?

Build a test matrix that covers every network path and every inspection mode. On each path, record three things: whether the handshake completes, how long it takes, and which group was negotiated at each hop.

The NIST NCCoE Migration to Post-Quantum Cryptography project frames interoperability work around resolution of compatibility issues in a controlled, non-production environment, and its TLS results appear in NIST SP 1800-38C. The NCCoE also warns that maintaining connectivity and interoperability among organizations and organizational elements during the transition from quantum-vulnerable algorithms to quantum-resistant algorithms will require careful planning. In practice, the testing looks like this:

Run the matrix for site-to-site links, cloud egress, remote access gateways and any OT DMZ separately. These paths often go through different policies or inspection profiles even on the same hardware. Re-run it after every firmware or signature update, because inspection behaviour can change without any change to your rules.

Should post-quantum encrypted tunnels bypass existing firewall inspection?

Not as a blanket rule. Decide per path: keep inspection where your security policy depends on it, and only bypass where inspection adds nothing on that link.

A post-quantum tunnel between two of your own sites carries traffic that your firewall may still need to see: remote engineering sessions, historian replication, patch distribution. If the tunnel terminates behind the firewall, the firewall only sees ciphertext. One workable pattern is to terminate the post-quantum layer in front of the firewall, so traffic is protected across the untrusted network and inspected in cleartext once it enters the site. The firewall then doesn’t need to decrypt anything.

Quantum Network’s QPN gateway follows this placement logic. It encrypts at Layer 3 with a hybrid key agreement that combines a classical part with ML-KEM (FIPS 203). It sits in front of the existing firewall when that firewall has to keep inspecting traffic, and can sit behind it when inspection isn’t needed on that path. It doesn’t replace the firewall’s own filtering functions.

Whichever pattern you choose, write down the reason for each bypass and review it as you would any other firewall exception. If a post-quantum path skips inspection only because the inspection engine couldn’t parse its handshake, that is a technical constraint to fix, not a design decision.