Post-quantum IKEv2 denial of service: are gateways at risk?

Post-quantum IKEv2 denial of service covers the availability risks that appear when VPN gateways add a post-quantum key exchange such as ML-KEM to IKEv2. The cryptography itself is cheap. The risk comes from larger messages, an extra round trip and more half-open state. Synchronised reconnects or rekeys across many sites can make all three worse.

For organisations running OT and critical infrastructure, a slow or failed tunnel is an operational problem as well as a security one. Availability expectations such as the resource-availability requirements in IEC 62443-3-3 make it necessary to treat handshake capacity as an engineering parameter. This article explains how hybrid IKEv2 behaves under load and which protocol mechanisms help.

Does ML-KEM make VPN gateways more vulnerable to handshake flooding attacks?

Only a little, and mostly not through CPU load. ML-KEM increases message size and the number of exchanges, so the exposure moves from computation to packet handling and half-open state.

What actually changes in the handshake

RFC 7296 (IKEv2) sets up a session with IKE_SA_INIT and IKE_AUTH. RFC 9370 extends this so that multiple key exchanges are performed when an IKE SA is being established using the IKE_INTERMEDIATE exchange. In practice this means typically, one classical (EC)DH and the other one post-quantum algorithm. The post-quantum part is ML-KEM, standardised by NIST in FIPS 203, and NIST recommends using ML-KEM-768 as the default parameter set.

The IETF ML-KEM profile for IKEv2 states that these key exchanges could require two or three network IP packets from both the initiator and the responder, so IKE_INTERMEDIATE messages carrying ML-KEM public keys and ciphertexts may be IKEv2 fragmented as per [RFC7383]. A recent protocol survey notes that IKEv2 was designed around the assumption of compact key exchange payloads, and that its fragmentation support is less mature than TLS 1.3.

Where the real exposure sits

Three things deserve attention. First, the responder keeps each half-open SA for an extra round trip, so memory and SA table slots stay occupied for longer. Second, fragment reassembly buffers become a resource of their own. Third, the responder sends a 1088-byte ciphertext in reply to a 1184-byte key. That means the post-quantum step does not amplify traffic, but it does double the bytes involved.

The ordering is useful here. ML-KEM runs in IKE_INTERMEDIATE, after IKE_SA_INIT, so a responder that demands a cookie checks that the initiator’s address is real before doing any post-quantum work.

How many hybrid IKEv2 or TLS handshakes per second can an encryption gateway handle?

No single published figure applies: the rate depends on CPU, authentication method, packet handling and SA table limits. Measure it on your own hardware and configuration, and size it against your recovery-time target, not a datasheet peak.

The authors of the IETF ML-KEM draft describe ML-KEM-768 and ML-KEM-1024 as parameters which will not have material performance impact on IKEv2/IPsec tunnels which usually stay up for long periods of time. That is true for steady state. Recovery after an outage is a burst problem, though, and in a burst the costs add up: classical (EC)DH, ML-KEM encapsulation or decapsulation, authentication signatures in IKE_AUTH, fragment handling and per-SA kernel state. In many configurations authentication, not ML-KEM, is the largest single cost.

A practical sizing method

• Define the target: how many tunnels must be back, and within what time after power or WAN returns.
• Derive the required rate: 600 sites inside 60 seconds needs a sustained 10 completed handshakes per second, plus headroom for retransmissions. This is an illustrative calculation, not a benchmark.
• Test with real message sizes: run the load test with ML-KEM payloads, fragmentation and the WAN MTU in place, including any path that drops IP fragments.
• Record the knee: note the rate at which completion time rises sharply or half-open SAs pile up. That figure is your real capacity.

The same reasoning applies to hybrid TLS 1.3 for remote users. In the TLS ML-KEM specification, ML-KEM-768 has encapsulation keys of size 1184 bytes, which makes the ClientHello noticeably larger than a classical one.

What happens when hundreds of OT sites reconnect at the same time after a power or WAN outage?

They form a self-inflicted handshake flood: every site starts IKE_SA_INIT within seconds of the others. If the hub’s half-open limits or cookie thresholds are tuned for normal traffic, legitimate sites can be slowed down by the same defences meant for hostile traffic.

The usual sequence looks like this. A regional power restoration brings substations or pumping stations back almost at the same moment. Each initiator sends its first message, and the hub’s half-open count rises past its cookie threshold. Every site now needs an extra round trip before the IKE_INTERMEDIATE exchange with its fragmented ML-KEM payload. Initiators that time out retransmit. Under RFC 7296 the initiator is responsible for retransmission, and if many devices share identical timers, their retries arrive in waves too.

Mitigations that work in OT estates

• Randomise the first attempt: add a random start delay on each remote gateway after boot or link-up, separate from the retransmission timer.
• Stagger by priority: give protection and control sites a shorter random window than non-critical monitoring sites.
• Spread responders: divide large estates over several hub gateways or addresses so that no single responder gets the whole burst.
• Check the MTU path: make sure IKEv2 fragmentation is enabled and that the WAN does not silently drop fragments. Otherwise every ML-KEM exchange turns into repeated timeouts.

Behaviour after a failure differs per product. As an example, the Quantum Network QPN gateway retries automatically after a line failure and never falls back to unencrypted traffic. Its vendor also states plainly that it is not a defence against denial of service, so upstream filtering and capacity planning remain separate tasks.

How do IKEv2 cookies and puzzles protect post-quantum gateways from DDoS?

Cookies prove that the initiator can receive traffic at its claimed address before the responder stores state. Puzzles then make each new handshake cost the initiator measurable work. Together they mean the responder only spends post-quantum effort on initiators that have proven they are real.

Stateless cookies (RFC 7296, section 2.6)

According to RFC 8019, the stateless cookie mechanism from Section 2.6 of [RFC7296] prevents an attack with spoofed source addresses. This doesn’t completely solve the issue, but it makes the limiting of half-open SAs by address or prefix work. For a hybrid gateway this is especially useful, because the cookie round trip happens in IKE_SA_INIT, before any ML-KEM payload is processed.

Client puzzles (RFC 8019)

RFC 8019, written by Yoav Nir and Valery Smyslov, adds puzzles on top of cookies. The PUZZLE notification contains the difficulty level of the puzzle and the PRF the Initiator should use. A responder can also use puzzles in IKE_AUTH. With fragmented messages, the responder SHOULD NOT try to verify authenticity of the kept fragments until the first fragment with the PS payload is received and the solution to the puzzle is verified. This matters for post-quantum gateways, where large fragmented messages are now normal.

The RFC also advises restraint. In the words of Nir and Smyslov: “When there is no general DDoS attack, it is suggested that no cookie or puzzles be used.” In OT terms, configure both as load-triggered, not always on, and keep puzzle difficulty low enough that constrained edge devices can still solve them during a reconnect storm.

How should rekey timers be randomised to avoid synchronised post-quantum rekeying peaks?

Draw a fresh random offset for every rekey, on every peer, within a window that is wide compared with the time a hybrid exchange takes under load. RFC 7296 requires the jitter but does not set its size, so you have to engineer the value.

The RFC 7296 authors (Kaufman, Hoffman, Nir, Eronen and Kivinen) write that the timing of rekeying requests SHOULD be jittered (delayed by a random amount of time after the need for rekeying is noticed). The original aim was to avoid simultaneous rekey collisions between two peers. In a hub-and-spoke OT estate the larger effect is at fleet level: tunnels that came up together after an outage, with identical lifetimes, will rekey together indefinitely unless something breaks the alignment.

With RFC 9370, a rekey that includes the post-quantum component uses CREATE_CHILD_SA followed by the new IKE_FOLLOWUP_KE exchange. That makes every rekey a multi-packet event, so short lifetimes multiply the effect of any synchronisation.

Practical rules for jitter

• Randomise per event, not per device: a fixed offset per device keeps the fleet’s pattern stable over time. A fresh random draw per rekey keeps it scattered.
• Make the window proportional: with short rekey intervals, even a few seconds of jitter spreads the load. With hour-scale lifetimes, use minutes.
• Bias towards earlier rekeying: subtract the jitter from the soft lifetime so that a rekey never runs past the hard lifetime.
• Verify in logs: plot rekey timestamps across the estate. Clusters mean the randomisation is too narrow or is being reset at reconnect.

What should OT operators check before enabling hybrid IKEv2 at scale?

Before rollout, make sure capacity, fragmentation and timer behaviour have been tested under burst conditions, not only that tunnels come up. The algorithm choice is well settled: NIST states that at present, ML-KEM is believed to be secure, even against adversaries who possess a quantum computer. The operational settings around it are less settled.

• Measured handshake completion rate at the hub, using real ML-KEM payloads and your WAN MTU.
• Half-open SA limits and cookie thresholds tested against a full-estate reconnect.
• RFC 8019 puzzle support and difficulty confirmed on both ends, including constrained edge hardware.
• Random start delays after boot and link-up, separate from retransmission timers.
• Per-event rekey jitter, checked against logged rekey timestamps.
• Upstream DDoS controls in place, since encryption gateways protect data in transit, not service availability.