netty-handler SniHandler 16MB allocation

Summary

The SniHandler can allocate up to 16MB of heap for each channel during the TLS handshake. When the handler or the channel does not have an idle timeout, it can be used to make a TCP server using the SniHandler to allocate 16MB of heap.

Details

The SniHandler class is a handler that waits for the TLS handshake to configure a SslHandler according to the indicated server name by the ClientHello record. For this matter it allocates a ByteBuf using the value defined in the ClientHello record.

Normally the value of the packet should be smaller than the handshake packet but there are not checks done here and the way the code is written, it is possible to craft a packet that makes the SslClientHelloHandler

1/ allocate a 16MB ByteBuf
2/ not fail decode method in buffer
3/ get out of the loop without an exception

The combination of this without the use of a timeout makes easy to connect to a TCP server and allocate 16MB of heap memory per connection.

Impact

If the user has no idle timeout handler configured it might be possible for a remote peer to send a client hello packet which lead the server to buffer up to 16MB of data per connection. This could lead to a OutOfMemoryError and so result in a DDOS.