Jetty SslConnection does not release pooled ByteBuffers in case of errors

Impact

SslConnection does not release ByteBuffers in case of error code paths.
For example, TLS handshakes that require client-auth with clients that send expired certificates will trigger a TLS handshake errors and the ByteBuffers used to process the TLS handshake will be leaked.

Workarounds

Configure explicitly a RetainableByteBufferPool with max[Heap|Direct]Memory to limit the amount of memory that is leaked.
Eventually the pool will be full of "active" entries (the leaked ones) and will provide ByteBuffers that will be GCed normally.

With embedded-jetty

int maxBucketSize = 1000;
long maxHeapMemory = 128 * 1024L * 1024L; // 128 MB
long maxDirectMemory = 128 * 1024L * 1024L; // 128 MB
RetainableByteBufferPool rbbp = new ArrayRetainableByteBufferPool(0, -1, -1, maxBucketSize, maxHeapMemory, maxDirectMemory);

server.addBean(rbbp); // make sure the ArrayRetainableByteBufferPool is added before the server is started
server.start();

With jetty-home/jetty-base

Create a ${jetty.base}/etc/retainable-byte-buffer-config.xml

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://www.eclipse.org/jetty/configure_10_0.dtd">

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Call name="addBean">
    <Arg>
      <New class="org.eclipse.jetty.io.ArrayRetainableByteBufferPool">
        <Arg type="int"><Property name="jetty.byteBufferPool.minCapacity" default="0"/></Arg>
        <Arg type="int"><Property name="jetty.byteBufferPool.factor" default="-1"/></Arg>
        <Arg type="int"><Property name="jetty.byteBufferPool.maxCapacity" default="-1"/></Arg>
        <Arg type="int"><Property name="jetty.byteBufferPool.maxBucketSize" default="1000"/></Arg>
        <Arg type="long"><Property name="jetty.byteBufferPool.maxHeapMemory" default="128000000"/></Arg>
        <Arg type="long"><Property name="jetty.byteBufferPool.maxDirectMemory" default="128000000"/></Arg>
      </New>
    </Arg>
  </Call>
</Configure>

And then reference it in ${jetty.base}/start.d/retainable-byte-buffer-config.ini

etc/retainable-byte-buffer-config.xml

References

https://github.com/eclipse/jetty.project/issues/8161

For more information

• Email us at security@webtide.com