[Solved] ConnectionPoolTimeoutException: Timeout waiting for connection from pool

background

An error was found in the monitoring system today. The error is as follows

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

action

At first, it was suspected that the connection pool setting was too small, so check the code’s setting of the connection pool size

It is found that the connection pool setting is large enough that our business can not use so many connections at all, so it is suspected that the connections have not been released after running out. View the number of connections on the server

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Check the code and find that the connection is not released, as shown below

HttpResponse response = httpclient.execute(httpPost);

List<String> lines = IOUtils.readLines(response.getEntity().getContent(), "utf-8");
StringBuffer res = new StringBuffer();
for (String line : lines) {
   res.append(line);
}
       
ChatMessage chatMessage = getChatMessage(sendChatMessageDTO, replyMessageInfo, sendMessageesResultBO);
return chatMessage;

Solution:

Once the problem is found, we can return the connection to the connection pool.

HttpPost httpPost = new HttpPost(connectUrl);
InputStream inputStream = null;
try {
            
    HttpResponse response = httpclient.execute(httpPost);
    inputStream = response.getEntity().getContent();
    List<String> lines = IOUtils.readLines(inputStream, "utf-8");
} catch (Exception e){
    logger.error("connect error", e);
    if(httpPost != null){
    	httpPost.abort();
     }
} finally {
    if(inputStream != null){
        try {
            inputStream.close();
         } catch (Exception e){
            logger.error("", e);
         }
    }
}

Thinking (based on version 4.5)

What is the principle of httpclient connection pooling?

There are three main attributes in the connection pool, which are used for connections in different states

//Store the connection in use
private final Set<E> leased;
// holds the available connections
private final LinkedList<E> available;
// holds pending connections 
private final LinkedList<PoolEntryFuture<E>> pending;

What is the default connection pool size?

We check the official code (version 4.5) and find that the default total size is 20, and the default maximum size of each route is 2

 public PoolingHttpClientConnectionManager(
        final HttpClientConnectionOperator httpClientConnectionOperator,
        final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
        final long timeToLive, final TimeUnit timeUnit) {
        super();
        this.configData = new ConfigData();
        this.pool = new CPool(new InternalConnectionFactory(
                this.configData, connFactory), 2, 20, timeToLive, timeUnit);
        this.pool.setValidateAfterInactivity(2000);
        this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator");
        this.isShutDown = new AtomicBoolean(false);
    }

How does close work?

There are two kinds of close

CloseableHttpResponse.Close(): this method is to put the exhausted connections back to the available collection of the connection pool, rather than actually closing the connections.Release() is called internally

httpClient.Close(): This is to close the connection pool and release all resources in the pool, that is, to really close the connection

Similar Posts: