Hi,
I've submitted a Jersey-related question on Stackoverflow. I'm posting it here hoping it will get answered by some of the experts guys on this mailing list. :)
Will appreciate any hint on this.
---
I have an asynchronous JAX-RS API for long-polling clients put together in Jersey Container Servlet 2.22 and hosted on Tomcat 7.
It looks similar to the snippet shown below. It works well in production.
On average 150 long-polling requests are being executed at the same time. It results in almost thesame number of live Tomcat HTTP connections (according to JMX metrics). For this low traffic scenario plain-old HTTP-BIO connector has been used without problems. No runtime connection leak can be detected provided you use only managed threads :)
@POST
@Path("/liveEvents")
@ManagedAsync
public void getResult(@Suspended final AsyncResponse asyncResponse, RequestPayload payload) {
asyncResponse.setTimeout(longPollTimeoutMs, TimeUnit.MILLISECONDS);
asyncResponse.setTimeoutHandler(new TimeoutHandler() {
@Override
public void handleTimeout(AsyncResponse asyncResponseArg) {
try {
asyncResponseArg.cancel();
} finally {
cleanupResources();
}
}
});
startListeningForExternalEventsAndReturn(payload);
}
private void startListeningForExternalEventsAndReturn(RequestPayload payload) {
externalResource.register(new Listener() {
@Override
public void onEvent(Event event) {
respond(event);
}
});
}
private void respond(Event event) {
try {
asyncResponse.resume(event);
} catch (RuntimeException exception) {
asyncResponse.resume(exception);
} finally {
cleanupResources();
}
}
The problem I'm facing is that after a successful Tomcat redeploy process the number of live connections will apparently increase to about 300 then to 450 and after some further redeploys it will hit the maxConnection limit configured for the container.
The clients of the API handle the redeploy by waiting for a client-side timeout (which is of course bigger than the one set at servlet-side) and start polling the API again. But they are guaranteed to send only one request at the same time.
The shape of the monitoring graph on connection count gives a hint. Connection count remains constant after undeployment (connections are not released back to the pool even by TimeOutHandler) and starts to increase (allocate new connections) as clients start long-polling again. In fact, ongoing (suspended) asnyc requests started in the previous context are never relesed until JVM termination!
After some digging around it's not difficult to find out by analyzing heap dumps made after few redeployments that unreleased, suspended AsyncResponse (AsyncResponder) instances remain in the memory from previous web application contexts (easily filterable by JQL queries grouped by Classloader instances). It's very suspicious too that the same number of out-dated org.apache.coyote.Request instances are present in the memory from previous contexts.
I started to look around the undeployment-related source code of the Jersey Container hoping that some graceful shutdown process is implemented for async requests with some cleanup actions executed at @PreDestroy-time or in close() or dispose() methods of Providers.
I had an optimistic guess that by running each scheduled TimeOutHandlers right before undeployment would solve the problem. But replacing the default @BackgroundScheduler provider (DefaultBackgroundSchedulerProvider) to a custom implementation and collecting all queued TimeoutHandlers of the Executor and eventually invoking AsyncResponse.resume() or AsyncResponse.cancel() on them did not help. This stage might be too late for this cleaning up because request-scope is already shut down.
Any ideas on what my async setup is missing or how Jersey can be configured to release the Servlet Container's connections that are still suspended at redeploy-time?
Locked
