I’ve had some bad experiences lately with the Tomcat 6 Comet APIs with the NIO Connector. I have a simple COMET servlet that spawns a thread and waits 5 seconds to respond to a Firefox browser. I also call event.setTimeout(10000) (timeout on the event of 10 seconds). After the response is finished writing I call event.close(); When using the Tomcat NIO, I don’t see a response until after event.setTimeout() time. So, if I set the timeout to 100 seconds I don’t see a response in the browser for 100 seconds.
Maybe I’m not adding some undocumented secret sauce, but when I test this out with JBoss Web and our APR connector, it works as expected: I see a response in the browser after 5 seconds. I also had a go with Servlet 3.0 through Jetty 7.0 pre3. Works as expected there as well.
This isn’t the only problems I’ve had. I also had a ClassCastException when I add a ServletFilter in front of a COMET Servlet. The APIs assume that all your filters implement CometFilter. Granted, I experienced this particular problem in the version of Tomcat 6 that ships with JBoss 4.2.3.GA, but after trying out the timeout problem described above with Tomcat 6 binaries from apache.org, I didn’t bother going any further to see if this was a JBoss specific problem or not. I did test to see if JBossWeb has the same problem…As expected it DID NOT.
So what’s the deal here?
- I’m just stupid?
- Nobody uses Tomcat 6 COMET + NIO APIs?
- Everybody who is doing asynchronous HTTP is using Jetty?
- Asynchronous HTTP is just hype generated by Greg Wilkins to breath new life into his increasingly irrelevant company?
My personal guess is that 2-3 are the main problems, with #1 a high possibility, with a sprinkle of 4. Anybody out there have any experiences with Tomcat 6 + COMET + NIO?
As a result of this, I’m not going to support COMET + Tomcat NIO with RESTEasy’s Async HTTP APIs. Only Jetty and JBossWeb.
Bill the Plumber 
Nov 13, 2008 @ 06:07:54
Bill,
Well if Async HTTP is just hype, then thanks for the puff of hot air for my irrelevant company.
But async is hard and only of benefit to a few specific use-cases. Most of the first attempts at APIs have not been the best. Jetty-6 Continuations definitely have their problems, but Tomcat CometProcessors almost completely missed the point. Even the current draft of Servlet 3.0 leaves a bit to be desired. But I’m hopeful the next draft will address the few remaining issues.
Eventually Framework developers should be able to hide the complexities from the average developer. DWR, JSR, Cometd and BlazeDS have all been ported to the draft servlet 3.0 API (and to Jetty-6 Continuations), and hopefully that trend will continue. Developers should be able to use messaging APIs and not have to deal with the details of the transport, so that eventually things like WebSocket will improve the infrastructure below applications.
Nov 13, 2008 @ 12:49:48
From the little I’ve dealt with it the past few weeks, I think the difficulties of async HTTP in Java are self inflicted. Jetty 6 continuations were just so bizarrely implemented (using exceptions as control flow?) that I refused to integrate it with Resteasy. The Tomcat COMET APIs are at least functional and straightforward, its too bad the documentation (and implementation, well, except for Remy’s of course) is so horrible. Servlet 3.0 is much better, but all the retry stuff is just bizarre and I don’t see why somebody would need or want the complication. I’m glad at least you guys have a pure Servlet 3.0 implementation or I’d be forced to use JBossWeb with a native plugin to use async HTTP at all. (Remy refuses to write a NIO transport).
As far as COMET itself goes, I also don’t like it as it is tunneling a proprietary protocol over HTTP, which is an architectural no-no in my book. You can still get 90% of the performance benefits using pure HTTP over the async APIs. WebSocket will probably take forever to take hold and will meet strong resistance from IT.
Still, I believe the performance benefits of async HTTP are real. The few specific use-cases you talk about will crop up more and more as AJAX applications come online. As REST becomes more popular, you’ll also see HTTP being used as a messaging protocol.
Nov 13, 2008 @ 21:55:01
Bill,
the retry request stuff is purely for applications that wish to use filters, servlets and JEE style stuff to generate responses. Ie asynchronous wait for a webservice response and then generate a JSF page more or less normally.
there is definitely another style of async HTTP that does not want or need to deal with filters, servlets and JEE stuff. For them, servlets are just a PITA and retry even more so. I certainly encourage new HTTP api to be developed that are more suitable for async. But for developers that are invested in servlet technology, the retry style of async makes many of the benefits available with out the need to switch to an entirely new way of generating content.
Nov 16, 2008 @ 08:40:43
Bill,
I would definitely like your feedback on the Servlet 3.0 async API. I am working on the Public Review of the spec (don’t look at the EDR that API is no longer valid). You can get access to the current proposal from Remy or wait for a few days and hopefully it will be public. I also agree that the retry is bizarre but that is a use case that Greg has been bringing up over and over again. I would like to get more feedback on the whole API.
Nov 17, 2008 @ 13:20:36
Rajiv, unrelated, but something I’d *really* like to see is JAX-RS expressions available for security, servlet, and filter mappings. I’ll look at Remy’s proposal.
Nov 17, 2008 @ 21:19:01
Bill are you referring to the URI templating stuff? Send me email about what you are thinking.
Nov 17, 2008 @ 22:52:11
Also get the proposal from Remy. It isn’t Remy’s proposal :).
Dec 17, 2008 @ 16:49:03
GraniteDS (Flex/JEE alternative to BlazeDS/LCDS) has Comet support for Tomcat 6 (CometProcessor/APR) and Jetty 6 (Continuation).
Tomcat 6 implementation (with APR) has been hell: unexpected invalid CometEvents, very complicated race conditions, no relevant support in Tomcat forums, etc. On the other hand, Jetty 6 was much more easier, mainly because it provides *true* samples. The Tomcat sample doesn’t even compile and, when compilation problems are fixed, doesn’t work (not to say that it is so basic that nobody can really start from it for a real world application)…
So, after 2/3 weeks of nightmare, I was also thinking that #1 was highly probable, but, because it’s (seems to be?) working now, I would say that #2 is the true problem (Tomcat developers included)…
Dec 17, 2008 @ 16:55:02
I talked to Remy about this earlier. When he refactored “Comet” within Jboss web, he found a number of bugs of which I mentioned in this blog and fixed them in JBoss Web.
Dec 24, 2008 @ 10:38:11
I agree with you, Comet using servlets seems to be immature.
The good news is Comet (long polling) is being to be standardized on Servlet 3.0 and project Atmosphere promises to bring us an unified API on top of the different servlet based asynchronous approaches for Comet.
Meanwhile we can do long polling using a synchronous approach (by holding the thread). This works and seems to scale too.
Servlet 3.0 Public Review Sparks a Debate | Xingyu.zhang's Technology Circle
Dec 25, 2008 @ 15:30:36
Jan 02, 2009 @ 17:02:08
We’ve had great performance results with Jetty/BlazeDS/Servlet3.0. You can read more at http://flexblog.faratasystems.com/?p=368
Feb 12, 2009 @ 05:02:06
A possible answer to the problem (i.e. secret sauce :). I am working on implementing a comet/server-push based solution, and use Tomcat and Jboss, so am trying to get Tomcat 6 working. I am wondering if after you finish writing the response, do you call writer.flush()? If not, this may be the answer…….on timeout I am guessing the response is automatically flushed, which would explain why the response is not received until timeout.
Thought I would mention one other thing as I find all this comet stuff to be an amazing deja vu…..we implemented servlet-push ten years ago for a http based chat-presence system we built, using Java 1.1 and an applet for the client interface, where we implemented a long-response approach to deliver sent messages and presence updates, and then applet would reconnect. We also implemented nio (yes, with JDK 1.1……we modified src.jar/rt.jar and modified Java base classes to use JNI and native socks/winsock library to implement socket select). It tickles me to read this stuff now as it appears we weren’t so crazy back then, eh?
Feb 12, 2009 @ 15:07:30
Paul,
Yes, i flushed the toilet.
Apr 01, 2009 @ 12:31:31
I have been trying to debug comet problems in Tomcat 6 (6.0.16).
It works OK most of the time but there are race conditions in both APR and NIO connectors which lead to Tomcat closing down sockets when it shouldn’t. If you set up clients and the Tomcat config to use HTTP keepalives you will see that occasionally a client request will fail because of a TCP RST. If you use the APR connector with HTTPS this will also show up as a SEGV in openssl function: ssl3_write_pending.
If you look at the Tomcat code in org.apache.coyote.http11.Http11AprProcessor#proces(), for example, you can see that the SocketState to return can depend on ‘comet’ boolean value. This value can be set from within process() or event() or in the action() callback. It is set in the action() callback from Response.finish() when closing a writer / stream.
The comet flag is not protected by locking but can be set from multiple threads if the response is sent on a separate thread. It is not thread safe. Code in the process() and event() functions appears to assume that the comet flag will not be set by another thread but that is possible. Also, because it is not protected by locking changes made in one thread may not be visible to another thread, or only become visible after an indeterminate interval (esp. on multi-processor machines).
Implementing async API – which is inherently multi-threaded – in a non-threadsafe way is crazy.
This one problem can be worked-around by making Http11AprProcessor methods process(), event() and action() synchronized. This locking protects the ‘comet’ flag. May be similar fix for the NIO connector.
Apr 01, 2009 @ 13:31:22
@Vlad
Try out JBossWeb. Remy has done a lot of work to stabilize this stuff. Would be interesting to know whether or not this is true 🙂
Apr 06, 2009 @ 17:03:33
But Remy is one of the most incompetent programmers around.
the code he produces show very silly beginner mistakes, and a complete lack of understanding of any more advanced concepts, such as basic threadsafety.
he shoud be draged outside and shot in the head.
Apr 06, 2009 @ 19:12:15
@Nisse:
I guess the whole Tomcat community is screwed then. The Tomcat Comet guys don’t seem to know what they are doing or how to fix the problem. And Remy is an idiot 😉 Not my experience with him, but to each is own.
What java comet-style framework is the most mature and robust? | DeveloperQuestion.com
Oct 14, 2010 @ 16:31:23
Oct 29, 2010 @ 06:20:40
I’ve just been trying to use RestEasy + JBoss 5.1 + Tomcat6 APR wrapper. Running into problems when using RestEasy Async stuff via JBossWebDispatcherServlet, as well as JAX-WS separately. In other words, one’s using async, while all the others are using sync servlet handling. When an async response is sent at roughly the same time as a sync response, I keep getting ‘Stream closed’. I’m putting this down to the race conditions described here. Any ideas at a workaround?
Oct 29, 2010 @ 13:33:05
Got a test to reproduce this? also, please move discussions to resteasy-dev list.
Jan 21, 2011 @ 17:09:38
I have had a mixed experience with Tomcat Comet/NIO. It is certainly very buggy.
The bug with “closing down sockets when it shouldn’t” (https://issues.apache.org/bugzilla/show_bug.cgi?id=50044) is actually fixed in Tomcat 7. But somehow they manage to introduce a new bugs very often. The latest version 7.0.6, for example, does not detect client disconnects: https://issues.apache.org/bugzilla/show_bug.cgi?id=50627
One other major problem is that writes block when client buffer reaches 32K. If I have a thread sending updates to multiple clients, it may block if one of the clients is “slow consumer” stopping data flow to multiple clients. There does not seem to be a workaround for it as Tomcat Comet API does not expose the size of this client queue. The only thing I can do is to allocate a thread per client – very ugly.
So my question is: are there any other “production ready” APIs in Java that handle Comet? I am not interested in long poll implementations or beta releases. I am starting to believe that there is nothing available in the Java world. Am I wrong?
Any input is appreciated. Thanks
Jan 21, 2011 @ 19:28:20
You could take a look at our HornetQ-REST interface. You’ll have to tell me if it falls under your “long poll” implementation. It is beta, but going final when HornetQ 2.2 comes out next month.