we have three servers running resin as web service, I send request from http_load and then all three servers run out of thread the same time.
Take a look at your adentry.java line 62.
From the thread dump, there are a large number of threads writing data to something like System.out or System.err.
yes, there is System.out writing something to file, and at that time we run out of harddisk, is this the reason?
In a production system, System.out should be very limited. Normally, you would want to use java.util.logging for that kind of information and use the "fine" level, so it's not normally getting logged.
I'll check the logging routines to see if they should handle out-of-disk issues better, but this doesn't directly look like a Resin bug.
I'm changing the flushing to reduce contention, which should help this case a bit, but the main change should be to reduce the System.out logging and use a finer java.util.logging level instead.