Question: We are facing an unusual problem in our application, in the last one month our application reached an unrecoverable state, It was recovered post application restart.

Background : Our application makes a DB query to fetch some information and this Database is hosted on a separate node.

Problematic case : When the thread dump was analyzed we see all the threads are in runnable state fetching the data from the database, but it didn't finished even after 20 minutes.

Post the application restart as expected all threads recovered. And the CPU usage was also normal.

Below is the thread dump

> 
> ThreadPool:2:47" prio=3 tid=0x0000000007334000 nid=0x5f runnable
>  [0xfffffd7fe9f54000] java.lang.Thread.State: RUNNABLE at
>  oracle.jdbc.driver.T2CStatement.t2cParseExecuteDescribe(Native Method)
>  at
>  oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:518)
>  at
>  oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:764)
>  at ora
> 
> 
> 

```
All threads in the same state.

```

Questions: 

1. what could be the reason for this state?
2. how to recover under this case ?

Answer: 
As others mentioned already, that native methods are always in runnable, as the JVM doesn't know/care about them.

The Oracle drivers on the client side have no socket timeout by default. This means if you have network issues, the client's low level socket may "stuck" there for ever, resulting in a maxxed out connection pool. You could also check the network trafic towards the Oracle server to see if it even transmits data or not.

When using the thin client, you can set `oracle.jdbc.ReadTimeout`, but I don't know how to do that for the thick (oci) client you use, I'm not familiar with it.

What to do? Research how can you specify read timeout for the thick ojdbc driver, and watch for exceptions related to the connection timeout, that will clearly signal network issues. If you can change the source, you can wrap the calls and retry the session when you catch timeout-related SQLExceptions.

To quickly address the issue, terminate the connection on the Oracle server manually.

Worth checking the session contention, maybe a query blocks these sessions. If you find one, you'll see which database object causes the problem.
Is the system or JVM getting hanged? 
If configurable and if possible, reduce the number of threads/ parallel connections.

The thread simply waste CPU cycles when waiting for IO. 
Yes your CPU is unfortunately kept busy by the threads who are awaiting a response from DB.