title: Unable to execute HTTP request: Timeout waiting for connection from pool in Flink
body: <p>I'm working on an <strong>app which uploads some files to an s3 bucket</strong> and at a later point, it <strong>reads files from s3 bucket and pushes it to my database</strong>. </p>
  
  <p>I'm using <strong>Flink 1.4.2</strong> and <strong>fs.s3a API</strong> for reading and write files from the s3 bucket.</p>
  
  <p><strong>Uploading files to s3 bucket works fine without any problem but when the second phase of my app that is reading those uploaded files from s3 starts, my app is throwing following error</strong>:</p>
  
  <pre><code>Caused by: java.io.InterruptedIOException: Reopen at position 0 on s3a://myfilepath/a/b/d/4: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
  at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:125)
  at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:155)
  at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
  at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
  at java.io.DataInputStream.read(DataInputStream.java:149)
  at org.apache.flink.fs.s3hadoop.shaded.org.apache.flink.runtime.fs.hdfs.HadoopDataInputStream.read(HadoopDataInputStream.java:94)
  at org.apache.flink.api.common.io.DelimitedInputFormat.fillBuffer(DelimitedInputFormat.java:702)
  at org.apache.flink.api.common.io.DelimitedInputFormat.open(DelimitedInputFormat.java:490)
  at org.apache.flink.api.common.io.GenericCsvInputFormat.open(GenericCsvInputFormat.java:301)
  at org.apache.flink.api.java.io.CsvInputFormat.open(CsvInputFormat.java:53)
  at org.apache.flink.api.java.io.PojoCsvInputFormat.open(PojoCsvInputFormat.java:160)
  at org.apache.flink.api.java.io.PojoCsvInputFormat.open(PojoCsvInputFormat.java:37)
  at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:145)
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
  at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>I was <strong>able to control this error by increasing the max connection</strong> parameter for s3a API.</p>
  
  <p>As of now, I have around <strong>1000 files in the s3 bucket</strong> which is <strong>pushed and pulled by my app</strong> in the s3 bucket and <strong>my max connection is 3000</strong>. I'm using Flink's parallelism to upload/download these files from s3 bucket. My <strong>task manager count is 14</strong>. 
  This is an <strong>intermittent failure</strong>, I'm having success cases also for this scenario.</p>
  
  <p>My query is, </p>
  
  <ol>
  <li>Why I'm getting an intermittent failure? If the max connection I set was low, then my app should be throwing this error every time I run.</li>
  <li>Is there any way to calculate the optimal number of max connection required for my app to work without facing the connection pool timeout error? Or Is this error related to something else that I'm not aware of?</li>
  </ol>
  
  <p>Thanks 
  In Advance</p>