title: Migrated to Flink 1.10.0, now my job fails and kills my TaskManager
body: <p>I migrated from version 1.9.1 to 1.10.0 and now my program doesn't run. I'm running a simple computing topology represented in the image below.
  <a href="https://i.stack.imgur.com/LxUew.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LxUew.png" alt="Topology"></a></p>
  
  <p>I tried tuning the dependencies in my program and in the flink dist folder but to no avail. I moved the hadoop dependencies to <code>plugins/s3</code> but it seems that Flink's unable to find the jar. The JobManager reports:</p>
  
  <pre><code>2020-02-14 18:52:02,376 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - --------------------------------------------------------------------------------
  2020-02-14 18:52:02,376 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Starting TaskManager (Version: 1.10.0, Rev:aa4eb8f, Date:07.02.2020 @ 19:18:19 CET)
  2020-02-14 18:52:02,376 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  OS current user: ubuntu
  2020-02-14 18:52:02,376 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Current Hadoop/Kerberos user: &lt;no hadoop dependency found&gt;
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JVM: OpenJDK 64-Bit Server VM - Private Build - 1.8/25.242-b08
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Maximum heap size: 4432 MiBytes
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  No Hadoop Dependency available
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JVM Options:
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -XX:+UseG1GC
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Xmx4647288761
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Xms4647288761
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -XX:MaxDirectMemorySize=1090519054
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -XX:MaxMetaspaceSize=100663296
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlog.file=/home/ubuntu/flink-1.10.0/log/flink-ubuntu-taskexecutor-0-ip-10-0-1-68.log
  2020-02-14 18:52:02,377 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlog4j.configuration=file:/home/ubuntu/flink-1.10.0/conf/log4j.properties
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlogback.configurationFile=file:/home/ubuntu/flink-1.10.0/conf/logback.xml
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Program Arguments:
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     --configDir
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     /home/ubuntu/flink-1.10.0/conf
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.framework.off-heap.size=134217728b
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.network.max=956301326b
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.network.min=956301326b
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.framework.heap.size=134217728b
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,378 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.managed.size=3825205305b
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.cpu.cores=8.0
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.task.heap.size=4513071033b
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -D
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     taskmanager.memory.task.off-heap.size=0b
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Classpath: /home/ubuntu/flink-1.10.0/lib/flink-metrics-datadog-1.10.0.jar:/home/ubuntu/flink-1.10.0/lib/flink-metrics-statsd-1.10.0.jar:/home/ubuntu/flink-1.10.0/lib/flink-table-blink_2.11-1.10.0.jar:/home/ubuntu/flink-1.10.0/lib/flink-table_2.11-1.10.0.jar:/home/ubuntu/flink-1.10.0/lib/log4j-1.2.17.jar:/home/ubuntu/flink-1.10.0/lib/slf4j-log4j12-1.7.15.jar:/home/ubuntu/flink-1.10.0/lib/flink-dist_2.11-1.10.0.jar::/home/ubuntu/app/conf:
  2020-02-14 18:52:02,379 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - --------------------------------------------------------------------------------
  </code></pre>
  
  <p>I'm using a StreamingFileSink. I get the following exception after a minute or so of now activity once I start the job:</p>
  
  <pre><code>2020-02-14 20:06:24
  org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
      at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:110)
      at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:76)
      at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
      at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:186)
      at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:180)
      at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:484)
      at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49)
      at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1703)
      at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1252)
      at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1220)
      at org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:955)
      at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.signalPayloadRelease(SingleLogicalSlot.java:173)
      at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.release(SingleLogicalSlot.java:165)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:732)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
      at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseTaskManagerInternal(SlotPoolImpl.java:815)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseTaskManager(SlotPoolImpl.java:774)
      at org.apache.flink.runtime.jobmaster.JobMaster.disconnectTaskManager(JobMaster.java:429)
      at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1147)
      at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:109)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
      at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
      at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
      at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
      at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
      at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
      at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
      at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
      at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
      at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
      at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
      at akka.actor.ActorCell.invoke(ActorCell.scala:561)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
      at akka.dispatch.Mailbox.run(Mailbox.scala:225)
      at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
      at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  Caused by: java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 56a63a7493bc527e7af383dc008a4800 timed out.
      ... 26 more
  </code></pre>