<p dir="auto">An indexing task might fail due to various reasons, and one of them is node failures. If an indexing task fails due to node failures, it might be completed by reissuing the failed task on another node. In this case, it would be better if the overlord can resubmit failed tasks automatically without manual handling.</p>
<p dir="auto">I'm thinking to introduce a new taskContext, called <code class="notranslate">retryCount</code>, to specify how many times each task should be retried on failures.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="114372588" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1896" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/1896/hovercard" href="https://github.com/apache/druid/pull/1896">#1896</a> introduced an addition of <a class="commit-link" href="https://github.com/apache/druid/commit/e4e5f0375b538ac7e2a2f7c36d00bd861e584c44#diff-e677e1ba7e3cf3b5b97660cfc17749beR144"><tt>e4e5f03</tt>#diff-e677e1ba7e3cf3b5b97660cfc17749beR144</a> which causes errors on duplicate primary key entries to be retried way more than they should.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-04-05T20:52:57,269 WARN [qtp726762476-61] com.metamx.common.RetryUtils - Failed on try 1, retrying in 1,976ms.
org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'index_realtime_REDACTED_2016-04-05T20:00:0' for key 'PRIMARY' [statement:&quot;INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (:id, :created_date, :datasource, :payload, :active, :status_payload)&quot;, located:&quot;INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (:id, :created_date, :datasource, :payload, :active, :status_payload)&quot;, rewritten:&quot;INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (?, ?, ?, ?, ?, ?)&quot;, arguments:{ positional:{}, named:{payload:[
...
        at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataConnector$2.call(SQLMetadataConnector.java:110) ~[druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:115) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataStorageActionHandler.insert(SQLMetadataStorageActionHandler.java:97) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.MetadataTaskStorage.insert(MetadataTaskStorage.java:134) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.TaskQueue.add(TaskQueue.java:321) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:124) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:119) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:518) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:116) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_60]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_60]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
..."><pre class="notranslate"><code class="notranslate">2016-04-05T20:52:57,269 WARN [qtp726762476-61] com.metamx.common.RetryUtils - Failed on try 1, retrying in 1,976ms.
org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'index_realtime_REDACTED_2016-04-05T20:00:0' for key 'PRIMARY' [statement:"INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (:id, :created_date, :datasource, :payload, :active, :status_payload)", located:"INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (:id, :created_date, :datasource, :payload, :active, :status_payload)", rewritten:"INSERT INTO REDACTED (id, created_date, datasource, payload, active, status_payload) VALUES (?, ?, ?, ?, ?, ?)", arguments:{ positional:{}, named:{payload:[
...
        at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataConnector$2.call(SQLMetadataConnector.java:110) ~[druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:115) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.metadata.SQLMetadataStorageActionHandler.insert(SQLMetadataStorageActionHandler.java:97) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.MetadataTaskStorage.insert(MetadataTaskStorage.java:134) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.TaskQueue.add(TaskQueue.java:321) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:124) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:119) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:518) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:116) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_60]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_60]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) [druid-selfcontained-0.9.0-rc3-mmx2.jar:0.9.0-rc3-mmx2]
...
</code></pre></div>
<p dir="auto">This retry holds onto the big lock in TaskQueue while it exhausts its attempts, causing a big backlog of things wanting to use the big lock in TaskQueue</p>