<p dir="auto">Hi,</p>
<p dir="auto">since a few days our Historical Nodes on AWS have temporarily problems connecting to S3. The Data are still written in S3, but from time to time follwoing error occurs (I replaced sensitive Information with &lt;&gt;):</p>
<p dir="auto"><strong>ERROR-Message:</strong></p>
<blockquote>
<p dir="auto">Failed on try 1, retrying in 739ms.<br>
org.jets3t.service.ServiceException: Request Error: &lt;BUCKETNAME&gt;.s3.eu-central-1.amazonaws.com: Name or service not known<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:625) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[jets3t-0.9.4.jar:0.9.4]<br>
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[jets3t-0.9.4.jar:0.9.4]<br>
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]<br>
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]<br>
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]<br>
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) [java-util-0.12.3.jar:0.12.3]<br>
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) [java-util-0.12.3.jar:0.12.3]<br>
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) [druid-s3-extensions-0.12.3.jar:0.12.3]<br>
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) [druid-s3-extensions-0.12.3.jar:0.12.3]<br>
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) [druid-s3-extensions-0.12.3.jar:0.12.3]<br>
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) [druid-s3-extensions-0.12.3.jar:0.12.3]<br>
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.12.3.jar:0.12.3]<br>
at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.12.3.jar:0.12.3]<br>
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]<br>
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]<br>
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]<br>
at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]<br>
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]<br>
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]<br>
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]<br>
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]<br>
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]<br>
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]<br>
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]<br>
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]<br>
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]<br>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]<br>
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]<br>
Caused by: java.net.UnknownHostException: &lt;BUCKETNAME&gt;.s3.eu-central-1.amazonaws.com: Name or service not known<br>
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_181]<br>
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_181]<br>
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_181]<br>
at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_181]<br>
at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_181]<br>
at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_181]<br>
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.1.jar:4.5.1]<br>
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:328) ~[jets3t-0.9.4.jar:0.9.4]<br>
... 41 more</p>
</blockquote>
<p dir="auto">We are using currently Druid 0.12.3 with following configuration:</p>
<p dir="auto"><strong>common.runtime.properties:</strong></p>
<blockquote>
<p dir="auto"># Extensions<br>
druid.extensions.loadList=["druid-kafka-indexing-service", "druid-histogram", "druid-datasketches", <br>
    "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-s3-extensions", "druid-avro-extensions", <br>
    "graphite-emitter"]</p>
<p dir="auto"># Zookeeper<br>
druid.zk.service.host=&lt;ZK_HOST&gt;<br>
druid.zk.paths.base=/druid</p>
<p dir="auto"># Metadata storage<br>
druid.metadata.storage.type=postgresql<br>
druid.metadata.storage.connector.connectURI=jdbc:postgresql://&lt;POSTGRES_URL&gt;:5432/druid<br>
druid.metadata.storage.connector.user=&lt;USER&gt;<br>
druid.metadata.storage.connector.password=&lt;PASSWORD&gt;</p>
<p dir="auto"># Deep storage<br>
druid.storage.type=s3<br>
druid.storage.bucket=&lt;BUCKET&gt;<br>
druid.storage.baseKey=segments<br>
druid.s3.accessKey=&lt;ACCESS_KEY&gt;<br>
druid.s3.secretKey=&lt;SECRET_KEY&gt;</p>
<p dir="auto"># Logging<br>
druid.startup.logging.logProperties=true<br>
druid.indexer.logs.type=noop</p>
<p dir="auto"># Service discovery<br>
druid.selectors.indexing.serviceName=druid/overlord<br>
druid.selectors.coordinator.serviceName=druid/coordinator</p>
<p dir="auto"># Monitoring<br>
druid.monitoring.monitors=["io.druid.java.util.metrics.JvmMonitor"]<br>
druid.emitter=graphite<br>
druid.emitter.logging.logLevel=info<br>
druid.emitter.graphite.hostname=&lt;GRAPHITE_HOST&gt;<br>
druid.emitter.graphite.port=9109<br>
druid.emitter.graphite.eventConverter={"type":"all", "namespacePrefix": "druid"}<br>
druid.emitter.graphite.protocol=plaintext</p>
<p dir="auto"># Caching<br>
druid.cache.type=caffeine<br>
druid.cache.sizeInBytes=1073741824</p>
<p dir="auto"># Storage type of double columns<br>
druid.indexing.doubleStorage=double</p>
<p dir="auto"># Misc<br>
druid.javascript.enabled=true<br>
druid.sql.enable=true</p>
<p dir="auto"># Maximum Amount of Heap space to use for the string dictionary during merging (broker, historical, middlemanager)<br>
druid.query.groupBy.maxMergingDictionarySize = 250000000<br>
druid.query.groupBy.maxOnDiskStorage = 4294967296</p>
</blockquote>
<p dir="auto"><strong>jets3t.properties:</strong></p>
<blockquote>
<p dir="auto">s3service.s3-endpoint=s3.eu-central-1.amazonaws.com<br>
storage-service.request-signature-version=AWS4-HMAC-SHA256</p>
</blockquote>
<p dir="auto">Does anybody else experience this behavior? I gladly provide more information/configuration to solve this issue.</p>
<p dir="auto">Thanks,<br>
Stephan</p>
<p dir="auto">We have seen problem scenarios like the following:</p>
<ol dir="auto">
<li>A user has load rules like: load 30 days, drop forever.</li>
<li>The user loads some data from slightly in the future (maybe some clocks are running a bit fast or slow) using streaming ingestion. This creates a segment with an interval that is in the future.</li>
<li>The coordinator disables the segment immediately upon noticing it (since it is not within the last 30 days).</li>
<li>The Kafka tasks time out during handoff (because the segments are never loaded).</li>
<li>And after that timeout, the data that was slightly in the future is <em>still</em> not available!</li>
</ol>
<p dir="auto">It would be better, I think, if period-based load rules included an unbounded time into the future too. I bet this lines up better with users' intuitive idea of what "retain 30 days" means. Of course it could be changeable with some optional parameters (or a different kind of rule?) but I think the default behavior should be the intuitive one.</p>
<p dir="auto">NB: A workaround to this is to use rules like "load 30 days" followed by "drop 1000 years" followed by "load forever". That has the same effect as what I am suggesting for the new default behavior: we'll drop anything older than 30 days, but retain the last 30 days and also anything that is in the future.</p>
<p dir="auto">Related, but different: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="331028349" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/5868" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/5868/hovercard" href="https://github.com/apache/druid/issues/5868">#5868</a>.</p>