<p dir="auto">Though we can specify query granularity and intervals in search query spec.,<br>
search query runner ignore those and scans whole the interval of the given segment and works with <code class="notranslate">QueryGranularity.ALL</code>.</p>
<p dir="auto">For example, in <code class="notranslate">testSearchWithCardinality()</code> of <code class="notranslate">SearchQueryRunnerTest</code>,<br>
it requests search for interval [2011-01-12, 2011-02-28) and [2011-03-01, 2011-04-15) within a segment of interval [2011-01-12, 2011-04-15] and unions the results.<br>
But, it actually scans whole the segment for each interval and results in duplicated counting of matching entries.</p>
<p dir="auto">As a result, you can see that the result count from <code class="notranslate">testSearch()</code> is much smaller than that of <code class="notranslate">testSearchWithCardinality()</code> even though <code class="notranslate">testSearch()</code> has larger query interval.</p>
<p dir="auto">This issue tracks all the "related" efforts targeted towards making realtime ingestion better in one way or the other.<br>
Here is a wishlist of items we should try to solve for. I understand that some of this is already solved by tranquility (probably not in kafka based ingestion), but we should ensure that those are not broken as we make changes and are supported by kafka based ingestion as well.</p>
<ol dir="auto">
<li>Exactly once semantics: There should be no duplication introduced and no dropping of events (except when input event is malformed/unparseable). This directly relates to both "No window period" and "FirehoseV2 proposal"</li>
<li>We should be able to scale realtime ingestion by adding more nodes.</li>
<li>Realtime Ingestion and Querying on that data should continue to happen successfully in the event of some process/node failures. It only happens for standalone realtime nodes today. With ingestion done via "tasks", there is possibility of node dying leading to data loss (replication of ingestion tasks should potentially solve this).</li>
<li>Query result consistency: Given query should get exactly same result independent of which realtime nodes/replicants served the query.</li>
<li>No downtime upgrade: Realtime ingestion and Queries should continue to make progress while upgrade.</li>
<li>Operational simplicity: For example kafka ingestion should be able to automatically handle kafka partition addition/removal</li>
</ol>
<p dir="auto">(Realtime Delta Ingestion: ability to ingest late events as they come would probably happen as a side effect of 1st)</p>
<p dir="auto"><strong>Related Refs:</strong><br>
<a href="https://groups.google.com/forum/#!msg/druid-development/kHgHTgqKFlQ/fXvtsNxWzlMJ" rel="nofollow">https://groups.google.com/forum/#!msg/druid-development/kHgHTgqKFlQ/fXvtsNxWzlMJ</a> (No window period proposal)<br>
<a href="https://groups.google.com/forum/#!msg/druid-development/9HB9hCcqvuI/L59RgsloZfoJ" rel="nofollow">https://groups.google.com/forum/#!msg/druid-development/9HB9hCcqvuI/L59RgsloZfoJ</a> (FirehoseV2 proposal)<br>
<a href="https://docs.google.com/document/d/1PUG3crI2jiPa_u926R0KrkZVM7t706rXp1IuUxVXB5E/edit?usp=sharing" rel="nofollow">https://docs.google.com/document/d/1PUG3crI2jiPa_u926R0KrkZVM7t706rXp1IuUxVXB5E/edit?usp=sharing</a> (doc covering design details for both above)<br>
<a href="https://groups.google.com/forum/#!searchin/druid-development/tier/druid-development/1I3CmxlOipM/e3-SpWqG170J" rel="nofollow">https://groups.google.com/forum/#!searchin/druid-development/tier/druid-development/1I3CmxlOipM/e3-SpWqG170J</a> (Task Tiering proposal)</p>
<p dir="auto"><strong>Related PRs:</strong><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="99725073" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1609" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/1609/hovercard" href="https://github.com/apache/druid/pull/1609">#1609</a> (kafka simple consumer based firehose and initial FirehoseV2 updates)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="101709878" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1639" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/1639/hovercard" href="https://github.com/apache/druid/pull/1639">#1639</a> (new plumber)</p>
<p dir="auto"><strong>Related Issues:</strong><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="27918978" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/401" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/401/hovercard" href="https://github.com/apache/druid/issues/401">#401</a> (log management for long-running tasks)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="94845926" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1513" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/1513/hovercard" href="https://github.com/apache/druid/issues/1513">#1513</a> (preemption for indexing service locks)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="94845936" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1514" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/1514/hovercard" href="https://github.com/apache/druid/issues/1514">#1514</a> (aggregatorFactories in segment metadata)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="94845941" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1515" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/1515/hovercard" href="https://github.com/apache/druid/issues/1515">#1515</a> (AllocateSegmentAction)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="94845943" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1516" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/1516/hovercard" href="https://github.com/apache/druid/issues/1516">#1516</a> (ElasticShardSpec)<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="94845965" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1517" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/1517/hovercard" href="https://github.com/apache/druid/issues/1517">#1517</a> (user-friendly Hadoop-based re-indexing/compaction)</p>