<p dir="auto">Apache Druid 0.21.0 contains around 120 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 36 contributors. Refer to the complete list of changes and everything tagged to the milestone for further details.</p>
<h1 dir="auto"><a name="user-content-21-new-features" href="#21-new-features">#</a> New features</h1>
<h2 dir="auto"><a name="user-content-21-operation" href="#21-operation">#</a> Operation</h2>
<h3 dir="auto"><a name="user-content-21-k8s-extension" href="#21-k8s-extension">#</a> Service discovery and leader election based on Kubernetes</h3>
<p dir="auto">The new Kubernetes extension supports service discovery and leader election based on Kubernetes. This extension works in conjunction with the HTTP-based server view (<code class="notranslate">druid.serverview.type=http</code>) and task management (<code class="notranslate">druid.indexer.runner.type=httpRemote</code>) to allow you to run a Druid cluster <em>with zero ZooKeeper dependencies</em>. This extension is still <strong>experimental</strong>. See <a href="https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html" rel="nofollow">Kubernetes extension</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="733350445" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10544" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10544/hovercard" href="https://github.com/apache/druid/pull/10544">#10544</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="579718925" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/9507" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/9507/hovercard" href="https://github.com/apache/druid/pull/9507">#9507</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="730754190" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10537" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10537/hovercard" href="https://github.com/apache/druid/pull/10537">#10537</a></p>
<h3 dir="auto"><a name="user-content-21-segment-count-limit-for-balancing" href="#21-segment-count-limit-for-balancing">#</a> New dynamic coordinator configuration to limit the number of segments when finding a candidate segment for segment balancing</h3>
<p dir="auto">You can set  the <code class="notranslate">percentOfSegmentsToConsiderPerMove</code> to limit the number of segments considered when picking a candidate segment to move. The candidates are searched up to <code class="notranslate">maxSegmentsToMove * 2</code> times. This new configuration prevents Druid from iterating through all available segments to speed up the segment balancing process, especially if you have lots of available segments in your cluster. See <a href="https://druid.apache.org/docs/latest/configuration/index.html#dynamic-configuration" rel="nofollow">Coordinator dynamic configuration</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="679301098" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10284" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10284/hovercard" href="https://github.com/apache/druid/pull/10284">#10284</a></p>
<h3 dir="auto"><a name="user-content-21-indexer-endpoints" href="#21-indexer-endpoints">#</a> <code class="notranslate">status</code> and <code class="notranslate">selfDiscovered</code> endpoints for Indexers</h3>
<p dir="auto">The Indexer now supports <code class="notranslate">status</code> and <code class="notranslate">selfDiscovered</code> endpoints. See <a href="https://druid.apache.org/docs/latest/operations/api-reference.html#process-information" rel="nofollow">Processor information APIs</a> for details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="766730261" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10679" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10679/hovercard" href="https://github.com/apache/druid/pull/10679">#10679</a></p>
<h2 dir="auto"><a name="user-content-21-querying" href="#21-querying">#</a> Querying</h2>
<h3 dir="auto"><a name="user-content-21-grouping-function" href="#21-grouping-function">#</a> New <code class="notranslate">grouping</code> aggregator function</h3>
<p dir="auto">You can use the new <code class="notranslate">grouping</code> aggregator SQL function with <code class="notranslate">GROUPING SETS</code> or <code class="notranslate">CUBE</code> to indicate which grouping dimensions are included in the current grouping set. See <a href="https://druid.apache.org/docs/latest/querying/sql.html#aggregation-functions" rel="nofollow">Aggregation functions</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="724533044" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10518" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10518/hovercard" href="https://github.com/apache/druid/pull/10518">#10518</a></p>
<h3 dir="auto"><a name="user-content-21-expr-type-handling" href="#21-expr-type-handling">#</a> Improved missing argument handling in expressions and functions</h3>
<p dir="auto">Expression processing now can be vectorized when inputs are missing. For example a non-existent column. When an argument is missing in an expression, Druid can now infer the proper type of result based on non-null arguments. For instance, for <code class="notranslate">longColumn + nonExistentColumn</code>, <code class="notranslate">nonExistentColumn</code> is treated as <code class="notranslate">(long) 0</code> instead of <code class="notranslate">(double) 0.0</code>. Finally, in default null handling mode, math functions can produce output properly by treating missing arguments as zeros.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="718393720" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10499" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10499/hovercard" href="https://github.com/apache/druid/pull/10499">#10499</a></p>
<h3 dir="auto"><a name="user-content-21-timestampadd" href="#21-timestampadd">#</a>  Allow zero period for <code class="notranslate">TIMESTAMPADD</code></h3>
<p dir="auto"><code class="notranslate">TIMESTAMPADD</code> function now allows zero period. This functionality is required for some BI tools such as Tableau.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="735045372" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10550" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10550/hovercard" href="https://github.com/apache/druid/pull/10550">#10550</a></p>
<h2 dir="auto"><a name="user-content-21-ingestion" href="#21-ingestion">#</a> Ingestion</h2>
<h3 dir="auto"><a name="user-content-21-parallel-ingestion-without-intervals" href="#21-parallel-ingestion-without-intervals">#</a> Native parallel ingestion no longer requires explicit intervals</h3>
<p dir="auto">Parallel task no longer requires you to set explicit intervals in <code class="notranslate">granularitySpec</code>. If intervals are missing, the parallel task executes an extra step for input sampling which collects the intervals to index.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="745272114" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10592" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10592/hovercard" href="https://github.com/apache/druid/pull/10592">#10592</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="758058186" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10647" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10647/hovercard" href="https://github.com/apache/druid/pull/10647">#10647</a></p>
<h3 dir="auto"><a name="user-content-21-old-kafka-support" href="#21-old-kafka-support">#</a> Old Kafka version support</h3>
<p dir="auto">Druid now supports Apache Kafka older than 0.11. To read from an old version of Kafka, set the <code class="notranslate">isolation.level</code> to <code class="notranslate">read_uncommitted</code> in <code class="notranslate">consumerProperties</code>. Only 0.10.2.1 have been tested up until this release. See <a href="https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#kafkasupervisorioconfig" rel="nofollow">Kafka supervisor configurations</a> for details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="735167823" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10551" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10551/hovercard" href="https://github.com/apache/druid/pull/10551">#10551</a></p>
<h3 dir="auto">Multi-phase segment merge for native batch ingestion</h3>
<p dir="auto">A new tuningConfig, <code class="notranslate">maxColumnsToMerge</code>, controls how many segments can be merged at the same time in the task. This configuration can be useful to avoid high memory pressure during the merge. See <a href="https://druid.apache.org/docs/latest/ingestion/native-batch.html#tuningconfig" rel="nofollow">tuningConfig for native batch ingestion</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="770353384" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10689" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10689/hovercard" href="https://github.com/apache/druid/pull/10689">#10689</a></p>
<h3 dir="auto"><a name="user-content-21-compaction" href="#21-compaction">#</a> Native re-ingestion is less memory intensive</h3>
<p dir="auto">Parallel tasks now sort segments by ID before assigning them to subtasks. This sorting minimizes the number of time chunks for each subtask to handle. As a result, each subtask is expected to use less memory, especially when a single Parallel task is issued to re-ingest segments covering a long time period.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="757790268" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10646" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10646/hovercard" href="https://github.com/apache/druid/pull/10646">#10646</a></p>
<h2 dir="auto"><a name="user-content-21-web-console" href="#21-web-console">#</a> Web console</h2>
<h3 dir="auto"><a name="user-content-21-refresh-console-style" href="#21-refresh-console-style">#</a> Updated and improved web console styles</h3>
<p dir="auto">The new web console styles make better use of the Druid brand colors and standardize paddings and margins throughout. The icon and background colors are now derived from the Druid logo.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/96206449-c3e58800-0f1d-11eb-977a-8fbdbe4dce77.png"><img src="https://user-images.githubusercontent.com/177816/96206449-c3e58800-0f1d-11eb-977a-8fbdbe4dce77.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="722839948" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10515" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10515/hovercard" href="https://github.com/apache/druid/pull/10515">#10515</a></p>
<h3 dir="auto"><a name="user-content-21-web-console-partitioning" href="#21-web-console-partitioning">#</a> Partitioning information is available in the web console</h3>
<p dir="auto">The web console now shows datasource partitioning information on the new <code class="notranslate">Segment granularity</code> and <code class="notranslate">Partitioning</code> columns.</p>
<h5 dir="auto"><code class="notranslate">Segment granularity</code> column in the <code class="notranslate">Datasources</code> tab</h5>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/2322288/104550926-b7d79800-55ea-11eb-824e-58a71b506139.png"><img src="https://user-images.githubusercontent.com/2322288/104550926-b7d79800-55ea-11eb-824e-58a71b506139.png" alt="97240667-1b9cb280-17ac-11eb-9c55-e312c24cd8fc" style="max-width: 100%;"></a></p>
<h5 dir="auto"><code class="notranslate">Partitioning</code> column in the <code class="notranslate">Segments</code> tab</h5>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/2322288/104550932-bc03b580-55ea-11eb-9196-c3f4c8203ede.png"><img src="https://user-images.githubusercontent.com/2322288/104550932-bc03b580-55ea-11eb-9196-c3f4c8203ede.png" alt="97240597-ebedaa80-17ab-11eb-976f-a0d49d6d1a40" style="max-width: 100%;"></a></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="730062911" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10533" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10533/hovercard" href="https://github.com/apache/druid/pull/10533">#10533</a></p>
<h3 dir="auto"><a name="user-content-21-web-console-dim-ordering" href="#21-web-console-dim-ordering">#</a> The column order in the <code class="notranslate">Schema</code> table matches the <code class="notranslate">dimensionsSpec</code></h3>
<p dir="auto">The <code class="notranslate">Schema</code> table now reflects the dimension ordering in the <code class="notranslate">dimensionsSpec</code>.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/2322288/104551169-35030d00-55eb-11eb-99ca-6aad43986b8b.png"><img src="https://user-images.githubusercontent.com/2322288/104551169-35030d00-55eb-11eb-99ca-6aad43986b8b.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="744252987" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10588" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10588/hovercard" href="https://github.com/apache/druid/pull/10588">#10588</a></p>
<h3 dir="auto"><a name="user-content-21-metrics" href="#21-metrics">#</a> Metrics</h3>
<h3 dir="auto"><a name="user-content-21-coordinator-duty-metrics" href="#21-coordinator-duty-metrics">#</a> Coordinator duty runtime metrics</h3>
<p dir="auto">The coordinator performs several 'duty' tasks. For example segment balancing, loading new segments, etc. Now there are two new metrics to help you analyze how fast the Coordinator is executing these duties.</p>
<ul dir="auto">
<li><code class="notranslate">coordinator/time</code>: the time for an individual duty to execute</li>
<li><code class="notranslate">coordinator/global/time</code>: the time for the whole duties runnable to execute</li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="750116085" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10603" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10603/hovercard" href="https://github.com/apache/druid/pull/10603">#10603</a></p>
<h3 dir="auto"><a name="user-content-21-query-timeout-metric" href="#21-query-timeout-metric">#</a> Query timeout metric</h3>
<p dir="auto">A new metric provides the number of timed out queries. Previously timed out queries were treated as interrupted and included in the <code class="notranslate">query/interrupted/count</code> (see <a href="#21-http-error-codes">Changed HTTP status codes for query errors</a> for more details).</p>
<p dir="auto"><code class="notranslate">query/timeout/count</code>: the number of timed out queries during the emission period</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="739280480" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10567" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10567/hovercard" href="https://github.com/apache/druid/pull/10567">#10567</a></p>
<h3 dir="auto"><a name="user-content-21-shuffle-metrics" href="#21-shuffle-metrics">#</a> Shuffle metrics for batch ingestion</h3>
<p dir="auto">Two new metrics provide shuffle statistics for MiddleManagers and Indexers. These metrics have the <code class="notranslate">supervisorTaskId</code> as their dimension.</p>
<ul dir="auto">
<li><code class="notranslate">ingest/shuffle/bytes</code>: number of bytes shuffled per emission period</li>
<li><code class="notranslate">ingest/shuffle/requests</code>: number of shuffle requests per emission period</li>
</ul>
<p dir="auto">To enable the shuffle metrics, add <code class="notranslate">org.apache.druid.indexing.worker.shuffle.ShuffleMonitor</code> in <code class="notranslate">druid.monitoring.monitors</code>. See <a href="https://druid.apache.org/docs/latest/operations/metrics.md" rel="nofollow">Shuffle metrics</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="693811103" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10359" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10359/hovercard" href="https://github.com/apache/druid/pull/10359">#10359</a></p>
<h3 dir="auto"><a name="user-content-21-cron-scheduler" href="#21-cron-scheduler">#</a> New clock-drift safe metrics monitor scheduler</h3>
<p dir="auto">The default metrics monitor scheduler is implemented based on <code class="notranslate">ScheduledThreadPoolExecutor</code> which is prone to unbounded <a href="https://en.wikipedia.org/wiki/Clock_drift" rel="nofollow">clock drift</a>. A new monitor scheduler, <code class="notranslate">ClockDriftSafeMonitorScheduler</code>, overcomes this limitation. To use the new scheduler, set <code class="notranslate">druid.monitoring.schedulerClassName</code> to <code class="notranslate">org.apache.druid.java.util.metrics.ClockDriftSafeMonitorScheduler</code> in the runtime.properties file.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="711000773" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10448" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10448/hovercard" href="https://github.com/apache/druid/pull/10448">#10448</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="780978092" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10732" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10732/hovercard" href="https://github.com/apache/druid/pull/10732">#10732</a></p>
<h3 dir="auto"><a name="user-content-21-others" href="#21-others">#</a> Others</h3>
<h3 dir="auto"><a name="user-content-21-rds-password-provider" href="#21-rds-password-provider">#</a> New extension for a password provider based on AWS RDS token</h3>
<p dir="auto">A new <code class="notranslate">PasswordProvider</code> type allows access to AWS RDS DB instances using temporary AWS tokens. This extension can be useful when an RDS is used as Druid's metadata store. See <a href="https://druid.apache.org/docs/latest/development/extensions-core/druid-aws-rds.html" rel="nofollow">AWS RDS extension</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="581066263" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/9518" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/9518/hovercard" href="https://github.com/apache/druid/pull/9518">#9518</a></p>
<h3 dir="auto"><a name="user-content-21-system-table-leader" href="#21-system-table-leader">#</a> The <code class="notranslate">sys.servers</code> table shows leaders</h3>
<p dir="auto">A new long-typed column <code class="notranslate">is_leader</code> in the <code class="notranslate">sys.servers</code> table indicates whether or not the server is the leader.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="767209636" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10680" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10680/hovercard" href="https://github.com/apache/druid/pull/10680">#10680</a></p>
<h3 dir="auto"><a name="user-content-21-https-influxdb" href="#21-https-influxdb">#</a> <code class="notranslate">druid-influxdb-emitter</code> extension supports the HTTPS protocol</h3>
<p dir="auto">See <a href="https://druid.apache.org/docs/latest/development/extensions-contrib/influxdb-emitter.html" rel="nofollow">Influxdb emitter extension</a> for new configurations.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="625805019" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/9938" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/9938/hovercard" href="https://github.com/apache/druid/pull/9938">#9938</a></p>
<h2 dir="auto"><a name="user-content-21-docker" href="#21-docker">#</a> Docker</h2>
<h3 dir="auto"><a name="user-content-21-small-docker" href="#21-small-docker">#</a> Small docker image</h3>
<p dir="auto">The docker image size is reduced by half by eliminating unnecessary duplication.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="719413192" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10506" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10506/hovercard" href="https://github.com/apache/druid/pull/10506">#10506</a></p>
<h2 dir="auto"><a name="user-content-21-development" href="#21-development">#</a> Development</h2>
<h3 dir="auto"><a name="user-content-21-dynamic-config-provider" href="#21-dynamic-config-provider">#</a> Extensible Kafka consumer properties via a new <code class="notranslate">DynamicConfigProvider</code></h3>
<p dir="auto">A new class <code class="notranslate">DynamicConfigProvider</code> enables fetching consumer properties at runtime. For instance, you can use <code class="notranslate">DynamicConfigProvider</code> fetch <code class="notranslate">bootstrap.servers</code> from location such as a local environment variable if it is not static. Currently, only a map-based config provider is supported by default. See <a href="https://druid.apache.org/docs/latest/operations/dynamic-config-provider.md" rel="nofollow">DynamicConfigProvider</a> for how to implement a custom config provider.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="683915395" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10309" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10309/hovercard" href="https://github.com/apache/druid/pull/10309">#10309</a></p>
<h1 dir="auto"><a name="user-content-21-bug-fixes" href="#21-bug-fixes">#</a> Bug fixes</h1>
<p dir="auto">Druid 0.21.0 contains 30 bug fixes, you can see <a href="https://github.com/apache/druid/pulls?q=is%3Apr+milestone%3A0.21.0+is%3Aclosed+label%3ABug+">the complete list here</a>.</p>
<h3 dir="auto"><a name="user-content-21-post-agg-subtotal" href="#21-post-agg-subtotal">#</a> Post-aggregator computation with subtotals</h3>
<p dir="auto">Before 0.21.0, the query fails with an error when you use post aggregators with sub-totals. Now this bug is fixed and you can use post aggregators with subtotals.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="759392455" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10653" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10653/hovercard" href="https://github.com/apache/druid/pull/10653">#10653</a></p>
<h3 dir="auto"><a name="user-content-21-indexer-announce" href="#21-indexer-announce">#</a> Indexers announce themselves as segment servers</h3>
<p dir="auto">In 0.19.0 and 0.20.0, Indexers could not process queries against streaming data as they did not announce themselves as segment servers. They are fixed to announce themselves properly in 0.21.0.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="756800514" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10631" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10631/hovercard" href="https://github.com/apache/druid/pull/10631">#10631</a></p>
<h3 dir="auto"><a name="user-content-21-historical-segment-loading" href="#21-historical-segment-loading">#</a> Validity check for segment files in historicals</h3>
<p dir="auto">Historicals now perform validity check after they download segment files and re-download automatically if those files are crashed.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="758349838" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10650" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10650/hovercard" href="https://github.com/apache/druid/pull/10650">#10650</a></p>
<h3 dir="auto"><a name="user-content-21-storage-location-selector" href="#21-storage-location-selector">#</a>  <code class="notranslate">StorageLocationSelectorStrategy</code> injection failure is fixed</h3>
<p dir="auto">The injection failure while reading the configurations of <code class="notranslate">StorageLocationSelectorStrategy</code> is fixed.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="695007257" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10363" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10363/hovercard" href="https://github.com/apache/druid/pull/10363">#10363</a></p>
<h1 dir="auto"><a name="user-content-21-upgrading-to-0.21.0" href="#21-upgrading-to-0.21.0">#</a> Upgrading to 0.21.0</h1>
<p dir="auto">Consider the following changes and updates when upgrading from Druid 0.20.0 to 0.21.0. If you're updating from an earlier version than 0.20.0, see the release notes of the relevant intermediate versions.</p>
<h3 dir="auto"><a name="user-content-21-http-error-codes" href="#21-http-error-codes">#</a> Improved HTTP status codes for query errors</h3>
<p dir="auto">Before this release, Druid returned the "internal error (500)" for most of the query errors. Now Druid returns different error codes based on their cause. The following table lists the errors and their corresponding codes that has changed:</p>
<table role="table">
<thead>
<tr>
<th>Exception</th>
<th>Description</th>
<th>Old code</th>
<th>New code</th>
</tr>
</thead>
<tbody>
<tr>
<td>SqlParseException and ValidationException from Calcite</td>
<td>Query planning failed</td>
<td>500</td>
<td><strong>400</strong></td>
</tr>
<tr>
<td>QueryTimeoutException</td>
<td>Query execution didn't finish in timeout</td>
<td>500</td>
<td><strong>504</strong></td>
</tr>
<tr>
<td>ResourceLimitExceededException</td>
<td>Query asked more resources than configured threshold</td>
<td>500</td>
<td><strong>400</strong></td>
</tr>
<tr>
<td>InsufficientResourceException</td>
<td>Query failed to schedule because of lack of merge buffers available at the time when it was submitted</td>
<td>500</td>
<td><strong>429</strong>, merged to QueryCapacityExceededException</td>
</tr>
<tr>
<td>QueryUnsupportedException</td>
<td>Unsupported functionality</td>
<td>400</td>
<td><strong>501</strong></td>
</tr>
</tbody>
</table>
<p dir="auto">There is also a new query metric for query timeout errors. See <a href="#21-query-timeout-metric">New query timeout metric</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="713736306" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10464" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10464/hovercard" href="https://github.com/apache/druid/pull/10464">#10464</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="784032276" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10746" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10746/hovercard" href="https://github.com/apache/druid/pull/10746">#10746</a></p>
<h3 dir="auto"><a name="user-content-21-query-interrupted-metric" href="#21-query-interrupted-metric">#</a> Query interrupted metric</h3>
<p dir="auto"><code class="notranslate">query/interrupted/count</code> no longer counts the queries that timed out. These queries are counted by <code class="notranslate">query/timeout/count</code>.</p>
<h3 dir="auto"><a name="user-content-21-context-dimension" href="#21-context-dimension">#</a> <code class="notranslate">context</code> dimension in query metrics</h3>
<p dir="auto"><code class="notranslate">context</code> is now a default dimension emitted for all query metrics. <code class="notranslate">context</code> is a JSON-formatted string containing the query context for the query that the emitted metric refers to. The addition of a dimension that was not previously alters some metrics emitted by Druid. You should plan to handle this new <code class="notranslate">context</code> dimension in your metrics pipeline. Since the dimension is a JSON-formatted string, a common solution is to parse the dimension and either flatten it or extract the bits you want and discard the full JSON-formatted string blob.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="742054647" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10578" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10578/hovercard" href="https://github.com/apache/druid/pull/10578">#10578</a></p>
<h3 dir="auto"><a name="user-content-21-deprecate-zk-3.4" href="#21-deprecate-zk-3.4">#</a> Deprecated support for Apache ZooKeeper 3.4</h3>
<p dir="auto"><a href="https://mail-archives.apache.org/mod_mbox/zookeeper-user/202004.mbox/%3C41A7EC67-D8F4-4C3A-B2DB-C2741C2EECA3%40apache.org%3E" rel="nofollow">As ZooKeeper 3.4 has been end-of-life for a while</a>, support for ZooKeeper 3.4 is deprecated in 0.21.0 and will be removed in the near future.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="789542120" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10780" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/10780/hovercard" href="https://github.com/apache/druid/issues/10780">#10780</a></p>
<h3 dir="auto"><a name="user-content-21-consistent-segments-table" href="#21-consistent-segments-table">#</a> Consistent serialization format and column naming convention for the <code class="notranslate">sys.segments</code> table</h3>
<p dir="auto">All columns in the <code class="notranslate">sys.segments</code> table are now serialized in the JSON format to make them consistent with other system tables. Column names now use the same "snake case" convention.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="715198285" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10481" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10481/hovercard" href="https://github.com/apache/druid/pull/10481">#10481</a></p>
<h1 dir="auto"><a name="user-content-21-known-issues" href="#21-known-issues">#</a> Known issues</h1>
<h3 dir="auto"><a name="user-content-21-cve-2020-13949" href="#21-cve-2020-13949">#</a> Known security vulnerability in the Thrift library</h3>
<p dir="auto">The <a href="https://druid.apache.org/docs/latest/development/extensions-contrib/thrift.html" rel="nofollow">Thrift extension</a> can be useful for ingesting files of the Thrift format into Druid. However, there is a <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-13949" rel="nofollow">known security vulnerability</a> in the version of the Thrift library that Druid uses. The vulerability can be exploitable by ingesting maliciously crafted Thrift files when you use Indexers. We recommend granting the <code class="notranslate">DATASOURCE WRITE</code> permission to only trusted users.</p>
<p dir="auto">For a full list of open issues, please see 
<a id="label-5191ae" href="https://github.com/apache/druid/labels/Bug" data-name="Bug" style="--label-r:252;--label-g:41;--label-b:41;--label-h:0;--label-s:97;--label-l:57;" data-view-component="true" class="IssueLabel hx_IssueLabel Label--inline">
  Bug</a>
.</p>
<h1 dir="auto"><a name="user-content-21-credits" href="#21-credits">#</a> Credits</h1>
<p dir="auto">Thanks to everyone who contributed to this release!</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/a2l007/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/a2l007">@a2l007</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhishekagarwal87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhishekagarwal87">@abhishekagarwal87</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/asdf2014/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/asdf2014">@asdf2014</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/AshishKapoor/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/AshishKapoor">@AshishKapoor</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/awelsh93/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/awelsh93">@awelsh93</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ayushkul2910/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ayushkul2910">@ayushkul2910</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bananaaggle/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bananaaggle">@bananaaggle</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/capistrant/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/capistrant">@capistrant</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ccaominh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ccaominh">@ccaominh</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/clintropolis/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/clintropolis">@clintropolis</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cloventt/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cloventt">@cloventt</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/FrankChen021/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/FrankChen021">@FrankChen021</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/harinirajendran/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/harinirajendran">@harinirajendran</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/himanshug/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/himanshug">@himanshug</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jihoonson/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jihoonson">@jihoonson</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jon-wei/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jon-wei">@jon-wei</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/kroeders/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/kroeders">@kroeders</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/liran-funaro/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/liran-funaro">@liran-funaro</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/martin-g/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/martin-g">@martin-g</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/maytasm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/maytasm">@maytasm</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mghosh4/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mghosh4">@mghosh4</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/michaelschiff/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/michaelschiff">@michaelschiff</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nishantmonu51/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nishantmonu51">@nishantmonu51</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pcarrier/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pcarrier">@pcarrier</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/QingdongZeng3/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/QingdongZeng3">@QingdongZeng3</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sthetland/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sthetland">@sthetland</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/suneet-s/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/suneet-s">@suneet-s</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/tdt17/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/tdt17">@tdt17</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/techdocsmith/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/techdocsmith">@techdocsmith</a><br>
@valdemar-giosg<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/viatcheslavmogilevsky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/viatcheslavmogilevsky">@viatcheslavmogilevsky</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/viongpanzi/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/viongpanzi">@viongpanzi</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vogievetsky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vogievetsky">@vogievetsky</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xvrl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xvrl">@xvrl</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhangyue19921010/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhangyue19921010">@zhangyue19921010</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a>, this is meant as a placeholder, feel free to alter title and content of this issue</p>
<p dir="auto">In <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="345406624" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/6066" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/6066/hovercard" href="https://github.com/apache/druid/issues/6066">#6066</a> we discussed the possibility of specifying the <code class="notranslate">sortOrder</code> on ingestion specs.</p>
<p dir="auto">Currently, rows in segments are sorted based on <code class="notranslate">[__time] + dimensions</code>. Knowing that Druid's bitmaps use some form of RLE compression, and that LZ4 operates by finding duplication on sliding windows (reminiscent of RLE-compression), allowing for a custom sort order could allow for significantly denser segments. Users may want to take the time to specify a sort order improves perf based on their data and workloads. This may mean sorting based on cardinality (lower dims first), one-to-many hierarchies across dims (country-&gt;region), or heavily used and/or commonly predicated column.</p>
<p dir="auto">A key item would be to remove the assumption that <code class="notranslate">__time</code> is the first item in the ordering to allow for more RLE-friendly ordering. As <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a> mentioned in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="345406624" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/6066" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/6066/hovercard" href="https://github.com/apache/druid/issues/6066">#6066</a>, many places in the codebase build upon this assumption, and we'd have to make sure that isn't the case anymore as a prerequisite.</p>