<p dir="auto">Apache Druid 25.0.0 contains over 300 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 51 contributors.</p>
<p dir="auto"><a href="https://github.com/apache/druid/milestone/48">See the complete set of changes for additional details</a>.</p>
<h1 dir="auto"><a name="user-content-25-highlights" href="#25-highlights">#</a> Highlights</h1>
<h2 dir="auto"><a name="user-content-25-highlights-msq" href="#25-highlights-msq-">#</a> MSQ task engine now production ready</h2>
<p dir="auto">The multi-stage query (MSQ) task engine used for SQL-based ingestion is now production ready. Use it for any supported workloads. For more information, see the following pages:</p>
<ul dir="auto">
<li><a href="https://druid.apache.org/docs/latest/ingestion/index.html" rel="nofollow">Ingestion</a></li>
<li><a href="https://druid.apache.org/docs/latest/multi-stage-query/index.html" rel="nofollow">SQL-based ingestion</a></li>
</ul>
<h2 dir="auto"><a name="user-content-25-highlights-deploying-druid" href="#25-highlights-deploying-druid">#</a> Simplified Druid deployments</h2>
<p dir="auto">The new <code class="notranslate">start-druid</code> script greatly simplifies deploying any combination of Druid services on a single-server. It comes pre-packaged with the required configs and can be used to launch a fully functional Druid cluster simply by invoking <code class="notranslate">./start-druid</code>. For experienced Druids, it also gives complete control over the runtime properties and JVM arguments to have a cluster that exactly fits your needs.</p>
<p dir="auto">The <code class="notranslate">start-druid</code> script deprecates the existing profiles such as <code class="notranslate">start-micro-quickstart</code> and <code class="notranslate">start-nano-quickstart</code>. These profiles may be removed in future releases. For more information, see <a href="https://druid.apache.org/docs/latest/operations/single-server.html" rel="nofollow">Single server deployment</a>.</p>
<h2 dir="auto"><a name="user-content-25-highlights-string-dictionary-compression" href="#25-highlights-string-dictionary-compression">#</a> String dictionary compression (experimental)</h2>
<p dir="auto">Added support for front coded string dictionaries for smaller string columns, leading to reduced segment sizes with only minor performance penalties for most Druid queries.</p>
<p dir="auto">This can be enabled by setting <code class="notranslate">IndexSpec.stringDictionaryEncoding</code> to <code class="notranslate">{"type":"frontCoded", "bucketSize": 4}</code> , where <code class="notranslate">bucketSize</code> is any power of 2 less than or equal to 128. Setting this property instructs indexing tasks to write segments using compressed dictionaries of the specified bucket size.</p>
<blockquote>
<p dir="auto">Any segment written using string dictionary compression is not readable by older versions of Druid.</p>
</blockquote>
<p dir="auto">For more information, see <a href="https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#front-coding" rel="nofollow">Front coding</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1147967678" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12277" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12277/hovercard" href="https://github.com/apache/druid/pull/12277">#12277</a></p>
<h2 dir="auto"><a name="user-content-25-highlights-kubernetes-native-tasks" href="#25-highlights-kubernetes-native-tasks">#</a> Kubernetes-native tasks</h2>
<p dir="auto">Druid can now use Kubernetes to launch and manage tasks, eliminating the need for middle managers.</p>
<p dir="auto">To use this feature, enable the <a href="../extensions.md#loading-extensions">druid-kubernetes-overlord-extensions</a> in the extensions load list for your Overlord process.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1391541066" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13156" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13156/hovercard" href="https://github.com/apache/druid/pull/13156">#13156</a></p>
<h2 dir="auto"><a name="user-content-25-highlights-hadoop-3-compatible-binary" href="#25-highlights-hadoop-3-compatible-binary">#</a> Hadoop-3 compatible binary</h2>
<p dir="auto">Druid now comes packaged as a dedicated binary for Hadoop-3 users, which contains Hadoop-3 compatible jars. If you do not use Hadoop-3 with your Druid cluster, you may continue using the classic binary.</p>
<h1 dir="auto"><a name="user-content-25-multi-stage-query-task-engine" href="#25-multi-stage-query-task-engine">#</a> Multi-stage query (MSQ) task engine</h1>
<h2 dir="auto"><a name="user-content-25-msq-docker-enabled" href="#25-msq-docker-enabled">#</a> MSQ enabled for Docker</h2>
<p dir="auto">MSQ task query engine is now enabled for Docker by default.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1368833994" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13069" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13069/hovercard" href="https://github.com/apache/druid/pull/13069">#13069</a></p>
<h2 dir="auto"><a name="user-content-25-msq-query-history" href="#25-msq-query-history">#</a> Query history</h2>
<p dir="auto">Multi-stage queries no longer show up in the Query history dialog. They are still available in the <strong>Recent query tasks</strong> panel.</p>
<h2 dir="auto"><a name="user-content-25-msq-clustered-by-limit" href="#25-msq-clustered-by-limit">#</a> Limit on CLUSTERED BY columns</h2>
<p dir="auto">When using the MSQ task engine to ingest data, the number of columns that can be passed in the CLUSTERED BY clause is now limited to 1500.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1444995162" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13352" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13352/hovercard" href="https://github.com/apache/druid/pull/13352">#13352</a></p>
<h2 dir="auto"><a name="user-content-25-msq-string-dict" href="#25-msq-string-dict">#</a> Support for string dictionary compression</h2>
<p dir="auto">The MSQ task engine supports the front-coding of String dictionaries for better compression. This can be enabled for INSERT or REPLACE statements by setting <code class="notranslate">indexSpec</code> to a valid json string in the query context.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1426957873" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13275" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13275/hovercard" href="https://github.com/apache/druid/pull/13275">#13275</a></p>
<h2 dir="auto"><a name="user-content-25-msq-sketch-merging-mode" href="#25-msq-sketch-merging-mode">#</a> Sketch merging mode</h2>
<p dir="auto">Workers can now gather key statistics, used to generate partition boundaries, either sequentially or in parallel. Set <code class="notranslate">clusterStatisticsMergeMode</code> to <code class="notranslate">PARALLEL</code>, <code class="notranslate">SEQUENTIAL</code> or <code class="notranslate">AUTO</code> in the query context to use the corresponding sketch merging mode. For more information, see <a href="https://druid.apache.org/docs/latest/multi-stage-query/reference.html#sketch-merging-mode" rel="nofollow">Sketch merging mode</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1403913688" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13205" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13205/hovercard" href="https://github.com/apache/druid/pull/13205">#13205</a></p>
<h2 dir="auto"><a name="user-content-25-msq-op-improvements" href="#25-msq-op-improvements">#</a> Performance and operational improvements</h2>
<ul dir="auto">
<li><strong>Error messages</strong>: For disallowed MSQ warnings of certain types, the warning is now surfaced as the error. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1402559207" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13198" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13198/hovercard" href="https://github.com/apache/druid/pull/13198">#13198</a></li>
<li><strong>Secrets</strong>: For tasks containing SQL with sensitive keys, Druid now masks the keys while logging with the help regular expressions. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1410961596" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13231" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13231/hovercard" href="https://github.com/apache/druid/pull/13231">#13231</a></li>
<li><strong>Downsampling accuracy</strong>: MSQ task engine now uses the number of bytes instead of number of keys when downsampling data. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1355724816" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12998" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12998/hovercard" href="https://github.com/apache/druid/pull/12998">#12998</a></li>
<li><strong>Memory usage</strong>: When determining partition boundaries, the heap footprint of internal sketches used by MSQ is now capped at 10% of available memory or 300 MB, whichever is lower. Previously, the cap was strictly 300 MB. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1426858396" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13274" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13274/hovercard" href="https://github.com/apache/druid/pull/13274">#13274</a></li>
<li><strong>Task reports</strong>: Added fields <code class="notranslate">pendingTasks</code> and <code class="notranslate">runningTasks</code> to the worker report. See <a href="#query-task-status-information">Query task status information</a> for related web console changes. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1423372162" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13263" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13263/hovercard" href="https://github.com/apache/druid/pull/13263">#13263</a></li>
</ul>
<h1 dir="auto"><a name="user-content-25-querying" href="#25-querying">#</a> Querying</h1>
<h2 dir="auto"><a name="user-content-25-querying-enabled-async-reads-for-jdbc" href="#25-querying-enabled-async-reads-for-jdbc">#</a> Async reads for JDBC</h2>
<p dir="auto">Prevented JDBC timeouts on long queries by returning empty batches when a batch fetch takes too long. Uses an async model to run the result fetch concurrently with JDBC requests.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1402149022" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13196" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13196/hovercard" href="https://github.com/apache/druid/pull/13196">#13196</a></p>
<h2 dir="auto"><a name="user-content-25-querying-in-filter" href="#25-querying-in-filter">#</a> Improved algorithm to check values of an IN filter</h2>
<p dir="auto">To accommodate large value sets arising from large IN filters or from joins pushed down as IN filters, Druid now uses a sorted merge algorithm for merging the set and dictionary for larger values.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1380768908" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13133" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13133/hovercard" href="https://github.com/apache/druid/pull/13133">#13133</a></p>
<h2 dir="auto"><a name="user-content-25-querying-context-security" href="#25-querying-context-security">#</a> Enhanced query context security</h2>
<p dir="auto">Added the following configuration properties that refine the query context security model controlled by <code class="notranslate">druid.auth.authorizeQueryContextParams</code>:</p>
<ul dir="auto">
<li><code class="notranslate">druid.auth.unsecuredContextKeys</code>: A JSON list of query context keys that do not require a security check.</li>
<li><code class="notranslate">druid.auth.securedContextKeys</code>: A JSON list of query context keys that do require a security check.</li>
</ul>
<p dir="auto">If both are set, <code class="notranslate">unsecuredContextKeys</code> acts as exceptions to <code class="notranslate">securedContextKeys</code>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1369059825" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13071" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13071/hovercard" href="https://github.com/apache/druid/pull/13071">#13071</a></p>
<h2 dir="auto"><a name="user-content-25-querying-http-response-headers" href="#25-querying-http-response-headers">#</a> HTTP response headers</h2>
<p dir="auto">The HTTP response for a SQL query now correctly sets response headers, same as a native query.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1365421322" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13052" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13052/hovercard" href="https://github.com/apache/druid/pull/13052">#13052</a></p>
<h1 dir="auto"><a name="user-content-25-metrics" href="#25-metrics">#</a> Metrics</h1>
<h2 dir="auto"><a name="user-content-25-metrics-new-metrics" href="#25-metrics-new-metrics">#</a> New metrics</h2>
<p dir="auto">The following metrics have been newly added. For more details, see <a href="https://druid.apache.org/docs/latest/operations/metrics.html" rel="nofollow">the complete list of Druid metrics</a>.</p>
<h3 dir="auto"><a name="user-content-25-metrics-new-metrics-batch-alloc" href="#25-metrics-new-metrics-batch-alloc">#</a> Batched segment allocation</h3>
<p dir="auto">These metrics pertain to <a href="#25-operations-batch-allocation">batched segment allocation</a>.</p>
<table role="table">
<thead>
<tr>
<th>Metric</th>
<th>Description</th>
<th>Dimensions</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="notranslate">task/action/batch/runTime</code></td>
<td>Milliseconds taken to execute a batch of task actions. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
<tr>
<td><code class="notranslate">task/action/batch/queueTime</code></td>
<td>Milliseconds spent by a batch of task actions in queue. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
<tr>
<td><code class="notranslate">task/action/batch/size</code></td>
<td>Number of task actions in a batch that was executed during the emission period. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
<tr>
<td><code class="notranslate">task/action/batch/attempts</code></td>
<td>Number of execution attempts for a single batch of task actions. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
<tr>
<td><code class="notranslate">task/action/success/count</code></td>
<td>Number of task actions that were executed successfully during the emission period. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskId</code>, <code class="notranslate">taskType</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
<tr>
<td><code class="notranslate">task/action/failed/count</code></td>
<td>Number of task actions that failed during the emission period. Currently only being emitted for <a href="#25-operations-batch-allocation">batched <code class="notranslate">segmentAllocate</code> actions</a></td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskId</code>, <code class="notranslate">taskType</code>, <code class="notranslate">taskActionType=segmentAllocate</code></td>
</tr>
</tbody>
</table>
<h3 dir="auto"><a name="user-content-25-metrics-new-metrics-streaming-ingest" href="#25-metrics-new-metrics-streaming-ingest">#</a> Streaming ingestion</h3>
<table role="table">
<thead>
<tr>
<th>Metric</th>
<th>Description</th>
<th>Dimensions</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="notranslate">ingest/kafka/partitionLag</code></td>
<td>Partition-wise lag between the offsets consumed by the Kafka indexing tasks and latest offsets in Kafka brokers. Minimum emission period for this metric is a minute.</td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">stream</code>, <code class="notranslate">partition</code></td>
</tr>
<tr>
<td><code class="notranslate">ingest/kinesis/partitionLag/time</code></td>
<td>Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.</td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">stream</code>, <code class="notranslate">partition</code></td>
</tr>
<tr>
<td><code class="notranslate">ingest/pause/time</code></td>
<td>Milliseconds spent by a task in a paused state without ingesting.</td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskId</code>, <code class="notranslate">taskType</code></td>
</tr>
<tr>
<td><code class="notranslate">ingest/handoff/time</code></td>
<td>Total time taken in milliseconds for handing off a given set of published segments.</td>
<td><code class="notranslate">dataSource</code>, <code class="notranslate">taskId</code>, <code class="notranslate">taskType</code></td>
</tr>
</tbody>
</table>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1413166465" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13238" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13238/hovercard" href="https://github.com/apache/druid/pull/13238">#13238</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1441326884" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13331" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13331/hovercard" href="https://github.com/apache/druid/pull/13331">#13331</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1436171319" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13313" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13313/hovercard" href="https://github.com/apache/druid/pull/13313">#13313</a></p>
<h2 dir="auto"><a name="user-content-25-metrics-improvements" href="#25-metrics-improvements">#</a> Other improvements</h2>
<ul dir="auto">
<li>New dimension <code class="notranslate">taskActionType</code> which may take values such as <code class="notranslate">segmentAllocate</code>, <code class="notranslate">segmentTransactionalInsert</code>, etc. This dimension is reported for <code class="notranslate">task/action/run/time</code> and the new <a href="#25-operations-batch-allocation">batched segment allocation</a> metrics. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1441402531" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13333" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13333/hovercard" href="https://github.com/apache/druid/pull/13333">#13333</a></li>
<li>Metric <code class="notranslate">namespace/cache/heapSizeInBytes</code> for global cached lookups now accounts for the <code class="notranslate">String</code> object overhead of 40 bytes. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1406721623" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13219" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13219/hovercard" href="https://github.com/apache/druid/pull/13219">#13219</a></li>
<li><code class="notranslate">jvm/gc/cpu</code> has been fixed to report nanoseconds instead of milliseconds. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1454036061" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13383" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13383/hovercard" href="https://github.com/apache/druid/pull/13383">#13383</a></li>
</ul>
<h1 dir="auto"><a name="user-content-25-nested-columns" href="#25-nested-columns">#</a> Nested columns</h1>
<h2 dir="auto"><a name="user-content-25-nested-columns-nested-columns-performance-improvement" href="#25-nested-columns-nested-columns-performance-improvement">#</a> Nested columns performance improvement</h2>
<p dir="auto">Improved <code class="notranslate">NestedDataColumnSerializer</code> to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1375834202" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13101" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13101/hovercard" href="https://github.com/apache/druid/pull/13101">#13101</a></p>
<h2 dir="auto"><a name="user-content-25-nested-columns-support-for-more-formats" href="#25-nested-columns-support-for-more-formats">#</a> Support for more formats</h2>
<p dir="auto">Druid nested columns and the associated JSON transform functions now support Avro, ORC, and Parquet.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1439285177" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13325" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13325/hovercard" href="https://github.com/apache/druid/pull/13325">#13325</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1451540965" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13375" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13375/hovercard" href="https://github.com/apache/druid/pull/13375">#13375</a></p>
<h2 dir="auto"><a name="user-content-25-nested-columns-refactored-a-datasource-before-unnest-" href="#25-nested-columns-refactored-a-datasource-before-unnest-">#</a> Refactored a datasource before unnest</h2>
<p dir="auto">When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1372235946" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13085" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13085/hovercard" href="https://github.com/apache/druid/pull/13085">#13085</a></p>
<h1 dir="auto"><a name="user-content-25-ingestion" href="#25-ingestion">#</a> Ingestion</h1>
<h2 dir="auto"><a name="user-content-25-ingestion-improved-filtering-for-cloud-objects" href="#25-ingestion-improved-filtering-for-cloud-objects">#</a> Improved filtering for cloud objects</h2>
<p dir="auto">You can now stop at arbitrary subfolders using glob syntax in the <code class="notranslate">ioConfig.inputSource.filter</code> field for native batch ingestion from cloud storage, such as S3.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1361183533" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13027" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13027/hovercard" href="https://github.com/apache/druid/pull/13027">#13027</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-async-task-client-for-streaming-ingestion" href="#25-ingestion-async-task-client-for-streaming-ingestion">#</a> Async task client for streaming ingestion</h2>
<p dir="auto">You can now enable asynchronous communication between the stream supervisor and indexing tasks by setting <code class="notranslate">chatAsync</code> to true in the <code class="notranslate">tuningConfig</code>. The async task client uses its own internal thread pool and thus ignrores the <code class="notranslate">chatThreads</code> property.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1445240326" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13354" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13354/hovercard" href="https://github.com/apache/druid/pull/13354">#13354</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-improved-control-for-how-druid-reads-json-data-for-streaming-ingestion" href="#25-ingestion-improved-control-for-how-druid-reads-json-data-for-streaming-ingestion">#</a> Improved handling of JSON data with streaming ingestion</h2>
<p dir="auto">You can now better control how Druid reads JSON data for streaming ingestion by setting the following fields in the input format specification:</p>
<ul dir="auto">
<li><code class="notranslate">assumedNewlineDelimited</code> to parse lines of JSON independently.</li>
<li><code class="notranslate">useJsonNodeReader</code> to retain valid JSON events when parsing multi-line JSON events when a parsing exception occurs.</li>
</ul>
<p dir="auto">The web console has been updated to include these options.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1373694669" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13089" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13089/hovercard" href="https://github.com/apache/druid/pull/13089">#13089</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-idle-supervisor" href="#25-ingestion-idle-supervisor">#</a> Ingesting from an idle Kafka stream</h2>
<p dir="auto">When a Kafka stream becomes inactive, the supervisor ingesting from it can be configured to stop creating new indexing tasks. The supervisor automatically resumes creation of new indexing tasks once the stream becomes active again. Set the property <code class="notranslate">dataSchema.ioConfig.idleConfig.enabled</code> to true in the respective supervisor spec or set <code class="notranslate">druid.supervisor.idleConfig.enabled</code> on the overlord to enable this behaviour. Please see the following for details:</p>
<ul dir="auto">
<li><a href="https://druid.apache.org/docs/latest/configuration/index.html#supervisors" rel="nofollow">Overlord configuration</a></li>
<li><a href="https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html#kafkasupervisorioconfig" rel="nofollow">Supervisor spec</a></li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1385820947" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13144" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13144/hovercard" href="https://github.com/apache/druid/pull/13144">#13144</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-kafka-consumer-improvement" href="#25-ingestion-kafka-consumer-improvement">#</a> Kafka Consumer improvement</h2>
<p dir="auto">You can now configure the Kafka Consumer's custom deserializer after its instantiation.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1375216967" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13097" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13097/hovercard" href="https://github.com/apache/druid/pull/13097">#13097</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-kafka-supervisor-logging" href="#25-ingestion-kafka-supervisor-logging">#</a> Kafka supervisor logging</h2>
<p dir="auto">Kafka supervisor logs are now less noisy. The supervisors now log events at the DEBUG level instead of INFO.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1457141070" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13392" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13392/hovercard" href="https://github.com/apache/druid/pull/13392">#13392</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-fixed-overlord-leader-election" href="#25-ingestion-fixed-overlord-leader-election">#</a> Fixed Overlord leader election</h2>
<p dir="auto">Fixed a problem where Overlord leader election failed due to lock reacquisition issues. Druid now fails these tasks and clears all locks so that the Overlord leader election isn't blocked.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1394603647" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13172" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13172/hovercard" href="https://github.com/apache/druid/pull/13172">#13172</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-support-for-inline-protobuf-descriptor" href="#25-ingestion-support-for-inline-protobuf-descriptor">#</a> Support for inline protobuf descriptor</h2>
<p dir="auto">Added a new <code class="notranslate">inline</code> type <code class="notranslate">protoBytesDecoder</code> that allows a user to pass inline the contents of a Protobuf descriptor file, encoded as a Base64 string.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1400379777" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13192" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13192/hovercard" href="https://github.com/apache/druid/pull/13192">#13192</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-duplicate-notices" href="#25-ingestion-duplicate-notices">#</a> Duplicate notices</h2>
<p dir="auto">For streaming ingestion, notices that are the same as one already in queue won't be enqueued. This will help reduce notice queue size.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1441408249" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13334" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13334/hovercard" href="https://github.com/apache/druid/pull/13334">#13334</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-sampling-from-stream-input-now-respects-the-configured-timeout" href="#25-ingestion-sampling-from-stream-input-now-respects-the-configured-timeout">#</a> Sampling from stream input now respects the configured timeout</h2>
<p dir="auto">Fixed a problem where sampling from a stream input, such as Kafka or Kinesis, failed to respect the configured timeout when the stream had no records available. You can now set the maximum amount of time in which the entry iterator will return results.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1432370468" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13296" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13296/hovercard" href="https://github.com/apache/druid/pull/13296">#13296</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-streaming-tasks-resume-on-overlord-switch" href="#25-ingestion-streaming-tasks-resume-on-overlord-switch">#</a> Streaming tasks resume on Overlord switch</h2>
<p dir="auto">Fixed a problem where streaming ingestion tasks continued to run until their duration elapsed after the Overlord leader had issued a pause to the tasks. Now, when the Overlord switch occurs right after it has issued a pause to the task, the task remains in a paused state even after the Overlord re-election.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1407739718" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13223" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13223/hovercard" href="https://github.com/apache/druid/pull/13223">#13223</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-fixed-parquet-list-conversion" href="#25-ingestion-fixed-parquet-list-conversion">#</a> Fixed Parquet list conversion</h2>
<p dir="auto">Fixed an issue with Parquet list conversion, where lists of complex objects could unexpectedly be wrapped in an extra object, appearing as <code class="notranslate">[{"element":&lt;actual_list_element&gt;},{"element":&lt;another_one&gt;}...]</code> instead of the direct list. This changes the behavior of the parquet reader for lists of structured objects to be consistent with other parquet logical list conversions. The data is now fetched directly, more closely matching its expected structure.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1432293473" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13294" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13294/hovercard" href="https://github.com/apache/druid/pull/13294">#13294</a></p>
<h2 dir="auto"><a name="user-content-25-ingestion-introduced-a-tree-type-to-flattenspec" href="#25-ingestion-introduced-a-tree-type-to-flattenspec">#</a> Introduced a tree type to flattenSpec</h2>
<p dir="auto">Introduced a <code class="notranslate">tree</code> type to <code class="notranslate">flattenSpec</code>. In the event that a simple hierarchical lookup is required, the <code class="notranslate">tree</code> type allows for faster JSON parsing than <code class="notranslate">jq</code> and <code class="notranslate">path</code> parsing types.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1108732435" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12177" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12177/hovercard" href="https://github.com/apache/druid/pull/12177">#12177</a></p>
<h1 dir="auto"><a name="user-content-25-operations" href="#25-operations">#</a> Operations</h1>
<h2 dir="auto"><a name="user-content-25-operations-compaction" href="#25-operations-compaction">#</a> Compaction</h2>
<p dir="auto">Compaction behavior has changed to improve the amount of time it takes and disk space it takes:</p>
<ul dir="auto">
<li>When segments need to be fetched, download them one at a time and delete them when Druid is done with them. This still takes time but minimizes the required disk space.</li>
<li>Don't fetch segments on the main compact task when they aren't needed. If the user provides a full <code class="notranslate">granularitySpec</code>, <code class="notranslate">dimensionsSpec</code>, and <code class="notranslate">metricsSpec</code>, Druid skips fetching segments.</li>
</ul>
<p dir="auto">For more information, see the documentation on <a href="https://druid.apache.org/docs/latest/data-management/compaction.html" rel="nofollow">Compaction</a> and <a href="https://druid.apache.org/docs/latest/data-management/automatic-compaction.html" rel="nofollow">Automatic compaction</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1427870970" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13280" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13280/hovercard" href="https://github.com/apache/druid/pull/13280">#13280</a></p>
<h2 dir="auto"><a name="user-content-25-operations-idle-configs-for-the-supervisor" href="#25-operations-idle-configs-for-the-supervisor">#</a> Idle configs for the Supervisor</h2>
<p dir="auto">You can now set the Supervisor to idle, which is useful in cases where freeing up slots so that autoscaling can be more effective.</p>
<p dir="auto">To configure the idle behavior, use the following properties:</p>
<table role="table">
<thead>
<tr>
<th>Property</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="notranslate">druid.supervisor.idleConfig.enabled</code></td>
<td>(Cluster wide) If <code class="notranslate">true</code>, supervisor can become idle if there is no data on input stream/topic for some time.</td>
<td>false</td>
</tr>
<tr>
<td><code class="notranslate">druid.supervisor.idleConfig.inactiveAfterMillis</code></td>
<td>(Cluster wide) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for <code class="notranslate">inactiveAfterMillis</code> milliseconds.</td>
<td><code class="notranslate">600_000</code></td>
</tr>
<tr>
<td><code class="notranslate">inactiveAfterMillis</code></td>
<td>(Individual Supervisor) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for <code class="notranslate">inactiveAfterMillis</code> milliseconds.</td>
<td>no (default == <code class="notranslate">600_000</code>)</td>
</tr>
</tbody>
</table>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1435747275" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13311" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13311/hovercard" href="https://github.com/apache/druid/pull/13311">#13311</a></p>
<h2 dir="auto"><a name="user-content-25-operations-improved-supervisor-termination" href="#25-operations-improved-supervisor-termination">#</a> Improved supervisor termination</h2>
<p dir="auto">Fixed issues with delayed supervisor termination during certain transient states.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1369353892" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13072" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13072/hovercard" href="https://github.com/apache/druid/pull/13072">#13072</a></p>
<h2 dir="auto"><a name="user-content-25-operations-backoff-for-httppostemitter" href="#25-operations-backoff-for-httppostemitter">#</a> Backoff for HttpPostEmitter</h2>
<p dir="auto">The <code class="notranslate">HttpPostEmitter</code> option now has a backoff. This means that there should be less noise in the logs and lower CPU usage if you use this option for logging.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1090172063" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12102" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12102/hovercard" href="https://github.com/apache/druid/pull/12102">#12102</a></p>
<h2 dir="auto"><a name="user-content-25-operations-dumpsegment-tool-for-nested-columns" href="#25-operations-dumpsegment-tool-for-nested-columns">#</a> DumpSegment tool for nested columns</h2>
<p dir="auto">The DumpSegment tool can now be used on nested columns with the <code class="notranslate">--dump nested</code> option.</p>
<p dir="auto">For more information, see <a href="https://druid.apache.org/docs/latest/operations/dump-segment" rel="nofollow">dump-segment tool</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1445297730" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13356" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13356/hovercard" href="https://github.com/apache/druid/pull/13356">#13356</a></p>
<h2 dir="auto"><a name="user-content-25-operations-segment-loading-and-balancing" href="#25-operations-segment-loading-and-balancing">#</a> Segment loading and balancing</h2>
<h3 dir="auto"><a name="user-content-25-operations-batch-allocation" href="#25-operations-batch-allocation">#</a> Batched segment allocation</h3>
<p dir="auto">Segment allocation on the Overlord can take some time to finish, which can cause ingestion lag while a task waits for segments to be allocated.  Performing segment allocation in batches can help improve performance.</p>
<p dir="auto">There are two new properties that affect how Druid performs segment allocation:</p>
<table role="table">
<thead>
<tr>
<th>Property</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="notranslate">druid.indexer.tasklock.batchSegmentAllocation</code></td>
<td>If set to true, Druid performs segment allocate actions in batches to improve throughput and reduce the average <code class="notranslate">task/action/run/time</code>. See <a href="https://druid.apache.org/docs/latest/ingestion/tasks.html#batching-segmentallocate-actions" rel="nofollow">batching <code class="notranslate">segmentAllocate</code> actions</a> for details.</td>
<td>false</td>
</tr>
<tr>
<td><code class="notranslate">druid.indexer.tasklock.batchAllocationWaitTime</code></td>
<td>Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if <code class="notranslate">batchSegmentAllocation</code> is enabled.</td>
<td>500</td>
</tr>
</tbody>
</table>
<p dir="auto">In addition to these properties, there are new metrics to track batch segment allocation. For more information, see <a href="#new-metrics-for-segment-allocation">New metrics for segment  allocation</a>.</p>
<p dir="auto">For more information, see the following:</p>
<ul dir="auto">
<li><a href="https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations" rel="nofollow">Overlord operations</a></li>
<li><a href="https://druid.apache.org/docs/latest/ingestion/tasks.html#task-actions" rel="nofollow">Task actions and Batching <code class="notranslate">segmentAllocate</code> actions</a></li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1449888957" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13369" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13369/hovercard" href="https://github.com/apache/druid/pull/13369">#13369</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1478246986" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13503" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13503/hovercard" href="https://github.com/apache/druid/pull/13503">#13503</a></p>
<h3 dir="auto"><a name="user-content-25-operations-segment-loading-and-balancing-cachingcost-balancer-strategy" href="#25-operations-segment-loading-and-balancing-cachingcost-balancer-strategy">#</a> Improved cachingCost balancer strategy</h3>
<p dir="auto">The <code class="notranslate">cachingCost</code> balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed:</p>
<ul dir="auto">
<li>Subtract the self cost of a segment if it is being served by the target server</li>
<li>Subtract the cost of segments that are marked to be dropped</li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1438694534" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13321" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13321/hovercard" href="https://github.com/apache/druid/pull/13321">#13321</a></p>
<h3 dir="auto"><a name="user-content-25-operations-segment-loading-and-balancing-segment-assignment" href="#25-operations-segment-loading-and-balancing-segment-assignment">#</a> Faster segment assignment</h3>
<p dir="auto">You can now use a round-robin segment strategy to speed up initial segment assignments. Set <code class="notranslate">useRoundRobinSegmentAssigment</code> to <code class="notranslate">true</code> in the Coordinator dynamic config to enable this feature.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1449242950" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13367" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13367/hovercard" href="https://github.com/apache/druid/pull/13367">#13367</a></p>
<h3 dir="auto"><a name="user-content-25-operations-segment-batch-sampling" href="#25-operations-segment-batch-sampling">#</a> Default to batch sampling for balancing segments</h3>
<p dir="auto">Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster.</p>
<p dir="auto">As part of this change, the following have been deprecated and will be removed in future releases:</p>
<ul dir="auto">
<li>coordinator dynamic config <code class="notranslate">useBatchedSegmentSampler</code></li>
<li>coordinator dynamic config <code class="notranslate">percentOfSegmentsToConsiderPerMove</code></li>
<li>old non-batch method of sampling segments</li>
</ul>
<h3 dir="auto"><a name="user-content-25-operations-segment-unused-property" href="#25-operations-segment-unused-property">#</a> Remove unused property</h3>
<p dir="auto">The unused coordinator property <code class="notranslate">druid.coordinator.loadqueuepeon.repeatDelay</code> has been removed. Use only <code class="notranslate">druid.coordinator.loadqueuepeon.http.repeatDelay</code> to configure repeat delay for the HTTP-based segment loading queue.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1456992752" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13391" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13391/hovercard" href="https://github.com/apache/druid/pull/13391">#13391</a></p>
<h3 dir="auto"><a name="user-content-25-operations-segment-replication" href="#25-operations-segment-replication">#</a> Avoid segment over-replication</h3>
<p dir="auto">Improved the process of checking server inventory to prevent over-replication of segments during segment balancing.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1377027344" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13114" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13114/hovercard" href="https://github.com/apache/druid/pull/13114">#13114</a></p>
<h2 dir="auto"><a name="user-content-25-operations-provided-service-specific-log4j-overrides-in-containerized-deployments" href="#25-operations-provided-service-specific-log4j-overrides-in-containerized-deployments">#</a> Provided service specific log4j overrides in containerized deployments</h2>
<p dir="auto">Provided an option to override log4j configs setup at the service level directories so that it works with Druid-operator based deployments.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1360686267" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13020" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13020/hovercard" href="https://github.com/apache/druid/pull/13020">#13020</a></p>
<h2 dir="auto"><a name="user-content-25-operations-various-docker-improvements" href="#25-operations-various-docker-improvements">#</a> Various Docker improvements</h2>
<ul dir="auto">
<li>Updated Docker to run with JRE 11 by default.</li>
<li>Updated Docker to use <a href="https://github.com/GoogleContainerTools/distroless"><code class="notranslate">gcr.io/distroless/java11-debian11</code></a> image as base by default.</li>
<li>Enabled Docker buildkit cache to speed up building.</li>
<li>Downloaded <a href="https://github.com/robxu9/bash-static"><code class="notranslate">bash-static</code></a> to the Docker image so that scripts that require bash can be executed.</li>
<li>Bumped builder image from <code class="notranslate">3.8.4-jdk-11-slim</code> to <code class="notranslate">3.8.6-jdk-11-slim</code>.</li>
<li>Switched busybox from <code class="notranslate">amd64/busybox:1.30.0-glibc</code> to <code class="notranslate">busybox:1.35.0-glibc</code>.</li>
<li>Added support to build arm64-based image.</li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1366795791" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13059" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13059/hovercard" href="https://github.com/apache/druid/pull/13059">#13059</a></p>
<h2 dir="auto"><a name="user-content-25-operations-enabled-cleaner-json-for-various-input-sources-and-formats" href="#25-operations-enabled-cleaner-json-for-various-input-sources-and-formats">#</a> Enabled cleaner JSON for various input sources and formats</h2>
<p dir="auto">Added <code class="notranslate">JsonInclude</code> to various properties, to avoid population of default values in serialized JSON.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1368143209" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13064" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13064/hovercard" href="https://github.com/apache/druid/pull/13064">#13064</a></p>
<h2 dir="auto"><a name="user-content-25-operations-improved-direct-memory-check-on-startup" href="#25-operations-improved-direct-memory-check-on-startup">#</a> Improved direct memory check on startup</h2>
<p dir="auto">Improved direct memory check on startup by providing better support for Java 9+ in <code class="notranslate">RuntimeInfo</code>, and clearer log messages where validation fails.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1403961849" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13207" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13207/hovercard" href="https://github.com/apache/druid/pull/13207">#13207</a></p>
<h2 dir="auto"><a name="user-content-25-operations-improved-the-run-time-of-the-markasunusedovershadowedsegments-duty" href="#25-operations-improved-the-run-time-of-the-markasunusedovershadowedsegments-duty">#</a> Improved the run time of the MarkAsUnusedOvershadowedSegments duty</h2>
<p dir="auto">Improved the run time of the <code class="notranslate">MarkAsUnusedOvershadowedSegments</code> duty by iterating over all overshadowed segments and marking segments as unused in batches.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1428367651" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13287" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13287/hovercard" href="https://github.com/apache/druid/pull/13287">#13287</a></p>
<h1 dir="auto"><a name="user-content-25-web-console" href="#25-web-console">#</a> Web console</h1>
<h2 dir="auto"><a name="user-content-25-web-console-delete-an-interval" href="#25-web-console-delete-an-interval">#</a> Delete an interval</h2>
<p dir="auto">You can now pick an interval to delete from a dropdown in the kill task dialog.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1462679298" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13431" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13431/hovercard" href="https://github.com/apache/druid/pull/13431">#13431</a></p>
<h2 dir="auto"><a name="user-content-25-web-console-removed-the-old-query-view" href="#25-web-console-removed-the-old-query-view">#</a> Removed the old query view</h2>
<p dir="auto">The old query view is removed. Use the new query view with tabs.<br>
For more information, see <a href="https://druid.apache.org/docs/latest/operations/web-console.html#query" rel="nofollow">Web console</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1394019727" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13169" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13169/hovercard" href="https://github.com/apache/druid/pull/13169">#13169</a></p>
<h2 dir="auto"><a name="user-content-25-web-console-filter-column-values-in-query-results" href="#25-web-console-filter-column-values-in-query-results">#</a> Filter column values in query results</h2>
<p dir="auto">The web console now allows you to add to existing filters for a selected column.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1394019727" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13169" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13169/hovercard" href="https://github.com/apache/druid/pull/13169">#13169</a></p>
<h2 dir="auto"><a name="user-content-25-web-console-support-for-kafka-lookups" href="#25-web-console-support-for-kafka-lookups">#</a> Support for Kafka lookups in the web-console</h2>
<p dir="auto">Added support for Kafka-based lookups rendering and input in the web console.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1375370412" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13098" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13098/hovercard" href="https://github.com/apache/druid/pull/13098">#13098</a></p>
<h2 dir="auto"><a name="user-content-25-web-console-query-task-status-information" href="#25-web-console-query-task-status-information">#</a> Query task status information</h2>
<p dir="auto">The web console now exposes a textual indication about running and pending tasks when a query is stuck due to lack of task slots.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1430256866" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13291" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13291/hovercard" href="https://github.com/apache/druid/pull/13291">#13291</a></p>
<h1 dir="auto"><a name="user-content-25-extensions" href="#25-extensions">#</a> Extensions</h1>
<h2 dir="auto"><a name="user-content-25-extensions-extension-optimization" href="#25-extensions-extension-optimization">#</a> Extension optimization</h2>
<p dir="auto">Optimized the <code class="notranslate">compareTo</code> function in <code class="notranslate">CompressedBigDecimal</code>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1372268744" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13086" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13086/hovercard" href="https://github.com/apache/druid/pull/13086">#13086</a></p>
<h2 dir="auto"><a name="user-content-25-extensions-compressedbigdecimal-cleanup-and-extension" href="#25-extensions-compressedbigdecimal-cleanup-and-extension">#</a> CompressedBigDecimal cleanup and extension</h2>
<p dir="auto">Removed unnecessary generic type from CompressedBigDecimal, added support for number input types, added support for reading aggregator input types directly (uningested data), and fixed scaling bug in buffer aggregator.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1365224141" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13048" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13048/hovercard" href="https://github.com/apache/druid/pull/13048">#13048</a></p>
<h2 dir="auto"><a name="user-content-25-extensions-support-for-kubernetes-discovery" href="#25-extensions-support-for-kubernetes-discovery">#</a> Support for Kubernetes discovery</h2>
<p dir="auto">Added <code class="notranslate">POD_NAME</code> and <code class="notranslate">POD_NAMESPACE</code> env variables to all Kubernetes Deployments and StatefulSets.<br>
Helm deployment is now compatible with <code class="notranslate">druid-kubernetes-extension</code>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1422601157" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13262" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13262/hovercard" href="https://github.com/apache/druid/pull/13262">#13262</a></p>
<h1 dir="auto"><a name="user-content-25-docs" href="#25-docs">#</a> Docs</h1>
<h2 dir="auto"><a name="user-content-25-docs-jupyter-notebook-tutorials" href="#25-docs-jupyter-notebook-tutorials">#</a> Jupyter Notebook tutorials</h2>
<p dir="auto">We released our first Jupyter Notebook-based tutorial to learn the basics of the Druid API. Download the notebook and follow along with the tutorial to learn how to get basic cluster information, ingest data, and query data.<br>
For more information, see <a href="https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-index.html" rel="nofollow">Jupyter Notebook tutorials</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1442902277" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13342" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13342/hovercard" href="https://github.com/apache/druid/pull/13342">#13342</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1442978247" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13345" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13345/hovercard" href="https://github.com/apache/druid/pull/13345">#13345</a></p>
<h1 dir="auto"><a name="user-content-25-dependency-updates" href="#25-dependency-updates">#</a> Dependency updates</h1>
<h2 dir="auto"><a name="user-content-25-dependency-updates-updated-kafka-version" href="#25-dependency-updates-updated-kafka-version">#</a> Updated Kafka version</h2>
<p dir="auto">Updated the Apache Kafka core dependency to version 3.3.1.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1395538430" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13176" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13176/hovercard" href="https://github.com/apache/druid/pull/13176">#13176</a></p>
<h2 dir="auto"><a name="user-content-25-dependency-updates-docker-improvements" href="#25-dependency-updates-docker-improvements">#</a> Docker improvements</h2>
<p dir="auto">Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1366795791" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13059" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13059/hovercard" href="https://github.com/apache/druid/pull/13059">#13059</a></p>
<h1 dir="auto"><a name="user-content-25-upgrade" href="#25-upgrade">#</a> Upgrading to 25.0.0</h1>
<p dir="auto">Consider the following changes and updates when upgrading from Druid 24.0.x to 25.0.0. If you're updating from an earlier version, see the release notes of the relevant intermediate versions.</p>
<h2 dir="auto"><a name="user-content-25-upgrade-segment-discovery" href="#25-upgrade-segment-discovery">#</a> Default HTTP-based segment discovery and task management</h2>
<p dir="auto">The default segment discovery method now uses HTTP instead of ZooKeeper.</p>
<p dir="auto">This update changes the defaults for the following properties:</p>
<table role="table">
<thead>
<tr>
<th>Property</th>
<th>New default</th>
<th>Previous default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="notranslate">druid.serverview.type</code> for segment management</td>
<td>http</td>
<td>batch</td>
</tr>
<tr>
<td><code class="notranslate">druid.coordinator.loadqueuepeon.type</code> for segment management</td>
<td>http</td>
<td>curator</td>
</tr>
<tr>
<td><code class="notranslate">druid.indexer.runner.type</code> for the Overlord</td>
<td>httpRemote</td>
<td>local</td>
</tr>
</tbody>
</table>
<p dir="auto">To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults. ZooKeeper-based implementations for these properties are deprecated and will be removed in a subsequent release.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1374470464" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13092" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13092/hovercard" href="https://github.com/apache/druid/pull/13092">#13092</a></p>
<h2 dir="auto"><a name="user-content-25-upgrade-hll-and-quantiles-sketches" href="#25-upgrade-hll-and-quantiles-sketches">#</a> Finalizing HLL and quantiles sketch aggregates</h2>
<p dir="auto">The aggregation functions for HLL and quantiles sketches returned sketches or numbers when they are finalized depending on where they were in the native query plan.</p>
<p dir="auto">Druid no longer finalizes aggregators in the following two cases:</p>
<ul dir="auto">
<li>aggregators appear in the outer level of a query</li>
<li>aggregators are used as input to an expression or finalizing-field-access post-aggregator</li>
</ul>
<p dir="auto">This change aligns the behavior of HLL and quantiles sketches with theta sketches.</p>
<p dir="auto">To restore old behaviour, you can set <code class="notranslate">sqlFinalizeOuterSketches=true</code> in the query context.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1417217739" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13247" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13247/hovercard" href="https://github.com/apache/druid/pull/13247">#13247</a></p>
<h2 dir="auto"><a name="user-content-25-upgrade-kill-tasks-mark-unused" href="#25-upgrade-kill-tasks-mark-unused">#</a> Kill tasks mark segments as unused only if specified</h2>
<p dir="auto">When you issue a kill task, Druid marks the underlying segments as unused only if explicitly specified. For more information, see the <a href="https://druid.apache.org/docs/latest/operations/api-reference.html#coordinator" rel="nofollow">API reference</a></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1376628479" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13104" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13104/hovercard" href="https://github.com/apache/druid/pull/13104">#13104</a></p>
<h2 dir="auto"><a name="user-content-25-upgrade-incompatible" href="#25-upgrade-incompatible">#</a> Incompatible changes</h2>
<h3 dir="auto"><a name="user-content-25-upgrade-incompatible-curator" href="#25-upgrade-incompatible-curator">#</a> Upgrade curator to 5.3.0</h3>
<p dir="auto">Apache Curator upgraded to the latest version, 5.3.0. This version drops support for ZooKeeper 3.4 but Druid has already officially dropped support in 0.22. In 5.3.0, Curator has removed support for Exhibitor so all related configurations and tests have been removed.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1347184665" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12939" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12939/hovercard" href="https://github.com/apache/druid/pull/12939">#12939</a></p>
<h3 dir="auto"><a name="user-content-25-upgrade-incompatible-parquet" href="#25-upgrade-incompatible-parquet">#</a> Fixed Parquet list conversion</h3>
<p dir="auto">The behavior of the parquet reader for lists of structured objects has been changed to be consistent with other parquet logical list conversions. The data is now fetched directly, more closely matching its expected structure. See <a href="#25-ingestion-fixed-parquet-list-conversion">parquet list conversion</a> for more details.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1432293473" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/13294" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/13294/hovercard" href="https://github.com/apache/druid/pull/13294">#13294</a></p>
<h1 dir="auto"><a name="user-content-25-credits" href="#25-credits">#</a> Credits</h1>
<p dir="auto">Thanks to everyone who contributed to this release!</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/317brian/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/317brian">@317brian</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/599166320/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/599166320">@599166320</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/a2l007/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/a2l007">@a2l007</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhagraw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhagraw">@abhagraw</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhishekagarwal87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhishekagarwal87">@abhishekagarwal87</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/adarshsanjeev/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/adarshsanjeev">@adarshsanjeev</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/adelcast/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/adelcast">@adelcast</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/AlexanderSaydakov/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/AlexanderSaydakov">@AlexanderSaydakov</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/amaechler/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/amaechler">@amaechler</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/AmatyaAvadhanula/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/AmatyaAvadhanula">@AmatyaAvadhanula</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ApoorvGuptaAi/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ApoorvGuptaAi">@ApoorvGuptaAi</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/arvindanugula/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/arvindanugula">@arvindanugula</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/asdf2014/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/asdf2014">@asdf2014</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/churromorales/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/churromorales">@churromorales</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/clintropolis/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/clintropolis">@clintropolis</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cloventt/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cloventt">@cloventt</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cristian-popa/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cristian-popa">@cristian-popa</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cryptoe/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cryptoe">@cryptoe</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dampcake/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dampcake">@dampcake</a><br>
<a class="user-mention notranslate" data-hovercard-type="organization" data-hovercard-url="/orgs/dependabot/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dependabot">@dependabot</a>[bot]<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/didip/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/didip">@didip</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ektravel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ektravel">@ektravel</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/eshengit/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/eshengit">@eshengit</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/findingrish/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/findingrish">@findingrish</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/FrankChen021/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/FrankChen021">@FrankChen021</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/hnakamor/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hnakamor">@hnakamor</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/hosswald/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hosswald">@hosswald</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/imply-cheddar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/imply-cheddar">@imply-cheddar</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jasonk000/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jasonk000">@jasonk000</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jon-wei/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jon-wei">@jon-wei</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Junge-401/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Junge-401">@Junge-401</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/kfaraz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/kfaraz">@kfaraz</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LakshSingla/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LakshSingla">@LakshSingla</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mcbrewster/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mcbrewster">@mcbrewster</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/paul-rogers/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/paul-rogers">@paul-rogers</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/petermarshallio/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/petermarshallio">@petermarshallio</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rash67/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rash67">@rash67</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohangarg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohangarg">@rohangarg</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sachidananda007/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sachidananda007">@sachidananda007</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/santosh-d3vpl3x/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/santosh-d3vpl3x">@santosh-d3vpl3x</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/senthilkv/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/senthilkv">@senthilkv</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/somu-imply/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/somu-imply">@somu-imply</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/techdocsmith/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/techdocsmith">@techdocsmith</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/tejaswini-imply/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/tejaswini-imply">@tejaswini-imply</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vogievetsky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vogievetsky">@vogievetsky</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vtlim/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vtlim">@vtlim</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wcc526/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wcc526">@wcc526</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/writer-jill/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/writer-jill">@writer-jill</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xvrl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xvrl">@xvrl</a><br>
@zachjsh</p>
<p dir="auto">Apache Druid 24.0.0 contains over 300 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 67 contributors. <a href="https://github.com/apache/druid/milestone/46">See the complete set of changes for additional details</a>.</p>
<h1 dir="auto"><a name="user-content-24.0.0-new-features" href="#24.0.0-new-features">#</a> New Features</h1>
<h2 dir="auto"><a name="user-content-24.0.0-new-features-multi-stage-query-task-engine" href="#24.0.0-new-features-multi-stage-query-task-engine">#</a> Multi-stage query task engine</h2>
<p dir="auto">SQL-based ingestion for Apache Druid uses a distributed multi-stage query architecture, which includes a query engine called the multi-stage query task engine (MSQ task engine). The MSQ task engine extends Druid's query capabilities, so you can write queries that reference external data as well as perform ingestion with SQL INSERT and REPLACE. Essentially, you can perform SQL-based ingestion instead of using JSON ingestion specs that Druid's native ingestion uses.</p>
<p dir="auto">SQL-based ingestion using the multi-stage query task engine is the recommended solution starting in Druid 24.0.0. Alternative ingestion solutions such as native batch and Hadoop-based ingestion systems will still be supported. We recommend you read all known issues and test the feature in a development environment before rolling out in production. Using the multi-stage query task engine with <code class="notranslate">SELECT</code> statements that do not write to a datasource is experimental.</p>
<p dir="auto">The extension for it (druid-multi-stage-query) is loaded by default. If you're upgrading from an earlier version of Druid, you'll need to add the extension to druid.extensions.loadlist in your common.runtime.properties file.</p>
<p dir="auto">For more information, see the overview for the multi-stage query architecture.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1236771890" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12524" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12524/hovercard" href="https://github.com/apache/druid/pull/12524">#12524</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1189549778" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12386" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12386/hovercard" href="https://github.com/apache/druid/pull/12386">#12386</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1236672587" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12523" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12523/hovercard" href="https://github.com/apache/druid/pull/12523">#12523</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1255824638" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12589" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12589/hovercard" href="https://github.com/apache/druid/pull/12589">#12589</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-new-features-nested-columns-" href="#24.0.0-new-features-nested-columns-">#</a> Nested columns</h2>
<p dir="auto">Druid now supports directly storing nested data structures in a newly added <code class="notranslate">COMPLEX&lt;json&gt;</code> column type. <code class="notranslate">COMPLEX&lt;json&gt;</code> columns store a copy of the structured data in JSON format as well as specialized internal columns and indexes for nested literal values—<code class="notranslate">STRING</code>, <code class="notranslate">LONG</code>, and <code class="notranslate">DOUBLE</code> types. An optimized virtual column allows Druid to read and filter these values at speeds consistent with standard Druid <code class="notranslate">LONG</code>, <code class="notranslate">DOUBLE</code>, and <code class="notranslate">STRING</code> columns.</p>
<p dir="auto">Newly added Druid SQL, native JSON functions, and virtual column allow you to extract, transform, and create <code class="notranslate">COMPLEX&lt;json&gt;</code> values in at query time. You can also use the JSON functions in <code class="notranslate">INSERT</code> and <code class="notranslate">REPLACE</code> statements in SQL-based ingestion, or in a <code class="notranslate">transformSpec</code> in native ingestion as an alternative to using a <code class="notranslate">flattenSpec</code> object to "flatten" nested data for ingestion.</p>
<p dir="auto">See <a href="docs/querying/sql-json-functions.md">SQL JSON functions</a>, <a href="docs/misc/math-expr.md#json-functions">native JSON functions</a>, <a href="docs/querying/nested-columns.md">Nested columns</a>, <a href="docs/querying/virtual-columns.md#nested-field-virtual-column">virtual columns</a>, and the <a href="https://github.com/apache/druid/issues/12695" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/12695/hovercard">feature summary</a> for more detail.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1296628212" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12753" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12753/hovercard" href="https://github.com/apache/druid/pull/12753">#12753</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1287844223" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12714" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12714/hovercard" href="https://github.com/apache/druid/pull/12714">#12714</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1296628212" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12753" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12753/hovercard" href="https://github.com/apache/druid/pull/12753">#12753</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1342964436" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12920" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12920/hovercard" href="https://github.com/apache/druid/pull/12920">#12920</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-new-features-updated-java-support" href="#24.0.0-new-features-updated-java-support">#</a> Updated Java support</h2>
<p dir="auto">Java 11 is fully supported is no longer experimental. Java 17 support is improved.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1323179155" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12839" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12839/hovercard" href="https://github.com/apache/druid/pull/12839">#12839</a></p>
<h1 dir="auto"><a name="user-content-24.0.0-query-engine-updates" href="#24.0.0-query-engine-updates">#</a> Query engine updates</h1>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-updated-column-indexes-and-query-processing-of-filters" href="#24.0.0-query-engine-updates-updated-column-indexes-and-query-processing-of-filters">#</a> Updated column indexes and query processing of filters</h3>
<p dir="auto">Reworked column indexes to be extraordinarily flexible, which will eventually allow us to model a wide range of index types. Added machinery to build the filters that use the updated indexes, while also allowing for other column implementations to implement the built-in index types to provide adapters to make use indexing in the current set filters that Druid provides.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1190250611" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12388" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12388/hovercard" href="https://github.com/apache/druid/pull/12388">#12388</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-time-filter-operator" href="#24.0.0-query-engine-updates-time-filter-operator">#</a> Time filter operator</h3>
<p dir="auto">You can now use the Druid SQL operator TIME_IN_INTERVAL to filter query results based on time. Prefer TIME_IN_INTERVAL over the SQL BETWEEN operator to filter on time. For more information, see <a href="docs/querying/sql-scalar.md#date-and-time-functions">Date and time functions</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1273137509" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12662" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12662/hovercard" href="https://github.com/apache/druid/pull/12662">#12662</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-null-values-and-the-%22in%22-filter" href="#24.0.0-query-engine-updates-null-values-and-the-%22in%22-filter">#</a> Null values and the "in" filter</h3>
<p dir="auto">If a <code class="notranslate">values</code> array contains <code class="notranslate">null</code>, the "in" filter matches null values. This differs from the SQL IN filter, which does not match null values.</p>
<p dir="auto">For more information, see <a href="/docs/querying/filters.md">Query filters</a> and <a href="/docs/querying/sql-data-types.md">SQL data types</a>.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1329092598" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12863" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12863/hovercard" href="https://github.com/apache/druid/pull/12863">#12863</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-virtual-columns-in-search-queries" href="#24.0.0-query-engine-updates-virtual-columns-in-search-queries">#</a> Virtual columns in search queries</h3>
<p dir="auto">Previously, a <a href="/docs/querying/searchquery.md">search query</a> could only search on dimensions that existed in the data source. Search queries now support <a href="/docs/querying/virtual-columns.md">virtual columns</a> as a parameter in the query.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1289754739" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12720" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12720/hovercard" href="https://github.com/apache/druid/pull/12720">#12720</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-optimize-simple-min-%2f-max-sql-queries-on-__time" href="#24.0.0-query-engine-updates-optimize-simple-min-%2F-max-sql-queries-on-__time">#</a> Optimize simple MIN / MAX SQL queries on __time</h3>
<p dir="auto">Simple queries like <code class="notranslate">select max(__time) from ds</code> now run as a <code class="notranslate">timeBoundary</code> queries to take advantage of the time dimension sorting in a segment. You can set a feature flag to enable this feature.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1212213021" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12472" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12472/hovercard" href="https://github.com/apache/druid/pull/12472">#12472</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1218411439" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12491" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12491/hovercard" href="https://github.com/apache/druid/pull/12491">#12491</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-string-aggregation-results" href="#24.0.0-query-engine-updates-string-aggregation-results">#</a> String aggregation results</h3>
<p dir="auto">The first/last string aggregator now only compares based on values. Previously, the first/last string aggregator’s values were compared based on the <code class="notranslate">_time</code> column first and then on values.</p>
<p dir="auto">If you have existing queries and want to continue using both the <code class="notranslate">_time</code> column and values, update your queries to use ORDER BY MAX(timeCol).</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1303307742" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12773" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12773/hovercard" href="https://github.com/apache/druid/pull/12773">#12773</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-reduced-allocations-due-to-jackson-serialization" href="#24.0.0-query-engine-updates-reduced-allocations-due-to-jackson-serialization">#</a> Reduced allocations due to Jackson serialization</h3>
<p dir="auto">Introduced and implemented new helper functions in <code class="notranslate">JacksonUtils</code> to enable reuse of<br>
<code class="notranslate">SerializerProvider</code> objects.</p>
<p dir="auto">Additionally, disabled backwards compatibility for map-based rows in the <code class="notranslate">GroupByQueryToolChest</code> by default, which eliminates the need to copy the heavyweight <code class="notranslate">ObjectMapper</code>. Introduced a configuration option to allow administrators to explicitly enable backwards compatibility.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1211482815" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12468" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12468/hovercard" href="https://github.com/apache/druid/pull/12468">#12468</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-updated-ipaddress-java-library-" href="#24.0.0-query-engine-updates-updated-ipaddress-java-library-">#</a> Updated IPAddress Java library</h3>
<p dir="auto">Added a new <a href="https://github.com/seancfoley/IPAddress">IPAddress</a> Java library dependency to handle IP addresses. The library includes IPv6 support. Additionally, migrated IPv4 functions to use the new library.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="979460652" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/11634" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/11634/hovercard" href="https://github.com/apache/druid/pull/11634">#11634</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-query-engine-updates-query-performance-improvements" href="#24.0.0-query-engine-updates-query-performance-improvements">#</a> Query performance improvements</h3>
<p dir="auto">Optimized SQL operations and functions as follows:</p>
<ul dir="auto">
<li>Vectorized numeric latest aggregators (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1205360729" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12439" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12439/hovercard" href="https://github.com/apache/druid/pull/12439">#12439</a>)</li>
<li>Optimized <code class="notranslate">isEmpty()</code> and <code class="notranslate">equals()</code> on RangeSets (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1213226111" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12477" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12477/hovercard" href="https://github.com/apache/druid/pull/12477">#12477</a>)</li>
<li>Optimized reuse of Yielder objects (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1212695856" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12475" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12475/hovercard" href="https://github.com/apache/druid/pull/12475">#12475</a>)</li>
<li>Operations on numeric columns with indexes are now faster (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1320361574" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12830" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12830/hovercard" href="https://github.com/apache/druid/pull/12830">#12830</a>)</li>
<li>Optimized GroupBy by reducing allocations. Reduced allocations by reusing entry and key holders (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1212596980" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12474" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12474/hovercard" href="https://github.com/apache/druid/pull/12474">#12474</a>)</li>
<li>Added a vectorized version of string last aggregator (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1220093622" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12493" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12493/hovercard" href="https://github.com/apache/druid/pull/12493">#12493</a>)</li>
<li>Added Direct UTF-8 access for IN filters (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1234652515" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12517" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12517/hovercard" href="https://github.com/apache/druid/pull/12517">#12517</a>)</li>
<li>Enabled virtual columns to cache their outputs in case Druid calls them multiple times on the same underlying row (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1251979306" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12577" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12577/hovercard" href="https://github.com/apache/druid/pull/12577">#12577</a>)</li>
<li>Druid now rewrites a join as a filter when possible in IN joins (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1122084186" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12225" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12225/hovercard" href="https://github.com/apache/druid/pull/12225">#12225</a>)</li>
<li>Added automatic sizing for GroupBy dictionaries (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1299561167" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12763" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12763/hovercard" href="https://github.com/apache/druid/pull/12763">#12763</a>)</li>
<li>Druid now distributes JDBC connections more evenly amongst brokers (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1316509865" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12817" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12817/hovercard" href="https://github.com/apache/druid/pull/12817">#12817</a>)</li>
</ul>
<h2 dir="auto"><a name="user-content-24.0.0-streaming-ingestion" href="#24.0.0-streaming-ingestion">#</a> Streaming ingestion</h2>
<h3 dir="auto"><a name="user-content-24.0.0-streaming-ingestion-kafka-consumers" href="#24.0.0-streaming-ingestion-kafka-consumers">#</a> Kafka consumers</h3>
<p dir="auto">Previously, consumers that were registered and used for ingestion persisted until Kafka deleted them. They were only used to make sure that an entire topic was consumed. There are no longer consumer groups that linger.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1323782021" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12842" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12842/hovercard" href="https://github.com/apache/druid/pull/12842">#12842</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-streaming-ingestion-kinesis-ingestion" href="#24.0.0-streaming-ingestion-kinesis-ingestion">#</a> Kinesis ingestion</h3>
<p dir="auto">You can now perform Kinesis ingestion even if there are empty shards. Previously, all shards had to have at least one record.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1305780684" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12792" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12792/hovercard" href="https://github.com/apache/druid/pull/12792">#12792</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-batch-ingestion" href="#24.0.0-batch-ingestion">#</a> Batch ingestion</h2>
<h3 dir="auto"><a name="user-content-24.0.0-batch-ingestion-batch-ingestion-from-s3" href="#24.0.0-batch-ingestion-batch-ingestion-from-s3">#</a> Batch ingestion from S3</h3>
<p dir="auto">You can now ingest data from endpoints that are different from your default S3 endpoint and signing region.<br>
For more information, see <a href="../development/extensions-core/s3.md#connecting-to-s3-configuration">S3 config</a>.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1025792140" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/11798" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/11798/hovercard" href="https://github.com/apache/druid/pull/11798">#11798</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-improvements-to-ingestion-in-general" href="#24.0.0-improvements-to-ingestion-in-general">#</a> Improvements to ingestion in general</h2>
<p dir="auto">This release includes the following  improvements for ingestion in general.</p>
<h3 dir="auto"><a name="user-content-24.0.0-improvements-to-ingestion-in-general-increased-robustness-for-task-management" href="#24.0.0-improvements-to-ingestion-in-general-increased-robustness-for-task-management">#</a> Increased robustness for task management</h3>
<p dir="auto">Added <code class="notranslate">setNumProcessorsPerTask</code> to prevent various automatically-sized thread pools from becoming unreasonably large. It isn't ideal for each task to size its pools as if it is the only process on the entire machine. On large machines, this solves a common cause of <code class="notranslate">OutOfMemoryError</code> due to "unable to create native thread".</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1256365896" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12592" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12592/hovercard" href="https://github.com/apache/druid/pull/12592">#12592</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-improvements-to-ingestion-in-general-avatica-jdbc-driver" href="#24.0.0-improvements-to-ingestion-in-general-avatica-jdbc-driver">#</a> Avatica JDBC driver</h3>
<p dir="auto">The JDBC driver now follows the JDBC standard and uses two kinds of statements, Statement and PreparedStatement.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1284829130" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12709" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12709/hovercard" href="https://github.com/apache/druid/pull/12709">#12709</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-improvements-to-ingestion-in-general-eight-hour-granularity" href="#24.0.0-improvements-to-ingestion-in-general-eight-hour-granularity">#</a> Eight hour granularity</h3>
<p dir="auto">Druid now accepts the <code class="notranslate">EIGHT_HOUR</code> granularity. You can segment incoming data to <code class="notranslate">EIGHT_HOUR</code> buckets as well as group query results by eight hour granularity.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1289324454" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12717" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12717/hovercard" href="https://github.com/apache/druid/pull/12717">#12717</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-ingestion-general" href="#24.0.0-ingestion-general">#</a> Ingestion general</h2>
<h3 dir="auto"><a name="user-content-24.0.0-ingestion-general-updated-avro-extension" href="#24.0.0-ingestion-general-updated-avro-extension">#</a> Updated Avro extension</h3>
<p dir="auto">The previous Avro extension leaked objects from the parser. If these objects leaked into your ingestion, you had objects being stored as a string column with the value as the .toString(). This string column will remain after you upgrade but will return <code class="notranslate">Map.toString()</code> instead of <code class="notranslate">GenericRecord.toString</code>. If you relied on the previous behavior, you can use the Avro extension from an earlier release.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1319676962" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12828" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12828/hovercard" href="https://github.com/apache/druid/pull/12828">#12828</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-ingestion-general-sampler-api" href="#24.0.0-ingestion-general-sampler-api">#</a> Sampler API</h3>
<p dir="auto">The sampler API has additional limits: <code class="notranslate">maxBytesInMemory</code> and <code class="notranslate">maxClientResponseBytes</code>. These options augment the existing options <code class="notranslate">numRows</code> and <code class="notranslate">timeoutMs</code>.  <code class="notranslate">maxBytesInMemory</code> can be used to control the memory usage on the Overlord while sampling.  <code class="notranslate">maxClientResponseBytes</code> can be used by clients to specify the maximum size of response they would prefer to handle.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1347809718" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12947" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12947/hovercard" href="https://github.com/apache/druid/pull/12947">#12947</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-sql" href="#24.0.0-sql">#</a> SQL</h2>
<h3 dir="auto"><a name="user-content-24.0.0-sql-column-order" href="#24.0.0-sql-column-order">#</a> Column order</h3>
<p dir="auto">The <code class="notranslate">DruidSchema</code> and <code class="notranslate">SegmentMetadataQuery</code> properties now preserve column order instead of ordering columns alphabetically. This means that query order better matches ingestion order.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1296712330" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12754" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12754/hovercard" href="https://github.com/apache/druid/pull/12754">#12754</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-sql-converting-joins-to-filter" href="#24.0.0-sql-converting-joins-to-filter">#</a> Converting JOINs to filter</h3>
<p dir="auto">You can improve performance by pushing JOINs partially or fully to the base table as a filter at runtime by setting the <code class="notranslate">enableRewriteJoinToFilter</code> context parameter to <code class="notranslate">true</code> for a query.</p>
<p dir="auto">Druid now pushes down join filters in case the query computing join references any columns from the right side.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1295910535" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12749" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12749/hovercard" href="https://github.com/apache/druid/pull/12749">#12749</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1329692382" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12868" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12868/hovercard" href="https://github.com/apache/druid/pull/12868">#12868</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-sql-add-is_active-to-sys.segments" href="#24.0.0-sql-add-is_active-to-sys.segments">#</a> Add is_active to sys.segments</h3>
<p dir="auto">Added <code class="notranslate">is_active</code> as shorthand for <code class="notranslate">(is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1)</code>. This represents "all the segments that should be queryable, whether or not they actually are right now".</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="961092430" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/11550" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/11550/hovercard" href="https://github.com/apache/druid/pull/11550">#11550</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-sql-%60usenativequeryexplain%60-now-defaults-to-true" href="#24.0.0-sql-%60usenativequeryexplain%60-now-defaults-to-true">#</a> <code class="notranslate">useNativeQueryExplain</code> now defaults to true</h3>
<p dir="auto">The <code class="notranslate">useNativeQueryExplain</code> property now defaults to <code class="notranslate">true</code>. This means that EXPLAIN PLAN FOR returns the explain plan as a JSON representation of equivalent native query(s) by default. For more information, see <a href="/docs/configuration/index.md">Broker Generated Query Configuration Supplementation</a>.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1346470370" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12936" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12936/hovercard" href="https://github.com/apache/druid/pull/12936">#12936</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-inline-query-engine" href="#24.0.0-inline-query-engine">#</a> Running queries with inline data using druid query engine</h3>
<p dir="auto">Some queries  that do not refer to any table, such as <code class="notranslate">select 1</code>, are now always translated to a native Druid query with <code class="notranslate">InlineDataSource</code> before execution. If translation is not possible, for queries such as <code class="notranslate">SELECT (1, 2)</code>, then an error occurs. In earlier versions, this query would still run.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1337007651" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12897" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12897/hovercard" href="https://github.com/apache/druid/pull/12897">#12897</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord" href="#24.0.0-coordinator%2Foverlord">#</a> Coordinator/Overlord</h2>
<h3 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord-you-can-configure-the-coordinator-to-kill-segments-in-the-future" href="#24.0.0-coordinator%2Foverlord-you-can-configure-the-coordinator-to-kill-segments-in-the-future">#</a> You can configure the Coordinator to kill segments in the future</h3>
<p dir="auto">You can now set <code class="notranslate">druid.coordinator.kill.durationToRetain</code> to a negative period to configure the Druid cluster to kill segments whose <code class="notranslate">interval_end</code> is a date in the future. For example, PT-24H would allow segments to be killed if their interval_end date was 24 hours or less into the future at the time that the kill task is generated by the system.<br>
A cluster operator can also disregard the <code class="notranslate">druid.coordinator.kill.durationToRetain</code> entirely by setting a new configuration, <code class="notranslate">druid.coordinator.kill.ignoreDurationToRetain=true</code>. This ignores <code class="notranslate">interval_end</code> date when looking for segments to kill, and can instead kill any segment marked unused. This new configuration is turned off by default, and a cluster operator should fully understand and accept the risks before enabling it.</p>
<h3 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord-improved-overlord-stability" href="#24.0.0-coordinator%2Foverlord-improved-overlord-stability">#</a> Improved Overlord stability</h3>
<p dir="auto">Reduced contention between the management thread and the reception of status updates from the cluster. This improves the stability of Overlord and all tasks in a cluster when there are large (1000+) task counts.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1089495362" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12099" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12099/hovercard" href="https://github.com/apache/druid/pull/12099">#12099</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord-improved-coordinator-segment-logging" href="#24.0.0-coordinator%2Foverlord-improved-coordinator-segment-logging">#</a> Improved Coordinator segment logging</h3>
<p dir="auto">Updated Coordinator load rule logging to include current replication levels. Added missing segment ID and tier information from some of the log messages.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1232406118" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12511" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12511/hovercard" href="https://github.com/apache/druid/pull/12511">#12511</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord-optimized-overlord-get-tasks-memory-usage" href="#24.0.0-coordinator%2Foverlord-optimized-overlord-get-tasks-memory-usage">#</a> Optimized overlord GET tasks memory usage</h3>
<p dir="auto">Addressed the significant memory overhead caused by the web-console indirectly calling the Overlord’s GET tasks API. This could cause unresponsiveness or Overlord failure when the ingestion tab was opened multiple times.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1194698511" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12404" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12404/hovercard" href="https://github.com/apache/druid/pull/12404">#12404</a></p>
<h3 dir="auto"><a name="user-content-24.0.0-coordinator%2foverlord-reduced-time-to-create-intervals" href="#24.0.0-coordinator%2Foverlord-reduced-time-to-create-intervals">#</a> Reduced time to create intervals</h3>
<p dir="auto">In order to optimize segment cost computation time by reducing time taken for interval creation, store segment interval instead of creating it each time from primitives and reduce memory overhead of storing intervals by interning them. The set of intervals for segments is low in cardinality.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1274953108" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12670" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12670/hovercard" href="https://github.com/apache/druid/pull/12670">#12670</a></p>
<h2 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord" href="#24.0.0--brokers%2Foverlord">#</a>  Brokers/Overlord</h2>
<p dir="auto">Brokers now have a default of 25MB maximum queued per query. Previously, there was no default limit. Depending on your use case, you may need to increase the value, especially if you have large result sets or large amounts of intermediate data. To adjust the maximum memory available, use the <code class="notranslate">druid.broker.http.maxQueuedBytes</code> property.<br>
For more information, see <a href="docs/configuration/index.md">Configuration reference</a>.</p>
<h2 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console" href="#24.0.0--brokers%2Foverlord-web-console">#</a> Web console</h2>
<blockquote>
<p dir="auto">Prepare to have your Web Console experience elevated! - <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vogievetsky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vogievetsky">@vogievetsky</a></p>
</blockquote>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support">#</a> New query view (WorkbenchView) with tabs and long running query support</h3>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309077-8840ff85-19a9-4fc0-8398-4f2446ff29b3.png"><img src="https://user-images.githubusercontent.com/177816/185309077-8840ff85-19a9-4fc0-8398-4f2446ff29b3.png" alt="" style="max-width: 100%;"></a></p>
<p dir="auto">You can use the new query view to execute multi-stage, task based, queries with the /druid/v2/sql/task and /druid/indexer/v1/task/* APIs as well as native and sql-native queries just like the old Query view. A key point of the sql-msq-task based queries is that they may run for a long time. This inspired / necessitated many UX changes including, but not limited to the following:</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-tabs" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-tabs">#</a> Tabs</h4>
<p dir="auto">You can now have many queries stored and running at the same time, significantly improving the query view UX.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309114-fe82cccd-a917-415c-a394-a3485403226d.png"><img src="https://user-images.githubusercontent.com/177816/185309114-fe82cccd-a917-415c-a394-a3485403226d.png" alt="" style="max-width: 100%;"></a></p>
<p dir="auto">You can open several tabs, duplicate them, and copy them as text to paste into any console and reopen there.</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-progress-reports-%28counter-reports%29" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-progress-reports-%28counter-reports%29">#</a> Progress reports (counter reports)</h4>
<p dir="auto">Queries run with the multi-stage query task engine have detailed progress reports shown in the summary progress bar and the in detail execution table that provides summaries of the counters for every step.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309244-cba3d640-c48a-49bd-8c72-ed3e842b0cb2.png"><img src="https://user-images.githubusercontent.com/177816/185309244-cba3d640-c48a-49bd-8c72-ed3e842b0cb2.png" alt="" style="max-width: 100%;"></a></p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-error-and-warning-reports" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-error-and-warning-reports">#</a> Error and warning reports</h4>
<p dir="auto">Queries run with the multi-stage query task engine present user friendly warnings and errors should anything go wrong.<br>
The new query view has components to visualize these with their full detail including a stack-trace.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309488-421e7410-251d-4590-88f1-92fb23f4be13.png"><img src="https://user-images.githubusercontent.com/177816/185309488-421e7410-251d-4590-88f1-92fb23f4be13.png" alt="" style="max-width: 100%;"></a></p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-recent-query-tasks-panel" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-recent-query-tasks-panel">#</a> Recent query tasks panel</h4>
<p dir="auto">Queries run with the multi-stage query task engine are tasks. This makes it possible to show queries that are executing currently and that have executed in the recent past.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309579-e2ce021c-6bf6-4576-bdee-ff552e1c4c3b.png"><img src="https://user-images.githubusercontent.com/177816/185309579-e2ce021c-6bf6-4576-bdee-ff552e1c4c3b.png" alt="" style="max-width: 100%;"></a></p>
<p dir="auto">For any query in the Recent query tasks  panel you can view the execution details for it and you can also attach it as a new tab and continue iterating on the query. It is also possible to download the "query detail archive", a JSON file containing all the important details for a given query to use  for troubleshooting.</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-connect-external-data-flow" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-connect-external-data-flow">#</a> Connect external data flow</h4>
<p dir="auto">Connect external data flow lets you use the sampler to sample your source data to, determine its schema and generate a fully formed SQL query that you can edit to fit your use case before you launch your ingestion job. This point-and-click flow will save you much  typing.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/177816/185309631-5ceed7d0-2bb2-43b9-83ed-fdcad4a152be.png"><img src="https://user-images.githubusercontent.com/177816/185309631-5ceed7d0-2bb2-43b9-83ed-fdcad4a152be.png" alt="" style="max-width: 100%;"></a></p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-preview-button" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-preview-button">#</a> Preview button</h4>
<p dir="auto">The Preview button appears when you type in an INSERT or REPLACE SQL query. Click the button to remove the INSERT or REPLACE clause and  execute your query as an "inline" query with a limi). This gives you a sense of the shape of your data after Druid applies all your transformations from your SQL query.</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-results-table" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-results-table">#</a> Results table</h4>
<p dir="auto">The query results table has been improved in style and function. It now shows you type icons for the column types and supports the ability to manipulate nested columns with ease.</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-helper-queries" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-helper-queries">#</a> Helper queries</h4>
<p dir="auto">The Web Console now has some UI affordances for notebook and CTE users. You can reference helper queries, collapsable elements that hold a query, from the main query just like they were defined with a WITH statement. When you are composing a complicated query, it is helpful to break it down into multiple queries to preview the parts individually.</p>
<h4 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-additional-web-console-tools" href="#24.0.0--brokers%2Foverlord-web-console-new-query-view-%28workbenchview%29-with-tabs-and-long-running-query-support-additional-web-console-tools">#</a> Additional Web Console tools</h4>
<p dir="auto">More tools are available from the  ... menu:</p>
<ul dir="auto">
<li>Explain query - show the query plan for  sql-native and multi-stage query task engine queries.</li>
<li>Convert ingestion spec to SQL - Helps you migrate your native batch and Hadoop based specs to the SQL-based format.</li>
<li>Open query detail archive - lets you open a query detail archive downloaded earlier.</li>
<li>Load demo queries - lets you load a set of pre-made queries to play around with multi-stage query task engine functionality.</li>
</ul>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-new-sql-based-data-loader" href="#24.0.0--brokers%2Foverlord-web-console-new-sql-based-data-loader">#</a> New SQL-based data loader</h3>
<p dir="auto">The data loader exists as a GUI wizard to help users craft a JSON ingestion spec using point and click and quick previews. The SQL data loader is the SQL-based ingestion analog of that.</p>
<p dir="auto">Like the native based data loader, the SQL-based data loader stores all the state in the SQL query itself. You can opt to manipulate the query directly at any stage. See (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1342627344" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12919" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12919/hovercard" href="https://github.com/apache/druid/pull/12919">#12919</a>) for more information about how the data loader differs from the <strong>Connect external data</strong> workflow.</p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-web-console-other-changes-and-improvements" href="#24.0.0--brokers%2Foverlord-web-console-other-changes-and-improvements">#</a> Other changes and improvements</h3>
<ul dir="auto">
<li>The query view has so much new functionality that it has moved to the far left as the first view available in the header.</li>
<li>You can now click on a datasource or segment to see a preview of the data within.</li>
<li>The task table now explicitly shows if a task has been canceled in a different color than a failed task.</li>
<li>The user experience when you view a JSON payload in the Druid console has been improved. There’s now syntax highlighting and a search.</li>
<li>The Druid console can now use the column order returned by a scan query to determine the column order for reindexing data.</li>
<li>The way errors are displayed in the Druid console has been improved. Errors no longer appear as a single long line.</li>
</ul>
<p dir="auto">See  (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1342627344" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12919" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12919/hovercard" href="https://github.com/apache/druid/pull/12919">#12919</a>) for more details and other improvements</p>
<h2 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics" href="#24.0.0--brokers%2Foverlord-metrics">#</a> Metrics</h2>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-sysmonitor-stats-for-peons" href="#24.0.0--brokers%2Foverlord-metrics-sysmonitor-stats-for-peons">#</a> Sysmonitor stats for Peons</h3>
<p dir="auto">Sysmonitor stats, like memory or swap, are no longer reported since Peons always run on the same host as MiddleManagerse. This means that duplicate stats will no longer be reported.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1309337591" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12802" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12802/hovercard" href="https://github.com/apache/druid/pull/12802">#12802</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-prometheus-" href="#24.0.0--brokers%2Foverlord-metrics-prometheus-">#</a> Prometheus</h3>
<p dir="auto">You can now include the host and service as labels for Prometheus by setting the following properties to true:</p>
<ul dir="auto">
<li><code class="notranslate">druid.emitter.prometheus.addHostAsLabel</code></li>
<li><code class="notranslate">druid.emitter.prometheus.addServiceAsLabel</code></li>
<li></li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1300524227" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12769" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12769/hovercard" href="https://github.com/apache/druid/pull/12769">#12769</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-rows-per-segment" href="#24.0.0--brokers%2Foverlord-metrics-rows-per-segment">#</a> Rows per segment</h3>
<p dir="auto">(Experimental) You can now see the average number of rows in a segment and the distribution of segments in predefined buckets with the following metrics: <code class="notranslate">segment/rowCount/avg</code> and <code class="notranslate">segment/rowCount/range/count</code>.<br>
Enable the metrics with the following property: <code class="notranslate">org.apache.druid.server.metrics.SegmentStatsMonitor</code><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1291706327" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12730" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12730/hovercard" href="https://github.com/apache/druid/pull/12730">#12730</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-new-%60sqlquery%2fplanningtimems%60-metric" href="#24.0.0--brokers%2Foverlord-metrics-new-%60sqlquery%2Fplanningtimems%60-metric">#</a> New <code class="notranslate">sqlQuery/planningTimeMs</code> metric</h3>
<p dir="auto">There’s a new <code class="notranslate">sqlQuery/planningTimeMs</code> metric for SQL queries that computes the time it takes to build a native query from a SQL query.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1343713350" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12923" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12923/hovercard" href="https://github.com/apache/druid/pull/12923">#12923</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-statsd-metrics-reporter" href="#24.0.0--brokers%2Foverlord-metrics-statsd-metrics-reporter">#</a> StatsD metrics reporter</h3>
<p dir="auto">The StatsD metrics reporter extension now includes the following metrics:</p>
<ul dir="auto">
<li>coordinator/time</li>
<li>coordinator/global/time</li>
<li>tier/required/capacity</li>
<li>tier/total/capacity</li>
<li>tier/replication/factor</li>
<li>tier/historical/count</li>
<li>compact/task/count</li>
<li>compactTask/maxSlot/count</li>
<li>compactTask/availableSlot/count</li>
<li>segment/waitCompact/bytes</li>
<li>segment/waitCompact/count</li>
<li>interval/waitCompact/count</li>
<li>segment/skipCompact/bytes</li>
<li>segment/skipCompact/count</li>
<li>interval/skipCompact/count</li>
<li>segment/compacted/bytes</li>
<li>segment/compacted/count</li>
<li>interval/compacted/count<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1299553392" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12762" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12762/hovercard" href="https://github.com/apache/druid/pull/12762">#12762</a></li>
</ul>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-new-worker-level-task-metrics" href="#24.0.0--brokers%2Foverlord-metrics-new-worker-level-task-metrics">#</a> New worker level task metrics</h3>
<p dir="auto">Added a new monitor, <code class="notranslate">WorkerTaskCountStatsMonitor</code>, that allows each middle manage worker to report metrics for successful / failed tasks, and task slot usage.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1206749338" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12446" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12446/hovercard" href="https://github.com/apache/druid/pull/12446">#12446</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-improvements-to-the-jvmmonitor" href="#24.0.0--brokers%2Foverlord-metrics-improvements-to-the-jvmmonitor">#</a> Improvements to the JvmMonitor</h3>
<p dir="auto">The JvmMonitor can now handle more generation and collector scenarios. The monitor is more robust and works properly for ZGC on both Java 11 and 15.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1211484579" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12469" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12469/hovercard" href="https://github.com/apache/druid/pull/12469">#12469</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-garbage-collection" href="#24.0.0--brokers%2Foverlord-metrics-garbage-collection">#</a> Garbage collection</h3>
<p dir="auto">Garbage collection metrics now use MXBeans.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1214379882" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12481" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12481/hovercard" href="https://github.com/apache/druid/pull/12481">#12481</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-metric-for-task-duration-in-the-pending-queue" href="#24.0.0--brokers%2Foverlord-metrics-metric-for-task-duration-in-the-pending-queue">#</a> Metric for task duration in the pending queue</h3>
<p dir="auto">Introduced the metric <code class="notranslate">task/pending/time</code> to measure how long a task stays in the pending queue.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1219462241" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12492" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12492/hovercard" href="https://github.com/apache/druid/pull/12492">#12492</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-emit-metrics-object-for-scan%2c-timeseries%2c-and-groupby-queries-during-cursor-creation" href="#24.0.0--brokers%2Foverlord-metrics-emit-metrics-object-for-scan%2C-timeseries%2C-and-groupby-queries-during-cursor-creation">#</a> Emit metrics object for Scan, Timeseries, and GroupBy queries during cursor creation</h3>
<p dir="auto">Adds vectorized metric for scan, timeseries and groupby queries.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1215624959" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12484" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12484/hovercard" href="https://github.com/apache/druid/pull/12484">#12484</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-emit-state-of-replace-and-append-for-native-batch-tasks" href="#24.0.0--brokers%2Foverlord-metrics-emit-state-of-replace-and-append-for-native-batch-tasks">#</a> Emit state of replace and append for native batch tasks</h3>
<p dir="auto">Druid now emits metrics so you can monitor and assess the use of  different types of batch ingestion, in particular replace and tombstone creation.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1218106564" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12488" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12488/hovercard" href="https://github.com/apache/druid/pull/12488">#12488</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1323574408" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12840" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12840/hovercard" href="https://github.com/apache/druid/pull/12840">#12840</a></p>
<h3 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-metrics-kafkaemitter-emits-%60querytype%60" href="#24.0.0--brokers%2Foverlord-metrics-kafkaemitter-emits-%60querytype%60">#</a> KafkaEmitter emits <code class="notranslate">queryType</code></h3>
<p dir="auto">The KafkaEmitter now properly emits the <code class="notranslate">queryType</code> property for native queries.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1341820467" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12915" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12915/hovercard" href="https://github.com/apache/druid/pull/12915">#12915</a></p>
<h2 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-security" href="#24.0.0--brokers%2Foverlord-security">#</a> Security</h2>
<p dir="auto">You can now hide properties that are sensitive in the API response from <code class="notranslate">/status/properties</code>, such as S3 access keys. Use the <code class="notranslate">druid.server.hiddenProperties</code> property in <code class="notranslate">common.runtime.properties</code> to specify the properties (case insensitive) you want to hide.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1348473346" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12950" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12950/hovercard" href="https://github.com/apache/druid/pull/12950">#12950</a></p>
<h2 dir="auto"><a name="user-content-24.0.0--brokers%2foverlord-other-changes" href="#24.0.0--brokers%2Foverlord-other-changes">#</a> Other changes</h2>
<ul dir="auto">
<li>You can now configure the retention period for request logs stored on disk with the  <code class="notranslate">druid.request.logging.durationToRetain</code> property. Set the retention period to be longer than <code class="notranslate">P1D</code> (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1244467642" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12559" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12559/hovercard" href="https://github.com/apache/druid/pull/12559">#12559</a>)</li>
<li>You can now specify liveness and readiness probe delays for the historical StatefulSet in your values.yaml file. The default is 60 seconds (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1310370387" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12805" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12805/hovercard" href="https://github.com/apache/druid/pull/12805">#12805</a>)</li>
<li>Improved exception message for native binary operators (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1169781450" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12335" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12335/hovercard" href="https://github.com/apache/druid/pull/12335">#12335</a>)</li>
<li>​​Improved error messages when URI points to a file that doesn't exist (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1218306969" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12490" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12490/hovercard" href="https://github.com/apache/druid/pull/12490">#12490</a>)</li>
<li>​​Improved build performance of modules (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1216689779" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12486" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12486/hovercard" href="https://github.com/apache/druid/pull/12486">#12486</a>)</li>
<li>Improved lookups made using the druid-kafka-extraction-namespace extension to handle records that have been deleted from a kafka topic (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1317720242" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12819" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12819/hovercard" href="https://github.com/apache/druid/pull/12819">#12819</a>)</li>
<li>Updated core Apache Kafka dependencies to 3.2.0 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1240594543" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12538" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12538/hovercard" href="https://github.com/apache/druid/pull/12538">#12538</a>)</li>
<li>Updated ORC to 1.7.5 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1274556347" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12667" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12667/hovercard" href="https://github.com/apache/druid/pull/12667">#12667</a>)</li>
<li>Updated Jetty to 9.4.41.v20210516 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1267390528" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12629" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12629/hovercard" href="https://github.com/apache/druid/pull/12629">#12629</a>)</li>
<li>Added <code class="notranslate">Zstandard</code> compression library to <code class="notranslate">CompressionStrategy</code> (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1195275557" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12408" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12408/hovercard" href="https://github.com/apache/druid/pull/12408">#12408</a>)</li>
<li>Updated the default gzip buffer size to 8 KB to for improved performance (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1251980114" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12579" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12579/hovercard" href="https://github.com/apache/druid/pull/12579">#12579</a>)</li>
<li>Updated the default <code class="notranslate">inputSegmentSizeBytes</code> in Compaction configuration to 100,000,000,000,000 (~100TB)</li>
</ul>
<h1 dir="auto"><a name="user-content-24.0.0-bug-fixes" href="#24.0.0-bug-fixes">#</a> Bug fixes</h1>
<p dir="auto">Druid 24.0 contains over 68 bug fixes. You can find the complete list <a href="https://github.com/apache/druid/issues?q=is%3Aclosed+label%3ABug+milestone%3A24.0.0+">here</a></p>
<h1 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0" href="#24.0.0-upgrading-to-24.0">#</a> Upgrading to 24.0</h1>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-permissions-for-multi-stage-query-engine" href="#24.0.0-upgrading-to-24.0-permissions-for-multi-stage-query-engine">#</a> Permissions for multi-stage query engine</h2>
<p dir="auto">To read external data using the multi-stage query task engine, you must have READ permissions for the <a href="/docs/operations/security-user-auth.md">EXTERNAL resource type</a>. Users without the correct permission encounter a 403 error when trying to run SQL queries that include EXTERN.</p>
<p dir="auto">The way you assign the permission depends on your authorizer. For example, with [basic security]((/docs/development/extensions-core/druid-basic-security.md) in Druid, add the <code class="notranslate">EXTERNAL READ</code> permission by sending a <code class="notranslate">POST</code> request to the <a href="/docs/development/extensions-core/druid-basic-security.md#permissions">roles API</a>.</p>
<p dir="auto">The example adds permissions for users with the <code class="notranslate">admin</code> role using a basic authorizer named <code class="notranslate">MyBasicMetadataAuthorizer</code>. The following permissions are granted:</p>
<ul dir="auto">
<li>DATASOURCE READ</li>
<li>DATASOURCE WRITE</li>
<li>CONFIG READ</li>
<li>CONFIG WRITE</li>
<li>STATE READ</li>
<li>STATE WRITE</li>
<li>EXTERNAL READ</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --location --request POST 'http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/roles/admin/permissions' \
--header 'Content-Type: application/json' \
--data-raw '[
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;DATASOURCE&quot;
  },
  &quot;action&quot;: &quot;READ&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;DATASOURCE&quot;
  },
  &quot;action&quot;: &quot;WRITE&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;CONFIG&quot;
  },
  &quot;action&quot;: &quot;READ&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;CONFIG&quot;
  },
  &quot;action&quot;: &quot;WRITE&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;STATE&quot;
  },
  &quot;action&quot;: &quot;READ&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;.*&quot;,
    &quot;type&quot;: &quot;STATE&quot;
  },
  &quot;action&quot;: &quot;WRITE&quot;
},
{
  &quot;resource&quot;: {
    &quot;name&quot;: &quot;EXTERNAL&quot;,
    &quot;type&quot;: &quot;EXTERNAL&quot;
  },
  &quot;action&quot;: &quot;READ&quot;
}
]'"><pre class="notranslate"><code class="notranslate">curl --location --request POST 'http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/roles/admin/permissions' \
--header 'Content-Type: application/json' \
--data-raw '[
{
  "resource": {
    "name": ".*",
    "type": "DATASOURCE"
  },
  "action": "READ"
},
{
  "resource": {
    "name": ".*",
    "type": "DATASOURCE"
  },
  "action": "WRITE"
},
{
  "resource": {
    "name": ".*",
    "type": "CONFIG"
  },
  "action": "READ"
},
{
  "resource": {
    "name": ".*",
    "type": "CONFIG"
  },
  "action": "WRITE"
},
{
  "resource": {
    "name": ".*",
    "type": "STATE"
  },
  "action": "READ"
},
{
  "resource": {
    "name": ".*",
    "type": "STATE"
  },
  "action": "WRITE"
},
{
  "resource": {
    "name": "EXTERNAL",
    "type": "EXTERNAL"
  },
  "action": "READ"
}
]'
</code></pre></div>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-behavior-for-unused-segments" href="#24.0.0-upgrading-to-24.0-behavior-for-unused-segments">#</a> Behavior for unused segments</h2>
<p dir="auto">Druid automatically retains any segments marked as unused. Previously, Druid permanently deleted unused segments from metadata store and deep storage after their duration to retain passed. This behavior was reverted from <code class="notranslate">0.23.0</code>.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1280786117" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12693" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12693/hovercard" href="https://github.com/apache/druid/pull/12693">#12693</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-default-for-%60druid.processing.fifo%60" href="#24.0.0-upgrading-to-24.0-default-for-%60druid.processing.fifo%60">#</a> Default for <code class="notranslate">druid.processing.fifo</code></h2>
<p dir="auto">The default for <code class="notranslate">druid.processing.fifo</code> is now true. This means that tasks of equal priority are treated in a FIFO manner. For most use cases, this change can improve performance on heavily loaded clusters.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1249128095" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12571" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12571/hovercard" href="https://github.com/apache/druid/pull/12571">#12571</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-update-to-jdbc-statement-closure" href="#24.0.0-upgrading-to-24.0-update-to-jdbc-statement-closure">#</a> Update to JDBC statement closure</h2>
<p dir="auto">In previous releases, Druid automatically closed the JDBC Statement when the ResultSet was closed. Druid closed the ResultSet on EOF. Druid closed the statement on any exception. This behavior is, however, non-standard.<br>
In this release, Druid's JDBC driver follows the JDBC standards more closely:<br>
The ResultSet closes automatically on EOF, but does not close the Statement or PreparedStatement. Your code must close these statements, perhaps by using a try-with-resources block.<br>
The PreparedStatement can now be used multiple times with different parameters. (Previously this was not true since closing the ResultSet closed the PreparedStatement.)<br>
If any call to a Statement or PreparedStatement raises an error, the client code must still explicitly close the statement. According to the JDBC standards, statements are not closed automatically on errors. This allows you to obtain information about a failed statement before closing it.<br>
If you have code that depended on the old behavior, you may have to change your code to add the required close statement.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1284829130" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/12709" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/12709/hovercard" href="https://github.com/apache/druid/pull/12709">#12709</a></p>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-known-issues" href="#24.0.0-upgrading-to-24.0-known-issues">#</a> Known issues</h2>
<h2 dir="auto"><a name="user-content-24.0.0-upgrading-to-24.0-credit" href="#24.0.0-upgrading-to-24.0-credit">#</a> Credits</h2>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/2bethere/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/2bethere">@2bethere</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/317brian/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/317brian">@317brian</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/a2l007/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/a2l007">@a2l007</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhagraw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhagraw">@abhagraw</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhishekagarwal87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhishekagarwal87">@abhishekagarwal87</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abhishekrb19/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abhishekrb19">@abhishekrb19</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/adarshsanjeev/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/adarshsanjeev">@adarshsanjeev</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aggarwalakshay/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aggarwalakshay">@aggarwalakshay</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/AmatyaAvadhanula/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/AmatyaAvadhanula">@AmatyaAvadhanula</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/BartMiki/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/BartMiki">@BartMiki</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/capistrant/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/capistrant">@capistrant</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/chenrui333/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/chenrui333">@chenrui333</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/churromorales/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/churromorales">@churromorales</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/clintropolis/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/clintropolis">@clintropolis</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cloventt/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cloventt">@cloventt</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/CodingParsley/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/CodingParsley">@CodingParsley</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cryptoe/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cryptoe">@cryptoe</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dampcake/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dampcake">@dampcake</a><br>
<a class="user-mention notranslate" data-hovercard-type="organization" data-hovercard-url="/orgs/dependabot/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dependabot">@dependabot</a>[bot]<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dherg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dherg">@dherg</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/didip/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/didip">@didip</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dongjoon-hyun/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dongjoon-hyun">@dongjoon-hyun</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ektravel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ektravel">@ektravel</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/EsoragotoSpirit/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/EsoragotoSpirit">@EsoragotoSpirit</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/exherb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/exherb">@exherb</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/FrankChen021/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/FrankChen021">@FrankChen021</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/hellmarbecker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hellmarbecker">@hellmarbecker</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/hwball/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hwball">@hwball</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/iandr413/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/iandr413">@iandr413</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/imply-cheddar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/imply-cheddar">@imply-cheddar</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jarnoux/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jarnoux">@jarnoux</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jasonk000/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jasonk000">@jasonk000</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jihoonson/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jihoonson">@jihoonson</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jon-wei/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jon-wei">@jon-wei</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/kfaraz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/kfaraz">@kfaraz</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LakshSingla/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LakshSingla">@LakshSingla</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/liujianhuanzz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/liujianhuanzz">@liujianhuanzz</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/liuxiaohui1221/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/liuxiaohui1221">@liuxiaohui1221</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lmsurpre/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lmsurpre">@lmsurpre</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/loquisgon/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/loquisgon">@loquisgon</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/machine424/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/machine424">@machine424</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/maytasm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/maytasm">@maytasm</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/MC-JY/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/MC-JY">@MC-JY</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Mihaylov93/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Mihaylov93">@Mihaylov93</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nishantmonu51/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nishantmonu51">@nishantmonu51</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/paul-rogers/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/paul-rogers">@paul-rogers</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/petermarshallio/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/petermarshallio">@petermarshallio</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pjfanning/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pjfanning">@pjfanning</a><br>
@rockc2020<br>
@rohangarg<br>
@somu-imply<br>
@suneet-s<br>
@superivaj<br>
@techdocsmith<br>
@tejaswini-imply<br>
@TSFenwick<br>
@vimil-saju<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vogievetsky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vogievetsky">@vogievetsky</a><br>
@vtlim<br>
@williamhyun<br>
@wiquan<br>
@writer-jill<br>
@xvrl<br>
@yuanlihan<br>
@zachjsh<br>
@zemin-piao</p>