<p dir="auto">It is often a pain to track down dependency problems, especially transient dependencies. It is helpful if Druid only ever include a jar ONCE if possible, and not shade it so that the exact version can be tracked. Unfortunately we seem to have missed a few duplicates, or they are in extensions and might be better served to be brought into core somehow.</p>
<p dir="auto">here is a handy little shell command I use to help find duplicates. I ran it against master at <code class="notranslate">e1033bb412244fc9dff9dea1c89d43a8e0081fd3</code> (today):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ TARBALL=&quot;/Users/charles.allen/src/druid/distribution/target/apache-druid-0.13.0-incubating-SNAPSHOT-bin.tar.gz&quot;; cat &lt;(tar -t  --include='*/lib/*.jar' -f &quot;${TARBALL}&quot; | cut -d / -f 3 ) &lt;(tar -t  --include='*/extensions/*.jar' -f &quot;${TARBALL}&quot; | cut -d / -f 4 ) | sed -e 's/-[0-9].*\.jar//' | sort | uniq -c | sort -n -r --stable
   5 snappy-java
   4 commons-beanutils
   4 commons-collections
   4 commons-compress
   4 gson
   4 objenesis
   4 slf4j-api
   3 commons-codec
   3 commons-digester
   3 commons-lang
   3 commons-logging
   3 commons-math3
   3 curator-framework
   3 curator-recipes
   3 guava
   3 jetty-util
   3 json-smart
   3 kafka-clients
   3 metrics-core
   3 servlet-api
   2 apacheds-i18n
   2 apacheds-kerberos-codec
   2 api-asn1-api
   2 api-util
   2 aws-java-sdk-core
   2 commons-beanutils-core
   2 commons-configuration
   2 commons-net
   2 druid-lookups-cached-global
   2 hadoop-auth
   2 hadoop-common
   2 htrace-core4
   2 httpclient
   2 httpcore
   2 ion-java
   2 jackson-annotations
   2 jackson-core
   2 jackson-databind
   2 jackson-dataformat-cbor
   2 jackson-jaxrs
   2 jackson-xc
   2 javax.servlet-api
   2 jaxb-api
   2 jcip-annotations
   2 jersey-client
   2 jersey-core
   2 jersey-server
   2 jetty
   2 jetty-client
   2 jetty-continuation
   2 jetty-http
   2 jetty-io
   2 jetty-proxy
   2 jetty-security
   2 jetty-server
   2 jetty-servlet
   2 jetty-servlets
   2 jetty-sslengine
   2 jmespath-java
   2 joda-time
   2 jopt-simple
   2 jsp-api
   2 jsr311-api
   2 kafka_2.10
   2 mapdb
   2 netty
   2 nimbus-jose-jwt
   2 okhttp
   2 protobuf-java
   2 scala-library
   2 stax-api
   2 xmlenc
   2 xz
   2 zkclient
   1 RoaringBitmap
   1 accessors-smart
   1 aether-api
   1 aether-connector-file
   1 aether-connector-okhttp
   1 aether-impl
   1 aether-spi
   1 aether-util
   1 aggdesigner-algorithm
   1 airline
   1 antlr
   1 antlr4-runtime
   1 aopalliance
   1 asm
   1 asm-commons
   1 asm-tree
   1 async-http-client
   1 async-http-client-netty-utils
   1 audience-annotations
   1 avatica-core
   1 avatica-metrics
   1 avatica-server
   1 avro
   1 avro-ipc
   1 avro-mapred
   1 aws-java-sdk-ec2
   1 aws-java-sdk-kinesis
   1 aws-java-sdk-kms
   1 aws-java-sdk-s3
   1 aws-java-sdk-sts
   1 caffeine
   1 calcite-core
   1 calcite-linq4j
   1 checker
   1 classmate
   1 commons-cli
   1 commons-collections4
   1 commons-compiler
   1 commons-dbcp2
   1 commons-io
   1 commons-lang3
   1 commons-pool
   1 commons-pool2
   1 commons-text
   1 commons-validator
   1 compress-lzf
   1 config-magic
   1 curator-client
   1 curator-x-discovery
   1 derby
   1 derbyclient
   1 derbynet
   1 disruptor
   1 druid-avro-extensions
   1 druid-aws-common
   1 druid-basic-security
   1 druid-bloom-filter
   1 druid-console
   1 druid-core
   1 druid-datasketches
   1 druid-examples
   1 druid-hdfs-storage
   1 druid-histogram
   1 druid-hll
   1 druid-indexing-hadoop
   1 druid-indexing-service
   1 druid-kafka-eight
   1 druid-kafka-extraction-namespace
   1 druid-kafka-indexing-service
   1 druid-kerberos
   1 druid-kinesis-indexing-service
   1 druid-lookups-cached-single
   1 druid-parquet-extensions
   1 druid-processing
   1 druid-protobuf-extensions
   1 druid-s3-extensions
   1 druid-server
   1 druid-services
   1 druid-sql
   1 druid-stats
   1 error_prone_annotations
   1 esri-geometry-api
   1 extendedset
   1 fastutil
   1 geoip2
   1 guice
   1 guice-multibindings
   1 guice-servlet
   1 hadoop-client
   1 hadoop-hdfs-client
   1 hadoop-mapreduce-client-app
   1 hadoop-mapreduce-client-common
   1 hadoop-mapreduce-client-core
   1 hadoop-mapreduce-client-jobclient
   1 hadoop-mapreduce-client-shuffle
   1 hadoop-yarn-api
   1 hadoop-yarn-client
   1 hadoop-yarn-common
   1 hadoop-yarn-server-common
   1 hibernate-validator
   1 hive-storage-api
   1 icu4j
   1 irc-api
   1 jackson-core-asl
   1 jackson-dataformat-smile
   1 jackson-datatype-guava
   1 jackson-datatype-joda
   1 jackson-jaxrs-base
   1 jackson-jaxrs-json-provider
   1 jackson-jaxrs-smile-provider
   1 jackson-jq
   1 jackson-mapper-asl
   1 jackson-module-jaxb-annotations
   1 janino
   1 java-xmlbuilder
   1 javax.activation
   1 javax.el
   1 javax.el-api
   1 javax.inject
   1 jaxb-impl
   1 jboss-logging
   1 jcl-over-slf4j
   1 jcodings
   1 jdbi
   1 jersey-guice
   1 jersey-json
   1 jersey-servlet
   1 jets3t
   1 jettison
   1 jline
   1 jna
   1 joni
   1 jsch
   1 json-path
   1 jsr305
   1 jvm-attach-api
   1 kafka-schema-registry-client
   1 leveldbjni-all
   1 log4j
   1 log4j-api
   1 log4j-core
   1 log4j-jul
   1 log4j-slf4j-impl
   1 lz4-java
   1 maven-aether-provider
   1 maven-artifact
   1 maven-model
   1 maven-model-builder
   1 maven-repository-metadata
   1 maven-settings
   1 maven-settings-builder
   1 maxminddb
   1 memory
   1 mysql-metadata-storage
   1 netty-all
   1 netty-buffer
   1 netty-codec
   1 netty-codec-dns
   1 netty-codec-http
   1 netty-codec-socks
   1 netty-common
   1 netty-handler
   1 netty-handler-proxy
   1 netty-reactive-streams
   1 netty-resolver
   1 netty-resolver-dns
   1 netty-transport
   1 netty-transport-native-epoll
   1 netty-transport-native-unix-common
   1 okio
   1 opencsv
   1 paranamer
   1 parquet-avro
   1 parquet-column
   1 parquet-common
   1 parquet-encoding
   1 parquet-format
   1 parquet-hadoop
   1 parquet-jackson
   1 plexus-interpolation
   1 plexus-utils
   1 postgresql
   1 postgresql-metadata-storage
   1 protobuf-dynamic
   1 protobuf-java-util
   1 reactive-streams
   1 rhino
   1 schema-repo-api
   1 schema-repo-avro
   1 schema-repo-client
   1 schema-repo-common
   1 shims
   1 sigar
   1 simple-client-sslcontext
   1 sketches-core
   1 spymemcached
   1 stringtemplate
   1 tesla-aether
   1 twitter4j-async
   1 twitter4j-core
   1 twitter4j-stream
   1 validation-api
   1 velocity
   1 wagon-provider-api
   1 zookeeper
   1 zstd-jni"><pre class="notranslate"><code class="notranslate">$ TARBALL="/Users/charles.allen/src/druid/distribution/target/apache-druid-0.13.0-incubating-SNAPSHOT-bin.tar.gz"; cat &lt;(tar -t  --include='*/lib/*.jar' -f "${TARBALL}" | cut -d / -f 3 ) &lt;(tar -t  --include='*/extensions/*.jar' -f "${TARBALL}" | cut -d / -f 4 ) | sed -e 's/-[0-9].*\.jar//' | sort | uniq -c | sort -n -r --stable
   5 snappy-java
   4 commons-beanutils
   4 commons-collections
   4 commons-compress
   4 gson
   4 objenesis
   4 slf4j-api
   3 commons-codec
   3 commons-digester
   3 commons-lang
   3 commons-logging
   3 commons-math3
   3 curator-framework
   3 curator-recipes
   3 guava
   3 jetty-util
   3 json-smart
   3 kafka-clients
   3 metrics-core
   3 servlet-api
   2 apacheds-i18n
   2 apacheds-kerberos-codec
   2 api-asn1-api
   2 api-util
   2 aws-java-sdk-core
   2 commons-beanutils-core
   2 commons-configuration
   2 commons-net
   2 druid-lookups-cached-global
   2 hadoop-auth
   2 hadoop-common
   2 htrace-core4
   2 httpclient
   2 httpcore
   2 ion-java
   2 jackson-annotations
   2 jackson-core
   2 jackson-databind
   2 jackson-dataformat-cbor
   2 jackson-jaxrs
   2 jackson-xc
   2 javax.servlet-api
   2 jaxb-api
   2 jcip-annotations
   2 jersey-client
   2 jersey-core
   2 jersey-server
   2 jetty
   2 jetty-client
   2 jetty-continuation
   2 jetty-http
   2 jetty-io
   2 jetty-proxy
   2 jetty-security
   2 jetty-server
   2 jetty-servlet
   2 jetty-servlets
   2 jetty-sslengine
   2 jmespath-java
   2 joda-time
   2 jopt-simple
   2 jsp-api
   2 jsr311-api
   2 kafka_2.10
   2 mapdb
   2 netty
   2 nimbus-jose-jwt
   2 okhttp
   2 protobuf-java
   2 scala-library
   2 stax-api
   2 xmlenc
   2 xz
   2 zkclient
   1 RoaringBitmap
   1 accessors-smart
   1 aether-api
   1 aether-connector-file
   1 aether-connector-okhttp
   1 aether-impl
   1 aether-spi
   1 aether-util
   1 aggdesigner-algorithm
   1 airline
   1 antlr
   1 antlr4-runtime
   1 aopalliance
   1 asm
   1 asm-commons
   1 asm-tree
   1 async-http-client
   1 async-http-client-netty-utils
   1 audience-annotations
   1 avatica-core
   1 avatica-metrics
   1 avatica-server
   1 avro
   1 avro-ipc
   1 avro-mapred
   1 aws-java-sdk-ec2
   1 aws-java-sdk-kinesis
   1 aws-java-sdk-kms
   1 aws-java-sdk-s3
   1 aws-java-sdk-sts
   1 caffeine
   1 calcite-core
   1 calcite-linq4j
   1 checker
   1 classmate
   1 commons-cli
   1 commons-collections4
   1 commons-compiler
   1 commons-dbcp2
   1 commons-io
   1 commons-lang3
   1 commons-pool
   1 commons-pool2
   1 commons-text
   1 commons-validator
   1 compress-lzf
   1 config-magic
   1 curator-client
   1 curator-x-discovery
   1 derby
   1 derbyclient
   1 derbynet
   1 disruptor
   1 druid-avro-extensions
   1 druid-aws-common
   1 druid-basic-security
   1 druid-bloom-filter
   1 druid-console
   1 druid-core
   1 druid-datasketches
   1 druid-examples
   1 druid-hdfs-storage
   1 druid-histogram
   1 druid-hll
   1 druid-indexing-hadoop
   1 druid-indexing-service
   1 druid-kafka-eight
   1 druid-kafka-extraction-namespace
   1 druid-kafka-indexing-service
   1 druid-kerberos
   1 druid-kinesis-indexing-service
   1 druid-lookups-cached-single
   1 druid-parquet-extensions
   1 druid-processing
   1 druid-protobuf-extensions
   1 druid-s3-extensions
   1 druid-server
   1 druid-services
   1 druid-sql
   1 druid-stats
   1 error_prone_annotations
   1 esri-geometry-api
   1 extendedset
   1 fastutil
   1 geoip2
   1 guice
   1 guice-multibindings
   1 guice-servlet
   1 hadoop-client
   1 hadoop-hdfs-client
   1 hadoop-mapreduce-client-app
   1 hadoop-mapreduce-client-common
   1 hadoop-mapreduce-client-core
   1 hadoop-mapreduce-client-jobclient
   1 hadoop-mapreduce-client-shuffle
   1 hadoop-yarn-api
   1 hadoop-yarn-client
   1 hadoop-yarn-common
   1 hadoop-yarn-server-common
   1 hibernate-validator
   1 hive-storage-api
   1 icu4j
   1 irc-api
   1 jackson-core-asl
   1 jackson-dataformat-smile
   1 jackson-datatype-guava
   1 jackson-datatype-joda
   1 jackson-jaxrs-base
   1 jackson-jaxrs-json-provider
   1 jackson-jaxrs-smile-provider
   1 jackson-jq
   1 jackson-mapper-asl
   1 jackson-module-jaxb-annotations
   1 janino
   1 java-xmlbuilder
   1 javax.activation
   1 javax.el
   1 javax.el-api
   1 javax.inject
   1 jaxb-impl
   1 jboss-logging
   1 jcl-over-slf4j
   1 jcodings
   1 jdbi
   1 jersey-guice
   1 jersey-json
   1 jersey-servlet
   1 jets3t
   1 jettison
   1 jline
   1 jna
   1 joni
   1 jsch
   1 json-path
   1 jsr305
   1 jvm-attach-api
   1 kafka-schema-registry-client
   1 leveldbjni-all
   1 log4j
   1 log4j-api
   1 log4j-core
   1 log4j-jul
   1 log4j-slf4j-impl
   1 lz4-java
   1 maven-aether-provider
   1 maven-artifact
   1 maven-model
   1 maven-model-builder
   1 maven-repository-metadata
   1 maven-settings
   1 maven-settings-builder
   1 maxminddb
   1 memory
   1 mysql-metadata-storage
   1 netty-all
   1 netty-buffer
   1 netty-codec
   1 netty-codec-dns
   1 netty-codec-http
   1 netty-codec-socks
   1 netty-common
   1 netty-handler
   1 netty-handler-proxy
   1 netty-reactive-streams
   1 netty-resolver
   1 netty-resolver-dns
   1 netty-transport
   1 netty-transport-native-epoll
   1 netty-transport-native-unix-common
   1 okio
   1 opencsv
   1 paranamer
   1 parquet-avro
   1 parquet-column
   1 parquet-common
   1 parquet-encoding
   1 parquet-format
   1 parquet-hadoop
   1 parquet-jackson
   1 plexus-interpolation
   1 plexus-utils
   1 postgresql
   1 postgresql-metadata-storage
   1 protobuf-dynamic
   1 protobuf-java-util
   1 reactive-streams
   1 rhino
   1 schema-repo-api
   1 schema-repo-avro
   1 schema-repo-client
   1 schema-repo-common
   1 shims
   1 sigar
   1 simple-client-sslcontext
   1 sketches-core
   1 spymemcached
   1 stringtemplate
   1 tesla-aether
   1 twitter4j-async
   1 twitter4j-core
   1 twitter4j-stream
   1 validation-api
   1 velocity
   1 wagon-provider-api
   1 zookeeper
   1 zstd-jni
</code></pre></div>
<h1 dir="auto">Proposal Adding Double precision metrics.</h1>
<p dir="auto">Currently Druid stores double aggregated numbers as floats which leads to lose of precision and not expected results set for the case of Min/Max where you sort of expect to see the same precision of the actual indexed data.<br>
The goal of this work is to add support of actual double indexed columns.<br>
There is several ways on how to get there would like to poll the community about which way to go.</p>
<h1 dir="auto">Option 1</h1>
<p dir="auto">Make the double aggregator store numbers as double period !</p>
<ul dir="auto">
<li>This might be radical and surprise druid user since the storage footprint of double columns will double.</li>
<li>Less code duplication (will need to keep float ser/deser code to ensure backward compatible).</li>
</ul>
<h1 dir="auto">Option 1+</h1>
<p dir="auto">If storage footprint is a deal breaker, do the same as option 1 and add new suite of FloatsXXXAggregators.</p>
<ul dir="auto">
<li>Make it more flexible if storage is an issue.</li>
<li>Lot of code duplication.</li>
</ul>
<h1 dir="auto">Option 2</h1>
<p dir="auto">Instead of modifying existing Double aggregator,  create new set of aggregator called DoubleSum64...etc.</p>
<ul dir="auto">
<li>lot of code duplication</li>
<li>some how confusing and ugly to have DoubleSum64 and DoubleSum etc</li>
<li>probably the most transparent for current druid users.</li>
</ul>
<h1 dir="auto">Option 3</h1>
<p dir="auto">Add a flag to indexSpec ie "StoreDoubleAsFloat" and set it to true/false</p>
<ul dir="auto">
<li>less code duplication that 1.1 and 1.2</li>
<li>need to have a new precision flag within the column to find out how to deserialize the column (or Maybe not haven't fleshed out this yet but it is doable).</li>
<li>same as option 2 don't see why index double numbers then store it as floats.</li>
</ul>