<p dir="auto"><strong>Elasticsearch version</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# elasticsearch --version
Version: 2.3.4, Build: e455fd0/2016-06-30T11:24:31Z, JVM: 1.8.0_91"><pre class="notranslate"><code class="notranslate"># elasticsearch --version
Version: 2.3.4, Build: e455fd0/2016-06-30T11:24:31Z, JVM: 1.8.0_91
</code></pre></div>
<p dir="auto"><strong>JVM version</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# java -version
openjdk version &quot;1.8.0_91&quot;
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)"><pre class="notranslate"><code class="notranslate"># java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
</code></pre></div>
<p dir="auto"><strong>OS version</strong>: Debian Jessie, elasticsearch is running in the container from the official docker image.</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Index ~1 billion documents (I used 1 241 584 977 for 12h) in ~1kb doc average.</li>
<li>Run topN query for low cardinality field with zero matches -&gt; get instant response. Good!</li>
<li>Run topN query for high cardinality field with zero matches -&gt; get slow response. Bad!</li>
</ol>
<p dir="auto">The index in question has 50 shards and all fields are <code class="notranslate">doc_values</code>.</p>
<p dir="auto">Query for low cardinality field:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;size&quot;: 0,
  &quot;query&quot;: {
    &quot;filtered&quot;: {
      &quot;filter&quot;: {
        &quot;bool&quot;: {
          &quot;must&quot;: [
            {
              &quot;query&quot;: {
                &quot;query_string&quot;: {
                  &quot;query&quot;: &quot;zoneName:foo.bar&quot;
                }
              }
            },
            {
              &quot;range&quot;: {
                &quot;@timestamp&quot;: {
                  &quot;gte&quot;: &quot;2016-08-03T00:00:00Z&quot;,
                  &quot;lte&quot;: &quot;2016-08-03T12:00:00Z&quot;
                }
              }
            }
          ]
        }
      }
    }
  },
  &quot;aggs&quot;: {
    &quot;topn&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;status&quot;,
        &quot;size&quot;: 10,
        &quot;order&quot;: {
          &quot;_count&quot;: &quot;desc&quot;
        }
      }
    }
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"size"</span>: <span class="pl-c1">0</span>,
  <span class="pl-ent">"query"</span>: {
    <span class="pl-ent">"filtered"</span>: {
      <span class="pl-ent">"filter"</span>: {
        <span class="pl-ent">"bool"</span>: {
          <span class="pl-ent">"must"</span>: [
            {
              <span class="pl-ent">"query"</span>: {
                <span class="pl-ent">"query_string"</span>: {
                  <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>zoneName:foo.bar<span class="pl-pds">"</span></span>
                }
              }
            },
            {
              <span class="pl-ent">"range"</span>: {
                <span class="pl-ent">"@timestamp"</span>: {
                  <span class="pl-ent">"gte"</span>: <span class="pl-s"><span class="pl-pds">"</span>2016-08-03T00:00:00Z<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"lte"</span>: <span class="pl-s"><span class="pl-pds">"</span>2016-08-03T12:00:00Z<span class="pl-pds">"</span></span>
                }
              }
            }
          ]
        }
      }
    }
  },
  <span class="pl-ent">"aggs"</span>: {
    <span class="pl-ent">"topn"</span>: {
      <span class="pl-ent">"terms"</span>: {
        <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>status<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"size"</span>: <span class="pl-c1">10</span>,
        <span class="pl-ent">"order"</span>: {
          <span class="pl-ent">"_count"</span>: <span class="pl-s"><span class="pl-pds">"</span>desc<span class="pl-pds">"</span></span>
        }
      }
    }
  }
}</pre></div>
<p dir="auto">Response:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;took&quot;: 148,
  &quot;timed_out&quot;: false,
  &quot;_shards&quot;: {
    &quot;total&quot;: 50,
    &quot;successful&quot;: 50,
    &quot;failed&quot;: 0
  },
  &quot;hits&quot;: {
    &quot;total&quot;: 0,
    &quot;max_score&quot;: 0,
    &quot;hits&quot;: []
  },
  &quot;aggregations&quot;: {
    &quot;topn&quot;: {
      &quot;doc_count_error_upper_bound&quot;: 0,
      &quot;sum_other_doc_count&quot;: 0,
      &quot;buckets&quot;: []
    }
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"took"</span>: <span class="pl-c1">148</span>,
  <span class="pl-ent">"timed_out"</span>: <span class="pl-c1">false</span>,
  <span class="pl-ent">"_shards"</span>: {
    <span class="pl-ent">"total"</span>: <span class="pl-c1">50</span>,
    <span class="pl-ent">"successful"</span>: <span class="pl-c1">50</span>,
    <span class="pl-ent">"failed"</span>: <span class="pl-c1">0</span>
  },
  <span class="pl-ent">"hits"</span>: {
    <span class="pl-ent">"total"</span>: <span class="pl-c1">0</span>,
    <span class="pl-ent">"max_score"</span>: <span class="pl-c1">0</span>,
    <span class="pl-ent">"hits"</span>: []
  },
  <span class="pl-ent">"aggregations"</span>: {
    <span class="pl-ent">"topn"</span>: {
      <span class="pl-ent">"doc_count_error_upper_bound"</span>: <span class="pl-c1">0</span>,
      <span class="pl-ent">"sum_other_doc_count"</span>: <span class="pl-c1">0</span>,
      <span class="pl-ent">"buckets"</span>: []
    }
  }
}</pre></div>
<p dir="auto">Query for high cardinality field:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;size&quot;: 0,
  &quot;query&quot;: {
    &quot;filtered&quot;: {
      &quot;filter&quot;: {
        &quot;bool&quot;: {
          &quot;must&quot;: [
            {
              &quot;query&quot;: {
                &quot;query_string&quot;: {
                  &quot;query&quot;: &quot;zoneName:foo.bar&quot;
                }
              }
            },
            {
              &quot;range&quot;: {
                &quot;@timestamp&quot;: {
                  &quot;gte&quot;: &quot;2016-08-03T00:00:00Z&quot;,
                  &quot;lte&quot;: &quot;2016-08-03T12:00:00Z&quot;
                }
              }
            }
          ]
        }
      }
    }
  },
  &quot;aggs&quot;: {
    &quot;topn&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;request&quot;,
        &quot;size&quot;: 10,
        &quot;order&quot;: {
          &quot;_count&quot;: &quot;desc&quot;
        }
      }
    }
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"size"</span>: <span class="pl-c1">0</span>,
  <span class="pl-ent">"query"</span>: {
    <span class="pl-ent">"filtered"</span>: {
      <span class="pl-ent">"filter"</span>: {
        <span class="pl-ent">"bool"</span>: {
          <span class="pl-ent">"must"</span>: [
            {
              <span class="pl-ent">"query"</span>: {
                <span class="pl-ent">"query_string"</span>: {
                  <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>zoneName:foo.bar<span class="pl-pds">"</span></span>
                }
              }
            },
            {
              <span class="pl-ent">"range"</span>: {
                <span class="pl-ent">"@timestamp"</span>: {
                  <span class="pl-ent">"gte"</span>: <span class="pl-s"><span class="pl-pds">"</span>2016-08-03T00:00:00Z<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"lte"</span>: <span class="pl-s"><span class="pl-pds">"</span>2016-08-03T12:00:00Z<span class="pl-pds">"</span></span>
                }
              }
            }
          ]
        }
      }
    }
  },
  <span class="pl-ent">"aggs"</span>: {
    <span class="pl-ent">"topn"</span>: {
      <span class="pl-ent">"terms"</span>: {
        <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>request<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"size"</span>: <span class="pl-c1">10</span>,
        <span class="pl-ent">"order"</span>: {
          <span class="pl-ent">"_count"</span>: <span class="pl-s"><span class="pl-pds">"</span>desc<span class="pl-pds">"</span></span>
        }
      }
    }
  }
}</pre></div>
<p dir="auto">Response:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;took&quot;: 10634,
  &quot;timed_out&quot;: false,
  &quot;_shards&quot;: {
    &quot;total&quot;: 50,
    &quot;successful&quot;: 50,
    &quot;failed&quot;: 0
  },
  &quot;hits&quot;: {
    &quot;total&quot;: 0,
    &quot;max_score&quot;: 0,
    &quot;hits&quot;: []
  },
  &quot;aggregations&quot;: {
    &quot;topn&quot;: {
      &quot;doc_count_error_upper_bound&quot;: 0,
      &quot;sum_other_doc_count&quot;: 0,
      &quot;buckets&quot;: []
    }
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"took"</span>: <span class="pl-c1">10634</span>,
  <span class="pl-ent">"timed_out"</span>: <span class="pl-c1">false</span>,
  <span class="pl-ent">"_shards"</span>: {
    <span class="pl-ent">"total"</span>: <span class="pl-c1">50</span>,
    <span class="pl-ent">"successful"</span>: <span class="pl-c1">50</span>,
    <span class="pl-ent">"failed"</span>: <span class="pl-c1">0</span>
  },
  <span class="pl-ent">"hits"</span>: {
    <span class="pl-ent">"total"</span>: <span class="pl-c1">0</span>,
    <span class="pl-ent">"max_score"</span>: <span class="pl-c1">0</span>,
    <span class="pl-ent">"hits"</span>: []
  },
  <span class="pl-ent">"aggregations"</span>: {
    <span class="pl-ent">"topn"</span>: {
      <span class="pl-ent">"doc_count_error_upper_bound"</span>: <span class="pl-c1">0</span>,
      <span class="pl-ent">"sum_other_doc_count"</span>: <span class="pl-c1">0</span>,
      <span class="pl-ent">"buckets"</span>: []
    }
  }
}</pre></div>
<p dir="auto">Field cardinality for the same timespan:</p>
<ul dir="auto">
<li><code class="notranslate">request</code>: 168154206</li>
<li><code class="notranslate">status</code>: 127</li>
</ul>
<p dir="auto">Problem is: these 10 seconds came out of nowhere. I don't have any matching documents, there's nothing to do. I'm afraid that I have to pay the price for every request trying to do aggregation for index-wise high cardinality field, even though query-wise the field is not that diverse.</p>
<p dir="auto">With <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="194577362" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/22073" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/22073/hovercard" href="https://github.com/elastic/elasticsearch/pull/22073">#22073</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="196054923" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/22225" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/22225/hovercard" href="https://github.com/elastic/elasticsearch/pull/22225">#22225</a> we enabled strict duplicate checks for XContent. Although our benchmarks only showed a negligible performance drop of 1 - 2% (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="167803334" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/19614" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/19614/hovercard" href="https://github.com/elastic/elasticsearch/issues/19614">#19614</a>), Elasticsearch 6.0 still allows to disable the strict duplicate checks with an undocumented flag. Hence, we also needed to keep the existing hand-coded duplicate checks around in case Elasticsearch is started with disabled XContent parser duplicate checks.</p>
<p dir="auto">In Elasticsearch 7.0, we should remove the undocumented fallback to disable strict duplicate checks and also remove our hand-coded duplicate checks.</p>