<p dir="auto">Currently you can mix BM25 and the classic similarity in the same index:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;book&quot; : {
    &quot;properties&quot; : {
      &quot;title&quot; : { &quot;type&quot; : &quot;text&quot;, &quot;similarity&quot; : &quot;BM25&quot; },
      &quot;text&quot; : { &quot;type&quot; : &quot;text&quot;, &quot;similarity&quot; : &quot;classic&quot; }
    }
}"><pre class="notranslate"><code class="notranslate">{
  "book" : {
    "properties" : {
      "title" : { "type" : "text", "similarity" : "BM25" },
      "text" : { "type" : "text", "similarity" : "classic" }
    }
}
</code></pre></div>
<p dir="auto">... this can be buggy since the base similarity cannot be picked properly. The base similarity is used to compute a factor for the coordinates and the query norm (cf Similarity.coord and Similarity.queryNorm).<br>
It is a per query setting that can change the score completely. For <code class="notranslate">BM25</code> the coord and the queryNorm are disabled (returns 1.0f) whereas they are important factors in the <code class="notranslate">classic</code> similarity. To resolve this we have two settings for the index similarity <code class="notranslate">index.similarity.default.type</code> and <code class="notranslate">index.similarity.base.type</code>.<br>
In order to make the things easier we could have a single setting:<br>
<code class="notranslate">index.similarity.type</code><br>
... that is dynamically updatable if the similarities are compatible:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="37157836" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/6727" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/6727/hovercard" href="https://github.com/elastic/elasticsearch/issues/6727">#6727</a><br>
This would not make the "are compatible?" easier but this would prevent the user to shoot themselves in the foot.</p>
<p dir="auto">This issue addresses a few topics:</p>
<ul dir="auto">
<li>separating <code class="notranslate">string</code> fields out into <code class="notranslate">text</code> and <code class="notranslate">keyword</code> fields (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91511789" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/11901" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/11901/hovercard" href="https://github.com/elastic/elasticsearch/issues/11901">#11901</a>)</li>
<li>deprecating in-memory fielddata for field-types that support doc-values (to remove in 3.x)</li>
<li><code class="notranslate">fielddata</code> and <code class="notranslate">doc_values</code> settings (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="50318581" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8693" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8693/hovercard" href="https://github.com/elastic/elasticsearch/issues/8693">#8693</a>) and <code class="notranslate">norms</code></li>
<li>good out-of-the-box dynamic mappings for string fields</li>
</ul>
<h2 dir="auto"><code class="notranslate">string</code> → <code class="notranslate">text</code>/ <code class="notranslate">keyword</code></h2>
<p dir="auto">Today, we use <code class="notranslate">string</code> both for full-text and for structured keywords.  We don't support doc-values on <code class="notranslate">analyzed</code> string fields, which means that strings which are essentially keywords (but eg need to be lowercased) cannot use doc-values.</p>
<p dir="auto"><strong>Proposal:</strong></p>
<ul dir="auto">
<li>deprecate <code class="notranslate">string</code> fields</li>
<li>add <code class="notranslate">text</code> fields which support the full analysis chain and don't support doc-values</li>
<li>add <code class="notranslate">keywords</code> fields which support only the <code class="notranslate">keyword</code> tokenizer, and have doc-values enabled by default</li>
<li>change <code class="notranslate">index</code> to accept <code class="notranslate">true</code> | <code class="notranslate">false</code></li>
</ul>
<p dir="auto">Question:  Should <code class="notranslate">keyword</code> fields allow token filters that introduce new tokens?</p>
<h2 dir="auto">Deprecating fielddata for fields that support doc values</h2>
<p dir="auto">In-memory fielddata is limited by the size of the heap, and has been one of the biggest pain-points for users. Doc-values are slightly slower but: (1) don't suffer from the same latency as fielddata, (2) are not limited by heap size, (3) don't impact garbage collection, (4) allow much greater scaling.</p>
<p dir="auto">All fields that support doc values already have them enabled by default.</p>
<p dir="auto">Proposal:</p>
<ul dir="auto">
<li>Deprecate fielddata implementations (except for analyzed string fields) in 2.x</li>
<li>Remove them in 3.x.</li>
</ul>
<p dir="auto">The question arises: what happens if the user disables doc values then decides that actually they DO want to aggregate on that field after all?  The answer is the same as if they have set a field to <code class="notranslate">index:false</code> - they have to reindex.</p>
<h2 dir="auto">Fielddata and doc values settings</h2>
<p dir="auto">Today we have these settings:</p>
<ul dir="auto">
<li><code class="notranslate">doc_values</code>: <code class="notranslate">true</code>|<code class="notranslate">false</code></li>
<li><code class="notranslate">fielddata.format</code>: <code class="notranslate">disabled</code> | <code class="notranslate">doc_values</code> | <code class="notranslate">paged_bytes</code> | <code class="notranslate">array</code></li>
<li><code class="notranslate">fielddata.loading</code>: <code class="notranslate">lazy</code> | <code class="notranslate">eager</code> | <code class="notranslate">eager_global_ordinals</code></li>
<li><code class="notranslate">fielddata.filters</code>: <code class="notranslate">frequency:{}</code>, <code class="notranslate">regex:{}</code></li>
</ul>
<p dir="auto">These become a lot easier to simplify if we deprecate fielddata for all but analyzed string fields.</p>
<p dir="auto"><strong>Proposal for fields that support doc values:</strong></p>
<ul dir="auto">
<li><code class="notranslate">doc_values</code> : <code class="notranslate">true</code> (default) | <code class="notranslate">false</code></li>
<li><code class="notranslate">global_ordinals</code> : <code class="notranslate">lazy</code> (default) | <code class="notranslate">eager</code></li>
</ul>
<p dir="auto"><strong>Proposal for analyzed string fields:</strong></p>
<ul dir="auto">
<li><code class="notranslate">fielddata</code>: <code class="notranslate">disabled</code> (default) | <code class="notranslate">lazy</code> | <code class="notranslate">eager</code></li>
<li><code class="notranslate">global_ordinals</code> : <code class="notranslate">lazy</code> (default) | <code class="notranslate">eager</code></li>
<li><code class="notranslate">fielddata.filters</code>: <code class="notranslate">frequency:{}</code>, <code class="notranslate">regex:{}</code></li>
</ul>
<p dir="auto">If, in the future, we can automatically figure out which global ordinals need to be built eagerly, then we can remove the <code class="notranslate">global_ordinals</code> setting.</p>
<h2 dir="auto">Norms settings</h2>
<p dir="auto">Similar to the above, we have:</p>
<ul dir="auto">
<li><code class="notranslate">norms.enabled</code> : <code class="notranslate">true</code> | <code class="notranslate">false</code></li>
<li><code class="notranslate">norms.loading</code> : <code class="notranslate">lazy</code> | <code class="notranslate">eager</code></li>
</ul>
<p dir="auto">In Lucene 5.3, norms are disk based, so the lazy/eager issue is less important (eager in this case would mean force-loading the norms into the file system cache, a decision which we can probably make automatically in the future).</p>
<p dir="auto"><strong>Proposal:</strong></p>
<ul dir="auto">
<li><code class="notranslate">norms</code>: <code class="notranslate">true</code> | <code class="notranslate">false</code></li>
<li>only supported on <code class="notranslate">text</code> fields</li>
</ul>
<h2 dir="auto">Good out-of-the-box dynamic mappings for string fields</h2>
<p dir="auto">Today, when we detect a new string field, we add it as an <code class="notranslate">analyzed</code> <code class="notranslate">string</code>, with <code class="notranslate">lazy</code> fielddata loading enabled.  While this allows users to get going with full text search, sorting and aggregations (with limitations, eg <code class="notranslate">new</code> + <code class="notranslate">york</code>), it's a poor default for heap usage.</p>
<p dir="auto"><strong>Proposal:</strong></p>
<p dir="auto">Add a <code class="notranslate">text</code> main field (with fielddata loading disabled) and a <code class="notranslate">keyword</code> multi-field by default, ie:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;my_string&quot;: {
    &quot;type&quot;: &quot;text&quot;,
    &quot;fields&quot;: {
      &quot;keyword&quot;: {
        &quot;type&quot;: &quot;keyword&quot;,
        &quot;ignore_above&quot;: 256
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "my_string": {
    "type": "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}
</code></pre></div>
<p dir="auto">With the default settings these fields would look like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;my_string&quot;: {
    &quot;type&quot;:                &quot;text&quot;,
    &quot;analyzer&quot;:            &quot;default&quot;,
    &quot;boost&quot;:               1,
    &quot;fielddata&quot;:           &quot;disabled&quot;,
    &quot;fielddata_filters&quot;:   {},
    &quot;ignore_above&quot;:        -1,
    &quot;include_in_all&quot;:      true,
    &quot;index&quot;:               true,
    &quot;index_options&quot;:       &quot;positions&quot;,
    &quot;norms&quot;:               true,
    &quot;null_value&quot;:          null,
    &quot;position_offset_gap&quot;: 0,
    &quot;search_analyzer&quot;:     &quot;default&quot;,
    &quot;similarity&quot;:          &quot;default&quot;,
    &quot;store&quot;:               false,
    &quot;term_vector&quot;:         &quot;no&quot;
  }
}

{
  &quot;my_string.keyword&quot;: {
    &quot;type&quot;:                &quot;keyword&quot;,
    &quot;analyzer&quot;:            &quot;keyword&quot;,
    &quot;boost&quot;:               1,
    &quot;doc_values&quot;:          true,
    &quot;ignore_above&quot;:        256,
    &quot;include_in_all&quot;:      true,
    &quot;index&quot;:               true,
    &quot;index_options&quot;:       &quot;docs&quot;,
    &quot;null_value&quot;:          null,
    &quot;position_offset_gap&quot;: 0,
    &quot;search_analyzer&quot;:     &quot;keyword&quot;,
    &quot;similarity&quot;:          &quot;default&quot;,
    &quot;store&quot;:               false,
    &quot;term_vector&quot;:         &quot;no&quot;
  }
}"><pre class="notranslate"><code class="notranslate">{
  "my_string": {
    "type":                "text",
    "analyzer":            "default",
    "boost":               1,
    "fielddata":           "disabled",
    "fielddata_filters":   {},
    "ignore_above":        -1,
    "include_in_all":      true,
    "index":               true,
    "index_options":       "positions",
    "norms":               true,
    "null_value":          null,
    "position_offset_gap": 0,
    "search_analyzer":     "default",
    "similarity":          "default",
    "store":               false,
    "term_vector":         "no"
  }
}

{
  "my_string.keyword": {
    "type":                "keyword",
    "analyzer":            "keyword",
    "boost":               1,
    "doc_values":          true,
    "ignore_above":        256,
    "include_in_all":      true,
    "index":               true,
    "index_options":       "docs",
    "null_value":          null,
    "position_offset_gap": 0,
    "search_analyzer":     "keyword",
    "similarity":          "default",
    "store":               false,
    "term_vector":         "no"
  }
}
</code></pre></div>