<p dir="auto">To recreate, start with a freshly started cluster (1.5.2) with nothing stored in fielddata.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET /_cat/fielddata?v

  id                     host                      ip         node  total 
  oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown    0b "><pre class="notranslate"><code class="notranslate">GET /_cat/fielddata?v

  id                     host                      ip         node  total 
  oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown    0b 
</code></pre></div>
<p dir="auto">Now create a document that contains fields with the same name as the 'meta' fields that appear in the output to fielddata.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST /lul/wat/1
{
  &quot;id&quot;: &quot;hah&quot;,
  &quot;host&quot;: &quot;yeah&quot;,
  &quot;ip&quot;: 1,
  &quot;node&quot;: &quot;lulwat&quot;,
  &quot;total&quot;: 5
}"><pre class="notranslate"><code class="notranslate">POST /lul/wat/1
{
  "id": "hah",
  "host": "yeah",
  "ip": 1,
  "node": "lulwat",
  "total": 5
}
</code></pre></div>
<p dir="auto">Then search with aggregations against those fields, to load them into fielddata.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST /lul/wat/_search
{
  &quot;query&quot;: {
    &quot;match_all&quot;: {}
  },
  &quot;aggs&quot;: {
    &quot;one&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;ip&quot;
      }
    },
    &quot;two&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;node&quot;
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">POST /lul/wat/_search
{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "one": {
      "terms": {
        "field": "ip"
      }
    },
    "two": {
      "terms": {
        "field": "node"
      }
    }
  }
}
</code></pre></div>
<p dir="auto">Now check the _cat/fielddata endpoint again.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET /_cat/fielddata?v

  id                     host                      ip         node     total  node       ip
  Bt6uoLNcS0mlVSdY1xGgSA Olivers-MacBook-Pro.local 10.4.1.106 Crown    8.9kb  Crown      10.4.1.106"><pre class="notranslate"><code class="notranslate">GET /_cat/fielddata?v

  id                     host                      ip         node     total  node       ip
  Bt6uoLNcS0mlVSdY1xGgSA Olivers-MacBook-Pro.local 10.4.1.106 Crown    8.9kb  Crown      10.4.1.106
</code></pre></div>
<p dir="auto">Note that <code class="notranslate">node</code> and <code class="notranslate">ip</code> are both repeated, but we can't actually see the usage for those fields - instead we get the metadata again. For the sake of completeness...</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST /lul/wat/_search
{
  &quot;query&quot;: {
    &quot;match_all&quot;: {}
  },
  &quot;aggs&quot;: {
    &quot;one&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;ip&quot;
      }
    },
    &quot;two&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;node&quot;
      }
    },
    &quot;three&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;host&quot;
      }
    },
    &quot;four&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;id&quot;
      }
    },
    &quot;five&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;total&quot;
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">POST /lul/wat/_search
{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "one": {
      "terms": {
        "field": "ip"
      }
    },
    "two": {
      "terms": {
        "field": "node"
      }
    },
    "three": {
      "terms": {
        "field": "host"
      }
    },
    "four": {
      "terms": {
        "field": "id"
      }
    },
    "five": {
      "terms": {
        "field": "total"
      }
    }
  }
}
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET /_cat/fielddata?v

  id                     host                      ip         node   total node   total ip         host                      id
  oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown 18.4kb Crown 18.4kb 10.4.1.106 Olivers-MacBook-Pro.local oXbVJr3_QNqRhTSkKB1WSw"><pre class="notranslate"><code class="notranslate">GET /_cat/fielddata?v

  id                     host                      ip         node   total node   total ip         host                      id
  oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown 18.4kb Crown 18.4kb 10.4.1.106 Olivers-MacBook-Pro.local oXbVJr3_QNqRhTSkKB1WSw
</code></pre></div>
<p dir="auto">Note that locally I can recreate the field duplication, but in our production environment (1.4.0) we see even stranger things:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  id                     host                           ip             node          total content.source name.raw delivery.id timestamp id                      _type sections    _id created delivery.channel content.id campaign published content.sections content.topics ip             referrer topics content.created source title.raw       p property.type _parent property.id 
  NcxVMomeTC6TOTEVH9lt3Q [REDACTED]                     [REDACTED]     prd-elastic-c 2.7gb           16mb       0b     805.9kb   172.3mb NcxVMomeTC6TOTEVH9lt3Q 32.8mb    9.9mb 52.2mb  34.4mb          491.3kb    116.7mb   82.5mb       3mb          113.5mb          1.5gb [REDACTED]     110.8mb 29.9mb            31mb 10.6mb     111kb 400.7mb        26.8mb 741.5mb       1.3mb 
                 152.6kb [REDACTED]                     0b             prd-elastic-a 1.5gb          6.4mb       0b     645.1kb    86.7mb                152.6kb 17.8mb    4.7mb 36.1mb    24mb          458.4kb     79.1mb   48.3mb     1.9mb           75.1mb        859.9mb          0b    84.3mb 12.7mb              0b  4.4mb   836.9kb 197.5mb          11mb 493.9mb     861.5kb 
  CqC9hO3tT2O_j2gMPrc2Uw [REDACTED]                     [REDACTED]     prd-elastic-f 3.1gb         23.1mb       0b     865.4kb   160.3mb CqC9hO3tT2O_j2gMPrc2Uw 18.1mb    8.5mb 49.3mb  35.2mb          559.1kb    135.2mb   80.2mb       3mb          128.7mb          1.9gb [REDACTED]     124.1mb 22.8mb          35.3mb  8.1mb   577.3kb 368.8mb        22.8mb 671.8mb       1.2mb 
                   5.6kb [REDACTED]                     0b             prd-elastic-d   3gb         18.7mb       0b       1.2mb   159.9mb                  5.6kb 12.5mb    9.2mb 51.3mb    38mb          769.9kb    134.1mb     82mb     3.6mb          132.4mb          1.7gb          0b    141mb 25.7mb          58.8mb  9.5mb   399.7kb 408.7mb          23mb 709.8mb       1.3mb 
  WkXcjwj8T6yQ1X2Pb_M2ow [REDACTED]                     [REDACTED]     prd-elastic-b 1.9gb         15.8mb       0b       1.1mb    87.3mb WkXcjwj8T6yQ1X2Pb_M2ow     0b    5.1mb 34.5mb  35.9mb          593.9kb     86.5mb   60.7mb     1.9mb           79.1mb          1.1gb [REDACTED]     87.7mb 19.2mb            31mb  7.2mb   110.6kb 202.9mb        20.3mb 505.8mb       1.2mb 
                 116.6kb [REDACTED]                     38.2mb         prd-elastic-e 3.5gb         25.9mb   21.6kb         1mb   204.5mb                116.6kb     0b   11.1mb 57.2mb    34mb          724.2kb    161.6mb   90.7mb     3.6mb          152.8mb          2.1gb      38.2mb    150mb 31.5mb          29.9mb 10.2mb   615.9kb 368.8mb        32.8mb 764.5mb       1.7mb "><pre class="notranslate"><code class="notranslate">  id                     host                           ip             node          total content.source name.raw delivery.id timestamp id                      _type sections    _id created delivery.channel content.id campaign published content.sections content.topics ip             referrer topics content.created source title.raw       p property.type _parent property.id 
  NcxVMomeTC6TOTEVH9lt3Q [REDACTED]                     [REDACTED]     prd-elastic-c 2.7gb           16mb       0b     805.9kb   172.3mb NcxVMomeTC6TOTEVH9lt3Q 32.8mb    9.9mb 52.2mb  34.4mb          491.3kb    116.7mb   82.5mb       3mb          113.5mb          1.5gb [REDACTED]     110.8mb 29.9mb            31mb 10.6mb     111kb 400.7mb        26.8mb 741.5mb       1.3mb 
                 152.6kb [REDACTED]                     0b             prd-elastic-a 1.5gb          6.4mb       0b     645.1kb    86.7mb                152.6kb 17.8mb    4.7mb 36.1mb    24mb          458.4kb     79.1mb   48.3mb     1.9mb           75.1mb        859.9mb          0b    84.3mb 12.7mb              0b  4.4mb   836.9kb 197.5mb          11mb 493.9mb     861.5kb 
  CqC9hO3tT2O_j2gMPrc2Uw [REDACTED]                     [REDACTED]     prd-elastic-f 3.1gb         23.1mb       0b     865.4kb   160.3mb CqC9hO3tT2O_j2gMPrc2Uw 18.1mb    8.5mb 49.3mb  35.2mb          559.1kb    135.2mb   80.2mb       3mb          128.7mb          1.9gb [REDACTED]     124.1mb 22.8mb          35.3mb  8.1mb   577.3kb 368.8mb        22.8mb 671.8mb       1.2mb 
                   5.6kb [REDACTED]                     0b             prd-elastic-d   3gb         18.7mb       0b       1.2mb   159.9mb                  5.6kb 12.5mb    9.2mb 51.3mb    38mb          769.9kb    134.1mb     82mb     3.6mb          132.4mb          1.7gb          0b    141mb 25.7mb          58.8mb  9.5mb   399.7kb 408.7mb          23mb 709.8mb       1.3mb 
  WkXcjwj8T6yQ1X2Pb_M2ow [REDACTED]                     [REDACTED]     prd-elastic-b 1.9gb         15.8mb       0b       1.1mb    87.3mb WkXcjwj8T6yQ1X2Pb_M2ow     0b    5.1mb 34.5mb  35.9mb          593.9kb     86.5mb   60.7mb     1.9mb           79.1mb          1.1gb [REDACTED]     87.7mb 19.2mb            31mb  7.2mb   110.6kb 202.9mb        20.3mb 505.8mb       1.2mb 
                 116.6kb [REDACTED]                     38.2mb         prd-elastic-e 3.5gb         25.9mb   21.6kb         1mb   204.5mb                116.6kb     0b   11.1mb 57.2mb    34mb          724.2kb    161.6mb   90.7mb     3.6mb          152.8mb          2.1gb      38.2mb    150mb 31.5mb          29.9mb 10.2mb   615.9kb 368.8mb        32.8mb 764.5mb       1.7mb 
</code></pre></div>
<p dir="auto"><code class="notranslate">ip</code> and <code class="notranslate">id</code> are both repeated, but in the output from <em>some</em> nodes the id and ip fields display the fielddata size for the fields, and <em>some</em> of the nodes display the metadata as I guess they should. I haven't been able to reproduce this locally (it might have been fixed since 1.4.0?), nor have I haven't yet been able to find any pattern to which nodes display fielddata sizes and which display the metadata.</p>
<p dir="auto">Thanks!</p>
<p dir="auto">We should explore the idea of disabling the <code class="notranslate">_all</code> field by default. For<br>
background, the <code class="notranslate">_all</code> field contains the contents of each of a document's<br>
field. During indexing, these contents are copied to the <code class="notranslate">_all</code> field, analyzed<br>
with a specific analyzer, and then indexed. At query time, some queries such as<br>
the <code class="notranslate">query_string</code> and <code class="notranslate">simple_query_string</code> queries search the <code class="notranslate">_all</code> field if<br>
no fields are specified.</p>
<p dir="auto">There are a number of issues with the <code class="notranslate">_all</code> field:</p>
<ul dir="auto">
<li>It uses a fair amount of disk space, due to duplicating the values from all<br>
other fields to an additional field</li>
<li><code class="notranslate">_all</code> has its own analyzer, which is confusing when expecting to query<br>
text analyzed a certain way (or with synonyms, for example) and discovering<br>
that it does not match due to a analysis difference</li>
<li>Additional indexing overhead caused by data duplication</li>
<li>Since the <code class="notranslate">_all</code> field is not retrievable or part of <code class="notranslate">_source</code>, its contents<br>
cannot easily be inspected for debugging purposes, some users do not know it<br>
even exists, which causes confusion at query time</li>
<li>Better alternatives exist, such as the <code class="notranslate">copy_to</code> value on mappings which can<br>
be used to create custom <code class="notranslate">_all</code> fields</li>
</ul>
<p dir="auto">If we were to change the <code class="notranslate">_all</code> field to be disabled by default, some queries<br>
would have to be handled differently. The <code class="notranslate">query_string</code> and<br>
<code class="notranslate">simple_query_string</code> queries would have to know a default field or fields to<br>
query. Perhaps we would be able to change these queries to send a <code class="notranslate">"fields": ["*"]</code> parameter if no fields are specified in the JSON query, so that all field<br>
values can still be queried.</p>