<p dir="auto">As part of the Great Mapping Refactoring (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51553271" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8870" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8870/hovercard" href="https://github.com/elastic/elasticsearch/issues/8870">#8870</a>), we had to reject field names containing dots (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="93414385" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/12068" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/12068/hovercard" href="https://github.com/elastic/elasticsearch/pull/12068">#12068</a>), eg:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{ 
  &quot;foo.bar&quot;: &quot;val1&quot;,  
  &quot;foo&quot;: {
    &quot;bar&quot;: &quot;val2&quot;
  }
}"><pre class="notranslate"><code class="notranslate">{ 
  "foo.bar": "val1",  
  "foo": {
    "bar": "val2"
  }
}
</code></pre></div>
<p dir="auto">The behaviour was undefined and resulted in ambiguities when trying to reference fields with the dot notation used in queries and aggregations.</p>
<p dir="auto">Removing support for dots has caused pain for a number of users and especially as Elasticsearch is being used more and more for the metrics use case (where dotted fields are common), we should consider what we can do to improve this situation.  Now that mappings are much stricter (and immutable), it becomes feasible to revisit the question of whether to allow dots to occur in field names.</p>
<h1 dir="auto">Replace dots with another character</h1>
<p dir="auto">The first and simplest solution is to simply replace dots in field names with another character (eg <code class="notranslate">_</code>) as is done by the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html" rel="nofollow">Logstash de_dot filter</a> and which will be supported natively in Elasticsearch by the <a href="https://github.com/elastic/elasticsearch/issues/15944" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/15944/hovercard">node ingest <code class="notranslate">de_dot</code> processor</a>.</p>
<h1 dir="auto">Treat dots as paths</h1>
<p dir="auto">Another solution would be to treat fields with dots in them as "paths" rather than field names.  In other words, these two documents would be equivalent:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{ &quot;foo.bar&quot;: &quot;value&quot; }
{ &quot;foo&quot;: { &quot;bar&quot;: &quot;value&quot; }}"><pre class="notranslate"><code class="notranslate">{ "foo.bar": "value" }
{ "foo": { "bar": "value" }}
</code></pre></div>
<p dir="auto">To use an edge case as an example, the following document:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo.bar&quot; : {
    &quot;baz&quot;: &quot;val1&quot;
  },
  &quot;foo&quot;: {
    &quot;bar.baz&quot;: &quot;val2&quot;
  }"><pre class="notranslate"><code class="notranslate">{
  "foo.bar" : {
    "baz": "val1"
  },
  "foo": {
    "bar.baz": "val2"
  }
</code></pre></div>
<p dir="auto">}</p>
<p dir="auto">would result in the following mapping:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;properties&quot;: {
    &quot;foo&quot;: {
      &quot;type&quot;: &quot;object&quot;,
      &quot;properties&quot;: {
        &quot;bar&quot;: {
          &quot;type&quot;: &quot;object&quot;,
          &quot;properties&quot;: {
            &quot;baz&quot;: {
              &quot;type&quot;: &quot;string&quot;
            }
          }
        }
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "properties": {
    "foo": {
      "type": "object",
      "properties": {
        "bar": {
          "type": "object",
          "properties": {
            "baz": {
              "type": "string"
            }
          }
        }
      }
    }
  }
}
</code></pre></div>
<p dir="auto">The lucene field would be called <code class="notranslate">foo.bar.baz</code> and would contain the terms <code class="notranslate">["val1", "val2]</code>.  Stored fields or doc values (for supported datatypes), would both contain <code class="notranslate">["val1", "val2"]</code>.</p>
<h2 dir="auto">Issues with this approach</h2>
<p dir="auto">This solution works well for search and aggregations, but leaves us with two incongruities:</p>
<h3 dir="auto"><code class="notranslate">_source=</code></h3>
<p dir="auto">The first occurs when using the <code class="notranslate">_source=</code> parameter to do source filtering on the response. The reason for this is that the <code class="notranslate">_source</code> field is stored as provided - it is not normalized before being stored  For instance:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET _search?_source=foo.*"><pre class="notranslate"><code class="notranslate">GET _search?_source=foo.*
</code></pre></div>
<p dir="auto">would return:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo.bar&quot; : {
    &quot;baz&quot;: &quot;val1&quot;
  },
  &quot;foo&quot;: {
    &quot;bar.baz&quot;: &quot;val2&quot;
  }"><pre class="notranslate"><code class="notranslate">{
  "foo.bar" : {
    "baz": "val1"
  },
  "foo": {
    "bar.baz": "val2"
  }
</code></pre></div>
<p dir="auto">}</p>
<p dir="auto">rather than:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo&quot;: {
    &quot;bar&quot;: {
      &quot;baz&quot;: [
        &quot;val1&quot;,
        &quot;val2&quot;
      ]
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "foo": {
    "bar": {
      "baz": [
        "val1",
        "val2"
      ]
    }
  }
}
</code></pre></div>
<h3 dir="auto">Update requests</h3>
<p dir="auto">The second occurs during update requests, which uses the <code class="notranslate">_source</code> as a map-of-maps.  Running an update like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST index/type/id/_update
{
  &quot;doc&quot;: {
    &quot;foo&quot;: {
      &quot;bar&quot;: {
        &quot;baz&quot;: &quot;val3&quot;
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">POST index/type/id/_update
{
  "doc": {
    "foo": {
      "bar": {
        "baz": "val3"
      }
    }
  }
}
</code></pre></div>
<p dir="auto">could result (depending on how it is implemented) in any of the following:</p>
<p dir="auto">Version 1:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo&quot;: {
    &quot;bar&quot;: {
      &quot;baz&quot;: &quot;val3&quot;
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "foo": {
    "bar": {
      "baz": "val3"
    }
  }
}
</code></pre></div>
<p dir="auto">Version 2:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo&quot;: {
    &quot;bar&quot;: {
      &quot;baz&quot;: [
        &quot;val1&quot;,
        &quot;val2&quot;,
        &quot;val3&quot;
      ]
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "foo": {
    "bar": {
      "baz": [
        "val1",
        "val2",
        "val3"
      ]
    }
  }
}
</code></pre></div>
<p dir="auto">Version 3:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;foo.bar&quot;: {
    &quot;baz&quot;: &quot;val1&quot;
  },
  &quot;foo&quot;: {
    &quot;bar.baz&quot;: &quot;val2&quot;,
    &quot;bar&quot;: {
      &quot;baz&quot;: &quot;val3&quot;
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "foo.bar": {
    "baz": "val1"
  },
  "foo": {
    "bar.baz": "val2",
    "bar": {
      "baz": "val3"
    }
  }
}
</code></pre></div>
<p dir="auto"><strong>Elasticsearch version</strong>: 2.4.1</p>
<p dir="auto"><strong>Plugins installed</strong>: []</p>
<p dir="auto"><strong>JVM version</strong>: 1.8.0_91</p>
<p dir="auto"><strong>OS version</strong>: OS X 10.11.6 and Amazon Linux 2016.03</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>: In 1.7.5, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-templates.html#multiple-templates" rel="nofollow">multiple template matching</a> worked as documented. In 2.4.1, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#multiple-templates" rel="nofollow">multiple template matching</a> does not work if the template refers to an analyzer defined in a previous template.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Create a template that defines a custom analyzer. This template matches all indices ("template": "*") and is ordered first ("order": 0).</li>
<li>Create a template that defines a mapping and uses the custom analyzer from the first template.</li>
<li>The second template cannot be created with Elasticsearch 2.4.1, but this has worked in all prior 1.x releases of Elasticsearch.</li>
</ol>
<p dir="auto">A gist of the recreate steps is here: <a href="https://gist.github.com/seallison/a4b916a1606c8f8ecc1450a021996509">https://gist.github.com/seallison/a4b916a1606c8f8ecc1450a021996509</a></p>