<p dir="auto">Sometimes we want to use aggregations to understand the details of how certain entities are connected:</p>
<ul dir="auto">
<li>When did these IP addresses talk to each other?</li>
<li>How much money was sent per day between a selection of bank accounts?</li>
</ul>
<p dir="auto">Right now we can structure this with an agg tree as follows:</p>
<ul dir="auto">
<li>query terms [A,B,C] with <code class="notranslate">min_should_match:2</code></li>
<li>terms agg 1 with <code class="notranslate">include: [A,B,C]</code>
<ul dir="auto">
<li>terms agg 2 with <code class="notranslate">include [A,B,C]</code>
<ul dir="auto">
<li>detail aggs  eg <code class="notranslate">date_histogram</code> or <code class="notranslate">sum</code>...</li>
</ul>
</li>
</ul>
</li>
</ul>
<p dir="auto">This will generate leaf aggregations for all of the possible connections but <em>will generate too many of them.</em> If the number of terms is N the number of leaf buckets created in a worst-case scenario is N² when it need only be (N²-N)/2. We don't want duplicate results for both A-&gt;B and B-&gt;A buckets. Similarly, we don't want A-&gt;A buckets. This trimming can be achieved if <code class="notranslate">terms agg 2</code> only collects terms that are less than the parent bucket's term in <code class="notranslate">terms agg 1</code>. (We could use "greater than" as the filter but the choice is immaterial).</p>
<p dir="auto">We could implement this by either:</p>
<ol dir="auto">
<li>Adding extra parameters to existing <code class="notranslate">terms</code> aggs or</li>
<li>Creating a new dedicated <code class="notranslate">adjacency</code> aggregation</li>
</ol>
<p dir="auto">Option 1 feels clumsy and introduces a need for changes in the agg framework to allow a child to see the keys used by the parent bucket.<br>
The advantage of option 2 is that clients would only need to specify the list of terms once as a parameter and it could provide a more compact results format.</p>
<p dir="auto">Would appreciate your thoughts <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/colings86/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/colings86">@colings86</a> and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jpountz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jpountz">@jpountz</a></p>
<p dir="auto"><strong>Elasticsearch version</strong>: 2.4.1</p>
<p dir="auto">After upgrading to the said version from 2.3 we were greeted with a nasty surprise. Templates are now validated upfront, as if they were <em>complete</em> objects, which they are certainly <strong>not</strong>. This design error implemented in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51177669" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8802" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/8802/hovercard" href="https://github.com/elastic/elasticsearch/pull/8802">#8802</a> is also cooked for 5.0.</p>
<p dir="auto">When a new index is created, all templates are scanned for a match between their pattern and the index name. All applicable templates are then merged in the order indicated by the <code class="notranslate">order</code> property and it's the resulting settings and mappings that are applied to the new index.</p>
<p dir="auto">Then why validate templates for completeness?! It's plain silly, and <em>not even documented</em>.</p>
<p dir="auto"><strong>Example:</strong><br>
All analyzers are defined in a common template that has pattern <code class="notranslate">*</code> and order <code class="notranslate">0</code>. These analyzers can used throughout all indices and it's up to the higher order templates to make actual use of them.<br>
This arrangement is now impossible, as you're using the same validator for templates as for an index mapping (which rightfully checks that all referred anlyzers are present).</p>
<p dir="auto">The mapping validation code should be more insightful and refrain from such checks when working on templates.</p>