<p dir="auto">Sometimes "child" docs have a common parent ID but are missing a physical parent doc - <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/paullovessearch/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/paullovessearch">@paullovessearch</a> would like to see these "orphans" grouped in search results to achieve pagination of top-ranking groups.</p>
<h3 dir="auto">Background</h3>
<p dir="auto">Number-one-voted issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="26393327" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/4915" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/4915/hovercard" href="https://github.com/elastic/elasticsearch/issues/4915">#4915</a> contained several requests for <strong>paging</strong> in aggregations. Several different use cases were tied up on that issue so it was closed and this issue has been opened to focus on the use case of <em>paginating top-ranked groups</em>. However, my assumption here is that pagination of ranked groups should be achieved by search and not the aggregations framework.</p>
<h3 dir="auto">Pagination = search API</h3>
<p dir="auto">What we established in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="191563378" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/21785" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/21785/hovercard" href="https://github.com/elastic/elasticsearch/issues/21785">#21785</a> is that efficient search (and especially efficient <em>paginated</em> search) depends on heavy down-selection of results at the local level. "Local" can mean same document, same segment (nested docs) or same shard (parent/child). We can rank and page related items only if they are grouped locally (i.e. we do not attempt inefficient pagination on related content that is spread across multiple shards).</p>
<p dir="auto">The existing parent/Child construct is our means of:</p>
<ol dir="auto">
<li>Routing related items to the same shard at index-time</li>
<li>Ensuring a child has exactly one parent and therefore shard</li>
<li>Producing paginated search results for the grouped items</li>
</ol>
<p dir="auto">This is why I think Parent/Child might be the answer to tackling the "top group pagination" requirement.</p>
<h3 dir="auto">Support for "Orphans"</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/paullovessearch/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/paullovessearch">@paullovessearch</a> suggested that in some systems the parent ID for child docs may serve only as a grouping key and there is no physical parent document to group child docs around at query-time. Some suggested using <code class="notranslate">terms</code> aggs and <code class="notranslate">top_hits</code> but this does not provide the pagination support that is required (or the required indexing discipline which means we can rely on the fact related items are held on the same shard).</p>
<h3 dir="auto">Next steps.</h3>
<p dir="auto">I think we have two possible solutions:</p>
<ol dir="auto">
<li>Parent/Child queries are adapted to work with missing physical parent docs</li>
<li>Users wanting pagination should provide physical parent docs</li>
</ol>
<p dir="auto">As with most things this is a cost/benefit analysis:</p>
<p dir="auto">Re costs: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/martijnvg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/martijnvg">@martijnvg</a> - we spoke briefly. How insanely complex is 1) to implement?</p>
<p dir="auto">Re benefits: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/paullovessearch/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/paullovessearch">@paullovessearch</a> - I'd assume parent IDs nearly always have an authoritative data source simply to generate the unique numbers (order numbers, transaction IDs..) and are likely to hold other useful "who/when?" type info e.g. inception dates and instigators. Can you list the use cases where a solution is unlikely to incorporate a data source for parent info?</p>
<p dir="auto">running a term facet on a large string field runs into an OutOfMemoryError (-Xmx50g -Xms50g).</p>
<p dir="auto">Would be great to have the the field cache access disk or that a facet query does not load all the terms of a field into the cache but only the terms of a query.<br>
The second would allow to run multiple facet term query's within different forms and aggregate at the end.</p>
<p dir="auto">Opened issue as discussed in irc.</p>