<p dir="auto">Hello,</p>
<p dir="auto">I've got an application which uses match_phrase queries to look for, and highlight, exact matches within file texts. It worked fantastically on ElasticSearch version 2.0.0 and 2.0.1, but breaks on 2.1.0.</p>
<p dir="auto">The query appears to run correctly but the highlighter highlights ALL instances of terms in the query instead of just those that appear within matched phrases. For example, if the query contains the word "is", then every instance of the word "is" - throughout the entire document - is highlighted.</p>
<h1 dir="auto">Example:</h1>
<p dir="auto">As an example, I've prepared a series of commands in the Sense chrome plugin to replicate the problem:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST /test_index
{
   &quot;settings&quot;: {
      &quot;number_of_shards&quot;: 1,
      &quot;number_of_replicas&quot;: 0
   },
       &quot;mappings&quot; : {
        &quot;file&quot; : {
            &quot;properties&quot; : {
                &quot;text&quot; : { &quot;type&quot; : &quot;string&quot;}
            }
        }
    }
}

POST /test_index/file
{
    &quot;text&quot;: &quot;The quick brown fox jumped over the other, lazier, fox.&quot;
}

POST /test_index/file
{
    &quot;text&quot;: &quot;Doc Brown is one quick fox.&quot;
}

POST /test_index/file/_search
{
   &quot;query&quot;: {
      &quot;match_phrase&quot;: {
         &quot;text&quot;: &quot;quick brown fox&quot;
      }
   },
   &quot;highlight&quot;: {
      &quot;fields&quot;: {
         &quot;text&quot;: {
            &quot;number_of_fragments&quot;: 0
         }
      }
   }
}"><pre class="notranslate"><code class="notranslate">POST /test_index
{
   "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
   },
       "mappings" : {
        "file" : {
            "properties" : {
                "text" : { "type" : "string"}
            }
        }
    }
}

POST /test_index/file
{
    "text": "The quick brown fox jumped over the other, lazier, fox."
}

POST /test_index/file
{
    "text": "Doc Brown is one quick fox."
}

POST /test_index/file/_search
{
   "query": {
      "match_phrase": {
         "text": "quick brown fox"
      }
   },
   "highlight": {
      "fields": {
         "text": {
            "number_of_fragments": 0
         }
      }
   }
}
</code></pre></div>
<h2 dir="auto">Version 2.0.1 results</h2>
<p dir="auto">Under version 2.0.1, the search call returns the following hits array:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&quot;hits&quot;: [
    {
        &quot;_index&quot;: &quot;test_index&quot;,
        &quot;_type&quot;: &quot;file&quot;,
        &quot;_id&quot;: &quot;AVF-PCoeysoVw9xNBYa3&quot;,
        &quot;_score&quot;: 0.55737644,
        &quot;_source&quot;:{
            &quot;text&quot;: &quot;The quick brown fox jumped over the other, lazier, fox.&quot;
        },
        &quot;highlight&quot;: {
            &quot;text&quot;: [
                &quot;The &lt;em&gt;quick&lt;/em&gt; &lt;em&gt;brown&lt;/em&gt; &lt;em&gt;fox&lt;/em&gt; jumped over the other, lazier, fox.&quot;
            ]
        }
    }
]"><pre class="notranslate"><span class="pl-ent">"hits"</span>: [
    {
        <span class="pl-ent">"_index"</span>: <span class="pl-s"><span class="pl-pds">"</span>test_index<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_type"</span>: <span class="pl-s"><span class="pl-pds">"</span>file<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>AVF-PCoeysoVw9xNBYa3<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_score"</span>: <span class="pl-c1">0.55737644</span>,
        <span class="pl-ent">"_source"</span>:{
            <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>The quick brown fox jumped over the other, lazier, fox.<span class="pl-pds">"</span></span>
        },
        <span class="pl-ent">"highlight"</span>: {
            <span class="pl-ent">"text"</span>: [
                <span class="pl-s"><span class="pl-pds">"</span>The &lt;em&gt;quick&lt;/em&gt; &lt;em&gt;brown&lt;/em&gt; &lt;em&gt;fox&lt;/em&gt; jumped over the other, lazier, fox.<span class="pl-pds">"</span></span>
            ]
        }
    }
]</pre></div>
<p dir="auto">This is the expected and desired output.</p>
<h2 dir="auto">Version 2.1.0 results</h2>
<p dir="auto">Under version 2.1.0, the search call returns the following hits array:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&quot;hits&quot;: [
    {
        &quot;_index&quot;: &quot;test_index&quot;,
        &quot;_type&quot;: &quot;file&quot;,
        &quot;_id&quot;: &quot;AVF-MdXsEcdhEnh7dUcO&quot;,
        &quot;_score&quot;: 0.55737644,
        &quot;_source&quot;: {
            &quot;text&quot;: &quot;The quick brown fox jumped over the other, lazier, fox.&quot;
        },
        &quot;highlight&quot;: {
            &quot;text&quot;: [
                &quot;The &lt;em&gt;quick&lt;/em&gt; &lt;em&gt;brown&lt;/em&gt; &lt;em&gt;fox&lt;/em&gt; jumped over the other, lazier, &lt;em&gt;fox&lt;/em&gt;.&quot;
            ]
        }
    }
]"><pre class="notranslate"><span class="pl-ent">"hits"</span>: [
    {
        <span class="pl-ent">"_index"</span>: <span class="pl-s"><span class="pl-pds">"</span>test_index<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_type"</span>: <span class="pl-s"><span class="pl-pds">"</span>file<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>AVF-MdXsEcdhEnh7dUcO<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"_score"</span>: <span class="pl-c1">0.55737644</span>,
        <span class="pl-ent">"_source"</span>: {
            <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>The quick brown fox jumped over the other, lazier, fox.<span class="pl-pds">"</span></span>
        },
        <span class="pl-ent">"highlight"</span>: {
            <span class="pl-ent">"text"</span>: [
                <span class="pl-s"><span class="pl-pds">"</span>The &lt;em&gt;quick&lt;/em&gt; &lt;em&gt;brown&lt;/em&gt; &lt;em&gt;fox&lt;/em&gt; jumped over the other, lazier, &lt;em&gt;fox&lt;/em&gt;.<span class="pl-pds">"</span></span>
            ]
        }
    }
]</pre></div>
<p dir="auto">This is the same match, and even has the same score, but the word "fox" is incorrectly highlighted when it occurs without the rest of the phrase.</p>
<h2 dir="auto">Tested solutions</h2>
<p dir="auto">In attempt to rectify this problem, I attempted two things:</p>
<ol dir="auto">
<li>First, on the theory that perhaps it had switched to the postings highlighter rather than the plain highlighter, I attempted to force the use of the plain version with the <code class="notranslate">"type": "plain"</code> flag. Unfortunately, this had no visible effect.</li>
<li>Second, duplicated the original query within a <code class="notranslate">"highlight_query"</code> block. This too, unfortunately, had no visible effect.</li>
</ol>
<h1 dir="auto">Theory</h1>
<p dir="auto">Currently, my theory is that this problem is related to the upgrade to lucene-5.3.0. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="104231337" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/13239" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/13239/hovercard" href="https://github.com/elastic/elasticsearch/pull/13239">#13239</a>)<br>
Specifically, I wonder if it has to do with how NearSpansOrdered are processed. (<a href="https://issues.apache.org/jira/browse/LUCENE-6537" rel="nofollow">https://issues.apache.org/jira/browse/LUCENE-6537</a>)</p>
<p dir="auto">That said, this is purely conjecture. It's quite possible - or even likely - that the problem lies elsewhere.<br>
I'm hoping that an elasticsearch expert might have a better idea.</p>
<p dir="auto">Thank you.</p>
<p dir="auto">Hello,</p>
<p dir="auto">After upgrading our Production cluster to Version 1.3.5 , the cluster is  constantly  in red state with :<br>
{<br>
"cluster_name" : "totango_prod_hist",<br>
"status" : "red",<br>
"timed_out" : false,<br>
"number_of_nodes" : 26,<br>
"number_of_data_nodes" : 20,<br>
"active_primary_shards" : 100,<br>
"active_shards" : 200,<br>
"relocating_shards" : 0,<br>
"initializing_shards" : 2,<br>
"unassigned_shards" : 2<br>
}</p>
<p dir="auto">When i  went over the logs i found this exception:</p>
<p dir="auto">2014-11-19 16:21:01,253][WARN ][index.engine.internal    ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]<br>
[2014-11-19 16:21:01,401][WARN ][cluster.action.shard     ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@739f95fa]]]<br>
[2014-11-19 16:23:52,294][WARN ][index.engine.internal    ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]<br>
[2014-11-19 16:23:52,569][WARN ][cluster.action.shard     ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@503839bb]]]<br>
[2014-11-19 16:26:42,678][WARN ][index.engine.internal    ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]<br>
[2014-11-19 16:26:42,934][WARN ][cluster.action.shard     ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@3674e0a0]]]<br>
[2014-11-19 16:29:32,599][WARN ][index.engine.internal    ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]<br>
[2014-11-19 16:29:32,855][WARN ][cluster.action.shard     ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@5a27fe79]]]<br>
[2014-11-19 16:32:23,194][WARN ][index.engine.internal    ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]<br>
[2014-11-19 16:32:23,442][WARN ][cluster.action.shard     ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@39c4890a]]]</p>
<p dir="auto">Please advise ?</p>
<p dir="auto">Thanks,</p>
<p dir="auto">Costya.</p>