<p dir="auto">Hello,</p>
<p dir="auto">I was trying to create czech lemmatisation analyzer using <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/stemmer-override-tokenfilter.html" rel="nofollow">stemmer override filter</a> and czech dictionary from aspell.</p>
<p dir="auto">This dictionary contains around <code class="notranslate">300 000</code> words in base form and some suffix/prefix rules. After expansion format file looks like this</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Aakjaer Aakjaerech Aakjaery Aakjaerům Aakjaerů Aakjaerem Aakjaere Aakjaerovi Aakjaeru Aakjaera Aakjaerové
Aakjaerová Aakjaerovými Aakjaerovým Aakjaerových Aakjaerovou Aakjaerové
Aakjaerův Aakjaerovýma Aakjaerovými Aakjaerových Aakjaerovou Aakjaerovo Aakjaerovy Aakjaerovi Aakjaerovým "><pre class="notranslate"><code class="notranslate">Aakjaer Aakjaerech Aakjaery Aakjaerům Aakjaerů Aakjaerem Aakjaere Aakjaerovi Aakjaeru Aakjaera Aakjaerové
Aakjaerová Aakjaerovými Aakjaerovým Aakjaerových Aakjaerovou Aakjaerové
Aakjaerův Aakjaerovýma Aakjaerovými Aakjaerových Aakjaerovou Aakjaerovo Aakjaerovy Aakjaerovi Aakjaerovým 
</code></pre></div>
<p dir="auto">each line is one word with its forms.</p>
<p dir="auto">Because of rules format <code class="notranslate">form =&gt; lemma</code> the final rule set is expanded from <code class="notranslate">300 000</code> to <code class="notranslate">4 364 674</code> lines.</p>
<p dir="auto">When I was trying on my local machine to index czech wikipedia pages (around <code class="notranslate">400 000</code> documents) <code class="notranslate">java.lang.OutOfMemoryError: Java heap space</code> error occured after approx 10 minutes of indexing (log file <a href="https://gist.github.com/vhyza/9f88b921a1d0f650ccd8#file-elasticsearch-log">here</a>)</p>
<p dir="auto">I'm using snapshot build of elasticsearch (<a href="https://github.com/elasticsearch/elasticsearch/commit/54e7e309a5d407b2fb1123a79e6af9d62e41ea1e">54e7e309a5d407b2fb1123a79e6af9d62e41ea1e</a>), <code class="notranslate">JAVA_OPTS -Xss200000 -Xms2g -Xmx2g</code> with no other indices.</p>
<p dir="auto">Index settings/mapping and river settings are in separate <a href="https://gist.github.com/vhyza/9f88b921a1d0f650ccd8#file-stemmer_override_test-sh">gist</a></p>
<p dir="auto">I was trying to achieve this functionality using <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/synonym-tokenfilter.html" rel="nofollow">synonym token filter</a>, because of better format of synonym rules - <code class="notranslate">form1, form2, form3, form4 =&gt; lemma</code> (so number of rules are only about <code class="notranslate">300 000</code>).</p>
<p dir="auto">But it's not the same. In the case of using <code class="notranslate">stemmer override filter</code>, when token was not found in rule set, stemmer was used. I probably can do the same by adding <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/keyword-marker-tokenfilter.html" rel="nofollow">keyword marker</a> and <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/stemmer-tokenfilter.html" rel="nofollow">stemmer</a> in the filter chain, but I don't think it is the right way to do that.</p>
<p dir="auto">Please, is there some better 'compressed' format of <code class="notranslate">stemmer override filter</code> rules? Any thoughts how to avoid <code class="notranslate">java.lang.OutOfMemoryError: Java heap space</code> error?</p>
<p dir="auto"><strong>Elasticsearch version</strong>: 5.0.0</p>
<p dir="auto"><strong>Plugins installed</strong>: None</p>
<p dir="auto"><strong>JVM version</strong>: 1.8.0_101</p>
<p dir="auto"><strong>OS version</strong>: Ubuntu 16.04</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:<br>
I have added multiple context mappings to a completion suggester field. In version 2.x and earlier, when filtering on those contexts, suggestions were only returned if they matched all of the contexts. But that does not seem to be the case with 5.0.0.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Mapping for my suggest field:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;suggest&quot;: {
  &quot;type&quot;: &quot;completion&quot;,
  &quot;contexts&quot;: [
      {
        &quot;name&quot;: &quot;companyContext&quot;,
        &quot;type&quot;: &quot;category&quot;
     },
     {
        &quot;name&quot;: &quot;usersContext&quot;, 
        &quot;type&quot;: &quot;category&quot;
     }                     
  ]
}"><pre class="notranslate"><code class="notranslate">"suggest": {
  "type": "completion",
  "contexts": [
      {
        "name": "companyContext",
        "type": "category"
     },
     {
        "name": "usersContext", 
        "type": "category"
     }                     
  ]
}
</code></pre></div>
<ol start="2" dir="auto">
<li>Index a document with suggest field contexts populated:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;suggest&quot;: {
    &quot;input&quot;: &quot;test team site&quot;,
    &quot;contexts&quot;: {
        &quot;companyContext&quot;: &quot;vendor&quot;,
        &quot;usersContext&quot;: [&quot;chad&quot;, &quot;fred&quot;]            
    }        
}"><pre class="notranslate"><code class="notranslate">"suggest": {
    "input": "test team site",
    "contexts": {
        "companyContext": "vendor",
        "usersContext": ["chad", "fred"]            
    }        
}
</code></pre></div>
<ol start="3" dir="auto">
<li>Run a query supplying only one matching context:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;suggest&quot;: {
   &quot;suggestions&quot;: {
        &quot;text&quot;: &quot;tes&quot;,
        &quot;completion&quot;: {
            &quot;field&quot;: &quot;suggest&quot;,
            &quot;contexts&quot;:{
                &quot;companyContext&quot;: &quot;vendor&quot;,
                &quot;usersContext&quot;: &quot;bill&quot;
            }
        }
    }
}"><pre class="notranslate"><code class="notranslate">"suggest": {
   "suggestions": {
        "text": "tes",
        "completion": {
            "field": "suggest",
            "contexts":{
                "companyContext": "vendor",
                "usersContext": "bill"
            }
        }
    }
}
</code></pre></div>
<p dir="auto">I would expect that this would not return the indexed document, because the usersContext is not a match. And that was the behavior in 2.x. But in 5.0.0, the document that matches only the companyContext is returned.</p>