<p dir="auto">Trying to run a test that extends <code class="notranslate">ESIntegTestCase</code> with Elasticsearch 2.0.0, getting this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="java.lang.RuntimeException: found jar hell in test classpath
    at org.elasticsearch.bootstrap.BootstrapForTesting.&lt;clinit&gt;(BootstrapForTesting.java:63)
    at org.elasticsearch.test.ESTestCase.&lt;clinit&gt;(ESTestCase.java:106)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:573)
Caused by: java.lang.IllegalStateException: jar hell!
class: jdk.packager.services.UserJvmOptionsService
jar1: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ant-javafx.jar
jar2: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/packager.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:267)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:185)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86)
    at org.elasticsearch.bootstrap.BootstrapForTesting.&lt;clinit&gt;(BootstrapForTesting.java:61)
    ... 4 more"><pre class="notranslate"><code class="notranslate">java.lang.RuntimeException: found jar hell in test classpath
    at org.elasticsearch.bootstrap.BootstrapForTesting.&lt;clinit&gt;(BootstrapForTesting.java:63)
    at org.elasticsearch.test.ESTestCase.&lt;clinit&gt;(ESTestCase.java:106)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:573)
Caused by: java.lang.IllegalStateException: jar hell!
class: jdk.packager.services.UserJvmOptionsService
jar1: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ant-javafx.jar
jar2: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/packager.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:267)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:185)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86)
    at org.elasticsearch.bootstrap.BootstrapForTesting.&lt;clinit&gt;(BootstrapForTesting.java:61)
    ... 4 more
</code></pre></div>
<p dir="auto">Not sure I can do much about that.</p>
<p dir="auto">ES' hit highlighting abilities are really sophisticated, but I'd like to suggest the API be made a bit more flexible.</p>
<p dir="auto">One use case we have is the user performs their query against the <code class="notranslate">_all</code> field, and our UI needs to highlight matching terms in whatever specific fields the user happens to be viewing.</p>
<p dir="auto">The fact that the <code class="notranslate">{"highlight"}</code> API lets you wildcard field names is pretty great, until you simply specify <code class="notranslate">"*"</code>.  What happens is that ES will complain that certain fields aren't necessarily compatible with the highlighting options specified.  For example, you might see:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2014-04-09 16:48:07,009][DEBUG][action.search.type       ] [Miles Warren] [6838] Failed to execute fetch phase
org.elasticsearch.ElasticsearchIllegalArgumentException: the field [edit_date] should be indexed with term vector with position offsets to be used with fast vector highlighter
        at org.elasticsearch.search.highlight.FastVectorHighlighter.highlight(FastVectorHighlighter.java:68)
        at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:126)
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:444)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:406)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:150)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:134)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)"><pre class="notranslate"><code class="notranslate">[2014-04-09 16:48:07,009][DEBUG][action.search.type       ] [Miles Warren] [6838] Failed to execute fetch phase
org.elasticsearch.ElasticsearchIllegalArgumentException: the field [edit_date] should be indexed with term vector with position offsets to be used with fast vector highlighter
        at org.elasticsearch.search.highlight.FastVectorHighlighter.highlight(FastVectorHighlighter.java:68)
        at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:126)
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:444)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:406)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:150)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:134)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
</code></pre></div>
<p dir="auto">("edit_date" is defined as a date field and of course has no "index_options" on it).</p>
<p dir="auto">And if you hack a bit at the ES sources to silently ignore these errors you'll see it also complains about ES' core fields such as <code class="notranslate">_id</code>, <code class="notranslate">_timestamp</code>, <code class="notranslate">_source</code>, etc.</p>
<p dir="auto">What I'd like to propose is an option in the <code class="notranslate">{ "highlight" }</code> API to be lenient about mismatches between field mapping definitions and highlight parameters.  So if the highlight parameters aren't compatible with the mapping, ES will just move on to the next field, rather than bomb the entire request.</p>
<p dir="auto">Call it a best-efforts highlighting for "all" fields.</p>
<p dir="auto">Maybe it could fall back to decreasingly less sophisticated highlighters, until it just can't do anything.  So if the request indicates the <code class="notranslate">fvh</code> be used, and that's not compatible with a particular field, it tries the <code class="notranslate">postings</code> highlighter, then the <code class="notranslate">plain</code> highlighter, then nothing.</p>
<p dir="auto">Additionally, I'd like to see a highlight parameter to exclude fields.  Another use case we have is, "I want to highlight all matched fields (whatever they are), except this array of 'large text' fields that I'm not currently showing the user" -- this would really be a bit of a performance optimization, from the client.  Our data model includes a couple of large text fields (10k+) per document that wouldn't need to be highlighted except at very specific times -- the other fields are small and generally need to be highlighted in one call to ES.</p>
<p dir="auto">Thoughts?</p>
<p dir="auto">If y'all think this is a good idea, I'm happy to fork, do the work, make a pull request, and work with the committer(s) until it's accepted.  I basically need to do this anyways, so I'd rather have buy-in instead of maintaining my own patched version of ES.</p>
<p dir="auto">I thought about doing this as a plugin, but it looked like I'd have to duplicate quite a bit of core ES classes, which doesn't seem maintainable in the long run.</p>