<p dir="auto">I have opened this pull request because to discuss if we can return an exception when filtering/querying a field that is not indexed, or that does not exist. As an example of this:</p>
<h2 dir="auto">Querying a non-indexed field</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PUT trial
{
  &quot;mappings&quot;: {
    &quot;document_type1&quot;: {
      &quot;properties&quot;: {
        &quot;prop1&quot;: {
          &quot;index&quot;: &quot;no&quot;,
          &quot;type&quot;: &quot;string&quot;
        }
      }
    }
  }
}

POST trial/document_type1
{
  &quot;prop1&quot;: &quot;sure&quot;
}

POST trial/_search
{
  &quot;query&quot;: {
    &quot;match&quot;: {
      &quot;prop1&quot;: &quot;sure&quot;
    }
  }
}"><pre class="notranslate"><code class="notranslate">PUT trial
{
  "mappings": {
    "document_type1": {
      "properties": {
        "prop1": {
          "index": "no",
          "type": "string"
        }
      }
    }
  }
}

POST trial/document_type1
{
  "prop1": "sure"
}

POST trial/_search
{
  "query": {
    "match": {
      "prop1": "sure"
    }
  }
}
</code></pre></div>
<p dir="auto">This returns:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
   &quot;took&quot;: 13,
   &quot;timed_out&quot;: false,
   &quot;_shards&quot;: {
      &quot;total&quot;: 5,
      &quot;successful&quot;: 5,
      &quot;failed&quot;: 0
   },
   &quot;hits&quot;: {
      &quot;total&quot;: 0,
      &quot;max_score&quot;: null,
      &quot;hits&quot;: []
   }
}"><pre class="notranslate"><code class="notranslate">{
   "took": 13,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 0,
      "max_score": null,
      "hits": []
   }
}
</code></pre></div>
<h2 dir="auto">Querying a non-existing field</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="POST trial/_search
{
  &quot;query&quot;: {
    &quot;match&quot;: {
      &quot;prop2&quot;: &quot;some text&quot;
    }
  }
}"><pre class="notranslate"><code class="notranslate">POST trial/_search
{
  "query": {
    "match": {
      "prop2": "some text"
    }
  }
}
</code></pre></div>
<p dir="auto">This returns:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
   &quot;took&quot;: 13,
   &quot;timed_out&quot;: false,
   &quot;_shards&quot;: {
      &quot;total&quot;: 5,
      &quot;successful&quot;: 5,
      &quot;failed&quot;: 0
   },
   &quot;hits&quot;: {
      &quot;total&quot;: 0,
      &quot;max_score&quot;: null,
      &quot;hits&quot;: []
   }
}"><pre class="notranslate"><code class="notranslate">{
   "took": 13,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 0,
      "max_score": null,
      "hits": []
   }
}
</code></pre></div>
<p dir="auto">Both cases could return an exception, indicating that the field is not indexed, or does not exist. I've marked this issue as a discussion, since there are some cases such as querying multiple indices that could be a bit tricky to add consistency to the result of mixed results of existing fields on certain indices plus non-indexed/existing fields on other indices.</p>
<p dir="auto">Officially filing what I've seen here: <a href="https://groups.google.com/forum/#!topic/elasticsearch/LPv_wPPVTJg" rel="nofollow">https://groups.google.com/forum/#!topic/elasticsearch/LPv_wPPVTJg</a></p>
<p dir="auto">In short, I'm attempting to configure a basic 2-node cluster with ES 0.90.0 on JRE 1.7.0_17, dedicated hardware, slightly different Linux distros. The baseline configuration isn't too complicated, simply 16 shards with predefined mappings and one index. Regardless of discovery mechanism, the nodes seem to discover/elect fine. If I run a single node and load it with the exact same harness, everything works fine. When I have two nodes, system errors in replication begin to happen but I'm not able to tell what the errors are due to what appears to be errors in the Netty transport layer being unable to parse the serialized exception. Stack traces appear similar to:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2013-06-05 09:35:20,480][WARN ][action.index             ] [es-1] Failed to perform index on replica [rules][4]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.StreamCorruptedException: unexpected end of block data
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
    at java.io.ObjectInputStream.access$300(ObjectInputStream.java:205)
    at java.io.ObjectInputStream$GetFieldImpl.readFields(ObjectInputStream.java:2132)
    at java.io.ObjectInputStream.readFields(ObjectInputStream.java:537)
    at java.net.InetSocketAddress.readObject(InetSocketAddress.java:282)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
    ... 23 more
[2013-06-05 09:35:20,483][WARN ][cluster.action.shard     ] [es-1] sending failed shard for [rules][4], node[Ed7LBQdzQMae69IzlCm-Dg], [R], s[STARTED], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2013-06-05 09:35:20,483][WARN ][cluster.action.shard     ] [es-1] received shard failed for [rules][4], node[Ed7LBQdzQMae69IzlCm-Dg], [R], s[STARTED], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]"><pre class="notranslate"><code class="notranslate">[2013-06-05 09:35:20,480][WARN ][action.index             ] [es-1] Failed to perform index on replica [rules][4]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.StreamCorruptedException: unexpected end of block data
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
    at java.io.ObjectInputStream.access$300(ObjectInputStream.java:205)
    at java.io.ObjectInputStream$GetFieldImpl.readFields(ObjectInputStream.java:2132)
    at java.io.ObjectInputStream.readFields(ObjectInputStream.java:537)
    at java.net.InetSocketAddress.readObject(InetSocketAddress.java:282)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
    ... 23 more
[2013-06-05 09:35:20,483][WARN ][cluster.action.shard     ] [es-1] sending failed shard for [rules][4], node[Ed7LBQdzQMae69IzlCm-Dg], [R], s[STARTED], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2013-06-05 09:35:20,483][WARN ][cluster.action.shard     ] [es-1] received shard failed for [rules][4], node[Ed7LBQdzQMae69IzlCm-Dg], [R], s[STARTED], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
</code></pre></div>
<p dir="auto">As additional context, I originally started with compression enabled and then disabled it. I've tested with both ZooKeeper and Zen discovery (unicast and multicast), same results. I've deleted the index as well as starting completely fresh, no change.</p>
<p dir="auto">I'll follow up with config files after I sanitize them. Haven't made any progress asking on the mailing list.</p>