title: ElasticSearch index used by Haystack is corrupted by an unknown cause
body: <p>I notice that running <code>rebuild_index</code> from the Python shell causes problems with sorting. Running <code>rebuild_index</code> from Bash does the right thing.</p>
  
  <pre><code>$ bin/manage rebuild_index
  WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
  Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
  Are you sure you wish to continue? [y/N] y
  
  Removing all documents from your index because you said so.
  Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
  All documents removed.
  Indexing 40 components.
  </code></pre>
  
  <p>--</p>
  
  <pre><code>In [1]: from haystack.query import SearchQuerySet
  
  In [2]: SearchQuerySet().order_by('name')
  Out[2]: [&lt;SearchResult: my_app.component (pk=u'2')&gt;, &lt;SearchResult: my_app.component (pk=u'1')&gt;, &lt;SearchResult: my_app.component (pk=u'5')&gt;, &lt;SearchResult: my_app.component (pk=u'4')&gt;, &lt;SearchResult: my_app.component (pk=u'3')&gt;, &lt;SearchResult: my_app.component (pk=u'6')&gt;, &lt;SearchResult: my_app.component (pk=u'7')&gt;, &lt;SearchResult: my_app.component (pk=u'8')&gt;, &lt;SearchResult: my_app.component (pk=u'9')&gt;, &lt;SearchResult: my_app.component (pk=u'11')&gt;, &lt;SearchResult: my_app.component (pk=u'14')&gt;, &lt;SearchResult: my_app.component (pk=u'13')&gt;, &lt;SearchResult: my_app.component (pk=u'15')&gt;, &lt;SearchResult: my_app.component (pk=u'19')&gt;, &lt;SearchResult: my_app.component (pk=u'20')&gt;, &lt;SearchResult: my_app.component (pk=u'21')&gt;, &lt;SearchResult: my_app.component (pk=u'22')&gt;, &lt;SearchResult: my_app.component (pk=u'23')&gt;, &lt;SearchResult: my_app.component (pk=u'24')&gt;, '...(remaining elements truncated)...']
  
  In [3]: from django.core import management
  
  In [4]: management.call_command('rebuild_index', interactive=False, verbosity=0)
  Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
  
  In [5]: SearchQuerySet().order_by('name')
  Failed to query Elasticsearch using '*:*': Non-OK status code returned (500) containing u'SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[SkgxFvdjRgyoiHv2qSpfbQ][my_index][4]: QueryPhaseExecutionException[[my_index][4]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))-&gt;cache(_type:modelresult)],from[0],size[20],sort[&lt;custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@3cb9b947&gt;]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][0]: QueryPhaseExecutionException[[my_index][0]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))-&gt;cache(_type:modelresult)],from[0],size[20],sort[&lt;custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@249cf580&gt;]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][3]: QueryPhaseExecutionException[[my_index][3]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))-&gt;cache(_type:modelresult)],from[0],size[20],sort[&lt;custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@4ff80724&gt;]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][2]: QueryPhaseExecutionException[[my_index][2]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))-&gt;cache(_type:modelresult)],from[0],size[20],sort[&lt;custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@99b8b51&gt;]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][1]: QueryPhaseExecutionException[[my_index][1]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))-&gt;cache(_type:modelresult)],from[0],size[20],sort[&lt;custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@42d777c4&gt;]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }]'.
  Out[5]: []
  </code></pre>
  
  <p>I inspected the index metadata using the elasticsearch-head frontend. I found that the settings structure was different depending on which shell I ran <code>rebuild_index</code> from.</p>
  
  <p>The good version (<code>rebuild_index</code> run from Bash):</p>
  
  <pre><code>settings: {
  index.analysis.filter.haystack_ngram.type: nGram
  index.analysis.tokenizer.haystack_edgengram_tokenizer.type: edgeNGram
  index.analysis.analyzer.edgengram_analyzer.filter.0: haystack_edgengram
  index.analysis.filter.haystack_edgengram.min_gram: 2
  index.analysis.tokenizer.haystack_edgengram_tokenizer.side: front
  index.analysis.filter.haystack_edgengram.type: edgeNGram
  index.analysis.filter.haystack_ngram.max_gram: 15
  index.analysis.analyzer.edgengram_analyzer.type: custom
  index.analysis.tokenizer.haystack_ngram_tokenizer.max_gram: 15
  index.analysis.tokenizer.haystack_ngram_tokenizer.type: nGram
  index.analysis.tokenizer.haystack_edgengram_tokenizer.max_gram: 15
  index.analysis.tokenizer.haystack_edgengram_tokenizer.min_gram: 2
  index.analysis.analyzer.ngram_analyzer.filter.0: haystack_ngram
  index.analysis.analyzer.ngram_analyzer.tokenizer: lowercase
  index.analysis.tokenizer.haystack_ngram_tokenizer.min_gram: 3
  index.analysis.filter.haystack_ngram.min_gram: 3
  index.analysis.analyzer.ngram_analyzer.type: custom
  index.analysis.analyzer.edgengram_analyzer.tokenizer: lowercase
  index.analysis.filter.haystack_edgengram.max_gram: 15
  index.number_of_shards: 5
  index.number_of_replicas: 1
  index.version.created: 190999
  }
  </code></pre>
  
  <p>The bad version (<code>rebuild_index</code> run from the Python shell):</p>
  
  <pre><code>settings: {
  index.number_of_shards: 5
  index.number_of_replicas: 1
  index.version.created: 190999
  }
  </code></pre>
  
  <p>What are all those <code>index.analysis.*</code> keys? Where do they come from?</p>
  
  <p>Haystack ticket: <a href="https://github.com/toastdriven/django-haystack/issues/645" rel="nofollow">https://github.com/toastdriven/django-haystack/issues/645</a></p>