title: How to handle dynamic index creation in Elasticsearch using Apache NiFi?
body: <p>I am routing data through to Elasticsearch using Nifi. I'm using NiFi to dynamically create indices based on a set of attributes.  I'm using Index Lifecycle Policy Management in Elasticsearch which requires all indices to be manually bootstrapped beforehand for ILM settings to be applied.  Since my NiFi flow automatically ingests messages into Elasticsearch any index created automatically will not have have ILM policies applied.</p>
  <p>Currently my flow is Nifi Consume from Kafka --&gt; Update Attribute --&gt; PutElasticsearch Record.</p>
  <p>A solution (I think) would be to call the <code>invokehttp</code> processor in front of the <code>PutElasticsearch</code> processor to bootstrap the indices dynamically via the attributes extracted before ingesting into elasticsearch.  Indices are dynamically created using the syntax: <code>index_${attribute_1}_${attribute_2}</code>. My only concern here is the invoke invokehttpprocessor would run with every new flowfile.  This could be thousands of calls to bootstrap an index.  And if the index already exists there could be collision there.</p>
  <p>Is this really the best way to do this?  Perhaps I could run the <code>QueryElasticsearchRecord</code> processor to get a list of indices and somehow match that against incoming flowfiles on the <code>attribute_1</code> and <code>attribute_2</code> field.  But that would still require a continuous query, I think?</p>