events.action: opened
events.author: mclate
events.masked_author: username_0
events.text: ES version: 7.1
  
  Error log:
  ```
  Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
  [2019-09-04T23:43:08,613][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
  [2019-09-04T23:43:08,742][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
  [2019-09-04T23:43:09,463][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.1.1"}
  [2019-09-04T23:43:09,512][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"81a1f248-f42e-4b21-942a-daf248b2573a", :path=>"/usr/share/logstash/data/uuid"}
  url template
  {:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
  [2019-09-04T23:43:21,304][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://.....us-east-1.es.amazonaws.com:443/]}}
  [2019-09-04T23:43:21,318][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://.....us-east-1.es.amazonaws.com:443/, :path=>"/"}
  [2019-09-04T23:43:22,476][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://.....us-east-1.es.amazonaws.com:443/"}
  [2019-09-04T23:43:22,583][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
  [2019-09-04T23:43:22,588][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
  [2019-09-04T23:43:22,624][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//.....us-east-1.es.amazonaws.com"]}
  [2019-09-04T23:43:22,663][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
  [2019-09-04T23:43:22,672][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x2c42be78 run>"}
  [2019-09-04T23:43:22,718][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"_doc"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
  [2019-09-04T23:43:22,851][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
  [2019-09-04T23:43:22,964][INFO ][logstash.outputs.elasticsearch] Installing amazon_es template to /_template/logstash
  [2019-09-04T23:43:22,965][INFO ][logstash.inputs.gelf     ] Starting gelf listener (udp) ... {:address=>"0.0.0.0:12201"}
  [2019-09-04T23:43:23,068][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
  [2019-09-04T23:43:23,162][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://.....us-east-1.es.amazonaws.com:443/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:126:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client.rb:365:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/http_client.rb:88:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/common.rb:118:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-6.4.2-java/lib/logstash/outputs/amazon_es/common.rb:49:in `block in install_template_after_successful_connection'"]}
  [2019-09-04T23:43:23,759][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
  ```
  
  I've tried to manually install template from `elasticsearch-template-es7x.json` and got next error:
  ```json
  {
    "error": {
      "root_cause": [
        {
          "type": "illegal_argument_exception",
          "reason": "The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."
        }
      ],
      "type": "illegal_argument_exception",
      "reason": "The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."
    },
    "status": 400
  }
  ```
events.title: Failed to install template on AWS ES 7.1
events.type: issue
events.action: created
events.author: asomir
events.masked_author: username_1
events.text: I have the same problem
events.type: comment
events.action: created
events.author: mclate
events.masked_author: username_0
events.text: So far i was able to fix it by manually applying logstash template using data from `elasticsearch-template-es7x.json` and applying it to `POST _template/logstash?include_type_name=true`
events.type: comment
events.action: created
events.author: ksemaev
events.masked_author: username_2
events.text: @username_1 @username_0  Thanks for the help guys! Still the console just hangs after that request with no output :(  any ` GET /_cat/templates?v&s=name` showes no templates at all 
  Tried with
  ```
  POST _template/logstash?include_type_name=true
  {
    "index_patterns" : "logstash-*",
    "version" : 60001,
    "settings" : {
      "index.refresh_interval" : "5s",
      "number_of_shards": 1
    },
    "mappings" : {
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword", "ignore_above": 256 }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date"},
        "@version": { "type": "keyword"},
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }
      }
    }
  }
  ```
  Any other advices are highly appreciated!
events.type: comment
events.action: created
events.author: briandicola
events.masked_author: username_3
events.text: Please correct me if I'm wrong, but I believe this plugin just does not support version 7 ELK.  I've gotten it to work with logstash 1:6.4.3-1 and AWS ES version 6.4.
events.type: comment
events.action: created
events.author: mclate
events.masked_author: username_0
events.text: @username_2 here is what worked for me (copied from kibana interface):
  ```json
  POST _template/logstash?include_type_name=true
  {
    "template" : "logstash-*",
    "version" : 60001,
    "settings" : {
      "index.refresh_interval" : "5s",
      "number_of_shards": 1
    },
    "mappings" : {
      "_doc" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text",
                "norms" : false
              }
            }
          },
          {
            "string_fields" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text", "norms" : false,
                "fields" : {
                  "keyword" : { "type": "keyword", "ignore_above": 256 }
                }
              }
            }
          }
        ],
        "properties" : {
          "@timestamp": { "type": "date"},
          "@version": { "type": "keyword"},
          "geoip"  : {
            "dynamic": true,
            "properties" : {
              "ip": { "type": "ip" },
              "location" : { "type" : "geo_point" },
              "latitude" : { "type" : "half_float" },
              "longitude" : { "type" : "half_float" }
            }
          }
        }
      }
    }
  }
  ```
  You should get something like `{"acknowledged": true}`
  
  @username_3 Guess it's not, but aside from applying template during startup it works fine for us - i can see records being sent to our ES cluster. No issues so far.
events.type: comment
events.action: created
events.author: malpani
events.masked_author: username_4
events.text: I will bring this up to the team to prioritize a fix.
  
  In the meantime, thanks for the workaround and if you already have a PR, happy to pull it in
events.type: comment
events.action: created
events.author: briandicola
events.masked_author: username_3
events.text: This worked for me.  Thanks.
events.type: comment
events.action: created
events.author: yadavcbala
events.masked_author: username_5
events.text: The pull request #130 should resolve this issue. 
  *From elasticsearch-7.0, types have been removed and request parameter include_type_name which omits types is defaulted to false. This will fail while installing the template with types. The PR #130 updates the `elasticsearch-template-es7x.json` template to remove types*
events.type: comment
events.action: closed
events.author: yadavcbala
events.masked_author: username_5
events.type: issue