Answers: <p>There are two ways to accomplish this, though one of them was only available recently.</p>
  
  <p>The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...</p>
  
  <pre><code>input {
    tcp {
      port =&gt; 1525
      codec =&gt; json_lines
      tags =&gt; [ 'tcp' ]
    }
  }
  
  input {
    file {
      path =&gt; '/var/log/app.log'
      codec =&gt; 'json'
      tags =&gt; [ 'file' ]
    }
  }
  
  output {
    if 'file' in [tags] {
      elasticsearch {
        host =&gt; 'logstash-es'
        index =&gt; 'files'
      }
    }
    if 'tcp' in [tags] {
      elasticsearch {
        host =&gt; 'logstash-es'
        index =&gt; 'tcp'
      }
    }
  }
  </code></pre>
  
  <p>This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with <a href="https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html" rel="nofollow noreferrer">a way to do multiple pipelines in separate files</a>.</p>
  
  <pre><code>- pipeline.id: tcp-inputs
    path.config: '/etc/logstash/pipelines/tcp.cfg'
    pipeline.workers: 3
  - pipeline.id: file-inputs
    path.config: '/etc/logstash/pipelines/files.cfg'
    pipeline.workers: 2
  </code></pre>
  
  <p>This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-08-14T12:59:59.787Z
Answers: 2018-08-14T12:59:59.787Z
Answers: sysadmin1138
Answers: <p>Once parsed your config create one and only one pipeline, with various inputs, various filters and various outputs. </p>
  
  <p>You have to use some conditional constructs to apply filter and output only to specific messages, usually using a special tag or field set on input.</p>
  
  <p>Some exemple are available in <a href="https://www.elastic.co/guide/en/logstash/current/config-examples.html#using-conditionals" rel="nofollow noreferrer">logstash documentation</a>:</p>
  
  <blockquote>
  <pre><code>output {
    if [type] == "apache" {
      if [status] =~ /^5\d\d/ {
        nagios { ...  }
      } else if [status] =~ /^4\d\d/ {
        elasticsearch { ... }
      }
      statsd { increment =&gt; "apache.%{status}" }
    }
  }
  </code></pre>
  </blockquote>
  
Answers: CC BY-SA 3.0
Answers: 2018-04-16T09:19:33.923Z
Answers: I tried it with conditionals and had the same outcome
Answers: Saar Grin
Answers: CC BY-SA 3.0
Answers: 2018-04-16T09:20:09.650Z
Answers: Turns out the proper way to do it is with pipelines.yml file which i somehow missed.
Answers: Saar Grin
Answers: CC BY-SA 3.0
Answers: 2018-04-16T09:30:33.983Z
Answers: Concerning your first comment, without what you tried, there's no way to tell what could have been wrong.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-04-16T09:14:12.773Z
Answers: 2018-04-16T09:14:12.773Z
Answers: Tensibai
Body: <p>I'm trying to set up a centralized syslog for multiple log sources.</p>
  
  <p>So I have a logstash that has two separate inputs and two separate outputs</p>
  
  <p>However for some reason the data from one of the inputs ends up in both indexes</p>
  
  <p>What am i doing wrong?</p>
  
  <p>Below are both pipelines' configs</p>
  
  <pre><code>input {
    tcp {
      port =&gt; 5052
      codec =&gt; "json_lines"
    }
  }
  output {
    elasticsearch {
      hosts =&gt; "10.50.6.116"
      index =&gt; "remote"
    }
    file {
      path =&gt; "/var/log/logstash/remote-tcp.log"
    }
    stdout { codec =&gt; rubydebug }
  }
  
  input {
    file {
      path =&gt; "/data/vmlist/*.csv"
      start_position =&gt; "beginning"
      sincedb_path =&gt; "/tmp/sincedb"
    }
  }
  filter {
    csv {
      separator =&gt; ","
      columns =&gt; ["VM Name","Creation Date","Owner","Type","Message"]
    }
  }
  output {
    elasticsearch {
      hosts =&gt; "http://10.50.6.116:9200"
      index =&gt; "vms"
      document_type =&gt; "csv"
    }
    stdout{ codec=&gt; rubydebug}
  }
  </code></pre>
  
Tags: <elastic><logstash>
Title: Logstash multiple pipelines going into same index
UserDisplayName: Saar Grin
ThreadText: # Logstash multiple pipelines going into same index
  
  (Asked by: username_0 on 2018-04-15)
  
  I'm trying to set up a centralized syslog for multiple log sources.
  
  So I have a logstash that has two separate inputs and two separate outputs
  
  However for some reason the data from one of the inputs ends up in both indexes
  
  What am i doing wrong?
  
  Below are both pipelines' configs
  
  ```
  input {
    tcp {
      port => 5052
      codec => "json_lines"
    }
  }
  output {
    elasticsearch {
      hosts => "10.50.6.116"
      index => "remote"
    }
    file {
      path => "/var/log/logstash/remote-tcp.log"
    }
    stdout { codec => rubydebug }
  }
  
  input {
    file {
      path => "/data/vmlist/*.csv"
      start_position => "beginning"
      sincedb_path => "/tmp/sincedb"
    }
  }
  filter {
    csv {
      separator => ","
      columns => ["VM Name","Creation Date","Owner","Type","Message"]
    }
  }
  output {
    elasticsearch {
      hosts => "http://10.50.6.116:9200"
      index => "vms"
      document_type => "csv"
    }
    stdout{ codec=> rubydebug}
  }
  
  ```
  
  **Tags:** elastic, logstash
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-14. Score: 2
  
  There are two ways to accomplish this, though one of them was only available recently.
  
  The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...
  
  ```
  input {
    tcp {
      port => 1525
      codec => json_lines
      tags => [ 'tcp' ]
    }
  }
  
  input {
    file {
      path => '/var/log/app.log'
      codec => 'json'
      tags => [ 'file' ]
    }
  }
  
  output {
    if 'file' in [tags] {
      elasticsearch {
        host => 'logstash-es'
        index => 'files'
      }
    }
    if 'tcp' in [tags] {
      elasticsearch {
        host => 'logstash-es'
        index => 'tcp'
      }
    }
  }
  
  ```
  
  This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with [a way to do multiple pipelines in separate files](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html).
  
  ```
  - pipeline.id: tcp-inputs
    path.config: '/etc/logstash/pipelines/tcp.cfg'
    pipeline.workers: 3
  - pipeline.id: file-inputs
    path.config: '/etc/logstash/pipelines/files.cfg'
    pipeline.workers: 2
  
  ```
  
  This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.
  
  
  ---
  
  ### Answer by username_2 on 2018-04-16. Score: 0
  
  Once parsed your config create one and only one pipeline, with various inputs, various filters and various outputs.
  
  You have to use some conditional constructs to apply filter and output only to specific messages, usually using a special tag or field set on input.
  
  Some exemple are available in [logstash documentation](https://www.elastic.co/guide/en/logstash/current/config-examples.html#using-conditionals):
  
  > ```
  > output {
  >   if [type] == "apache" {
  >     if [status] =~ /^5\d\d/ {
  >       nagios { ...  }
  >     } else if [status] =~ /^4\d\d/ {
  >       elasticsearch { ... }
  >     }
  >     statsd { increment => "apache.%{status}" }
  >   }
  > }
  >
  > ```
  
  > **Comments:**
  > **username_0**: I tried it with conditionals and had the same outcome  
  > **username_0**: Turns out the proper way to do it is with pipelines.yml file which i somehow missed.  
  > **username_2**: Concerning your first comment, without what you tried, there's no way to tell what could have been wrong.  
  
  ---
  
  