Answers: <p>I propose the following solution although it will require 1 extra metricbeat running per monitored entity, it's simple, not optimized. The approach will also work for other non-kubernetes contexts like docker-compose with a little bit of tweaking.</p>
  <ul>
  <li>Deploy sidecar (extra container per pod) to monitor something (in this example, it's logstash but doing same for elasticsearch or kibana should be trivial, just replacing the monitoring module and possbibly metrics port.</li>
  <li>Set xpack.monitoring.enabled: false in logstash.yml (or elastic/kibana)</li>
  <li>Have metricbeat fetch metrics via localhost on port 9600</li>
  <li>Configure metricbeat with mounted configmaps defined in yaml</li>
  <li>Example assumes no TLS/RBAC to elasticsearch but if you have tls modifying the below should hopefully be straight forward</li>
  </ul>
  <pre><code># Configmaps for metricbeat conf files
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: metricbeat-modules
    namespace: logstash
  data:
    logstash-xpack.yml: |-
      - module: logstash
        xpack.enabled: true # This is what turns on xpack monitoring metric export
        period: 10s
        hosts: [&quot;localhost:9600&quot;] # Connect to logstash metrics api using localhost inside pod
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: metricbeat-config
    namespace: logstash
  data:
    metricbeat.yml: |-
      metricbeat.config.modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 10s
      metricbeat.modules: # Disable system metrics
      - module: system
        enabled: false
  
      output.elasticsearch:
        hosts: [&quot;http://elasticsearch:9200&quot;] # You probably need to modify elasticsearch URL 
      setup.ilm.enabled: true
  
  ---
  ...
  Here goes the rest of your yaml for logstash/elasticsearch/kibana
  And then the sidecar container:
  ... 
        containers:
        - name: metricbeat-sidecar
          image: docker.elastic.co/beats/metricbeat:7.14.0
          volumeMounts:
          - name: config
            mountPath: /usr/share/metricbeat/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: modules
            mountPath: /usr/share/metricbeat/modules.d
            readOnly: true
  ...
  # and then finally in the volumes section:
  ...
        volumes:
        - name: config
          configMap:
            defaultMode: 0640
            name: metricbeat-config
        - name: modules
          configMap:
            defaultMode: 0640
            name: metricbeat-modules
  
  </code></pre>
  <p>Main sources:</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/67145400/creating-sidecar-metricbeat-with-aws-eks-fargate">https://stackoverflow.com/questions/67145400/creating-sidecar-metricbeat-with-aws-eks-fargate</a></li>
  <li><a href="https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html</a></li>
  </ul>
  <p>Update:
  Similarly for kibana:</p>
  <pre><code># Configmaps for metricbeat conf files
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kibana-metricbeat-modules
    namespace: elk
  data:
    xpack.yml: |-
      - module: kibana
        xpack.enabled: true # This is what turns on reporting
        period: 10s
        hosts: [&quot;localhost:5601&quot;]
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kibana-metricbeat-config
    namespace: elk
  data:
    metricbeat.yml: |-
      metricbeat.config.modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 10s
      metricbeat.modules: # Disable system metrics
      - module: system
        enabled: false
  
      output.elasticsearch:
        hosts: [&quot;http://elasticsearch:9200&quot;]
      setup.ilm.enabled: true
  ...
  container: 
    - name: metricbeat-sidecar
        image: docker.elastic.co/beats/metricbeat:7.14.0
        volumeMounts:
          - name: config
            mountPath: /usr/share/metricbeat/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: modules
            mountPath: /usr/share/metricbeat/modules.d
            readOnly: true
  ...
        volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: kibana-metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0640
          name: kibana-metricbeat-modules
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2021-08-17T11:28:48.973Z
Answers: 2021-08-23T13:38:03.150Z
Answers: 2021-08-23T13:38:03.150Z
Answers: Kristofer
Body: <p>For probably more than a year there's been a deprecation warning on xpack monitoring in elasticsearch, logstash, kibana etc. saying the built-in monitoring is going to disappear.</p>
  <p>Since I don't like running deprecated setups I figured it was time to fix it now, should be easy, right? Not really, what is 2 lines of config in ELK 7.x instead requires deploying an extra metricbeat instance. Personally I really don't like this approach since it's a hassle and seems like a waste of resources to deploy another container just to send off some monitoring metrics. But elastic has chosen the path so we can just follow.</p>
  <p>I read a bit in some official channels and there seems to be some official solutions in the works that require custom kubernetes ELK CRDs etc. This is not what I want, I just want a simple solution to deploy anywhere as a sidecar.</p>
  
Tags: <kubernetes><monitoring><logstash><elasticsearch><elk>
Title: How to monitor ELK using metricbeat on kubernetes (v8 compatible and simple solution)
UserDisplayName: Kristofer
ThreadText: # How to monitor ELK using metricbeat on kubernetes (v8 compatible and simple solution)
  
  (Asked by: username_0 on 2021-08-17)
  
  For probably more than a year there's been a deprecation warning on xpack monitoring in elasticsearch, logstash, kibana etc. saying the built-in monitoring is going to disappear.
  
  Since I don't like running deprecated setups I figured it was time to fix it now, should be easy, right? Not really, what is 2 lines of config in ELK 7.x instead requires deploying an extra metricbeat instance. Personally I really don't like this approach since it's a hassle and seems like a waste of resources to deploy another container just to send off some monitoring metrics. But elastic has chosen the path so we can just follow.
  
  I read a bit in some official channels and there seems to be some official solutions in the works that require custom kubernetes ELK CRDs etc. This is not what I want, I just want a simple solution to deploy anywhere as a sidecar.
  
  **Tags:** kubernetes, monitoring, logstash, elasticsearch, elk
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-08-17. Score: 2
  
  I propose the following solution although it will require 1 extra metricbeat running per monitored entity, it's simple, not optimized. The approach will also work for other non-kubernetes contexts like docker-compose with a little bit of tweaking.
  
  * Deploy sidecar (extra container per pod) to monitor something (in this example, it's logstash but doing same for elasticsearch or kibana should be trivial, just replacing the monitoring module and possbibly metrics port.
  * Set xpack.monitoring.enabled: false in logstash.yml (or elastic/kibana)
  * Have metricbeat fetch metrics via localhost on port 9600
  * Configure metricbeat with mounted configmaps defined in yaml
  * Example assumes no TLS/RBAC to elasticsearch but if you have tls modifying the below should hopefully be straight forward
  
  ```
  # Configmaps for metricbeat conf files
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: metricbeat-modules
    namespace: logstash
  data:
    logstash-xpack.yml: |-
      - module: logstash
        xpack.enabled: true # This is what turns on xpack monitoring metric export
        period: 10s
        hosts: ["localhost:9600"] # Connect to logstash metrics api using localhost inside pod
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: metricbeat-config
    namespace: logstash
  data:
    metricbeat.yml: |-
      metricbeat.config.modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 10s
      metricbeat.modules: # Disable system metrics
      - module: system
        enabled: false
  
      output.elasticsearch:
        hosts: ["http://elasticsearch:9200"] # You probably need to modify elasticsearch URL 
      setup.ilm.enabled: true
  
  ---
  ...
  Here goes the rest of your yaml for logstash/elasticsearch/kibana
  And then the sidecar container:
  ... 
        containers:
        - name: metricbeat-sidecar
          image: docker.elastic.co/beats/metricbeat:7.14.0
          volumeMounts:
          - name: config
            mountPath: /usr/share/metricbeat/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: modules
            mountPath: /usr/share/metricbeat/modules.d
            readOnly: true
  ...
  # and then finally in the volumes section:
  ...
        volumes:
        - name: config
          configMap:
            defaultMode: 0640
            name: metricbeat-config
        - name: modules
          configMap:
            defaultMode: 0640
            name: metricbeat-modules
  
  ```
  
  Main sources:
  
  * <https://stackoverflow.com/questions/67145400/creating-sidecar-metricbeat-with-aws-eks-fargate>
  * <https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html>
  
  Update:
  Similarly for kibana:
  
  ```
  # Configmaps for metricbeat conf files
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kibana-metricbeat-modules
    namespace: elk
  data:
    xpack.yml: |-
      - module: kibana
        xpack.enabled: true # This is what turns on reporting
        period: 10s
        hosts: ["localhost:5601"]
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kibana-metricbeat-config
    namespace: elk
  data:
    metricbeat.yml: |-
      metricbeat.config.modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 10s
      metricbeat.modules: # Disable system metrics
      - module: system
        enabled: false
  
      output.elasticsearch:
        hosts: ["http://elasticsearch:9200"]
      setup.ilm.enabled: true
  ...
  container: 
    - name: metricbeat-sidecar
        image: docker.elastic.co/beats/metricbeat:7.14.0
        volumeMounts:
          - name: config
            mountPath: /usr/share/metricbeat/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: modules
            mountPath: /usr/share/metricbeat/modules.d
            readOnly: true
  ...
        volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: kibana-metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0640
          name: kibana-metricbeat-modules
  
  ```
  
  
  ---
  
  