Answers: <p>Changing logging driver form journald to json has fixed it. 
  I edited <code>/etc/sysconfig/docker</code>, and set </p>
  
  <pre><code>OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'
  </code></pre>
  
  <p>Then restarted docker on the nodes I needed:</p>
  
  <pre><code>systemctl daemon-reload &amp;&amp; systemctl restart docker
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2019-12-03T12:15:54.753Z
Answers: Could you please explain _how_ you changed the logging driver?
Answers: Bruce Becker
Answers: CC BY-SA 4.0
Answers: 2019-12-03T12:21:00.847Z
Answers: vi /etc/sysconfig/docker: set OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false' and then systemctl daemon-reload && systemctl restart docker on the nodes I needed.
Answers: anVzdGFub3RoZXJodW1hbg
Answers: CC BY-SA 4.0
Answers: 2019-12-03T12:22:33.893Z
Answers: Great! but could you update the answer with this information? Thanks
Answers: Bruce Becker
Answers: CC BY-SA 4.0
Answers: 2019-12-03T11:47:00.933Z
Answers: 2019-12-03T12:53:52.610Z
Answers: 2019-12-03T12:53:52.610Z
Answers: anVzdGFub3RoZXJodW1hbg
Body: <p>I install a fluentd chart on OKD 3.11 with:</p>
  
  <pre><code>helm install kiwigrid/fluentd-elasticsearch \
  --name fluentd \
  -f ../../tools/fluentd/values.yaml \
  --namespace logging
  </code></pre>
  
  <p>The pod is created, I can enter in it and I can curl to external elasticsearch successfully.
  However the liveness probe keeps failing and the pods on timeout get recreated:</p>
  
  <pre><code>Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com 
  Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.
  </code></pre>
  
  <p>And I can't find any indexes being created in ES.</p>
  
  <p>The liveness config is the default for the chart:</p>
  
  <pre><code> Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
    exit 1;
  fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
    rm -rf /var/log/fluentd-buffers;
    exit 1;
  fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
    exit 1;
  fi;
  ] delay=600s timeout=1s period=60s #success=1 #failure=3
  </code></pre>
  
  <p>The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.</p>
  
Tags: <elastic><openshift><fluentd>
Title: elasticsearch-fluentd helm chart on Openshift 3.11 pod failing
UserDisplayName: anVzdGFub3RoZXJodW1hbg
ThreadText: # elasticsearch-fluentd helm chart on Openshift 3.11 pod failing
  
  (Asked by: username_0 on 2019-12-02)
  
  I install a fluentd chart on OKD 3.11 with:
  
  ```
  helm install kiwigrid/fluentd-elasticsearch \
  --name fluentd \
  -f ../../tools/fluentd/values.yaml \
  --namespace logging
  
  ```
  
  The pod is created, I can enter in it and I can curl to external elasticsearch successfully.
  However the liveness probe keeps failing and the pods on timeout get recreated:
  
  ```
  Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com 
  Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.
  
  ```
  
  And I can't find any indexes being created in ES.
  
  The liveness config is the default for the chart:
  
  ```
   Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
    exit 1;
  fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
    rm -rf /var/log/fluentd-buffers;
    exit 1;
  fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
    exit 1;
  fi;
  ] delay=600s timeout=1s period=60s #success=1 #failure=3
  
  ```
  
  The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.
  
  **Tags:** elastic, openshift, fluentd
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-12-03. Score: 2 | Accepted answer
  
  Changing logging driver form journald to json has fixed it.
  I edited `/etc/sysconfig/docker`, and set
  
  ```
  OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'
  
  ```
  
  Then restarted docker on the nodes I needed:
  
  ```
  systemctl daemon-reload && systemctl restart docker
  
  ```
  
  > **Comments:**
  > **username_1**: Could you please explain _how_ you changed the logging driver?  
  > **username_0**: vi /etc/sysconfig/docker: set OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false' and then systemctl daemon-reload && systemctl restart docker on the nodes I needed.  
  > **username_1**: Great! but could you update the answer with this information? Thanks  
  
  ---
  
  