Answers: <p>A co-worker ended up taking over deploying this a few days ago and managed to get it working. I'm not certain what he did differently, but I know discovered very similar issues I was facing with other Helm charts boiled down to the database they depended on not deploying properly which caused the other components to fail during initialization.</p>
  
  <p>Database issues were different between the other services I was deploying, but one was rejecting the mounting of a root NFS share. The other appeared to have a mismatch between the base-chart and its packaged-in database chart where the database name and password configuration names were different. Anchore's database was likely failing due to the root-NFS share issue, which then cascaded to other components.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-06-18T16:59:22.197Z
Answers: 2019-06-18T16:59:22.197Z
Answers: wrc3314
Body: <p>Kubernetes version 1.14.2 (Client) 1.14.1 (Server)
  Anchore Chart anchore-engine.1.0.5 App Version 0.4.0</p>
  
  <p>I'm trying to deploy Anchore via Helm to a Kubernetes cluster, but I'm getting failures in every pod.</p>
  
  <p>Helm is properly configured and I've confirmed this by installing several other releases. I install Anchore with</p>
  
  <pre><code>helm install --name anchore-release stable/anchore-engine
  </code></pre>
  
  <p>Checking pods confirms all of the pods install</p>
  
  <pre><code>anchore-release-anchore-engine-analyzer-659854799f-qkvdn     0/1     Running   0          108s
  anchore-release-anchore-engine-api-5d75bbd9d9-9hnms          0/1     Running   0          108s
  anchore-release-anchore-engine-catalog-58d7f8bfcd-7d2jz      0/1     Running   0          108s
  anchore-release-anchore-engine-policy-6d7579d494-zcg4g       0/1     Running   0          108s
  anchore-release-anchore-engine-simplequeue-d8f4f4dbc-62hst   0/1     Running   0          108s
  </code></pre>
  
  <p>However, none ever enter Ready. Taking a peek at any one of them reveals the issue.</p>
  
  <pre><code>     Type     Reason     Age                   From                Message
    ----     ------     ----                  ----                -------
    Normal   Pulled     17m                   kubelet, XXXXXX  Container image "docker.io/anchore/anchore-engine:v0.4.0" already present on machine
    Normal   Created    17m                   kubelet, XXXXXX  Created container anchore-engine-analyzer
    Normal   Started    17m                   kubelet, XXXXXX  Started container anchore-engine-analyzer
    Normal   Scheduled  17m                   default-scheduler   Successfully assigned default/anchore-release-anchore-engine-analyzer-659854799f-9qdpq to XXXXXX
    Warning  Unhealthy  15m (x5 over 15m)     kubelet, XXXXXX  Liveness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
    Warning  Unhealthy  2m50s (x91 over 17m)  kubelet, XXXXXX  Readiness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
  </code></pre>
  
  <p>All of the pod experience liveness/readiness probe failures. In order to examine this closer I moved onto the master node and attempted to query these ports manually. I found that all of the Anchore pods were pingable but had exposed no ports. There is no firewall running anywhere within the cluster, and I installed an Nginx pod to confirm that. The other pod was able to expose and access ports in the 8000 range. </p>
  
  <p>Flannel is serving as the networking manager within the cluster, and due to the Nginx test-deployment properly exposing its ports I'm inclined to believe it's set up properly.</p>
  
  <p>Is anyone familiar with what could be cause the Anchore pods to not expose any of their ports, or another issue which could give the appearance of that? As additional information, here's an example description of one of the pods.</p>
  
  <pre><code>Name:               anchore-release-anchore-engine-analyzer-659854799f-9qdpq
  Namespace:          default
  Priority:           0
  PriorityClassName:  &lt;none&gt;
  Node:               XXXXXX/XXX.XXX.XXX.XXX
  Start Time:         Wed, 05 Jun 2019 17:41:11 -0400
  Labels:             app=anchore-release-anchore-engine
                      component=analyzer
                      pod-template-hash=659854799f
  Annotations:        &lt;none&gt;
  Status:             Running
  IP:                 XXX.XXX.XXX.XXX
  Controlled By:      ReplicaSet/anchore-release-anchore-engine-analyzer-659854799f
  Containers:
    anchore-engine-analyzer:
      Container ID:  docker://977890fe98a69f6399db55a06e26911befa2e94f98a1dc2d102041132961364e
      Image:         docker.io/anchore/anchore-engine:v0.4.0
      Image ID:      docker-pullable://docker.io/anchore/anchore-engine@sha256:1db43d98a0895cb266658b1cee9db275227b585f19590824ece58ad497bb772f
      Port:          8084/TCP
      Host Port:     0/TCP
      Command:
        anchore-manager
      Args:
        service
        start
        analyzer
      State:          Running
        Started:      Wed, 05 Jun 2019 17:41:13 -0400
      Ready:          False
      Restart Count:  0
      Liveness:       http-get http://:analyzer-api/health delay=120s timeout=10s period=10s #success=1 #failure=6
      Readiness:      http-get http://:analyzer-api/health delay=0s timeout=10s period=10s #success=1 #failure=3
      Environment Variables from:
        anchore-release-anchore-engine  Secret     Optional: false
        anchore-release-anchore-engine  ConfigMap  Optional: false
      Environment:
        ANCHORE_POD_NAME:  anchore-release-anchore-engine-analyzer-659854799f-9qdpq (v1:metadata.name)
      Mounts:
        /analysis_scratch from analyzer-scratch (rw)
        /config/config.yaml from config-volume (rw,path="config.yaml")
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-nzsfd (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:
    config-volume:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      anchore-release-anchore-engine
      Optional:  false
    analyzer-scratch:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:
      SizeLimit:  &lt;unset&gt;
    default-token-nzsfd:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-nzsfd
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  </code></pre>
  
  <p>I appreciate any input. Thanks.</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-06-17T03:03:59.477Z
Comments: What does running `lsof -t -i:8084` reveal? Does the GCP dashboard help any?
Comments: Daniel
Tags: <kubernetes><helm>
Title: Deploying Anchore to Kubernetes Cluster using Helm
UserDisplayName: wrc3314
ThreadText: # Deploying Anchore to Kubernetes Cluster using Helm
  
  (Asked by: username_0 on 2019-06-05)
  
  Kubernetes version 1.14.2 (Client) 1.14.1 (Server)
  Anchore Chart anchore-engine.1.0.5 App Version 0.4.0
  
  I'm trying to deploy Anchore via Helm to a Kubernetes cluster, but I'm getting failures in every pod.
  
  Helm is properly configured and I've confirmed this by installing several other releases. I install Anchore with
  
  ```
  helm install --name anchore-release stable/anchore-engine
  
  ```
  
  Checking pods confirms all of the pods install
  
  ```
  anchore-release-anchore-engine-analyzer-659854799f-qkvdn     0/1     Running   0          108s
  anchore-release-anchore-engine-api-5d75bbd9d9-9hnms          0/1     Running   0          108s
  anchore-release-anchore-engine-catalog-58d7f8bfcd-7d2jz      0/1     Running   0          108s
  anchore-release-anchore-engine-policy-6d7579d494-zcg4g       0/1     Running   0          108s
  anchore-release-anchore-engine-simplequeue-d8f4f4dbc-62hst   0/1     Running   0          108s
  
  ```
  
  However, none ever enter Ready. Taking a peek at any one of them reveals the issue.
  
  ```
       Type     Reason     Age                   From                Message
    ----     ------     ----                  ----                -------
    Normal   Pulled     17m                   kubelet, XXXXXX  Container image "docker.io/anchore/anchore-engine:v0.4.0" already present on machine
    Normal   Created    17m                   kubelet, XXXXXX  Created container anchore-engine-analyzer
    Normal   Started    17m                   kubelet, XXXXXX  Started container anchore-engine-analyzer
    Normal   Scheduled  17m                   default-scheduler   Successfully assigned default/anchore-release-anchore-engine-analyzer-659854799f-9qdpq to XXXXXX
    Warning  Unhealthy  15m (x5 over 15m)     kubelet, XXXXXX  Liveness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
    Warning  Unhealthy  2m50s (x91 over 17m)  kubelet, XXXXXX  Readiness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
  
  ```
  
  All of the pod experience liveness/readiness probe failures. In order to examine this closer I moved onto the master node and attempted to query these ports manually. I found that all of the Anchore pods were pingable but had exposed no ports. There is no firewall running anywhere within the cluster, and I installed an Nginx pod to confirm that. The other pod was able to expose and access ports in the 8000 range.
  
  Flannel is serving as the networking manager within the cluster, and due to the Nginx test-deployment properly exposing its ports I'm inclined to believe it's set up properly.
  
  Is anyone familiar with what could be cause the Anchore pods to not expose any of their ports, or another issue which could give the appearance of that? As additional information, here's an example description of one of the pods.
  
  ```
  Name:               anchore-release-anchore-engine-analyzer-659854799f-9qdpq
  Namespace:          default
  Priority:           0
  PriorityClassName:  <none>
  Node:               XXXXXX/XXX.XXX.XXX.XXX
  Start Time:         Wed, 05 Jun 2019 17:41:11 -0400
  Labels:             app=anchore-release-anchore-engine
                      component=analyzer
                      pod-template-hash=659854799f
  Annotations:        <none>
  Status:             Running
  IP:                 XXX.XXX.XXX.XXX
  Controlled By:      ReplicaSet/anchore-release-anchore-engine-analyzer-659854799f
  Containers:
    anchore-engine-analyzer:
      Container ID:  docker://977890fe98a69f6399db55a06e26911befa2e94f98a1dc2d102041132961364e
      Image:         docker.io/anchore/anchore-engine:v0.4.0
      Image ID:      docker-pullable://docker.io/anchore/anchore-engine@sha256:1db43d98a0895cb266658b1cee9db275227b585f19590824ece58ad497bb772f
      Port:          8084/TCP
      Host Port:     0/TCP
      Command:
        anchore-manager
      Args:
        service
        start
        analyzer
      State:          Running
        Started:      Wed, 05 Jun 2019 17:41:13 -0400
      Ready:          False
      Restart Count:  0
      Liveness:       http-get http://:analyzer-api/health delay=120s timeout=10s period=10s #success=1 #failure=6
      Readiness:      http-get http://:analyzer-api/health delay=0s timeout=10s period=10s #success=1 #failure=3
      Environment Variables from:
        anchore-release-anchore-engine  Secret     Optional: false
        anchore-release-anchore-engine  ConfigMap  Optional: false
      Environment:
        ANCHORE_POD_NAME:  anchore-release-anchore-engine-analyzer-659854799f-9qdpq (v1:metadata.name)
      Mounts:
        /analysis_scratch from analyzer-scratch (rw)
        /config/config.yaml from config-volume (rw,path="config.yaml")
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-nzsfd (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:
    config-volume:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      anchore-release-anchore-engine
      Optional:  false
    analyzer-scratch:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:
      SizeLimit:  <unset>
    default-token-nzsfd:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-nzsfd
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  <none>
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  
  ```
  
  I appreciate any input. Thanks.
  
  **Tags:** kubernetes, helm
  
  > **Comments:**
  > **username_1**: What does running `lsof -t -i:8084` reveal? Does the GCP dashboard help any?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-06-18. Score: 1
  
  A co-worker ended up taking over deploying this a few days ago and managed to get it working. I'm not certain what he did differently, but I know discovered very similar issues I was facing with other Helm charts boiled down to the database they depended on not deploying properly which caused the other components to fail during initialization.
  
  Database issues were different between the other services I was deploying, but one was rejecting the mounting of a root NFS share. The other appeared to have a mismatch between the base-chart and its packaged-in database chart where the database name and password configuration names were different. Anchore's database was likely failing due to the root-NFS share issue, which then cascaded to other components.
  
  
  ---
  
  