Answers: <p>After days and tests here's the solution :</p>
  <p>Kubernetes worker and master got the wrong certificate.</p>
  <p>In my case, i have to :</p>
  <ul>
  <li>create a .pem certificate chain composed of the intermediate + wildcard certificate</li>
  <li>convert the .pem chain into a .pem fileas my tool (Nutanix Kubernetes Engine) need a .pem format.</li>
  </ul>
  <p>Finally, workers and master got certificate updated into /etc/containerd/certs.d/my_repo_folder.</p>
  <p>This makes AWX working and successfully pull image from my private registry.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-08-25T11:29:21.997Z
Answers: 2023-08-25T11:29:21.997Z
Answers: motorbass
Body: <p><strong>Context:</strong><br />
  I got a K8s cluster where I setup AWX using the official <a href="https://github.com/ansible/awx-operator/blob/devel/docs/installation/helm-install-on-existing-cluster.md" rel="nofollow noreferrer">Helm install</a>. On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.</p>
  <p>At the moment, the private registry works only with a HTTP listener configured on port 8080.
  I'm able to login and push/pull my custom image to this registry from another server using docker cli.</p>
  <p><strong>Configuration/AWX Setup for image pulling:</strong><br />
  In order to pull my custom image into my AWX project, I configure AWX the following way :</p>
  <ol>
  <li><p>Setup a custom EE
  <a href="https://i.sstatic.net/8Q1sD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8Q1sD.png" alt="enter image description here" /></a></p>
  </li>
  <li><p>Create a Container Registry credentials<br />
  <a href="https://i.sstatic.net/A5YtC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/A5YtC.png" alt="enter image description here" /></a></p>
  </li>
  <li><p>And finally setup Execution Environment on my templates.</p>
  </li>
  </ol>
  <p><strong>Problem:</strong></p>
  <p>When i launch a template, it keeps running for a long time, and here's error i got from kubectl :</p>
  <pre><code>PS &gt; kubectl describe pods automation-job-50-78mp5
  Name:             automation-job-50-78mp5
  Namespace:        awx
  Priority:         0
  Service Account:  default
  Node:             XXXXXX
  Start Time:       Tue, 22 Aug 2023 11:42:04 +0200
  Labels:           ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
                    ansible-awx-job-id=50
  Annotations:      cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
                    cni.projectcalico.org/podIP: 172.20.5.208/32
                    cni.projectcalico.org/podIPs: 172.20.5.208/32
  Status:           Pending
  IP:               172.20.5.208
  IPs:
    IP:  172.20.5.208
  Containers:
    worker:
      Container ID:
      Image:         awx-repo.mydomain.com/awx-repo
      Image ID:
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Args:
        ansible-runner
        worker
        --private-data-dir=/runner
      State:          Waiting
        Reason:       ImagePullBackOff
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:        250m
        memory:     100Mi
      Environment:  &lt;none&gt;
      Mounts:       &lt;none&gt;
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:            &lt;none&gt;
  QoS Class:          Burstable
  Node-Selectors:     &lt;none&gt;
  Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                      node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason     Age                From               Message
    ----     ------     ----               ----               -------
    Normal   Scheduled  20s                default-scheduler  Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
    Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image &quot;awx-repo.mydomain.com/awx-repo&quot;
    Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
    Normal   Pulling    5s (x2 over 20s)   kubelet            Pulling image &quot;awx-repo.mydomain.com/awx-repo&quot;
    Warning  Failed     5s (x2 over 20s)   kubelet            Failed to pull image &quot;awx-repo.mydomain.com/awx-repo&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;awx-repo.mydomain.com/awx-repo:latest&quot;: failed to resolve reference &quot;awx-repo.mydomain.com/awx-repo:latest&quot;: failed to do request: Head &quot;https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest&quot;: x509: certificate signed by unknown authority
    Warning  Failed     5s (x2 over 20s)   kubelet            Error: ErrImagePull
  </code></pre>
  <p>My question are :</p>
  <ul>
  <li>How can I double check the pod is using credentials I set from AWX GUI ?</li>
  <li>As credentials works from the server, is there a &quot;dedicated format&quot; we have to set from AWX GUI ?</li>
  <li>Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate&quot;</li>
  </ul>
  
Tags: <kubernetes><docker-registry><nexus><awx>
Title: AWX Failed to pull image on a private registry
UserDisplayName: motorbass
ThreadText: # AWX Failed to pull image on a private registry
  
  (Asked by: username_0 on 2023-08-22)
  
  **Context:**  
  I got a K8s cluster where I setup AWX using the official [Helm install](https://github.com/ansible/awx-operator/blob/devel/docs/installation/helm-install-on-existing-cluster.md). On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.
  
  At the moment, the private registry works only with a HTTP listener configured on port 8080.
  I'm able to login and push/pull my custom image to this registry from another server using docker cli.
  
  **Configuration/AWX Setup for image pulling:**  
  In order to pull my custom image into my AWX project, I configure AWX the following way :
  
  1. Setup a custom EE
     [![enter image description here](https://i.sstatic.net/8Q1sD.png)](https://i.sstatic.net/8Q1sD.png)
  2. Create a Container Registry credentials  
     [![enter image description here](https://i.sstatic.net/A5YtC.png)](https://i.sstatic.net/A5YtC.png)
  3. And finally setup Execution Environment on my templates.
  
  **Problem:**
  
  When i launch a template, it keeps running for a long time, and here's error i got from kubectl :
  
  ```
  PS > kubectl describe pods automation-job-50-78mp5
  Name:             automation-job-50-78mp5
  Namespace:        awx
  Priority:         0
  Service Account:  default
  Node:             XXXXXX
  Start Time:       Tue, 22 Aug 2023 11:42:04 +0200
  Labels:           ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
                    ansible-awx-job-id=50
  Annotations:      cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
                    cni.projectcalico.org/podIP: 172.20.5.208/32
                    cni.projectcalico.org/podIPs: 172.20.5.208/32
  Status:           Pending
  IP:               172.20.5.208
  IPs:
    IP:  172.20.5.208
  Containers:
    worker:
      Container ID:
      Image:         awx-repo.mydomain.com/awx-repo
      Image ID:
      Port:          <none>
      Host Port:     <none>
      Args:
        ansible-runner
        worker
        --private-data-dir=/runner
      State:          Waiting
        Reason:       ImagePullBackOff
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:        250m
        memory:     100Mi
      Environment:  <none>
      Mounts:       <none>
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:            <none>
  QoS Class:          Burstable
  Node-Selectors:     <none>
  Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                      node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason     Age                From               Message
    ----     ------     ----               ----               -------
    Normal   Scheduled  20s                default-scheduler  Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
    Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image "awx-repo.mydomain.com/awx-repo"
    Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
    Normal   Pulling    5s (x2 over 20s)   kubelet            Pulling image "awx-repo.mydomain.com/awx-repo"
    Warning  Failed     5s (x2 over 20s)   kubelet            Failed to pull image "awx-repo.mydomain.com/awx-repo": rpc error: code = Unknown desc = failed to pull and unpack image "awx-repo.mydomain.com/awx-repo:latest": failed to resolve reference "awx-repo.mydomain.com/awx-repo:latest": failed to do request: Head "https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest": x509: certificate signed by unknown authority
    Warning  Failed     5s (x2 over 20s)   kubelet            Error: ErrImagePull
  
  ```
  
  My question are :
  
  * How can I double check the pod is using credentials I set from AWX GUI ?
  * As credentials works from the server, is there a "dedicated format" we have to set from AWX GUI ?
  * Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate"
  
  **Tags:** kubernetes, docker-registry, nexus, awx
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-08-25. Score: 0 | Accepted answer
  
  After days and tests here's the solution :
  
  Kubernetes worker and master got the wrong certificate.
  
  In my case, i have to :
  
  * create a .pem certificate chain composed of the intermediate + wildcard certificate
  * convert the .pem chain into a .pem fileas my tool (Nutanix Kubernetes Engine) need a .pem format.
  
  Finally, workers and master got certificate updated into /etc/containerd/certs.d/my\_repo\_folder.
  
  This makes AWX working and successfully pull image from my private registry.
  
  
  ---
  
  