Answers: <p>I had two problems. @taleodor was correct though, I needed to connect using my domain name that I had setup my certificate for.
  I put a copy of this question on stackoverflow as well as it seemed there was a bigger body of kubernetes questions there already.  I outlined the other problem here <a href="https://stackoverflow.com/questions/76146713/unable-to-pull-image-from-a-private-docker-registry-with-lets-encrypt-certifica?noredirect=1#comment134287667_76146713">https://stackoverflow.com/questions/76146713/unable-to-pull-image-from-a-private-docker-registry-with-lets-encrypt-certifica?noredirect=1#comment134287667_76146713</a>
  For this problem I had to find this in my app's deployment yaml:</p>
  <pre><code>image: 192.168.42.50:5000/dev-backend:latest
  </code></pre>
  <p>and change it to</p>
  <pre><code>image: &lt;my domain&gt;:5000/dev-backend:latest
  </code></pre>
  <p>And apply it of course. Then I had to make sure that port 5000 was port forwarded to my machine at 192.168.42.50 but that got me to a 401 (bad password error) which is basically a success.</p>
  <p>You also want to make sure that the domain is resolving properly from the machines running k3s cluster with <code>nslookup &lt;your domain&gt;</code>.  If not you probably have a problem with your <code>A</code> record.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-05-01T18:03:22.153Z
Answers: 2023-05-01T18:04:21.440Z
Answers: 2023-05-01T18:04:21.440Z
Answers: Snorghma
Body: <p>I have set up a private Docker registry on my Kubernetes cluster (k3s) running on a Raspberry Pi 4. I have secured the registry with a Let's Encrypt certificate, and I can confirm that the certificate is valid when accessing the registry from a web browser. However, when trying to deploy an application in Kubernetes that uses an image from this private registry, I am encountering an error related to the certificate.</p>
  <pre><code>Failed to pull image &quot;192.168.42.50:5000/dev-backend:latest&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;192.168.42.50:5000/dev-backend:latest&quot;: failed to resolve reference &quot;192.168.42.50:5000/dev-backend:latest&quot;: failed to do request: Head &quot;https://192.168.42.50:5000/v2/dev-backend/manifests/latest&quot;: x509: certificate signed by unknown authority
  </code></pre>
  <p>I have created a Certificate resource in the &quot;docker&quot; namespace and successfully obtained a certificate from Let's Encrypt. The certificate request and its corresponding certificate can be seen with the following commands:</p>
  <pre><code>root@atlas00:~# kubectl get certificaterequests -n docker
  NAME                        APPROVED   DENIED   READY   ISSUER             REQUESTOR                                         AGE
  docker-registry-tls-hhm6j   True                True    letsencrypt-prod   system:serviceaccount:cert-manager:cert-manager   7h17m
  
  root@atlas00:~# kubectl get certificates -n docker
  NAME                  READY   SECRET                AGE
  docker-registry-tls   True    docker-registry-tls   7h17m
  </code></pre>
  <p>The registry is using the Let's Encrypt certificate correctly, and I have even added the Let's Encrypt root certificate (ISRG Root X1) to the trusted certificate store for the container runtime (Docker) on each node of my cluster.</p>
  <p>Here is my deployment YAML file:</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dev-backend
    namespace: dev
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: dev-backend
    template:
      metadata:
        labels:
          app: dev-backend
      spec:
        containers:
        - name: dev-backend
          image: 192.168.42.50:5000/dev-backend:latest
          ports:
          - containerPort: 80
          volumeMounts:
          - name: dev-backend-storage
            mountPath: /app/data
        volumes:
        - name: dev-backend-storage
          persistentVolumeClaim:
            claimName: dev-backend-pvc
        imagePullSecrets:
        - name: docker-registry-config
  </code></pre>
  <p>I have created a Kubernetes secret (docker-registry-config) containing the necessary credentials for accessing the private registry.</p>
  <p>Despite these configurations, Kubernetes still reports the certificate as signed by an unknown authority. How can I ensure that Kubernetes trusts the Let's Encrypt certificate and successfully pulls the image from my private registry?  Or is there a certificate authority that Kubernetes trusts out of the box?  I'd like to avoid using an 'insecure-registry' approach.  Thanks.</p>
  
Comments: CC BY-SA 4.0
Comments: 2023-05-01T12:35:12.680Z
Comments: You should have a domain name for let's encrypt cert. Then you should reference your registry by that domain name rather than by IP address (which is in private network space in your case btw). Alternatively, research support for insecure registries for your kubernetes flavor.
Comments: taleodor
Tags: <docker><kubernetes><docker-registry><k3s><tls>
Title: Unable to pull image from a private Docker registry with Let's Encrypt certificate in Kubernetes
UserDisplayName: Snorghma
ThreadText: # Unable to pull image from a private Docker registry with Let's Encrypt certificate in Kubernetes
  
  (Asked by: username_0 on 2023-05-01)
  
  I have set up a private Docker registry on my Kubernetes cluster (k3s) running on a Raspberry Pi 4. I have secured the registry with a Let's Encrypt certificate, and I can confirm that the certificate is valid when accessing the registry from a web browser. However, when trying to deploy an application in Kubernetes that uses an image from this private registry, I am encountering an error related to the certificate.
  
  ```
  Failed to pull image "192.168.42.50:5000/dev-backend:latest": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.42.50:5000/dev-backend:latest": failed to resolve reference "192.168.42.50:5000/dev-backend:latest": failed to do request: Head "https://192.168.42.50:5000/v2/dev-backend/manifests/latest": x509: certificate signed by unknown authority
  
  ```
  
  I have created a Certificate resource in the "docker" namespace and successfully obtained a certificate from Let's Encrypt. The certificate request and its corresponding certificate can be seen with the following commands:
  
  ```
  root@atlas00:~# kubectl get certificaterequests -n docker
  NAME                        APPROVED   DENIED   READY   ISSUER             REQUESTOR                                         AGE
  docker-registry-tls-hhm6j   True                True    letsencrypt-prod   system:serviceaccount:cert-manager:cert-manager   7h17m
  
  root@atlas00:~# kubectl get certificates -n docker
  NAME                  READY   SECRET                AGE
  docker-registry-tls   True    docker-registry-tls   7h17m
  
  ```
  
  The registry is using the Let's Encrypt certificate correctly, and I have even added the Let's Encrypt root certificate (ISRG Root X1) to the trusted certificate store for the container runtime (Docker) on each node of my cluster.
  
  Here is my deployment YAML file:
  
  ```
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dev-backend
    namespace: dev
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: dev-backend
    template:
      metadata:
        labels:
          app: dev-backend
      spec:
        containers:
        - name: dev-backend
          image: 192.168.42.50:5000/dev-backend:latest
          ports:
          - containerPort: 80
          volumeMounts:
          - name: dev-backend-storage
            mountPath: /app/data
        volumes:
        - name: dev-backend-storage
          persistentVolumeClaim:
            claimName: dev-backend-pvc
        imagePullSecrets:
        - name: docker-registry-config
  
  ```
  
  I have created a Kubernetes secret (docker-registry-config) containing the necessary credentials for accessing the private registry.
  
  Despite these configurations, Kubernetes still reports the certificate as signed by an unknown authority. How can I ensure that Kubernetes trusts the Let's Encrypt certificate and successfully pulls the image from my private registry? Or is there a certificate authority that Kubernetes trusts out of the box? I'd like to avoid using an 'insecure-registry' approach. Thanks.
  
  **Tags:** docker, kubernetes, docker-registry, k3s, tls
  
  > **Comments:**
  > **username_1**: You should have a domain name for let's encrypt cert. Then you should reference your registry by that domain name rather than by IP address (which is in private network space in your case btw). Alternatively, research support for insecure registries for your kubernetes flavor.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-05-01. Score: 1
  
  I had two problems. @username_1 was correct though, I needed to connect using my domain name that I had setup my certificate for.
  I put a copy of this question on stackoverflow as well as it seemed there was a bigger body of kubernetes questions there already. I outlined the other problem here <https://stackoverflow.com/questions/76146713/unable-to-pull-image-from-a-private-docker-registry-with-lets-encrypt-certifica?noredirect=1#comment134287667_76146713>
  For this problem I had to find this in my app's deployment yaml:
  
  ```
  image: 192.168.42.50:5000/dev-backend:latest
  
  ```
  
  and change it to
  
  ```
  image: <my domain>:5000/dev-backend:latest
  
  ```
  
  And apply it of course. Then I had to make sure that port 5000 was port forwarded to my machine at 192.168.42.50 but that got me to a 401 (bad password error) which is basically a success.
  
  You also want to make sure that the domain is resolving properly from the machines running k3s cluster with `nslookup <your domain>`. If not you probably have a problem with your `A` record.
  
  
  ---
  
  