Answers: <p>For anyone running into the same problem:
  cert-manager's issue tracker has an <a href="https://github.com/cert-manager/cert-manager/issues/466" rel="nofollow noreferrer">excellent discussion about the problem</a> dating back to 2018.</p>
  <p>The solution for me was to install compumike's <a href="https://github.com/compumike/hairpin-proxy" rel="nofollow noreferrer">hairpin-proxy</a> to enable PROXY protocol support for internal-to-LoadBalancer traffic.</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-04-02T19:09:14.357Z
Answers: 2024-04-02T19:09:14.357Z
Answers: cvk
Body: <p>I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.</p>
  <p>I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.</p>
  <p>I created an issuer:</p>
  <pre><code>apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    name: letsencrypt-prod
  spec:
    acme:
      server: https://acme-v02.api.letsencrypt.org/directory
      email: my@email.com
      privateKeySecretRef:
        name: letsencrypt-prod-key
      solvers:
      - selector: {}
        http01:
          ingress:
            ingressClassName: nginx
  
  </code></pre>
  <p>and annotated the ingress:</p>
  <pre><code>apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: frontend-ingress
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
      nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
      nginx.ingress.kubernetes.io/rewrite-target: /$1
  spec:
    ingressClassName: nginx
    rules:
    - http:
        paths:
        - path: /api/(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: backend
              port:
                number: 8080
        - path: /?(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: frontend
              port:
                number: 3000
    tls:
    - hosts:
      - example.com
      secretName: frontend-ingress-cert
  </code></pre>
  <p>This creates the cm-acme-http-solver:</p>
  <pre><code>NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
  cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m
  </code></pre>
  <p>This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:</p>
  <pre><code>Name:         frontend-ingress-cert-1-1328491808-3444115809
  Namespace:    default
  Labels:       &lt;none&gt;
  Annotations:  &lt;none&gt;
  API Version:  acme.cert-manager.io/v1
  Kind:         Challenge
  Metadata:
    Creation Timestamp:  2024-04-02T15:07:31Z
    Finalizers:
      finalizer.acme.cert-manager.io
    Generation:  1
    Owner References:
      API Version:           acme.cert-manager.io/v1
      Block Owner Deletion:  true
      Controller:            true
      Kind:                  Order
      Name:                  frontend-ingress-cert-1-1328491808
      UID:                   e85e8c53-5965-4be4-87fa-50359217539f
    Resource Version:        10398552697
    UID:                     73e25122-6340-406f-aa68-632e92adec4b
  Spec:
    Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
    Dns Name:           example.com
    Issuer Ref:
      Group:  cert-manager.io
      Kind:   Issuer
      Name:   letsencrypt-prod
    Key:      &lt;redacted&gt;&gt;.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
    Solver:
      http01:
        Ingress:
          Ingress Class Name:  nginx
      Selector:
    Token:     &lt;redacted&gt;&gt;
    Type:      HTTP-01
    URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
    Wildcard:  false
  Status:
    Presented:   true
    Processing:  true
    Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;': Get &quot;http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;&quot;: EOF
    State:       pending
  Events:
    Type    Reason     Age    From                     Message
    ----    ------     ----   ----                     -------
    Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
    Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism
  </code></pre>
  <p>Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:</p>
  <pre><code>❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
  /app $ wget http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;
  Connecting to mydomain.com (51.158.131.xxx:80)
  wget: error getting response: Resource temporarily unavailable
  </code></pre>
  <p>Nginx logs this:</p>
  <pre><code>2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: &quot;GET /.well-known/acme-challenge/&lt;redacted&gt;&gt; HTTP/1.1&quot; while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80
  </code></pre>
  
Tags: <kubernetes><cert-manager><tls>
Title: Cert-manager challenge failing self check
UserDisplayName: cvk
ThreadText: # Cert-manager challenge failing self check
  
  (Asked by: username_0 on 2024-04-02)
  
  I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.
  
  I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.
  
  I created an issuer:
  
  ```
  apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    name: letsencrypt-prod
  spec:
    acme:
      server: https://acme-v02.api.letsencrypt.org/directory
      email: my@email.com
      privateKeySecretRef:
        name: letsencrypt-prod-key
      solvers:
      - selector: {}
        http01:
          ingress:
            ingressClassName: nginx
  
  ```
  
  and annotated the ingress:
  
  ```
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: frontend-ingress
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
  spec:
    ingressClassName: nginx
    rules:
    - http:
        paths:
        - path: /api/(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: backend
              port:
                number: 8080
        - path: /?(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: frontend
              port:
                number: 3000
    tls:
    - hosts:
      - example.com
      secretName: frontend-ingress-cert
  
  ```
  
  This creates the cm-acme-http-solver:
  
  ```
  NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
  cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m
  
  ```
  
  This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:
  
  ```
  Name:         frontend-ingress-cert-1-1328491808-3444115809
  Namespace:    default
  Labels:       <none>
  Annotations:  <none>
  API Version:  acme.cert-manager.io/v1
  Kind:         Challenge
  Metadata:
    Creation Timestamp:  2024-04-02T15:07:31Z
    Finalizers:
      finalizer.acme.cert-manager.io
    Generation:  1
    Owner References:
      API Version:           acme.cert-manager.io/v1
      Block Owner Deletion:  true
      Controller:            true
      Kind:                  Order
      Name:                  frontend-ingress-cert-1-1328491808
      UID:                   e85e8c53-5965-4be4-87fa-50359217539f
    Resource Version:        10398552697
    UID:                     73e25122-6340-406f-aa68-632e92adec4b
  Spec:
    Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
    Dns Name:           example.com
    Issuer Ref:
      Group:  cert-manager.io
      Kind:   Issuer
      Name:   letsencrypt-prod
    Key:      <redacted>>.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
    Solver:
      http01:
        Ingress:
          Ingress Class Name:  nginx
      Selector:
    Token:     <redacted>>
    Type:      HTTP-01
    URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
    Wildcard:  false
  Status:
    Presented:   true
    Processing:  true
    Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/<redacted>>': Get "http://example.com/.well-known/acme-challenge/<redacted>>": EOF
    State:       pending
  Events:
    Type    Reason     Age    From                     Message
    ----    ------     ----   ----                     -------
    Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
    Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism
  
  ```
  
  Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:
  
  ```
  ❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
  /app $ wget http://example.com/.well-known/acme-challenge/<redacted>>
  Connecting to mydomain.com (51.158.131.xxx:80)
  wget: error getting response: Resource temporarily unavailable
  
  ```
  
  Nginx logs this:
  
  ```
  2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: "GET /.well-known/acme-challenge/<redacted>> HTTP/1.1" while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80
  
  ```
  
  **Tags:** kubernetes, cert-manager, tls
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-04-02. Score: 0
  
  For anyone running into the same problem:
  cert-manager's issue tracker has an [excellent discussion about the problem](https://github.com/cert-manager/cert-manager/issues/466) dating back to 2018.
  
  The solution for me was to install compumike's [hairpin-proxy](https://github.com/compumike/hairpin-proxy) to enable PROXY protocol support for internal-to-LoadBalancer traffic.
  
  
  ---
  
  