Answers: <h1>6/30/2024</h1>
  <p>I found what I posted on 6/27/2024 is misleading. Even it repaired the issue in the question, but only on the surface. The Certificate still failed to issue.</p>
  <p>It took me some time to make everything including cert-manager, ExternalDNS, Traefik work well.</p>
  <p>Also, it turns out <code>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;</code> annotation is not necessary to fix the issue in the question.</p>
  <p>Note for the solution in my case, I am using DNS01 challenge.</p>
  <p>I also published all my deployment codes <a href="https://github.com/hongbo-miao/hongbomiao.com/issues/17463#issuecomment-2180242019" rel="nofollow noreferrer">here</a>. Hopefully it can give a bigger picture how it get deployed.</p>
  <h2>cert-manager AWS IAM role <code>CertManagerRole</code></h2>
  <h3>Trust relationships</h3>
  <pre class="lang-json prettyprint-override"><code>{
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
          {
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Principal&quot;: {
                  &quot;Federated&quot;: &quot;arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
              },
              &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,
              &quot;Condition&quot;: {
                  &quot;StringEquals&quot;: {
                      &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud&quot;: &quot;sts.amazonaws.com&quot;,
                      &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub&quot;: &quot;system:serviceaccount:production-hm-cert-manager:hm-cert-manager&quot;
                  }
              }
          }
      ]
  }
  </code></pre>
  <h3>Policy</h3>
  <pre class="lang-json prettyprint-override"><code>{
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
          {
              &quot;Action&quot;: [
                  &quot;route53:GetChange&quot;
              ],
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Resource&quot;: &quot;arn:aws:route53:::change/*&quot;
          },
          {
              &quot;Action&quot;: [
                  &quot;route53:ListResourceRecordSets&quot;
              ],
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Resource&quot;: &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
          },
          {
              &quot;Action&quot;: [
                  &quot;route53:ChangeResourceRecordSets&quot;
              ],
              &quot;Condition&quot;: {
                  &quot;ForAllValues:StringEquals&quot;: {
                      &quot;route53:ChangeResourceRecordSetsNormalizedRecordNames&quot;: [
                          &quot;_acme-challenge.*&quot;
                      ],
                      &quot;route53:ChangeResourceRecordSetsRecordTypes&quot;: [
                          &quot;TXT&quot;
                      ]
                  }
              },
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Resource&quot;: &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
          },
          {
              &quot;Action&quot;: [
                  &quot;route53:ListHostedZonesByName&quot;
              ],
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Resource&quot;: &quot;*&quot;
          }
      ]
  }
  </code></pre>
  <h2>ExternalDNS AWS IAM role <code>ExternalDNSRole</code></h2>
  <h3>Trust relationships</h3>
  <pre class="lang-json prettyprint-override"><code>{
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
          {
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Principal&quot;: {
                  &quot;Federated&quot;: &quot;arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
              },
              &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,
              &quot;Condition&quot;: {
                  &quot;StringEquals&quot;: {
                      &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub&quot;: &quot;system:serviceaccount:production-hm-external-dns:hm-external-dns&quot;,
                      &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud&quot;: &quot;sts.amazonaws.com&quot;
                  }
              }
          }
      ]
  }
  </code></pre>
  <h3>Policy</h3>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
      {
        &quot;Action&quot;: [
          &quot;route53:ChangeResourceRecordSets&quot;
        ],
        &quot;Effect&quot;: &quot;Allow&quot;,
        &quot;Resource&quot;: [
          &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
        ]
      },
      {
        &quot;Action&quot;: [
          &quot;route53:ListHostedZones&quot;,
          &quot;route53:ListResourceRecordSets&quot;,
          &quot;route53:ListTagsForResource&quot;
        ],
        &quot;Effect&quot;: &quot;Allow&quot;,
        &quot;Resource&quot;: [
          &quot;*&quot;
        ]
      }
    ]
  }
  </code></pre>
  <h2>cert-manager Helm chart custom values.yaml</h2>
  <pre><code>crds:
    enabled: true
    keep: true
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/CertManagerRole
  securityContext:
    fsGroup: 1001
  </code></pre>
  <h2>ExternalDNS Helm chart custom values.yaml</h2>
  <pre class="lang-yaml prettyprint-override"><code>provider: aws
  aws:
    region: us-west-2
    zoneType: public
    txtOwnerId: Z1XXXXXXXXXXXXXXXXXXX
    domainFilters:
      - internal.example.com
    registry: txt
  policy: sync
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/ExternalDNSRole
  </code></pre>
  <h2>Traefik Helm chart custom values.yaml</h2>
  <pre class="lang-yaml prettyprint-override"><code>providers:
    kubernetesIngress:
      publishedService:
        enabled: true
  service:
    enabled: true
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: &quot;true&quot;
  </code></pre>
  <h2>ClusterIssuer</h2>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: production-lets-encrypt-cluster-issuer
    namespace: production-hm-cert-manager
  spec:
    acme:
      email: me@example.com
      server: https://acme-v02.api.letsencrypt.org/directory
      privateKeySecretRef:
        name: production-lets-encrypt-cluster-issuer-account-secret
      solvers:
        - selector:
            dnsZones:
              - internal.example.com
          dns01:
            route53:
              region: us-west-2
  </code></pre>
  <h2>Ingress</h2>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
      cert-manager.io/cluster-issuer: production-lets-encrypt-cluster-issuer
  spec:
    rules:
      - host: hm-airbyte.internal.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.internal.example.com
        secretName: production-hm-airbyte-certificate
  </code></pre>
  <h1>6/27/2024 (Misleading Answer)</h1>
  <p>After a lot of experiments, I found if I add <code>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;</code> annotation, it helps resolve the issue.</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
      cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
      acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;  # &lt;- Added this
    labels:
      app.kubernetes.io/name: hm-airbyte-ingress
      app.kubernetes.io/part-of: production-hm-airbyte
  spec:
    rules:
      - host: hm-airbyte.internal.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.internal.example.com
        secretName: hm-airbyte-ingress-tls
  </code></pre>
  <p>So now here is what happens after I apply this Ingress file</p>
  <ol>
  <li>A temporary <code>Secret: hm-airbyte-ingress-tls-7w9z7</code> with random suffix will be created</li>
  <li>After about half minute, a <code>Secret: hm-airbyte-ingress-tls</code> without suffix will be created</li>
  <li>The one will suffix will be automatically deleted.</li>
  </ol>
  <p>Without this annotation, based on my experiment, it will be stuck at step 1 forever.</p>
  <p>This is the <code>Secret: hm-airbyte-ingress-tls</code> without suffix, you can see the difference between the temporary secret I posted in the question.</p>
  <p><a href="https://i.sstatic.net/osEg2tA4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/osEg2tA4.png" alt="enter image description here" /></a></p>
  <p><a href="https://cert-manager.io/docs/usage/ingress/#supported-annotations" rel="nofollow noreferrer">Here</a> is the explanation of this annotation:</p>
  <blockquote>
  <p>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;: this controls whether the ingress is modified 'in-place', or a new one is created specifically for the HTTP01 challenge. If present, and set to &quot;true&quot;, the existing ingress will be modified. Any other value, or the absence of the annotation assumes &quot;false&quot;. This annotation will also add the annotation &quot;cert-manager.io/issue-temporary-certificate&quot;: &quot;true&quot; onto created certificates which will cause a temporary certificate to be set on the resulting Secret until the final signed certificate has been returned. This is useful for keeping compatibility with the ingress-gce component.</p>
  </blockquote>
  <p>P.S. I cannot explain why this annotation helps resolve the issue. Because I actually expect if with this annotation, the temporary secret will not be created at all. Hope someone can further explain in future, thanks!</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-06-30T12:51:22.760Z
Answers: 2024-07-01T04:40:17.777Z
Answers: 2024-07-01T04:40:17.777Z
Answers: Hongbo Miao
Body: <p>I am using &quot;Traefik&quot;, &quot;cert-mangaer&quot;, and &quot;Let's Encrypt&quot;.</p>
  <p>Here is my ClusterIssuer file</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: production-lets-encrypt-issuer
    namespace: production-hm-cert-manager
  spec:
    acme:
      email: me@example.com
      server: https://acme-v02.api.letsencrypt.org/directory
      privateKeySecretRef:
        name: production-lets-encrypt-issuer-account-secret
      solvers:
        - http01:
            ingress:
              ingressClassName: traefik
  </code></pre>
  <p>Here is my Ingress file</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
      cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
    labels:
      app.kubernetes.io/name: hm-airbyte-ingress
      app.kubernetes.io/part-of: production-hm-airbyte
  spec:
    rules:
      - host: hm-airbyte.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.example.com
        secretName: hm-airbyte-ingress-tls
  
  </code></pre>
  <p>My Traefik pod shows error</p>
  <blockquote>
  <p>&quot;2024-06-27T06:35:45Z ERR Error configuring TLS error=&quot;secret production-hm-airbyte/hm-airbyte-ingress-tls does not exist&quot; ingress=hm-airbyte-ingress namespace=production-hm-airbyte providerName=kubernetes&quot;</p>
  </blockquote>
  <p>But I found I actually have a secret <code>Secret: hm-airbyte-ingress-tls-7w9z7</code> inside namespace <code>production-hm-airbyte</code>. Note it has a random suffix.</p>
  <p><a href="https://i.sstatic.net/gwoZhxcI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gwoZhxcI.png" alt="enter image description here" /></a></p>
  <p>I searched online and found</p>
  <ul>
  <li><a href="https://github.com/cert-manager/cert-manager/issues/3300" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/3300</a></li>
  <li><a href="https://github.com/cert-manager/cert-manager/issues/3283" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/3283</a></li>
  </ul>
  <p>This issue existed for four years, but unfortunately, no solution has been posted.</p>
  
Tags: <kubernetes><eks><cert-manager><traefik><letsencrypt>
Title: Error configuring TLS error: secret xxx does not exist
UserDisplayName: Hongbo Miao
ThreadText: # Error configuring TLS error: secret xxx does not exist
  
  (Asked by: username_0 on 2024-06-30)
  
  I am using "Traefik", "cert-mangaer", and "Let's Encrypt".
  
  Here is my ClusterIssuer file
  
  ```yaml
  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: production-lets-encrypt-issuer
    namespace: production-hm-cert-manager
  spec:
    acme:
      email: me@example.com
      server: https://acme-v02.api.letsencrypt.org/directory
      privateKeySecretRef:
        name: production-lets-encrypt-issuer-account-secret
      solvers:
        - http01:
            ingress:
              ingressClassName: traefik
  
  ```
  
  Here is my Ingress file
  
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
      cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
    labels:
      app.kubernetes.io/name: hm-airbyte-ingress
      app.kubernetes.io/part-of: production-hm-airbyte
  spec:
    rules:
      - host: hm-airbyte.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.example.com
        secretName: hm-airbyte-ingress-tls
  
  ```
  
  My Traefik pod shows error
  
  > "2024-06-27T06:35:45Z ERR Error configuring TLS error="secret production-hm-airbyte/hm-airbyte-ingress-tls does not exist" ingress=hm-airbyte-ingress namespace=production-hm-airbyte providerName=kubernetes"
  
  But I found I actually have a secret `Secret: hm-airbyte-ingress-tls-7w9z7` inside namespace `production-hm-airbyte`. Note it has a random suffix.
  
  [![enter image description here](https://i.sstatic.net/gwoZhxcI.png)](https://i.sstatic.net/gwoZhxcI.png)
  
  I searched online and found
  
  * <https://github.com/cert-manager/cert-manager/issues/3300>
  * <https://github.com/cert-manager/cert-manager/issues/3283>
  
  This issue existed for four years, but unfortunately, no solution has been posted.
  
  **Tags:** kubernetes, eks, cert-manager, traefik, letsencrypt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-06-30. Score: 0 | Accepted answer
  
  # 6/30/2024
  
  I found what I posted on 6/27/2024 is misleading. Even it repaired the issue in the question, but only on the surface. The Certificate still failed to issue.
  
  It took me some time to make everything including cert-manager, ExternalDNS, Traefik work well.
  
  Also, it turns out `acme.cert-manager.io/http01-edit-in-place: "true"` annotation is not necessary to fix the issue in the question.
  
  Note for the solution in my case, I am using DNS01 challenge.
  
  I also published all my deployment codes [here](https://github.com/username_0/username_0.com/issues/17463#issuecomment-2180242019). Hopefully it can give a bigger picture how it get deployed.
  
  ## cert-manager AWS IAM role `CertManagerRole`
  
  ### Trust relationships
  
  ```json
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com",
                      "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-cert-manager:hm-cert-manager"
                  }
              }
          }
      ]
  }
  
  ```
  
  ### Policy
  
  ```json
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "route53:GetChange"
              ],
              "Effect": "Allow",
              "Resource": "arn:aws:route53:::change/*"
          },
          {
              "Action": [
                  "route53:ListResourceRecordSets"
              ],
              "Effect": "Allow",
              "Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
          },
          {
              "Action": [
                  "route53:ChangeResourceRecordSets"
              ],
              "Condition": {
                  "ForAllValues:StringEquals": {
                      "route53:ChangeResourceRecordSetsNormalizedRecordNames": [
                          "_acme-challenge.*"
                      ],
                      "route53:ChangeResourceRecordSetsRecordTypes": [
                          "TXT"
                      ]
                  }
              },
              "Effect": "Allow",
              "Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
          },
          {
              "Action": [
                  "route53:ListHostedZonesByName"
              ],
              "Effect": "Allow",
              "Resource": "*"
          }
      ]
  }
  
  ```
  
  ## ExternalDNS AWS IAM role `ExternalDNSRole`
  
  ### Trust relationships
  
  ```json
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-external-dns:hm-external-dns",
                      "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com"
                  }
              }
          }
      ]
  }
  
  ```
  
  ### Policy
  
  ```json
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": [
          "route53:ChangeResourceRecordSets"
        ],
        "Effect": "Allow",
        "Resource": [
          "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
        ]
      },
      {
        "Action": [
          "route53:ListHostedZones",
          "route53:ListResourceRecordSets",
          "route53:ListTagsForResource"
        ],
        "Effect": "Allow",
        "Resource": [
          "*"
        ]
      }
    ]
  }
  
  ```
  
  ## cert-manager Helm chart custom values.yaml
  
  ```
  crds:
    enabled: true
    keep: true
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/CertManagerRole
  securityContext:
    fsGroup: 1001
  
  ```
  
  ## ExternalDNS Helm chart custom values.yaml
  
  ```yaml
  provider: aws
  aws:
    region: us-west-2
    zoneType: public
    txtOwnerId: Z1XXXXXXXXXXXXXXXXXXX
    domainFilters:
      - internal.example.com
    registry: txt
  policy: sync
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/ExternalDNSRole
  
  ```
  
  ## Traefik Helm chart custom values.yaml
  
  ```yaml
  providers:
    kubernetesIngress:
      publishedService:
        enabled: true
  service:
    enabled: true
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  
  ```
  
  ## ClusterIssuer
  
  ```yaml
  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: production-lets-encrypt-cluster-issuer
    namespace: production-hm-cert-manager
  spec:
    acme:
      email: me@example.com
      server: https://acme-v02.api.letsencrypt.org/directory
      privateKeySecretRef:
        name: production-lets-encrypt-cluster-issuer-account-secret
      solvers:
        - selector:
            dnsZones:
              - internal.example.com
          dns01:
            route53:
              region: us-west-2
  
  ```
  
  ## Ingress
  
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
      cert-manager.io/cluster-issuer: production-lets-encrypt-cluster-issuer
  spec:
    rules:
      - host: hm-airbyte.internal.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.internal.example.com
        secretName: production-hm-airbyte-certificate
  
  ```
  
  # 6/27/2024 (Misleading Answer)
  
  After a lot of experiments, I found if I add `acme.cert-manager.io/http01-edit-in-place: "true"` annotation, it helps resolve the issue.
  
  ```yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: hm-airbyte-ingress
    namespace: production-hm-airbyte
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
      cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
      acme.cert-manager.io/http01-edit-in-place: "true"  # <- Added this
    labels:
      app.kubernetes.io/name: hm-airbyte-ingress
      app.kubernetes.io/part-of: production-hm-airbyte
  spec:
    rules:
      - host: hm-airbyte.internal.example.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: hm-airbyte-airbyte-webapp-svc
                  port:
                    number: 80
    tls:
      - hosts:
          - hm-airbyte.internal.example.com
        secretName: hm-airbyte-ingress-tls
  
  ```
  
  So now here is what happens after I apply this Ingress file
  
  1. A temporary `Secret: hm-airbyte-ingress-tls-7w9z7` with random suffix will be created
  2. After about half minute, a `Secret: hm-airbyte-ingress-tls` without suffix will be created
  3. The one will suffix will be automatically deleted.
  
  Without this annotation, based on my experiment, it will be stuck at step 1 forever.
  
  This is the `Secret: hm-airbyte-ingress-tls` without suffix, you can see the difference between the temporary secret I posted in the question.
  
  [![enter image description here](https://i.sstatic.net/osEg2tA4.png)](https://i.sstatic.net/osEg2tA4.png)
  
  [Here](https://cert-manager.io/docs/usage/ingress/#supported-annotations) is the explanation of this annotation:
  
  > acme.cert-manager.io/http01-edit-in-place: "true": this controls whether the ingress is modified 'in-place', or a new one is created specifically for the HTTP01 challenge. If present, and set to "true", the existing ingress will be modified. Any other value, or the absence of the annotation assumes "false". This annotation will also add the annotation "cert-manager.io/issue-temporary-certificate": "true" onto created certificates which will cause a temporary certificate to be set on the resulting Secret until the final signed certificate has been returned. This is useful for keeping compatibility with the ingress-gce component.
  
  P.S. I cannot explain why this annotation helps resolve the issue. Because I actually expect if with this annotation, the temporary secret will not be created at all. Hope someone can further explain in future, thanks!
  
  
  ---
  
  