Answers: <p>I learned that it's not required to set the ingress class to nginx as I did in <code>kubernetes.io/ingress.class: nginx</code>. All annotations prefixed with <code>nginx.ingress.kubernetes.io</code> apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is</p>
  
  <pre><code>nginx.ingress.kubernetes.io/proxy-body-size: "0"
  </code></pre>
  
  <p>because the default value is way too small for Docker image layers. Without this annotation, we get <code>413 Request Entity Too Large</code> errors.</p>
  
  <p>The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:</p>
  
  <pre><code>defaults
      # ...
      retries 3
      timeout connect     1000
      timeout client      1000
      timeout server      1000
  </code></pre>
  
  <p>According to <a href="https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#2.49" rel="nofollow noreferrer">HAProxys time formats</a>, this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems. </p>
  
Answers: CC BY-SA 4.0
Answers: 2019-06-21T19:09:25.813Z
Answers: 2019-06-21T19:09:25.813Z
Answers: Daniel
Body: <p>When pushing images to a self hosted <a href="https://github.com/helm/charts/tree/master/stable/docker-registry" rel="nofollow noreferrer">Docker registry installed using their official helm chart</a>, this works for some images. But for one image, I'm not able to push two layers. Docker retry it several times, without success.</p>
  
  <pre><code>3a6fab8d5d23: Retrying in 2 seconds
  </code></pre>
  
  <p>The logs of the Kubernetes pod gave me the following error: </p>
  
  <pre><code>unknown error reading request payload: unexpected EOF
  </code></pre>
  
  <p>Both layers were relatively large (252.5 MB/ 393.8 MB).</p>
  
  <h3>What I already tried on the Ingress</h3>
  
  <p>I had the Kubernetes Nginx Ingress behind my registy. Using the direct <code>NodePort</code> without Ingress, it works perfectly. So it seems that some limits of the Ingress break the upload. The following annotations were applied:</p>
  
  <pre><code>ingress:
    enabled: true
    hosts: 
      - my-docker-registry.domain
    tls:
      - hosts:
        - my-docker-registry.domain
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
      nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
  </code></pre>
  
  <p>This was taken from <a href="https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/docker-registry/ingress-without-tls.yaml" rel="nofollow noreferrer">the Nginx Ingress documentation</a>, where a Docker Registry behind Nginx acts as example. Are there any other limits I need to increase so that my registry will properly work behind the Ingress? </p>
  
  <h3>Background</h3>
  
  <p>I want to install the <a href="https://www.ibm.com/support/knowledgecenter/en/SSYGQH_6.0.0/admin/install/cp_install_config_intro.html" rel="nofollow noreferrer">Component Pack for IBM Connections</a>. IBM has a very special way to deliver this Kubernetes based application: You get a ~4GB Zip file. It contains all images and helm charts as well as a shell script, which tags and pushes all those images to a custom registry. The <code>cnx-ingress</code> image is relatively large:</p>
  
  <pre><code>:~/Downloads/IC-ComponentPack-6.0.0.8# l microservices_connections/hybridcloud/images/cnx-ingress.tar 
  -rw------- 1 daniel daniel 739M Mär 29 13:18 microservices_connections/hybridcloud/images/cnx-ingress.tar
  </code></pre>
  
Tags: <docker><kubernetes><helm><ingress><docker-registry>
Title: Increase Kubernetes Nginx Ingress limits for large images in Docker registry
UserDisplayName: Daniel
ThreadText: # Increase Kubernetes Nginx Ingress limits for large images in Docker registry
  
  (Asked by: username_0 on 2019-05-27)
  
  When pushing images to a self hosted [Docker registry installed using their official helm chart](https://github.com/helm/charts/tree/master/stable/docker-registry), this works for some images. But for one image, I'm not able to push two layers. Docker retry it several times, without success.
  
  ```
  3a6fab8d5d23: Retrying in 2 seconds
  
  ```
  
  The logs of the Kubernetes pod gave me the following error:
  
  ```
  unknown error reading request payload: unexpected EOF
  
  ```
  
  Both layers were relatively large (252.5 MB/ 393.8 MB).
  
  ### What I already tried on the Ingress
  
  I had the Kubernetes Nginx Ingress behind my registy. Using the direct `NodePort` without Ingress, it works perfectly. So it seems that some limits of the Ingress break the upload. The following annotations were applied:
  
  ```
  ingress:
    enabled: true
    hosts: 
      - my-docker-registry.domain
    tls:
      - hosts:
        - my-docker-registry.domain
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
      nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
  
  ```
  
  This was taken from [the Nginx Ingress documentation](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/docker-registry/ingress-without-tls.yaml), where a Docker Registry behind Nginx acts as example. Are there any other limits I need to increase so that my registry will properly work behind the Ingress?
  
  ### Background
  
  I want to install the [Component Pack for IBM Connections](https://www.ibm.com/support/knowledgecenter/en/SSYGQH_6.0.0/admin/install/cp_install_config_intro.html). IBM has a very special way to deliver this Kubernetes based application: You get a ~4GB Zip file. It contains all images and helm charts as well as a shell script, which tags and pushes all those images to a custom registry. The `cnx-ingress` image is relatively large:
  
  ```
  :~/Downloads/IC-ComponentPack-6.0.0.8# l microservices_connections/hybridcloud/images/cnx-ingress.tar 
  -rw------- 1 username_0 username_0 739M Mär 29 13:18 microservices_connections/hybridcloud/images/cnx-ingress.tar
  
  ```
  
  **Tags:** docker, kubernetes, helm, ingress, docker-registry
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-06-21. Score: 1
  
  I learned that it's not required to set the ingress class to nginx as I did in `kubernetes.io/ingress.class: nginx`. All annotations prefixed with `nginx.ingress.kubernetes.io` apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is
  
  ```
  nginx.ingress.kubernetes.io/proxy-body-size: "0"
  
  ```
  
  because the default value is way too small for Docker image layers. Without this annotation, we get `413 Request Entity Too Large` errors.
  
  The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:
  
  ```
  defaults
      # ...
      retries 3
      timeout connect     1000
      timeout client      1000
      timeout server      1000
  
  ```
  
  According to [HAProxys time formats](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#2.49), this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems.
  
  
  ---
  
  