<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): Everything 'ELB' for past few months</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug report</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.6</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Jessie</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.4.26-k8s</li>
<li><strong>Install tools</strong>: kops 1.4.1</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">My goal was to expose a k8s Service using an AWS ELB, using both HTTP and HTTPS. HTTPS should use an AWS cert, passing through both HTTP and HTTPS to an nginx container.</p>
<p dir="auto">Scenario A:</p>
<p dir="auto">With these three options set:<br>
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx<br>
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https<br>
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https</p>
<p dir="auto">The listeners created were:<br>
TCP/80 -&gt; TCP/(nodeport for container's 80)<br>
HTTPS/443 -&gt; HTTPS/(nodeport for container's 443), using AWS cert</p>
<p dir="auto">While this does move traffic on both 80 and 443, the source IP is lost on port 80. 443 works properly.</p>
<p dir="auto">Scenario B:</p>
<p dir="auto">Backend changed to http, otherwise the same:<br>
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http</p>
<p dir="auto">The listeners created this time were:<br>
HTTP/80 -&gt; HTTP/(nodeport for container's 80)<br>
HTTPS/443 -&gt; HTTP/(nodeport for container's 443), using AWS cert</p>
<p dir="auto">In this instance, 80 works properly, but 443 is completely broken as it's attempting to deliver HTTP traffic to the container over port 443.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">In scenario A (backend-protocol: https), I expected:<br>
HTTP/80 -&gt; HTTP/(nodeport for container's 80) [or alternatively, -&gt; HTTPS/(nodeport for 443)]<br>
HTTPS/443 -&gt; HTTPS/(nodeport for container's 443)</p>
<p dir="auto">In short, I expected HTTP on 80, not TCP, which would enable proper L7 on both ports.</p>
<p dir="auto">In scenario B (backend-protocol: http), I expected:<br>
HTTP/80 -&gt; HTTP/(nodeport for container's 80)<br>
HTTPS/443 -&gt; HTTP/(nodeport for container's 80)</p>
<p dir="auto">Here I expected the backend HTTP to be directed to the NodePort that forwards to container's port 80 in both cases.</p>
<p dir="auto">In my mind, in the presence of aws-load-balancer-ssl-cert, k8s already assumes traffic should now be L7. I expect L7 (and the benefits of the X-Forwarded-* set of headers) to happen on both 80 and 443, not just 443.</p>
<p dir="auto">Further, I expect the Instance Protocol and the NodePort assigned to Instance Port to match, and not get HTTP traffic directed to a NodePort for HTTPS, or vice versa.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
I installed Kubernetes via Kops 1.4.1, targeting K8s 1.4.6. Then I built a simple nginx container (that exposes 80 + 443) and used the following Service definition:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
      # or http
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
  - name: https
    port: 443
    protocol: TCP
  selector:
    app: web"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
      # or http
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
  - name: https
    port: 443
    protocol: TCP
  selector:
    app: web
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Ideally, I'd like backend-protocol of both http and https to be working, as we do run some sensitive services that require https. If only https worked, we'd accept the internal slowdown on less critical services.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): Yes</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): elasticsearch</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.6_coreos.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: on premise, vCloud</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS alpha (1221.0.0)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux f1 4.8.6-coreos <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Thu Nov 3 04:43:35 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>: None. Default installation</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
When I tried to install es-controller.yaml, The containers are created successfully and but terminated with the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Exception in thread &quot;main&quot; java.lang.IllegalStateException: Unable to access 'path.data' (/data/kubernetes-logging)
Likely root cause: java.nio.file.AccessDeniedException: /data/kubernetes-logging
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
        at java.nio.file.Files.createDirectory(Files.java:674)
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
        at java.nio.file.Files.createDirectories(Files.java:767)
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:259)
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
        at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
        at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:212)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details."><pre class="notranslate"><code class="notranslate">Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/data/kubernetes-logging)
Likely root cause: java.nio.file.AccessDeniedException: /data/kubernetes-logging
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
        at java.nio.file.Files.createDirectory(Files.java:674)
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
        at java.nio.file.Files.createDirectories(Files.java:767)
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:259)
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
        at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
        at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:212)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
The container runs successfully.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
in a running kube cluster, run</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl create -f cluster/addons/fluentd-elasticsearch/es-controller.yaml"><pre class="notranslate"><code class="notranslate">kubectl create -f cluster/addons/fluentd-elasticsearch/es-controller.yaml
</code></pre></div>
<p dir="auto">The pod will restart itself because of the error. The error message is in the container log.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
It seems that when container mount the volume /data, it mounted as root:root (755). However, elasticsearch is running under user elasticsearch. So it couldn't write the files under /data.<br>
After I run <code class="notranslate">chown elasticsearch:elasticsearch /data</code>, the container runs fine.</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/crassirostris/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/crassirostris">@crassirostris</a></p>