events.action: opened
events.author: kevincantu
events.masked_author: username_0
events.text: I've just gotten started setting up otel-collector for some Kubernetes clusters where we use [Envoy](https://www.envoyproxy.io/) (configured via [Contour](https://projectcontour.io/)) for routing, and discovered a detail that gave me fits, so I think it's worth laying it all out here.  I suspect it may be a gRPC server issue in the collector: some gnarly interaction with Envoy, perhaps?
  
  ## Expected
  What I hoped was that otel-collector could be set up much like this demo with [YAGES](https://github.com/mhausenblas/yages#yet-another-grpc-echo-server) (a gRPC echo server), where:
  * grpcurl sends gRPC TLS traffic,
  * Envoy terminates TLS,
  * Envoy sends traffic to the upstream as **HTTP/2 cleartext gRPC**.
  
  I set this up using a Contour HTTPProxy in [TCP proxying](https://projectcontour.io/docs/main/httpproxy/#tcp-proxying) mode, which relies on SNI to route traffic by domain name:
  ```yaml
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: yages
    namespace: monitoring
    labels:
      app: yages
  spec:
    selector:
      matchLabels:
        app: yages
    replicas: 1
    template:
      metadata:
        labels:
          app: yages
      spec:
        containers:
        - name: grpcsrv
          image: quay.io/mhausenblas/yages:0.1.0
          ports:
          - containerPort: 9000
            protocol: TCP
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: yages
    namespace: monitoring
    labels:
      app: yages
  spec:
    ports:
    - name: demo
      port: 55682
      protocol: TCP
      targetPort: 9000
    selector:
      app: yages
  ---
  apiVersion: projectcontour.io/v1
  kind: HTTPProxy
  metadata:
    name: yages
    namespace: monitoring
    labels:
      app: yages
  spec:
    virtualhost:
      fqdn: yages.staging.test
      tls:
        secretName: yages-wildcard
        #passthrough: true
    tcpproxy:
      services:
      - name: yages
        port: 55682
        # tls: HTTP/1 TLS
        # h2:  HTTP/2 TLS
        # h2c: HTTP/2 cleartext
        protocol: h2c
  ```
  
  You can exercise that yages app (to send a ping and receive a pong) with the following [grpcurl](https://github.com/fullstorydev/grpcurl#grpcurl) command:
  ```bash
  grpcurl --insecure -v yages.staging.test:443 yages.Echo.Ping
  ```
  
  I expected routing just like that to work for otel-collector:
  * my Python demo app using `opentelemetry-exporter-otlp` sends gRPC TLS traffic,
  * Envoy terminates TLS,
  * Envoy sends traffic to the upstream as **HTTP/2 cleartext gRPC**...
  
  
  ## Actual
  
  But that didn't work.
  
  Instead, when configuring Envoy (via Contour) like that, I saw TCP events in the [Envoy access logs](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage) like so, but no success:
  ```
  [2020-10-01T03:18:09.593Z] "- - -" 0 - 0 15 33 - "-" "-" "-" "-" "172.21.5.170:55680"
  ```
  
  My sample app (sending traffic to `otel-grpc.staging.test:443`) only received `StatusCode.UNAVAILABLE` error responses!  (I extended [this part](https://github.com/open-telemetry/opentelemetry-python/blob/v0.13b0/exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py#L181) of the opentelemetry-exporter-otlp Python library to log those codes.)
  
  
  ## Workaround
  
  To make things work, I had to configure Envoy to pass HTTP/2 TLS traffic to the upstream.
  
  Like so:
  * my Python demo app using `opentelemetry-exporter-otlp` sends gRPC TLS traffic,
  * Envoy passes TLS traffic through to the upstream as **HTTP/2 TLS gRPC**.
  
  ```
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: otel-collector-conf
    namespace: monitoring
    labels:
      app: opentelemetry
      component: otel-collector-conf
  data:
    otel-collector-config: |
      receivers:
        otlp:
          protocols:
            grpc:
              tls_settings:
                cert_file: /tls/cert.pem
                key_file: /tls/key.pem
            http:
      processors:
        batch:
        memory_limiter:
          # Same as --mem-ballast-size-mib CLI argument
          ballast_size_mib: 1024
          # 80% of maximum memory
          limit_mib: 1600
          # 25% of limit
          spike_limit_mib: 512
          check_interval: 5s
      extensions:
        health_check: {}
        zpages:
          endpoint: "0.0.0.0:55679"  # default was localhost only!
      exporters:
        logging:
          logLevel: debug
        honeycomb:
          api_key: "$HONEYCOMB_API_KEY"
          dataset: "apps"
          api_url: "https://api.honeycomb.io"
      service:
        extensions: [health_check, zpages]
        pipelines:
          traces:
            receivers: [otlp]
            processors: [memory_limiter, batch]
            exporters: [logging, honeycomb]
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: otel-collector
    namespace: monitoring
    labels:
      app: opentelemetry
      component: otel-collector
  spec:
    ports:
    - name: zpages
      port: 55679
      # when proxied: http://localhost:8001/api/v1/namespaces/monitoring/services/http:otel-collector:55679/proxy/debug/tracez
    - name: otlp-grpc # Default endpoint for OpenTelemetry receiver.
      port: 55680
    - name: otlp-http
      port: 55681
    - name: jaeger-grpc # Default endpoing for Jaeger gRPC receiver
      port: 14250
    - name: jaeger-thrift-http # Default endpoint for Jaeger HTTP receiver.
      port: 14268
    - name: zipkin # Default endpoint for Zipkin receiver.
      port: 9411
    - name: metrics # Default endpoint for querying metrics.
      port: 8888
    selector:
      component: otel-collector
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: otel-collector
    namespace: monitoring
    labels:
      app: opentelemetry
      component: otel-collector
  spec:
    selector:
      matchLabels:
        app: opentelemetry
        component: otel-collector
    minReadySeconds: 5
    progressDeadlineSeconds: 120
    replicas: 2
    template:
      metadata:
        labels:
          app: opentelemetry
          component: otel-collector
      spec:
        containers:
        - command:
            - "/otelcontribcol"
            - "--log-level=DEBUG"
            - "--config=/conf/otel-collector-config.yaml"
            # Memory Ballast size should be max 1/3 to 1/2 of memory.
            - "--mem-ballast-size-mib=1024"
          #image: otel/opentelemetry-collector-dev:latest
          image: otel/opentelemetry-collector-contrib:0.11.0
          name: otel-collector
          envFrom:
          - secretRef:
              name: otel-collector
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
          ports:
          - containerPort: 55679 # Default endpoint for ZPages.
          - containerPort: 55680 # OTLP gRPC receiver.
          - containerPort: 55681 # OTLP HTML/JSON receiver.
          - containerPort: 14250 # Default endpoint for Jaeger HTTP receiver.
          - containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
          - containerPort: 9411  # Default endpoint for Zipkin receiver.
          - containerPort: 8888  # Default endpoint for querying metrics.
          volumeMounts:
          - name: otel-collector-config-vol
            mountPath: /conf
          - name: otel-tls
            mountPath: /tls
          livenessProbe:
            httpGet:
              path: /
              port: 13133 # Health Check extension default port.
          readinessProbe:
            httpGet:
              path: /
              port: 13133 # Health Check extension default port.
        volumes:
          - name: otel-collector-config-vol
            configMap:
              name: otel-collector-conf
              items:
                - key: otel-collector-config
                  path: otel-collector-config.yaml
          - name: otel-tls
            secret:
              secretName: otel-wildcard
              items:
                - key: tls.crt
                  path: cert.pem
                - key: tls.key
                  path: key.pem
  ---
  apiVersion: projectcontour.io/v1
  kind: HTTPProxy
  metadata:
    name: otel-collector
    namespace: monitoring
    annotations:
      kubernetes.io/ingress.class: "contour"
    labels:
      app: opentelemetry
      component: otel-collector
  spec:
    virtualhost:
      fqdn: otel.staging.test
      tls:
        #secretName: otel-wildcard
        passthrough: true
    tcpproxy:
      services:
      - name: otel-collector
        port: 55680
        # tls: HTTP/1 TLS
        # h2:  HTTP/2 TLS
        # h2c: HTTP/2 cleartext
        protocol: h2
  ```
  
  
  ## Bug?
  
  Specifically, I found that when routing OTLP (gRPC) traffic wrapped in HTTP/2 TLS: 
  * the yages echo app works when terminating TLS at Envoy (h2c), but
  * otel-collector does not and needs a TLS passthrough (h2).
  
  I think that means that there's _something_ we could do here to make otel-collector's gRPC server play nicely with Envoy!
events.title: otel-collector requires HTTP/2 TLS passthrough from Envoy / Contour: should it?
events.type: issue
events.action: created
events.author: kevincantu
events.masked_author: username_0
events.text: Thanks, by the way, to @pjanotti and @flands who helped me in the Gitter channel, and to this old Contour ticket that pointed me at [yages](https://github.com/projectcontour/contour/issues/379)!
events.type: comment
events.action: created
events.author: kevincantu
events.masked_author: username_0
events.text: My spidey sense tells me [this cmux issue](https://github.com/soheilhy/cmux/issues/64) may be related... 🤷‍♀️
events.type: comment
events.action: created
events.author: andrewcheelightstep
events.masked_author: username_1
events.text: Hi folks. Just a quick check to see if there is a timeline with this fix since we are running into this as well.
events.type: comment
events.action: created
events.author: carlosalberto
events.masked_author: username_2
events.text: Hey @username_0 
  
  As I'm not a Countour expert, I tested against 'vanilla' Envoy and I [got it working](https://github.com/username_2/hello-envoy-otlp-tls):
  
  * Client using OTel Python 0.15 (patched to accept self-signed certificates, while doing full TLS verification).
  * Envoy 1.16 doing TLS termination
  * Collector receiving plain text.
  
  I'm wondering if there's something Contour specific or I'm missing something. Let me know ;)
events.type: comment
events.action: created
events.author: kevincantu
events.masked_author: username_0
events.text: Oh that's encouraging: perhaps something in Envoy 1.16 fixes this?  (The version of Contour I last tested with was using an earlier Envoy.)
events.type: comment
events.action: created
events.author: carlosalberto
events.masked_author: username_2
events.text: Hey @username_0 Any update on this? ;)
events.type: comment
events.action: created
events.author: kevincantu
events.masked_author: username_0
events.text: I'm no longer actively working on the same system which used this, so I haven't spun up a cluster to try any of this out again lately.
  
  What I'd try, though, is setting up something like my example above, with a newer version of Contour (and its [corresponding newer version of Envoy](https://projectcontour.io/resources/compatibility-matrix/)), and see whether the workaround I showed is still necessary!
  
  Specifically:
  * remove the TLS setttings in the configmap for otel-collector, here (so the collector service isn't expecting TLS connections):
  ```
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: otel-collector-conf
  ...
  data:
    otel-collector-config: |
      receivers:
        otlp:
          protocols:
            grpc:
              # remove me?
              tls_settings:
                cert_file: /tls/cert.pem
                key_file: /tls/key.pem
  ...
  ```
  * and configure Contour to terminate the TLS and forward unencrypted gRPC connections to the backend by altering this:
  ```
  ---
  apiVersion: projectcontour.io/v1
  kind: HTTPProxy
  metadata:
    name: otel-collector
  ...
  spec:
    ...
    tcpproxy:
      services:
      - name: otel-collector
        port: 55680
        # tls: HTTP/1 TLS
        # h2:  HTTP/2 TLS
        # h2c: HTTP/2 cleartext
        protocol: h2  # try making me "h2c"?
  ```
events.type: comment