Answers: <p>The issue was due to the cert-manager. After I install this way, it works:</p>
  <pre><code>kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml
  kubectl create namespace observability
  kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.57.0/jaeger-operator.yaml -n observability
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2024-08-16T05:42:26.847Z
Answers: 2024-08-16T05:42:26.847Z
Answers: Alpin Cleopatra
Body: <p>I installed opentelemery collector</p>
  <pre><code>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
  helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=deployment --set image.repository=&quot;otel/opentelemetry-collector-k8s&quot; --set command.name=&quot;otelcol-k8s&quot; -f values.yaml
  </code></pre>
  <p>values.yaml</p>
  <pre><code>mode: deployment
  
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      otlp:
        endpoint: jaeger-collector.default.svc.cluster.local:14250
        tls:
          insecure: true
      debug:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          exporters: [debug]
        logs:
          receivers: [otlp]
          exporters: [debug]
  </code></pre>
  <p>Also installed Jaeger</p>
  <pre><code>helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
  helm install jaeger jaegertracing/jaeger
  </code></pre>
  <p>Port forward</p>
  <pre><code>kubectl port-forward svc/opentelemetry-collector 4317:4317
  kubectl port-forward svc/opentelemetry-collector 4318:4318
  </code></pre>
  <p>Run an application to point to the port</p>
  <pre><code>/*instrumentation.ts*/
  import * as opentelemetry from '@opentelemetry/sdk-node';
  import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
  import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
  import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
  import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
  import { Resource } from '@opentelemetry/resources';
  import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
  
  const resource = new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'test-service',
  });
  
  const sdk = new opentelemetry.NodeSDK({
    traceExporter: new OTLPTraceExporter({
      url: 'http://localhost:4317',
      // optional - collection of custom headers to be sent with each request, empty by default
      headers: {},
    }),
    metricReader: new PeriodicExportingMetricReader({
      exporter: new OTLPMetricExporter({
        url: 'http://localhost:4317',
        headers: {}, // an optional object containing custom headers to be sent with each request
      }),
    }),
    resource: resource,
    instrumentations: [getNodeAutoInstrumentations()],
  });
  sdk.start();
  </code></pre>
  <p>After access the app several times, the <code>opentelemetry-collector</code> pod got error log:</p>
  <pre><code>2024-07-19T08:50:13.705Z    error   exporterhelper/queue_sender.go:90   Exporting failed. Dropping data.    {&quot;kind&quot;: &quot;exporter&quot;, &quot;data_type&quot;: &quot;traces&quot;, &quot;name&quot;: &quot;otlp&quot;, &quot;error&quot;: &quot;not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService&quot;, &quot;dropped_items&quot;: 2}
  go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
      go.opentelemetry.io/collector/exporter@v0.104.0/exporterhelper/queue_sender.go:90
  go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
      go.opentelemetry.io/collector/exporter@v0.104.0/internal/queue/bounded_memory_queue.go:52
  go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
      go.opentelemetry.io/collector/exporter@v0.104.0/internal/queue/consumers.go:43
  </code></pre>
  <p>The log in <code>jaeger-collector</code> pod:</p>
  <pre><code>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1721717166.6132424,&quot;caller&quot;:&quot;grpc@v1.60.0/server.go:983&quot;,&quot;msg&quot;:&quot;[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \&quot;transport: http2Server.HandleStreams received bogus greeting from client: \\\&quot;GET / HTTP/1.1\\\\r\\\\nHost: lo\\\&quot;\&quot;&quot;,&quot;system&quot;:&quot;grpc&quot;,&quot;grpc_log&quot;:true}
  </code></pre>
  <p>How to setup correctly?</p>
  
Comments: CC BY-SA 4.0
Comments: 2024-08-14T22:34:24.673Z
Comments: might not be anything, but the host for the gRPC request is `Host: lo`, check to see if that resolves as this could be as simple as a name resolution issue.
Comments: Richard Slater
Comments: CC BY-SA 4.0
Comments: 2024-08-15T02:22:59.200Z
Comments: @RichardSlater Thank you. The issue due to didn't install cert-manager for jaeger.
Comments: Alpin Cleopatra
Comments: CC BY-SA 4.0
Comments: 2024-08-15T10:36:03.260Z
Comments: Thanks for letting me know, if you are able to answer your own question below it would help future users searching for the same problem.
Comments: Richard Slater
Tags: <kubernetes>
Title: How to send application trace data to OpenTelemetry in Kubernetes?
UserDisplayName: Alpin Cleopatra
ThreadText: # How to send application trace data to OpenTelemetry in Kubernetes?
  
  (Asked by: username_0 on 2024-07-22)
  
  I installed opentelemery collector
  
  ```
  helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
  helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=deployment --set image.repository="otel/opentelemetry-collector-k8s" --set command.name="otelcol-k8s" -f values.yaml
  
  ```
  
  values.yaml
  
  ```
  mode: deployment
  
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      otlp:
        endpoint: jaeger-collector.default.svc.cluster.local:14250
        tls:
          insecure: true
      debug:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          exporters: [debug]
        logs:
          receivers: [otlp]
          exporters: [debug]
  
  ```
  
  Also installed Jaeger
  
  ```
  helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
  helm install jaeger jaegertracing/jaeger
  
  ```
  
  Port forward
  
  ```
  kubectl port-forward svc/opentelemetry-collector 4317:4317
  kubectl port-forward svc/opentelemetry-collector 4318:4318
  
  ```
  
  Run an application to point to the port
  
  ```
  /*instrumentation.ts*/
  import * as opentelemetry from '@opentelemetry/sdk-node';
  import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
  import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
  import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
  import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
  import { Resource } from '@opentelemetry/resources';
  import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
  
  const resource = new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'test-service',
  });
  
  const sdk = new opentelemetry.NodeSDK({
    traceExporter: new OTLPTraceExporter({
      url: 'http://localhost:4317',
      // optional - collection of custom headers to be sent with each request, empty by default
      headers: {},
    }),
    metricReader: new PeriodicExportingMetricReader({
      exporter: new OTLPMetricExporter({
        url: 'http://localhost:4317',
        headers: {}, // an optional object containing custom headers to be sent with each request
      }),
    }),
    resource: resource,
    instrumentations: [getNodeAutoInstrumentations()],
  });
  sdk.start();
  
  ```
  
  After access the app several times, the `opentelemetry-collector` pod got error log:
  
  ```
  2024-07-19T08:50:13.705Z    error   exporterhelper/queue_sender.go:90   Exporting failed. Dropping data.    {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", "dropped_items": 2}
  go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
      go.opentelemetry.io/collector/exporter@v0.104.0/exporterhelper/queue_sender.go:90
  go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
      go.opentelemetry.io/collector/exporter@v0.104.0/internal/queue/bounded_memory_queue.go:52
  go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
      go.opentelemetry.io/collector/exporter@v0.104.0/internal/queue/consumers.go:43
  
  ```
  
  The log in `jaeger-collector` pod:
  
  ```
  {"level":"info","ts":1721717166.6132424,"caller":"grpc@v1.60.0/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams received bogus greeting from client: \\\"GET / HTTP/1.1\\\\r\\\\nHost: lo\\\"\"","system":"grpc","grpc_log":true}
  
  ```
  
  How to setup correctly?
  
  **Tags:** kubernetes
  
  > **Comments:**
  > **username_1**: might not be anything, but the host for the gRPC request is `Host: lo`, check to see if that resolves as this could be as simple as a name resolution issue.  
  > **username_0**: @username_1 Thank you. The issue due to didn't install cert-manager for jaeger.  
  > **username_1**: Thanks for letting me know, if you are able to answer your own question below it would help future users searching for the same problem.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-08-16. Score: 0
  
  The issue was due to the cert-manager. After I install this way, it works:
  
  ```
  kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml
  kubectl create namespace observability
  kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.57.0/jaeger-operator.yaml -n observability
  
  ```
  
  
  ---
  
  