Answers: <p>I was getting this because I had a pod in <code>CrashLoopBackOff</code>. In my specific case because <code>yaml</code> file had changed (through the Grafana interface) and Grafana could no longer start.</p>
  <blockquote>
  <p>Error: ✗ Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default</p>
  </blockquote>
  <p><strong>So I couldn't port forward <code>:3000</code> because nothing was running on it.</strong></p>
  
Answers: CC BY-SA 4.0
Answers: 2024-09-23T04:19:52.707Z
Answers: 2024-09-25T02:04:22.060Z
Answers: 2024-09-25T02:04:22.060Z
Answers: Evan Carroll
Body: <p>I've been doing some work with k3s, and everything was working fine. Now I'm getting the following error when I restart and I try to run the port-forward command I had previously run to get to the service I want,</p>
  <blockquote>
  <p>E0921 19:32:10.085091 3083215 portforward.go:400] an error occurred forwarding 3000 -&gt; 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace &quot;/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5&quot;: failed to connect to localhost:3000 inside namespace &quot;ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df&quot;, IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused</p>
  </blockquote>
  <p>I can see my service running,</p>
  <pre><code>kubectl get svc -n monitoring prometheus-grafana
  NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
  prometheus-grafana   ClusterIP   10.43.34.93   &lt;none&gt;        80/TCP    45h
  </code></pre>
  <p>And when I run this command</p>
  <pre class="lang-shell prettyprint-override"><code>kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
  </code></pre>
  <p>I see</p>
  <pre><code>Forwarding from 127.0.0.1:3000 -&gt; 3000
  Forwarding from [::1]:3000 -&gt; 3000
  Handling connection for 3000
  E0921 19:37:03.012976 3085961 portforward.go:400] an error occurred forwarding 3000 -&gt; 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace &quot;/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5&quot;: failed to connect to localhost:3000 inside namespace &quot;ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df&quot;, IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused
  </code></pre>
  
Tags: <networking><k3s>
Title: k3s: Error E0921 failed to connect inside namespace ... connect: connection refused
UserDisplayName: Evan Carroll
ThreadText: # k3s: Error E0921 failed to connect inside namespace ... connect: connection refused
  
  (Asked by: username_0 on 2024-09-22)
  
  I've been doing some work with k3s, and everything was working fine. Now I'm getting the following error when I restart and I try to run the port-forward command I had previously run to get to the service I want,
  
  > E0921 19:32:10.085091 3083215 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused
  
  I can see my service running,
  
  ```
  kubectl get svc -n monitoring prometheus-grafana
  NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
  prometheus-grafana   ClusterIP   10.43.34.93   <none>        80/TCP    45h
  
  ```
  
  And when I run this command
  
  ```shell
  kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
  
  ```
  
  I see
  
  ```
  Forwarding from 127.0.0.1:3000 -> 3000
  Forwarding from [::1]:3000 -> 3000
  Handling connection for 3000
  E0921 19:37:03.012976 3085961 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused
  
  ```
  
  **Tags:** networking, k3s
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-09-23. Score: 1 | Accepted answer
  
  I was getting this because I had a pod in `CrashLoopBackOff`. In my specific case because `yaml` file had changed (through the Grafana interface) and Grafana could no longer start.
  
  > Error: ✗ Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default
  
  **So I couldn't port forward `:3000` because nothing was running on it.**
  
  
  ---
  
  