Answers: <p>Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific <code>Host:</code> header, that's why it's 403 instead of timeout or something else.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-07-07T20:28:34.673Z
Answers: 2023-07-07T20:28:34.673Z
Answers: Kokizzu
Body: <p>I have this terraform-kubernetes script <a href="https://github.com/kokizzu/terraform1" rel="nofollow noreferrer">terraform1</a></p>
  <p>so prometheus scraping list of pods, and hitting <code>/metrics</code> based on custom resource <code>Prometheus</code> and <code>PodMonitor</code> rules</p>
  <p>it's working fine locally using minikube,</p>
  <p>but in GKE, it doesn't work,</p>
  <p>podmonitor works normally:</p>
  <p><a href="https://i.sstatic.net/0zP0w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0zP0w.png" alt="service discovery works" /></a></p>
  <p>but scraping failed (this part is green when on local minikube):</p>
  <p><a href="https://i.sstatic.net/HatyM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HatyM.png" alt="scraping targets" /></a></p>
  <p>normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:</p>
  <p><a href="https://i.sstatic.net/3bruF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3bruF.png" alt="metrics exists using port-forwrding" /></a></p>
  <p>does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on 10.63.1.4 can scrape other pods in the target list (10.63.1.252, 10.63.1.5, 10.63.3.5, 10.63.4.210, 10.63.1.133, etc)</p>
  
Tags: <kubernetes><google-kubernetes-engine>
Title: Pod to pod communication GKE
UserDisplayName: Kokizzu
ThreadText: # Pod to pod communication GKE
  
  (Asked by: username_0 on 2023-07-07)
  
  I have this terraform-kubernetes script [terraform1](https://github.com/username_0/terraform1)
  
  so prometheus scraping list of pods, and hitting `/metrics` based on custom resource `Prometheus` and `PodMonitor` rules
  
  it's working fine locally using minikube,
  
  but in GKE, it doesn't work,
  
  podmonitor works normally:
  
  [![service discovery works](https://i.sstatic.net/0zP0w.png)](https://i.sstatic.net/0zP0w.png)
  
  but scraping failed (this part is green when on local minikube):
  
  [![scraping targets](https://i.sstatic.net/HatyM.png)](https://i.sstatic.net/HatyM.png)
  
  normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:
  
  [![metrics exists using port-forwrding](https://i.sstatic.net/3bruF.png)](https://i.sstatic.net/3bruF.png)
  
  does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on 10.63.1.4 can scrape other pods in the target list (10.63.1.252, 10.63.1.5, 10.63.3.5, 10.63.4.210, 10.63.1.133, etc)
  
  **Tags:** kubernetes, google-kubernetes-engine
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-07-07. Score: 0
  
  Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific `Host:` header, that's why it's 403 instead of timeout or something else.
  
  
  ---
  
  