I am running a Istio setup where my python flask service running behind gunicorn.
when debugging the logs from the service, the flask service successfully execute the api call while the calling client is receiving 503 error from the rest call. I suspect this might be some issue with side car proxy or gunicorn server where it is processing the request.
Also, I am hitting the service directly from another pod in the namespace and hence not going through ingress gateway and virtualservice
In my case it was the SSL between Google Front End (ILB) and Istio service mesh. Somehow the connection between GFE and Istio gateway over TLS was not reliable. I converted that to HTTP from HTTP2(https) and it started working.
I will debug later why this https between those 2 was not working but moving HTTP2 to HTTP solved my issue.
Istio - Gunicorn - Python getting 503 upstream connect error or disconnect/reset before headers. reset reason: connection failure