Answers: <p>This sounds like an asymmetric or n-path routing issue. Here is what is probably happening:</p>
  
  <p>Machine A at IP address 192.168.1.1 makes initiates a [SYN] request through the LB at 192.168.1.10. the LB then proxies the payload to Machine B at 192.168.1.2, so the payload now has source: 192.168.1.1 and has has destination: 192.168.1.2 (which used to be 192.168.1.10).</p>
  
  <p>So what happens now when 192.168.1.2 responds with a [SYN, ACK]? What <em>should</em> happen is that Machine B should respond to Machine A <em>through the load balancer</em> - typically because of a default route or gateway on the server that routes traffic through the LB. In this case however, the machine is on the same subnet, so the route/gateway is not used and the routing table ignored by the server. This means that when the server responds, the [SYN,ACK] appears to Machine A to come from an IP different than the IP that Machine A initiated the request with - it was expecting a source IP of 192.168.1.10 (the LB) but is seeing a [SYN,ACK] coming from 192.168.1.2 (machine B) and thus the LB is unable to establish a connection with the machine B in this scenario because the response went to the wrong device.</p>
  
  <p>The reason this works for external traffic is because of your default route - the responses to everyone else are routed through the ELB. The ELB sees that it was initiating a connection and automagically intercepts the response and swaps the source of 192.168.1.2 back to 192.168.1.10.</p>
  
  <p>So, for one solution to this issue, you could implement <a href="http://networkgeekstuff.com/networking/basic-load-balancer-scenarios-explained/" rel="nofollow noreferrer">one-armed load balancing</a> (also known as a load balancer on a stick). What this will do is use a Source NAT on the inside interface of the load balancer  (so assume you had outside interface 192.168.1.10 on your load balancer and 192.168.1.11 on the inside interface). This will make all traffic appear to be coming from 192.168.1.11 from the perspective of Machine B which should solve your connection issue.</p>
  
  <p>It appears however, that your <a href="https://forums.aws.amazon.com/thread.jspa?messageID=331340#331340" rel="nofollow noreferrer">AWS ELB doesn't support SNAT</a>, so you will either need to put your hosts and ELB on different subnets or use something that supports SNATs like F5's Virtual Edition which comes in <a href="https://aws.amazon.com/marketplace/pp/B00JL3UCQ4" rel="nofollow noreferrer">hourly</a> or <a href="https://aws.amazon.com/marketplace/pp/B00KXHNAPW" rel="nofollow noreferrer">BYOL</a> flavors. Beware connection limitations with SNATing though - if you need over about 30k simultaneous connections you will run into <a href="https://support.f5.com/csp/article/K8246" rel="nofollow noreferrer">SNAT port exhaustion and need to start using a SNAT pool.</a>.</p>
  
  <p>Hence, you best solution (for cost and to prevent future issues) would be to make sure the client and server are on different subnets.</p>
  
  <p>The best way to confirm would be to use <a href="https://danielmiessler.com/study/tcpdump/" rel="nofollow noreferrer">tcpdump</a> on the connecting host and/or back-end server and look for responses coming directly to/from the back-end server instead of going through the load balancer. You can then load your dump file into WireShark to figure out exactly what is going on.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-06-05T19:02:09.660Z
Answers: ELB doesn't forward packets.  It makes new TCP connections and forwards payloads.  Route asymmetry is one thing the problem can't be.
Answers: Michael - sqlbot
Answers: CC BY-SA 3.0
Answers: 2017-06-05T19:13:03.697Z
Answers: So do F5's and they still suffer from asymmetric routing issues. Even with a full proxy architecture and a separate, new TCP connection, an F5 load balancer will by default the source address of the connecting client, so the issue still occurs exactly as described above. I am assuming the ELB works in a similar manner. I know A10s do behave in the same manner.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-06T09:58:11.380Z
Answers: They don't, ELBs have a seperate IP. The
Answers: Robo
Answers: CC BY-SA 3.0
Answers: 2017-06-06T15:16:46.633Z
Answers: AWS ELB can act as a reverse proxy, not only as a TCP load balancer. As OP said the ELB is doing the SSL offloading, it can't be a TCP balancer and have to be a HTTP reverse proxy. Your answer is not applicable to the context and ELB are never used for outgoing packets (they are not routers at all). Moreover, if you try to proxy with an F5 with 2 interfaces and setting the same subnet on each interface you're indeed creating a problem, solving it with SNAT is just a bad solution.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-06-06T15:26:38.237Z
Answers: I am aware. The front side connection is independent of the connection on the back side. In other words, you can create a full connection including an SSL handshake between the client and the ELB, but If the source IP from the ELB to the server is on the same subnet as the server, the connection *will* fail. At what stage an ELB will fail, I can't say. On an F5, you would establish a Client <-> LB connection, but the SSL handshake would fail when the F5 fails to establish a connection between the server and LB. But, yes SNAT is not the recommended solution which is why I provided 2 options.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-06T15:30:25.727Z
Answers: I think people are hung up on terminology here because I used the word "routing." It is important to remember that servers have routes and routing does not refer to "router" in this context, it refers to the route that packets are taking through the network. I never understood the ELB to be routing traffic outbound. I understood it to be proxying traffic inbound and the route that that the server-side packets take is asymmetric.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-06T15:30:59.907Z
Answers: Having been an F5 SME for the last three years, half of which was spent working technical support for F5, I can tell you [this is the very common problem](https://goo.gl/qUQZ4n)
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-06T15:51:04.707Z
Answers: Well, here what the machine B from your example see is the ELB IP, the client IP will be in the X-Forwarded-Port header. The client could be aside the server that won't be a problem. When in HTTP mode an ELB doesn't act as a F5 with SSL termination. (even in TCP mode, it's still an nginx like load balancer, nothing remotely comparable at all). What I think you are hung on is "proxying", we're indeed talking about proxying packets and not forwarding packets. I can send you a tcpdump of an ELB within the same subnet as 2 machines if you want, it works.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-06-05T18:48:37.630Z
Answers: 2017-06-06T15:37:50.840Z
Answers: 2017-06-06T15:37:50.840Z
Answers: James Shewey
Answers: <p>So what you have actually is this:</p>
  
  <p><a href="https://i.sstatic.net/0PDeG.png" rel="noreferrer"><img src="https://i.sstatic.net/0PDeG.png" alt="Schema of OP&#39;s architecture"></a></p>
  
  <p>As your API ELB is in a private zone it can't be accessed from the internet.<br>
  Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.</p>
  
  <p>You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.</p>
  
  <p>The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-06-07T08:38:53.623Z
Answers: 2017-06-07T08:38:53.623Z
Answers: Tensibai
Body: <h2>Some background:</h2>
  
  <p>I've created a moderately complex network using Amazon's vpc. It's a three-tiered network across two availability zones. Each layer has a subnet in zone-a and zone-b. The presentation layer is at the top, there’s an application layer in the middle, and a core layer at the bottom.</p>
  
  <p>All of the security groups and ACL's for the subnets are currently allowing ALL inbound and outbound traffic to help me reduce surface area of the issue. </p>
  
  <p>The presentation layer’s routing table is pointing all traffic to an internet gateway. The NAT gateway is in a segregated subnet also pointing all traffic to the internet gateway.</p>
  
  <p>My application has two components, a UI (React.js) and an API (Node/Express). They are deployed as docker images. In front of each is a classic load balancer. </p>
  
  <p>The UI-ELB is internet facing and resides in the presentation layer, routing traffic from 80/443 to port 8080 and is associated with my app-ec2 that is placed in the application-layer subnet. </p>
  
  <p>My API has an internal load balancer in front of it. The API-ELB is in the application layer (in the same subnet as the app-ec2), and takes traffic on port 80/443 and routes it down to the api-ec2 in the core on port 3000.</p>
  
  <p>Both load balancers are offloading the certificate before passing traffic to their instances.</p>
  
  <p>I have both my load balancers associated as alias's in Route53, and referenced in the applications by their pretty url (<a href="https://app.website.com" rel="nofollow noreferrer">https://app.website.com</a>). Each load balancer passes the defined healthchecks and reports all ec2 instances in use.</p>
  
  <p>Lastly, on the API I have enabled cors using the cors nodejs package.</p>
  
  <p>Here's a <a href="https://drive.google.com/a/emeentsmedia.com/file/d/0B8gkXonu3RXqaGJHU1Y0U1ZRUHc/view?usp=drivesdk" rel="nofollow noreferrer">quick and dirty diagram</a> of my network.</p>
  
  <h2>The problem:</h2>
  
  <p>The APP-ELB successfully routes me to the application. However, when the app tries to send a GET request to the API-ELB, it first sends an OPTIONS request that tims out with the error code 408.</p>
  
  <p><strong>Where it gets weird</strong></p>
  
  <p>Some of the weirdest things I've encountered while debugging are:</p>
  
  <ol>
  <li>I can SSH into the app-ec2 instance and can run a successful curl against the API-ELB. I’ve tried many, and they all work. A few examples are: <code>curl -L https://api.website.com/system/healthcheck</code> and <code>curl -L -X OPTIONS https://api.website.com/system/healthcheck</code>. It always returns the desired information.</li>
  <li>I've moved the entire application out of my network into a public default vpc and it works as it's supposed to.</li>
  <li>I have the api-ec2 writing all network requests to the console. While it shows the healthcheck requests, it does not show any requests from the app-ec2. This leads me to believe traffic is not even reaching the api.</li>
  </ol>
  
  <p>Really the biggest thing that has me at a complete loss is that curling the internal api elb works, but the axios request to the same exact url does not. This doesn't make sense to me at all.</p>
  
  <h2>What I've tried</h2>
  
  <p>I originally spent a lot of time playing with ACL rules and security groups thinking I did something wrong. Eventually I just said, "screw it", and opened everything up to try and take that piece out of the equation.</p>
  
  <p>I've spent way to much time playing with Cors on my api. Eventually landing on the configuration I have now, that is the default <code>app.use(cors())</code> callback provided by the cors node package. I've also included the <code>app.options('*', cors())</code> that is recommended in the documentation.</p>
  
  <p>I've google everything under the sun, but specifically whether I need to define some special custom headers with the elbs? But can't seem to find anything. Plus, when I moved my app out of the network it worked just fine.</p>
  
  <p>I'm sure I've tried many other things, but these seem to be the most pertinent. What am I missing? I realize this is potentially a very vague and broad issue, and an enormous post, but I appreciate any insight and your time in reading in it!</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-06-05T20:16:58.417Z
Comments: *Thats two subnets per zone: a presentation layer, an application layer, and a core layer.*  That's three.  Can you clarify that?  *I have the route table for the presentation layer and NAT drive routing all traffic through an internet gateway. I have both the presentation and core layers routing all traffic through the nat drive.*  That seems self-contradictory.   If the presentation layer is routing through the NAT (drive?) (Gateway?), then it isn't also routing through the Internet gateway.  Which of your layers are on which of your subnets, and what's the default route for each subnet?
Comments: Michael - sqlbot
Comments: CC BY-SA 3.0
Comments: 2017-06-05T20:20:32.120Z
Comments: ...Specifically, your external-facing ELB must be on subnets whose default route points to the Internet Gateway, which almost always means it is **not** correct to place it on the same subnet(s) as the instances it is balancing traffic towards.  Those instances would be on a subnet whose default route is the NAT Gateway... and the NAT Gateway itself, in turn would **not** be on any of the same subnets as the instances for which it is providing outbound services but might be on the same subnet as the ELB.
Comments: Michael - sqlbot
Comments: CC BY-SA 3.0
Comments: 2017-06-05T20:21:39.170Z
Comments: Yeah, sorry about that, started getting some words mixed up. I have two subnets (one in zone-a and one in zone-b) in each of the three layers. The presentation layer is routing through the internet gateway, as is the nat. One thing I did fail to mention was that NAT is in it's own seperate subnet. Then the app and core route through the nat gateway.
Comments: David Meents
Comments: CC BY-SA 3.0
Comments: 2017-06-05T20:36:52.257Z
Comments: You may want to thoroughly review the question and clarify accordingly -- it's a good question, but there are some difficult convolutions to follow, and many things to check.  When you say *"The application-UI is timing out on the options request when making an api call"* who sees this error?  The external caller? `curl -X OPTIONS 127.0.0.1...` on app-ec2?   Only `OPTIONS` is broken?  The ELBs are "Classic" not "Application," correct?  Are all of the instances able to correctly access the Internet via the NAT, e.g. `curl ipv4.icanhazip.com`? (Yes, I ask for a reason that may seem obscure.)
Comments: Michael - sqlbot
Comments: CC BY-SA 3.0
Comments: 2017-06-05T20:55:59.783Z
Comments: I've tried clarifying it a bit better, thanks for pointing these things out to me. Let me try your curl command and I'll edit with the results. I have curled google on each the api and app without any issues.
Comments: David Meents
Comments: CC BY-SA 3.0
Comments: 2017-06-06T10:02:33.117Z
Comments: Can you please provide a network diagram?
Comments: Robo
Comments: CC BY-SA 3.0
Comments: 2017-06-06T14:59:23.960Z
Comments: Ahum... Your API ELB is in private network right ? And your Js app (running into customer browser somewhere on the internet) can't communicate with it ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-06-06T15:31:53.513Z
Comments: I've added a rough diagram to illustrate how its set up. Yes the api elb is in the private network, and so is the app. Its the app-elb that is communicating to the user.
Comments: David Meents
Comments: CC BY-SA 3.0
Comments: 2017-06-06T15:52:07.093Z
Comments: Unless I'm totally mistaken, react.js apps run in the browser and need to contact the API server, your frontend server just serve html and js files and does no routing/proxying of request toward the API
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-06-06T15:54:29.077Z
Comments: Holy hell, I think that might be it.... Wow. That never even occurred to me. Let me give that a spin tonight when I get back.
Comments: David Meents
Comments: CC BY-SA 3.0
Comments: 2017-06-06T19:40:55.927Z
Comments: Tensibai's observation totally worked. I can't thank you enough for pointing this out to me. I should have realized that weeks ago, my head was totally somewhere else. Thank you! You should make a quick answer so I can accept it!
Comments: David Meents
Tags: <amazon-web-services><amazon-ec2><infrastructure><software-engineering><cors>
Title: Application calling AWS internal load balancer in same subnet is timing out
UserDisplayName: David Meents
ThreadText: # Application calling AWS internal load balancer in same subnet is timing out
  
  (Asked by: username_0 on 2017-06-05)
  
  ## Some background:
  
  I've created a moderately complex network using Amazon's vpc. It's a three-tiered network across two availability zones. Each layer has a subnet in zone-a and zone-b. The presentation layer is at the top, there’s an application layer in the middle, and a core layer at the bottom.
  
  All of the security groups and ACL's for the subnets are currently allowing ALL inbound and outbound traffic to help me reduce surface area of the issue.
  
  The presentation layer’s routing table is pointing all traffic to an internet gateway. The NAT gateway is in a segregated subnet also pointing all traffic to the internet gateway.
  
  My application has two components, a UI (React.js) and an API (Node/Express). They are deployed as docker images. In front of each is a classic load balancer.
  
  The UI-ELB is internet facing and resides in the presentation layer, routing traffic from 80/443 to port 8080 and is associated with my app-ec2 that is placed in the application-layer subnet.
  
  My API has an internal load balancer in front of it. The API-ELB is in the application layer (in the same subnet as the app-ec2), and takes traffic on port 80/443 and routes it down to the api-ec2 in the core on port 3000.
  
  Both load balancers are offloading the certificate before passing traffic to their instances.
  
  I have both my load balancers associated as alias's in Route53, and referenced in the applications by their pretty url (<https://app.website.com>). Each load balancer passes the defined healthchecks and reports all ec2 instances in use.
  
  Lastly, on the API I have enabled cors using the cors nodejs package.
  
  Here's a [quick and dirty diagram](https://drive.google.com/a/emeentsmedia.com/file/d/0B8gkXonu3RXqaGJHU1Y0U1ZRUHc/view?usp=drivesdk) of my network.
  
  ## The problem:
  
  The APP-ELB successfully routes me to the application. However, when the app tries to send a GET request to the API-ELB, it first sends an OPTIONS request that tims out with the error code 408.
  
  **Where it gets weird**
  
  Some of the weirdest things I've encountered while debugging are:
  
  1. I can SSH into the app-ec2 instance and can run a successful curl against the API-ELB. I’ve tried many, and they all work. A few examples are: `curl -L https://api.website.com/system/healthcheck` and `curl -L -X OPTIONS https://api.website.com/system/healthcheck`. It always returns the desired information.
  2. I've moved the entire application out of my network into a public default vpc and it works as it's supposed to.
  3. I have the api-ec2 writing all network requests to the console. While it shows the healthcheck requests, it does not show any requests from the app-ec2. This leads me to believe traffic is not even reaching the api.
  
  Really the biggest thing that has me at a complete loss is that curling the internal api elb works, but the axios request to the same exact url does not. This doesn't make sense to me at all.
  
  ## What I've tried
  
  I originally spent a lot of time playing with ACL rules and security groups thinking I did something wrong. Eventually I just said, "screw it", and opened everything up to try and take that piece out of the equation.
  
  I've spent way to much time playing with Cors on my api. Eventually landing on the configuration I have now, that is the default `app.use(cors())` callback provided by the cors node package. I've also included the `app.options('*', cors())` that is recommended in the documentation.
  
  I've google everything under the sun, but specifically whether I need to define some special custom headers with the elbs? But can't seem to find anything. Plus, when I moved my app out of the network it worked just fine.
  
  I'm sure I've tried many other things, but these seem to be the most pertinent. What am I missing? I realize this is potentially a very vague and broad issue, and an enormous post, but I appreciate any insight and your time in reading in it!
  
  **Tags:** amazon-web-services, amazon-ec2, infrastructure, software-engineering, cors
  
  > **Comments:**
  > **username_1**: *Thats two subnets per zone: a presentation layer, an application layer, and a core layer.*  That's three.  Can you clarify that?  *I have the route table for the presentation layer and NAT drive routing all traffic through an internet gateway. I have both the presentation and core layers routing all traffic through the nat drive.*  That seems self-contradictory.   If the presentation layer is routing through the NAT (drive?) (Gateway?), then it isn't also routing through the Internet gateway.  Which of your layers are on which of your subnets, and what's the default route for each subnet?  
  > **username_1**: ...Specifically, your external-facing ELB must be on subnets whose default route points to the Internet Gateway, which almost always means it is **not** correct to place it on the same subnet(s) as the instances it is balancing traffic towards.  Those instances would be on a subnet whose default route is the NAT Gateway... and the NAT Gateway itself, in turn would **not** be on any of the same subnets as the instances for which it is providing outbound services but might be on the same subnet as the ELB.  
  > **username_0**: Yeah, sorry about that, started getting some words mixed up. I have two subnets (one in zone-a and one in zone-b) in each of the three layers. The presentation layer is routing through the internet gateway, as is the nat. One thing I did fail to mention was that NAT is in it's own seperate subnet. Then the app and core route through the nat gateway.  
  > **username_1**: You may want to thoroughly review the question and clarify accordingly -- it's a good question, but there are some difficult convolutions to follow, and many things to check.  When you say *"The application-UI is timing out on the options request when making an api call"* who sees this error?  The external caller? `curl -X OPTIONS 127.0.0.1...` on app-ec2?   Only `OPTIONS` is broken?  The ELBs are "Classic" not "Application," correct?  Are all of the instances able to correctly access the Internet via the NAT, e.g. `curl ipv4.icanhazip.com`? (Yes, I ask for a reason that may seem obscure.)  
  > **username_0**: I've tried clarifying it a bit better, thanks for pointing these things out to me. Let me try your curl command and I'll edit with the results. I have curled google on each the api and app without any issues.  
  > **username_2**: Can you please provide a network diagram?  
  > **username_3**: Ahum... Your API ELB is in private network right ? And your Js app (running into customer browser somewhere on the internet) can't communicate with it ?  
  > **username_0**: I've added a rough diagram to illustrate how its set up. Yes the api elb is in the private network, and so is the app. Its the app-elb that is communicating to the user.  
  > **username_3**: Unless I'm totally mistaken, react.js apps run in the browser and need to contact the API server, your frontend server just serve html and js files and does no routing/proxying of request toward the API  
  > **username_0**: Holy hell, I think that might be it.... Wow. That never even occurred to me. Let me give that a spin tonight when I get back.  
  > **username_0**: username_3's observation totally worked. I can't thank you enough for pointing this out to me. I should have realized that weeks ago, my head was totally somewhere else. Thank you! You should make a quick answer so I can accept it!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-06-07. Score: 9 | Accepted answer
  
  So what you have actually is this:
  
  [![Schema of OP's architecture](https://i.sstatic.net/0PDeG.png)](https://i.sstatic.net/0PDeG.png)
  
  As your API ELB is in a private zone it can't be accessed from the internet.  
  Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.
  
  You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.
  
  The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.
  
  
  ---
  
  ### Answer by username_4 on 2017-06-05. Score: 1
  
  This sounds like an asymmetric or n-path routing issue. Here is what is probably happening:
  
  Machine A at IP address 192.168.1.1 makes initiates a [SYN] request through the LB at 192.168.1.10. the LB then proxies the payload to Machine B at 192.168.1.2, so the payload now has source: 192.168.1.1 and has has destination: 192.168.1.2 (which used to be 192.168.1.10).
  
  So what happens now when 192.168.1.2 responds with a [SYN, ACK]? What *should* happen is that Machine B should respond to Machine A *through the load balancer* - typically because of a default route or gateway on the server that routes traffic through the LB. In this case however, the machine is on the same subnet, so the route/gateway is not used and the routing table ignored by the server. This means that when the server responds, the [SYN,ACK] appears to Machine A to come from an IP different than the IP that Machine A initiated the request with - it was expecting a source IP of 192.168.1.10 (the LB) but is seeing a [SYN,ACK] coming from 192.168.1.2 (machine B) and thus the LB is unable to establish a connection with the machine B in this scenario because the response went to the wrong device.
  
  The reason this works for external traffic is because of your default route - the responses to everyone else are routed through the ELB. The ELB sees that it was initiating a connection and automagically intercepts the response and swaps the source of 192.168.1.2 back to 192.168.1.10.
  
  So, for one solution to this issue, you could implement [one-armed load balancing](http://networkgeekstuff.com/networking/basic-load-balancer-scenarios-explained/) (also known as a load balancer on a stick). What this will do is use a Source NAT on the inside interface of the load balancer (so assume you had outside interface 192.168.1.10 on your load balancer and 192.168.1.11 on the inside interface). This will make all traffic appear to be coming from 192.168.1.11 from the perspective of Machine B which should solve your connection issue.
  
  It appears however, that your [AWS ELB doesn't support SNAT](https://forums.aws.amazon.com/thread.jspa?messageID=331340#331340), so you will either need to put your hosts and ELB on different subnets or use something that supports SNATs like F5's Virtual Edition which comes in [hourly](https://aws.amazon.com/marketplace/pp/B00JL3UCQ4) or [BYOL](https://aws.amazon.com/marketplace/pp/B00KXHNAPW) flavors. Beware connection limitations with SNATing though - if you need over about 30k simultaneous connections you will run into [SNAT port exhaustion and need to start using a SNAT pool.](https://support.f5.com/csp/article/K8246).
  
  Hence, you best solution (for cost and to prevent future issues) would be to make sure the client and server are on different subnets.
  
  The best way to confirm would be to use [tcpdump](https://danielmiessler.com/study/tcpdump/) on the connecting host and/or back-end server and look for responses coming directly to/from the back-end server instead of going through the load balancer. You can then load your dump file into WireShark to figure out exactly what is going on.
  
  > **Comments:**
  > **username_1**: ELB doesn't forward packets.  It makes new TCP connections and forwards payloads.  Route asymmetry is one thing the problem can't be.  
  > **username_4**: So do F5's and they still suffer from asymmetric routing issues. Even with a full proxy architecture and a separate, new TCP connection, an F5 load balancer will by default the source address of the connecting client, so the issue still occurs exactly as described above. I am assuming the ELB works in a similar manner. I know A10s do behave in the same manner.  
  > **username_2**: They don't, ELBs have a seperate IP. The  
  > **username_3**: AWS ELB can act as a reverse proxy, not only as a TCP load balancer. As OP said the ELB is doing the SSL offloading, it can't be a TCP balancer and have to be a HTTP reverse proxy. Your answer is not applicable to the context and ELB are never used for outgoing packets (they are not routers at all). Moreover, if you try to proxy with an F5 with 2 interfaces and setting the same subnet on each interface you're indeed creating a problem, solving it with SNAT is just a bad solution.  
  > **username_4**: I am aware. The front side connection is independent of the connection on the back side. In other words, you can create a full connection including an SSL handshake between the client and the ELB, but If the source IP from the ELB to the server is on the same subnet as the server, the connection *will* fail. At what stage an ELB will fail, I can't say. On an F5, you would establish a Client <-> LB connection, but the SSL handshake would fail when the F5 fails to establish a connection between the server and LB. But, yes SNAT is not the recommended solution which is why I provided 2 options.  
  > **username_4**: I think people are hung up on terminology here because I used the word "routing." It is important to remember that servers have routes and routing does not refer to "router" in this context, it refers to the route that packets are taking through the network. I never understood the ELB to be routing traffic outbound. I understood it to be proxying traffic inbound and the route that that the server-side packets take is asymmetric.  
  > **username_4**: Having been an F5 SME for the last three years, half of which was spent working technical support for F5, I can tell you [this is the very common problem](https://goo.gl/qUQZ4n)  
  > **username_3**: Well, here what the machine B from your example see is the ELB IP, the client IP will be in the X-Forwarded-Port header. The client could be aside the server that won't be a problem. When in HTTP mode an ELB doesn't act as a F5 with SSL termination. (even in TCP mode, it's still an nginx like load balancer, nothing remotely comparable at all). What I think you are hung on is "proxying", we're indeed talking about proxying packets and not forwarding packets. I can send you a tcpdump of an ELB within the same subnet as 2 machines if you want, it works.  
  
  ---
  
  