Weird TCP issue with amazon S3
I am having a weird TCP issue that makes no sense at all.  I am tearing my hair out.

When downloading a file from amazon s3 (and only amazon s3- every other site works fine) some percentage of the time the connections dies.   This only happens from our servers in Seattle with a Web Server behind a nat router.  Connecting directly from the router works fine. From our servers here in Victoria everything works fine and we can't replicate this at all.


Web Server Seattle (tried 10 different servers) ->  snat router (2 different routers; old kernels and new kernels) -> amazonS3 =  doesn't work ~10% of the time
Web Server Seattle  ->  snat router -> any other website =  works
Router box -> amazonS3 =  works
Web Server Victoria -> snat router Victoria -> amazonS3 = works


Things I've tried:
Disabling window scaling
Lowering the window size
upgrading the router to the newest kernel
Disabling the firewall entirely

In the failure case Amazon (or something along the way) sends us a 868 length packet instead of the expect 1500 byte packet.  The server upon seeing the packets replies with a RST packet (error has occurred) and the connection is torn down.

Any help would be greatly appreciated!


Below are two TCP dumps- 


------------ Begin Error Case TCP Dump -------------
tcpdump -r /tmp/s3doesntwork-died 
reading from file /tmp/s3doesntwork-died, link-type EN10MB (Ethernet)
19:58:42.542189 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: S 193799772:193799772(0) win 5840 <mss 1460,sackOK,timestamp 760821159 0,nop,wscale 5>
19:58:42.544115 IP 189-81.amazon.com.www > locum.sparklit.com.39491: S 3148664267:3148664267(0) ack 193799773 win 8190 <mss 1460,nop,wscale 6,nop,nop,sackOK>
19:58:42.545176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1 win 183
19:58:42.545184 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: P 1:212(211) ack 1 win 183
19:58:42.548113 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . ack 212 win 916
19:58:42.558108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1:1461(1460) ack 212 win 916
19:58:42.558117 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1461:2921(1460) ack 212 win 916
19:58:42.558123 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 2921:4381(1460) ack 212 win 916
19:58:42.558128 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 4381:5841(1460) ack 212 win 916
19:58:42.559108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 5841:7301(1460) ack 212 win 916
19:58:42.559118 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 7301:8129(828) ack 212 win 916
19:58:42.559138 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: R 193799984:193799984(0) win 0
19:58:42.559169 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1461 win 274
19:58:42.559176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 2921 win 365
19:58:42.559180 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 4381 win 457
19:58:42.559188 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 5841 win 548
19:58:42.560167 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 7301 win 639
19:58:45.308618 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: F 212:212(0) ack 7301 win 639
19:58:45.310512 IP 189-81.amazon.com.www > locum.sparklit.com.39491: R 3148671568:3148671568(0) win 8201

Weird TCP issue with amazon S3
===

I am having a weird TCP issue that makes no sense at all.  I am tearing my hair out.

When downloading a file from amazon s3 (and only amazon s3- every other site works fine) some percentage of the time the connections dies.   This only happens from our servers in Seattle with a Web Server behind a nat router.  Connecting directly from the router works fine. From our servers here in Victoria everything works fine and we can't replicate this at all.


Web Server Seattle (tried 10 different servers) ->  snat router (2 different routers; old kernels and new kernels) -> amazonS3 =  doesn't work ~10% of the time
Web Server Seattle  ->  snat router -> any other website =  works
Router box -> amazonS3 =  works
Web Server Victoria -> snat router Victoria -> amazonS3 = works


Things I've tried:
Disabling window scaling
Lowering the window size
upgrading the router to the newest kernel
Disabling the firewall entirely

In the failure case Amazon (or something along the way) sends us a 868 length packet instead of the expect 1500 byte packet.  The server upon seeing the packets replies with a RST packet (error has occurred) and the connection is torn down.

Any help would be greatly appreciated!


Below are two TCP dumps- 


------------ Begin Error Case TCP Dump -------------
tcpdump -r /tmp/s3doesntwork-died 
reading from file /tmp/s3doesntwork-died, link-type EN10MB (Ethernet)
19:58:42.542189 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: S 193799772:193799772(0) win 5840 <mss 1460,sackOK,timestamp 760821159 0,nop,wscale 5>
19:58:42.544115 IP 189-81.amazon.com.www > locum.sparklit.com.39491: S 3148664267:3148664267(0) ack 193799773 win 8190 <mss 1460,nop,wscale 6,nop,nop,sackOK>
19:58:42.545176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1 win 183
19:58:42.545184 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: P 1:212(211) ack 1 win 183
19:58:42.548113 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . ack 212 win 916
19:58:42.558108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1:1461(1460) ack 212 win 916
19:58:42.558117 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1461:2921(1460) ack 212 win 916
19:58:42.558123 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 2921:4381(1460) ack 212 win 916
19:58:42.558128 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 4381:5841(1460) ack 212 win 916
19:58:42.559108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 5841:7301(1460) ack 212 win 916
19:58:42.559118 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 7301:8129(828) ack 212 win 916
19:58:42.559138 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: R 193799984:193799984(0) win 0
19:58:42.559169 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1461 win 274
19:58:42.559176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 2921 win 365
19:58:42.559180 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 4381 win 457
19:58:42.559188 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 5841 win 548
19:58:42.560167 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 7301 win 639
19:58:45.308618 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: F 212:212(0) ack 7301 win 639
19:58:45.310512 IP 189-81.amazon.com.www > locum.sparklit.com.39491: R 3148671568:3148671568(0) win 8201
