title: What could delay the handshake between a Jenkins server and an SSL SVN server (neither has internet access)?
body: <p>We have a VM (in a VMWare cluster) running Jenkins (1.477, 1.480.3, and 1.508) to do builds for commits to our SVN repository (Collabnet SVN 1.7.5-3150.92).  The repository is accessed via an SSL connection.  For security reasons neither computer (the build server or the SVN server) have access to the internet.  When a Jenkins build begins the start of the SVN update the console for the job pauses on Updating "<a href="https://vcfs01.redacted-address.com/svn/MTCM/Trunk" rel="nofollow noreferrer">https://vcfs01.redacted-address.com/svn/MTCM/Trunk</a>" for 30 - 90 seconds.  Once the update begins it is fairly fast.</p>
  
  <p>To rule Jenkins out as the culprit, I reproduced the same issue by doing a check-out from the build server using TortoiseSVN.  The same delay occurs with Tortoise, and once files begin to transfer the transfer rate ranges from 50 - 70 KB/s (which is great).  </p>
  
  <p>We use Kaspersky, and have ruled it out as an issue, because the problem does not occur on programmers PCs that have Kaspersky.  We also tried excluding both servers just to be %100 sure.  </p>
  
  <p>For awhile I was convinced this was an issue with certificate revocation checking because I saw with WireShark an attempted HTTP GET from <a href="http://ctldl.windowsupdate.com/msdownload/update/v3/static/trustedr/en/disallowedcertstl.cab?dca976bb02bdc2e3" rel="nofollow noreferrer">http://ctldl.windowsupdate.com/msdownload/update/v3/static/trustedr/en/disallowedcertstl.cab?dca976bb02bdc2e3</a> .  Using the steps in <a href="http://support.microsoft.com/kb/2677070" rel="nofollow noreferrer">this KB article</a> I disabled certificate revocation checks on both the the Jenkins server and the SVN server (though I doubt the latter mattered).  Once I made this change I no longer so attempted connections to the windowsupdate server but instead saw an HTTP GET from <a href="http://crl.globalsign.com/gs/gsorganizationvalg2.crl" rel="nofollow noreferrer">http://crl.globalsign.com/gs/gsorganizationvalg2.crl</a> .   I stumbled across <a href="http://www.page-house.com/blog/2009/04/how-to-disable-crl-checking.html" rel="nofollow noreferrer">this article on disabling CRL checking</a>.  I followed the steps there for both server and no longer see HTTP GETs to foreign (internet) addresses.  </p>
  
  <p>When the Jenkins server has access to the internet the handshake takes ~5 seconds in Tortoise (as opposed to ~90 seconds when the firewall prevents access).  Despite the fast handshake for Tortoise the speed for Jenkins is the same as when the firewall was in place!  </p>
  
  <p>I did some research on Jenkins (I also updated Jenkins from version 1.477 to 1.508) and found <a href="http://wiki.svnkit.com/SVNKit_FAQ#Q:_I.27m_encountering_performance_issues_refreshing_a_working_copy_accessed_through_a_symbolic_link." rel="nofollow noreferrer">an article about SVNKit having problems with symbolic links</a>.  As far as I can tell, no symbolic links are being used.</p>
  
  <p>What I am seeing with WireShark is that there is some initial activity (creation of an encrypted connection) between the Jenkins server and SVN server.  After the initial activity ~30 seconds passes and then there is more activity (application data is sent).  After the application data there is another ~30 second delay, and then more application data is sent, the encrypted connection is reset, and the update begins. </p>
  
  <p>I talked to the networking group about what @Chris and @Barmar wrote and the networking group said:</p>
  
  <blockquote>
    <p>Our DNS servers already have a reverse 168.192 look up zone and it is populated with
    quite a few servers. I have rarely ever had to do anything with these zones with 
    the exception of searching for old rogue entries of internal servers.</p>
  </blockquote>
  
  <p>I assume that means it's not a lookup issue, but I'm over my head here.  Here is a filtered capture between the Jenkins machine (172.25.2.106) and the SVN server (172.25.2.106) that shows the pauses between packet transmission:</p>
  
  <p><img src="https://i.stack.imgur.com/HdkLj.jpg" alt="WireShark Trace"></p>
  
  <p>Both of these are Win2K8 R2 Datacenter VMware machines.  According to our networking group the DNS entries / lookups for these servers are configured and working correctly.</p>