title: Azure Service Fabric InvokeWithRetryAsync huge overhead
body: <p>I'm currently working on a Service Fabric microservice which needs to have a high throughput.</p>
  
  <p>I wondered why I'm not able to achieve more than 500 1KB messages per second on my workstation using loopback.</p>
  
  <p>I removed all the business logic and attached a performance profiler, just to measure end to end performance.</p>
  
  <p>It seems that ~96% of the time is spent resolving the Client and only ~2% doing the actual Http requests.</p>
  
  <p>I'm invoking "Send" in a tight loop for the test:</p>
  
  <pre><code>private HttpCommunicationClientFactory factory = new HttpCommunicationClientFactory();
  
  public async Task Send()
  {
      var client = new ServicePartitionClient&lt;HttpCommunicationClient&gt;(
           factory,
           new Uri("fabric:/MyApp/MyService"));
  
      await client.InvokeWithRetryAsync(c =&gt; c.HttpClient.GetAsync(c.Url + "/test"));
  }
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/m33Y4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m33Y4.png" alt="Profiler output"></a></p>
  
  <p>Any ideas on this? According to the documentation the way I call the Services seems to be Service Fabric best practice.</p>
  
  <p><strong>UPDATE</strong>: Caching the ServicePartioningClient does improve the Performance, but using partioned services, I'm unable to cache the client, since I don't know the partition for a give PartitionKey.</p>
  
  <p><strong>UPDATE 2</strong>: I'm sorry that I didn't include full details in my initial question.
  We noticed the huge overhead of InvokeWithRetry when initially implementing a socket based communication.</p>
  
  <p>You won't notice it that much if you are using http requests. A http request already takes ~1ms, so adding 0.5ms for the InvokeWithRetry isn't that noticable.</p>
  
  <p>But if you use raw sockets which takes in our case ~ 0.005ms adding 0.5ms overhead for the InvokeWithRetry is immense!</p>
  
  <p>Here is an http example, with InvokeAndRetry it takes 3x as long:</p>
  
  <pre><code>public async Task RunTest()
  {
      var factory = new HttpCommunicationClientFactory();
      var uri = new Uri("fabric:/MyApp/MyService");
      var count = 10000;
  
      // Example 1: ~6000ms
      for (var i = 0; i &lt; count; i++)
      {
          var pClient1 = new ServicePartitionClient&lt;HttpCommunicationClient&gt;(factory, uri, new ServicePartitionKey(1));
          await pClient1.InvokeWithRetryAsync(c =&gt; c.HttpClient.GetAsync(c.Url));
      }
  
      // Example 2: ~1800ms
      var pClient2 = new ServicePartitionClient&lt;HttpCommunicationClient&gt;(factory, uri, new ServicePartitionKey(1));
      HttpCommunicationClient resolvedClient = null;
      await pClient2.InvokeWithRetryAsync(
          c =&gt;
          {
              resolvedClient = c;
              return Task.FromResult(true);
          });
  
      for (var i = 0; i &lt; count; i++)
      {
          await resolvedClient.HttpClient.GetAsync(resolvedClient.Url);
      }
  }
  </code></pre>
  
  <p>I'm aware that InvokeWithRetry adds some nice stuff I don't want to miss from the clients. But does it need to resolve the partitions on every call?</p>