<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have searched the <a href="https://github.com/apache/dubbo/issues">issues</a> of this repository and believe that this is not a duplicate.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/apache/dubbo/blob/master/FAQ.md">FAQ</a> of this repository and believe that this is not a duplicate.</li>
</ul>
<h3 dir="auto">Environment</h3>
<ul dir="auto">
<li>Dubbo version: 2.6.2</li>
<li>Operating System version: Linux</li>
<li>Java version: 1.8</li>
</ul>
<h3 dir="auto">Steps to reproduce this issue</h3>
<ol dir="auto">
<li>Restart dubbo service.</li>
<li>Ping hostname becomes slow.</li>
</ol>
<h3 dir="auto">Expected Result</h3>
<p dir="auto">Start dubbo service quickly.</p>
<h3 dir="auto">Actual Result</h3>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/15684156/61106851-f4d04500-a4b0-11e9-898a-4c55a78f7654.png"><img src="https://user-images.githubusercontent.com/15684156/61106851-f4d04500-a4b0-11e9-898a-4c55a78f7654.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">Dubbo service startup is slow, it takes two hours.</p>
<p dir="auto">Thread dump like this:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/15684156/61106974-4678cf80-a4b1-11e9-9c03-67a9f39f5d99.png"><img src="https://user-images.githubusercontent.com/15684156/61106974-4678cf80-a4b1-11e9-9c03-67a9f39f5d99.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a href="https://crossoverjie.top/2019/07/05/troubleshoot/dubbo-start-slow/" rel="nofollow">More details</a></p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have searched the <a href="https://github.com/apache/incubator-dubbo/issues">issues</a> of this repository and believe that this is not a duplicate.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/apache/incubator-dubbo/blob/master/FAQ.md">FAQ</a> of this repository and believe that this is not a duplicate.</li>
</ul>
<h3 dir="auto">Environment</h3>
<ul dir="auto">
<li>Dubbo version: 2.6.3</li>
<li>Operating System version: Centos 7</li>
<li>Java version: 1.8</li>
</ul>
<h3 dir="auto">Steps to reproduce this issue</h3>
<p dir="auto">For some time we found some performance jitter under some high loads points. Generally like the chart below (Figure 1) showing the latency on provider side didn't change much but latency on consumer side had a big deterioration.</p>
<blockquote>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/17573073/50273714-575ee680-0476-11e9-8559-bd54b4cbf08d.png"><img width="982" alt="Figure 1" src="https://user-images.githubusercontent.com/17573073/50273714-575ee680-0476-11e9-8559-bd54b4cbf08d.png" style="max-width: 100%;"></a>
Figure 1
</blockquote>
<p dir="auto">The count of providers in the figure 1 is 3 in three nodes. Each nodes has 4 cores and 16G.</p>
<p dir="auto">We have tried:</p>
<ul dir="auto">
<li>Check the nat gateway band width and connections monitor<br>
30MB/s max and less then 3000 connections. They don't hit the limit.</li>
<li>Try to separate to a new connection for one service to see whether the problem is in ONE connection model.<br>
The chart is capture after that and issue continues.</li>
<li>Check the tomcat thread pool usage<br>
Though i think it's irrelevant to the issue but i have integrated Sentinel to make sure the tomcat thread pool will not be full.</li>
<li>Doubt if it's a vpc's performance problem (They are deployed in Aliyun.com) but we found similar issue in our IDC cluster as Figure 2.<br>
6 Providers deployed.</li>
</ul>
<blockquote>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/17573073/50274475-516a0500-0478-11e9-8bb4-4b54929fc1c7.png"><img width="976" alt="screen shot 2018-12-20 at 16 57 09" src="https://user-images.githubusercontent.com/17573073/50274475-516a0500-0478-11e9-8bb4-4b54929fc1c7.png" style="max-width: 100%;"></a>
Figure 2
</blockquote>
<p dir="auto">Now we are continue figuring out what the problem is. Did anyone hit this issue and kindly gives some advices?</p>
<p dir="auto">Thanks</p>