<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have searched the <a href="https://github.com/apache/dubbo/issues">issues</a> of this repository and believe that this is not a duplicate.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/apache/dubbo/blob/master/FAQ.md">FAQ</a> of this repository and believe that this is not a duplicate.</li>
</ul>
<h3 dir="auto">Environment</h3>
<ul dir="auto">
<li>Dubbo version: 2.5.3</li>
<li>Operating System version: CentOS Linux release 7.6.1810</li>
<li>Java version: 1.8</li>
<li>zkclient version is 0.1</li>
<li>zk version is 3.4.6</li>
</ul>
<h3 dir="auto">Steps to reproduce this issue</h3>
<ol dir="auto">
<li>在一次流量突增时，dubbo的用户线程池被打满</li>
</ol>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[2020-11-19 10:12:56.558 WARN ] [New I/O worker #12] (AbortPolicyWithReport:52) [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-127.0.0.1:28080, Pool Size: 500 (active: 500, core: 0, max: 500, largest: 500), Task: 161962029 (completed: 161961529), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://127.0.0.1:28080!, dubbo version: 2.5.3"><pre class="notranslate">[<span class="pl-c1">2020</span>-<span class="pl-c1">11</span>-<span class="pl-c1">19</span> <span class="pl-c1">10</span>:<span class="pl-c1">12</span>:<span class="pl-c1">56.558</span> <span class="pl-c1">WARN</span> ] [<span class="pl-smi">New</span> <span class="pl-s1">I</span>/<span class="pl-smi">O</span> <span class="pl-s1">worker</span> #<span class="pl-c1">12</span>] (<span class="pl-s1">AbortPolicyWithReport</span>:<span class="pl-c1">52</span>) [<span class="pl-c1">DUBBO</span>] <span class="pl-s1">Thread</span> <span class="pl-smi">pool</span> <span class="pl-s1">is</span> <span class="pl-c1">EXHAUSTED</span>! <span class="pl-smi">Thread</span> <span class="pl-s1">Name</span>: <span class="pl-s1">DubboServerHandler</span>-<span class="pl-c1">127.0</span>.<span class="pl-c1">0.1</span>:<span class="pl-c1">28080</span>, <span class="pl-s1">Pool</span> <span class="pl-s1">Size</span>: <span class="pl-c1">500</span> (<span class="pl-s1">active</span>: <span class="pl-c1">500</span>, <span class="pl-s1">core</span>: <span class="pl-c1">0</span>, <span class="pl-s1">max</span>: <span class="pl-c1">500</span>, <span class="pl-s1">largest</span>: <span class="pl-c1">500</span>), <span class="pl-s1">Task</span>: <span class="pl-c1">161962029</span> (<span class="pl-s1">completed</span>: <span class="pl-c1">161961529</span>), <span class="pl-smi">Executor</span> <span class="pl-s1">status</span>:(<span class="pl-s1">isShutdown</span>:<span class="pl-c1">false</span>, <span class="pl-s1">isTerminated</span>:<span class="pl-c1">false</span>, <span class="pl-s1">isTerminating</span>:<span class="pl-c1">false</span>), <span class="pl-smi">in</span> <span class="pl-s1">dubbo</span>:<span class="pl-c">//127.0.0.1:28080!, dubbo version: 2.5.3</span></pre></div>
<p dir="auto">随后Tomcat中发现有大量的 timedwaiting 状态的线程</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/13230226/99629455-fc273b80-2a72-11eb-85c7-89f52e8f419e.png"><img src="https://user-images.githubusercontent.com/13230226/99629455-fc273b80-2a72-11eb-85c7-89f52e8f419e.png" alt="image" style="max-width: 100%;"></a></p>
<ol start="2" dir="auto">
<li>在查看jstack后，有几千个名为 DubboRegistryFailedRetryTimer-thread-1 的线程，这个重连线程理论上应该只有一个，在启动注册中心时创建，但实际上被创建了几千个</li>
</ol>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231399 daemon prio=5 os_prio=0 tid=0x0000000001ddf000 nid=0x339f waiting on condition [0x00007f4ef5786000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f8e0250&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231397 daemon prio=5 os_prio=0 tid=0x0000000001ddd000 nid=0x339d waiting on condition [0x00007f4ef5887000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f8e4250&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231395 daemon prio=5 os_prio=0 tid=0x0000000001ddb000 nid=0x339b waiting on condition [0x00007f4ef5988000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f8f3358&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231393 daemon prio=5 os_prio=0 tid=0x0000000001dd9000 nid=0x3399 waiting on condition [0x00007f4ef5a89000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f902ae0&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231391 daemon prio=5 os_prio=0 tid=0x0000000001dd7000 nid=0x3397 waiting on condition [0x00007f4ef5b8a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f90d5f0&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231389 daemon prio=5 os_prio=0 tid=0x0000000001dd5000 nid=0x3395 waiting on condition [0x00007f4ef5c8b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f90f5f0&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

&quot;DubboRegistryFailedRetryTimer-thread-1&quot; #1231387 daemon prio=5 os_prio=0 tid=0x0000000001dd3000 nid=0x3393 waiting on condition [0x00007f4ef5d8c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  &lt;0x000000072f917140&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)"><pre class="notranslate"><span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231399</span> <span class="pl-smi">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001ddf000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x339f</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5786000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f8e0250</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231397</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001ddd000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x339d</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5887000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f8e4250</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231395</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001ddb000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x339b</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5988000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f8f3358</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231393</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001dd9000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x3399</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5a89000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f902ae0</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231391</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001dd7000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x3397</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5b8a000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f90d5f0</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231389</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001dd5000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x3395</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5c8b000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f90f5f0</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)

   <span class="pl-s1">Locked</span> <span class="pl-smi">ownable</span> <span class="pl-s1">synchronizers</span>:
	- <span class="pl-s1">None</span>

<span class="pl-s">"DubboRegistryFailedRetryTimer-thread-1"</span> #<span class="pl-c1">1231387</span> <span class="pl-s1">daemon</span> <span class="pl-s1">prio</span>=<span class="pl-c1">5</span> <span class="pl-s1">os_prio</span>=<span class="pl-c1">0</span> <span class="pl-s1">tid</span>=<span class="pl-c1">0x0000000001dd3000</span> <span class="pl-s1">nid</span>=<span class="pl-c1">0x3393</span> <span class="pl-s1">waiting</span> <span class="pl-smi">on</span> <span class="pl-s1">condition</span> [<span class="pl-c1">0x00007f4ef5d8c000</span>]
   <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">Thread</span>.<span class="pl-smi">State</span>: <span class="pl-c1">TIMED_WAITING</span> (<span class="pl-smi">parking</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">misc</span>.<span class="pl-s1">Unsafe</span>.<span class="pl-en">park</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
	- <span class="pl-smi">parking</span> <span class="pl-s1">to</span> <span class="pl-s1">wait</span> <span class="pl-k">for</span>  &lt;<span class="pl-c1">0x000000072f917140</span>&gt; (<span class="pl-smi">a</span> <span class="pl-s1">java</span>.<span class="pl-smi"></span><span class="pl-smi">util</span>.<span class="pl-smi">concurrent</span>.<span class="pl-smi">locks</span>.<span class="pl-smi">AbstractQueuedSynchronizer$ConditionObject</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">LockSupport</span>.<span class="pl-s1">parkNanos</span>(<span class="pl-smi">LockSupport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">215</span>)
	<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">locks</span>.<span class="pl-s1">AbstractQueuedSynchronizer$ConditionObject</span>.<span class="pl-en">awaitNanos</span>(<span class="pl-smi">AbstractQueuedSynchronizer</span>.<span class="pl-smi">java</span>:<span class="pl-c1">2078</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1093</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ScheduledThreadPoolExecutor$DelayedWorkQueue</span>.<span class="pl-en">take</span>(<span class="pl-smi">ScheduledThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">809</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">getTask</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1074</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor</span>.<span class="pl-en">runWorker</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1134</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">util</span>.<span class="pl-s1">concurrent</span>.<span class="pl-s1">ThreadPoolExecutor$Worker</span>.<span class="pl-en">run</span>(<span class="pl-smi">ThreadPoolExecutor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">624</span>)
	<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">Thread</span>.<span class="pl-en">run</span>(<span class="pl-smi">Thread</span>.<span class="pl-smi">java</span>:<span class="pl-c1">748</span>)</pre></div>
<ol start="3" dir="auto">
<li>而且日志有大量重新加载本地缓存文件的输出</li>
</ol>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(AbstractRegistry:232) [DUBBO] Load registry store file xxx.properties"><pre class="notranslate">(<span class="pl-s1">AbstractRegistry</span>:<span class="pl-c1">232</span>) [<span class="pl-c1">DUBBO</span>] <span class="pl-s1">Load</span> <span class="pl-smi">registry</span> <span class="pl-s1">store</span> <span class="pl-smi">file</span> <span class="pl-s1">xxx</span>.<span class="pl-s1">properties</span></pre></div>
<ol start="4" dir="auto">
<li>我们用的注册中心是zk， 因此我怀疑 com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry 对象在某个情况下被创建了多次，每次创建时都会启动 com.alibaba.dubbo.registry.support.FailbackRegistry 中的 retryExecutor 线程池</li>
</ol>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="private final ScheduledExecutorService retryExecutor = Executors.newScheduledThreadPool(1, new NamedThreadFactory(&quot;DubboRegistryFailedRetryTimer&quot;, true));
public FailbackRegistry(URL url) {
        super(url);
        int retryPeriod = url.getParameter(Constants.REGISTRY_RETRY_PERIOD_KEY, Constants.DEFAULT_REGISTRY_RETRY_PERIOD);
        this.retryFuture = retryExecutor.scheduleWithFixedDelay(new Runnable() {
            public void run() {
                // 检测并连接注册中心
                try {
                    retry();
                } catch (Throwable t) { // 防御性容错
                    logger.error(&quot;Unexpected error occur at failed retry, cause: &quot; + t.getMessage(), t);
                }
            }
        }, retryPeriod, retryPeriod, TimeUnit.MILLISECONDS);
    }"><pre class="notranslate"><span class="pl-k">private</span> <span class="pl-k">final</span> <span class="pl-smi">ScheduledExecutorService</span> <span class="pl-s1">retryExecutor</span> = <span class="pl-smi">Executors</span>.<span class="pl-en">newScheduledThreadPool</span>(<span class="pl-c1">1</span>, <span class="pl-k">new</span> <span class="pl-smi">NamedThreadFactory</span>(<span class="pl-s">"DubboRegistryFailedRetryTimer"</span>, <span class="pl-c1">true</span>));
<span class="pl-k">public</span> <span class="pl-s1">FailbackRegistry</span>(<span class="pl-smi">URL</span> <span class="pl-s1">url</span>) {
        <span class="pl-en">super</span>(<span class="pl-s1">url</span>);
        <span class="pl-smi">int</span> <span class="pl-s1">retryPeriod</span> = <span class="pl-s1">url</span>.<span class="pl-en">getParameter</span>(<span class="pl-smi">Constants</span>.<span class="pl-c1">REGISTRY_RETRY_PERIOD_KEY</span>, <span class="pl-smi">Constants</span>.<span class="pl-c1">DEFAULT_REGISTRY_RETRY_PERIOD</span>);
        <span class="pl-smi">this</span>.<span class="pl-s1">retryFuture</span> = <span class="pl-s1">retryExecutor</span>.<span class="pl-en">scheduleWithFixedDelay</span>(<span class="pl-k">new</span> <span class="pl-smi">Runnable</span>() {
            <span class="pl-k">public</span> <span class="pl-smi">void</span> <span class="pl-en">run</span>() {
                <span class="pl-c">// 检测并连接注册中心</span>
                <span class="pl-k">try</span> {
                    <span class="pl-en">retry</span>();
                } <span class="pl-k">catch</span> (<span class="pl-smi">Throwable</span> <span class="pl-s1">t</span>) { <span class="pl-c">// 防御性容错</span>
                    <span class="pl-s1">logger</span>.<span class="pl-en">error</span>(<span class="pl-s">"Unexpected error occur at failed retry, cause: "</span> + <span class="pl-s1">t</span>.<span class="pl-en">getMessage</span>(), <span class="pl-s1">t</span>);
                }
            }
        }, <span class="pl-s1">retryPeriod</span>, <span class="pl-s1">retryPeriod</span>, <span class="pl-smi">TimeUnit</span>.<span class="pl-c1">MILLISECONDS</span>);
    }</pre></div>
<ol start="5" dir="auto">
<li>并且 com.alibaba.dubbo.registry.support.AbstractRegistry 类会重新 loadProperties，<strong>所以有大量的Load registry 日志输出</strong></li>
</ol>
<blockquote>
<p dir="auto">(AbstractRegistry:232) [DUBBO] Load registry store file xxx.properties log output</p>
</blockquote>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="public AbstractRegistry(URL url) {
        setUrl(url);
        // 启动文件保存定时器
        syncSaveFile = url.getParameter(Constants.REGISTRY_FILESAVE_SYNC_KEY, false);
        String filename = url.getParameter(Constants.FILE_KEY, System.getProperty(&quot;user.home&quot;) + &quot;/.dubbo/dubbo-registry-&quot; + url.getHost() + &quot;.cache&quot;);
        File file = null;
        if (ConfigUtils.isNotEmpty(filename)) {
            file = new File(filename);
            if(! file.exists() &amp;&amp; file.getParentFile() != null &amp;&amp; ! file.getParentFile().exists()){
                if(! file.getParentFile().mkdirs()){
                    throw new IllegalArgumentException(&quot;Invalid registry store file &quot; + file + &quot;, cause: Failed to create directory &quot; + file.getParentFile() + &quot;!&quot;);
                }
            }
        }
        this.file = file;
        loadProperties();
        notify(url.getBackupUrls());
    }
private void loadProperties() {
        if (file != null &amp;&amp; file.exists()) {
            InputStream in = null;
            try {
                in = new FileInputStream(file);
                properties.load(in);
                if (logger.isInfoEnabled()) {
                    logger.info(&quot;Load registry store file &quot; + file + &quot;, data: &quot; + properties);
                }
            } catch (Throwable e) {
                logger.warn(&quot;Failed to load registry store file &quot; + file, e);
            } finally {
                if (in != null) {
                    try {
                        in.close();
                    } catch (IOException e) {
                        logger.warn(e.getMessage(), e);
                    }
                }
            }
        }
    }"><pre class="notranslate"><span class="pl-k">public</span> <span class="pl-s1">AbstractRegistry</span>(<span class="pl-smi">URL</span> <span class="pl-s1">url</span>) {
        <span class="pl-en">setUrl</span>(<span class="pl-s1">url</span>);
        <span class="pl-c">// 启动文件保存定时器</span>
        <span class="pl-s1">syncSaveFile</span> = <span class="pl-s1">url</span>.<span class="pl-en">getParameter</span>(<span class="pl-smi">Constants</span>.<span class="pl-c1">REGISTRY_FILESAVE_SYNC_KEY</span>, <span class="pl-c1">false</span>);
        <span class="pl-smi">String</span> <span class="pl-s1">filename</span> = <span class="pl-s1">url</span>.<span class="pl-en">getParameter</span>(<span class="pl-smi">Constants</span>.<span class="pl-c1">FILE_KEY</span>, <span class="pl-smi">System</span>.<span class="pl-en">getProperty</span>(<span class="pl-s">"user.home"</span>) + <span class="pl-s">"/.dubbo/dubbo-registry-"</span> + <span class="pl-s1">url</span>.<span class="pl-en">getHost</span>() + <span class="pl-s">".cache"</span>);
        <span class="pl-smi">File</span> <span class="pl-s1">file</span> = <span class="pl-c1">null</span>;
        <span class="pl-k">if</span> (<span class="pl-smi">ConfigUtils</span>.<span class="pl-en">isNotEmpty</span>(<span class="pl-s1">filename</span>)) {
            <span class="pl-s1">file</span> = <span class="pl-k">new</span> <span class="pl-smi">File</span>(<span class="pl-s1">filename</span>);
            <span class="pl-k">if</span>(! <span class="pl-s1">file</span>.<span class="pl-en">exists</span>() &amp;&amp; <span class="pl-s1">file</span>.<span class="pl-en">getParentFile</span>() != <span class="pl-c1">null</span> &amp;&amp; ! <span class="pl-s1">file</span>.<span class="pl-en">getParentFile</span>().<span class="pl-en">exists</span>()){
                <span class="pl-k">if</span>(! <span class="pl-s1">file</span>.<span class="pl-en">getParentFile</span>().<span class="pl-en">mkdirs</span>()){
                    <span class="pl-k">throw</span> <span class="pl-k">new</span> <span class="pl-smi">IllegalArgumentException</span>(<span class="pl-s">"Invalid registry store file "</span> + <span class="pl-s1">file</span> + <span class="pl-s">", cause: Failed to create directory "</span> + <span class="pl-s1">file</span>.<span class="pl-en">getParentFile</span>() + <span class="pl-s">"!"</span>);
                }
            }
        }
        <span class="pl-smi">this</span>.<span class="pl-s1">file</span> = <span class="pl-s1">file</span>;
        <span class="pl-en">loadProperties</span>();
        <span class="pl-en">notify</span>(<span class="pl-s1">url</span>.<span class="pl-en">getBackupUrls</span>());
    }
<span class="pl-k">private</span> <span class="pl-smi">void</span> <span class="pl-s1">loadProperties</span>() {
        <span class="pl-k">if</span> (<span class="pl-s1">file</span> != <span class="pl-c1">null</span> &amp;&amp; <span class="pl-s1">file</span>.<span class="pl-en">exists</span>()) {
            <span class="pl-smi">InputStream</span> <span class="pl-s1">in</span> = <span class="pl-c1">null</span>;
            <span class="pl-k">try</span> {
                <span class="pl-s1">in</span> = <span class="pl-k">new</span> <span class="pl-smi">FileInputStream</span>(<span class="pl-s1">file</span>);
                <span class="pl-s1">properties</span>.<span class="pl-en">load</span>(<span class="pl-s1">in</span>);
                <span class="pl-k">if</span> (<span class="pl-s1">logger</span>.<span class="pl-en">isInfoEnabled</span>()) {
                    <span class="pl-s1">logger</span>.<span class="pl-en">info</span>(<span class="pl-s">"Load registry store file "</span> + <span class="pl-s1">file</span> + <span class="pl-s">", data: "</span> + <span class="pl-s1">properties</span>);
                }
            } <span class="pl-k">catch</span> (<span class="pl-smi">Throwable</span> <span class="pl-s1">e</span>) {
                <span class="pl-s1">logger</span>.<span class="pl-en">warn</span>(<span class="pl-s">"Failed to load registry store file "</span> + <span class="pl-s1">file</span>, <span class="pl-s1">e</span>);
            } <span class="pl-k">finally</span> {
                <span class="pl-k">if</span> (<span class="pl-s1">in</span> != <span class="pl-c1">null</span>) {
                    <span class="pl-k">try</span> {
                        <span class="pl-s1">in</span>.<span class="pl-en">close</span>();
                    } <span class="pl-k">catch</span> (<span class="pl-smi">IOException</span> <span class="pl-s1">e</span>) {
                        <span class="pl-s1">logger</span>.<span class="pl-en">warn</span>(<span class="pl-s1">e</span>.<span class="pl-en">getMessage</span>(), <span class="pl-s1">e</span>);
                    }
                }
            }
        }
    }</pre></div>
<ol start="6" dir="auto">
<li>我翻阅了源码，从创建ZookeeperRegistry过程到zk断开后重连的recover机制后都没发现和复现这个问题。</li>
</ol>
<h3 dir="auto">Expected Result</h3>
<p dir="auto">在com.alibaba.dubbo.registry.support.FailbackRegistry类中名为<br>
DubboRegistryFailedRetryTimer这个线程应该在单注册中心的情况下应该是单例的，只应该被创建一个</p>
<h3 dir="auto">Actual Result</h3>
<p dir="auto">在com.alibaba.dubbo.registry.support.FailbackRegistry类中名为<br>
DubboRegistryFailedRetryTimer的线程被创建了几千个，产生了大量的timewaited线程。</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have searched the <a href="https://github.com/apache/dubbo/issues">issues</a> of this repository and believe that this is not a duplicate.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/apache/dubbo/blob/master/FAQ.md">FAQ</a> of this repository and believe that this is not a duplicate.</li>
</ul>
<h3 dir="auto">Environment</h3>
<ul dir="auto">
<li>Dubbo version: 2.7.5</li>
<li>Operating System version: linux</li>
<li>Java version: 1.8</li>
</ul>
<h3 dir="auto">Steps to reproduce this issue</h3>
<ol dir="auto">
<li>在2.7.5版本中的telnet已经取消了部分指令了吗？</li>
<li>ls、invoke、pwd指令没有出现。</li>
<li>只有help、status指令。</li>
</ol>
<p dir="auto">Pls. provide [GitHub address] to reproduce this issue.</p>
<h3 dir="auto">Expected Result</h3>
<p dir="auto">What do you expected from the above steps？</p>
<h3 dir="auto">Actual Result</h3>
<p dir="auto">What actually happens?</p>
<p dir="auto">If there is an exception, please attach the exception trace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="D:\projects\storevm\dubbo-demo\dubbo-provider&gt;telnet localhost 9085
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

dubbo&gt;help
Please input &quot;help [command]&quot; show detail.
 status [-l]    - Show status.
 clear [lines]  - Clear screen.
 log level      - Change log level or show log
 help [command] - Show help.
 exit           - Exit the telnet.
dubbo&gt;"><pre class="notranslate"><code class="notranslate">D:\projects\storevm\dubbo-demo\dubbo-provider&gt;telnet localhost 9085
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

dubbo&gt;help
Please input "help [command]" show detail.
 status [-l]    - Show status.
 clear [lines]  - Clear screen.
 log level      - Change log level or show log
 help [command] - Show help.
 exit           - Exit the telnet.
dubbo&gt;
</code></pre></div>