<p dir="auto">Recently, we keep receiving issues from the community reporting duplicate URL addresses in Registry, mostly zookeeper, after upgrading to 2.7.1. Those URLs are all pushed to the Consumer and because only one of the duplicated URLs is valid, it causes the consumer to fail when the consumer chooses the invalid one during load balance.</p>
<p dir="auto">The following are some typical issues: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="427652986" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3785" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3785/hovercard" href="https://github.com/apache/dubbo/issues/3785">#3785</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="426764906" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3770" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3770/hovercard" href="https://github.com/apache/dubbo/issues/3770">#3770</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="436470099" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3920" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3920/hovercard" href="https://github.com/apache/dubbo/issues/3920">#3920</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="442501640" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/4013" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/4013/hovercard" href="https://github.com/apache/dubbo/issues/4013">#4013</a></p>
<p dir="auto">After troubleshooting, we found the reason for the problem: in version 2.7.1, the URL addresses will be registered as a persistent node in zookeeper, which is different from the behavior in 2.7.0 or before, in which they are registered as ephemeral nodes. The meaning of the ephemeral node is that it will get removed from zookeeper when the server process went for a while. Now, after changed to persistent, the node cannot be removed in some exceptional scenarios, such as process crashed, process forced killed with kill -9, etc, eventually leading to the appearance of dirty data.</p>
<p dir="auto">Advices to solve or avoid the problem:</p>
<ol dir="auto">
<li>Never forcibly terminating the process to ensure that Dubbo enters it's elegant shutdown process.</li>
<li>Wait for 2.7.2 to <a href="https://github.com/apache/dubbo/pull/3824" data-hovercard-type="pull_request" data-hovercard-url="/apache/dubbo/pull/3824/hovercard">fix this issue</a>, going to be released in early June.</li>
<li>For dirty data that has already been generated, consider manual or ZK scripts to clean up, but be careful of the deletion criteria.</li>
<li>Add the following configuration for 2.7.1: &lt;dubbo:provider dynamic="true"/&gt; or &lt;dubbo:service dynamic="true"/&gt;</li>
</ol>
<p dir="auto">近期，关于升级到2.7.1版本后注册中心（多数是zookeeper）出现重复URL地址数据无法删除的情况，我们持续收到来自社区的issue报告。以下issue是几个典型的异常现象：<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="427652986" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3785" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3785/hovercard" href="https://github.com/apache/dubbo/issues/3785">#3785</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="426764906" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3770" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3770/hovercard" href="https://github.com/apache/dubbo/issues/3770">#3770</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="436470099" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/3920" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/3920/hovercard" href="https://github.com/apache/dubbo/issues/3920">#3920</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="442501640" data-permission-text="Title is private" data-url="https://github.com/apache/dubbo/issues/4013" data-hovercard-type="issue" data-hovercard-url="/apache/dubbo/issues/4013/hovercard" href="https://github.com/apache/dubbo/issues/4013">#4013</a></p>
<p dir="auto">经过问题排查我们定位到了问题的原因：在2.7.1版本中URL地址在zookeeper中被注册为持久persistent节点(2.7.0及之前注册的是临时emperal节点)，这样当server进程异常终止无法进入正常优雅下线流程时，如进程崩溃、强行kill -9等，导致zookeeper已失效持久节点无法清空，最终导致脏数据的出现。</p>
<p dir="auto">解决及规避方法：</p>
<ol dir="auto">
<li>上下线过程避免强行终止进程，保证让Dubbo进入优雅停机流程</li>
<li>官方2.7.2版本将会<a href="https://github.com/apache/dubbo/pull/3824" data-hovercard-type="pull_request" data-hovercard-url="/apache/dubbo/pull/3824/hovercard">修复此问题</a>，预计在6月初发布</li>
<li>对于已经产生的脏数据，考虑手动或ZK脚本予以清理，但此时要注意严格控制删选条件</li>
<li>2.7.1版本增加配置：&lt;dubbo:provider dynamic="true"/&gt; 或&lt;dubbo:service dynamic="true"/&gt;</li>
</ol>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have searched the <a href="https://github.com/apache/incubator-dubbo/issues">issues</a> of this repository and believe that this is not a duplicate.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/apache/incubator-dubbo/blob/master/FAQ.md">FAQ</a> of this repository and believe that this is not a duplicate.</li>
</ul>
<h3 dir="auto">Environment</h3>
<ul dir="auto">
<li>Dubbo version: 2.6.6 -&gt; 2.7.0</li>
<li>Operating System version: mac</li>
<li>Java version: 1.8</li>
</ul>
<h3 dir="auto">Steps to reproduce this issue</h3>
<ol dir="auto">
<li>start provider and consumer with dubbo version 2.6.6, consumer can connect to provider</li>
<li>just change the dubbo version in pom from 2.6.6 to 2.7.0, and then an exception occur</li>
</ol>
<p dir="auto">the provider code is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" @Override
    public String sayHello(String name) {
        System.out.println(&quot;[&quot; + new SimpleDateFormat(&quot;HH:mm:ss&quot;).format(new Date()) + &quot;] Hello &quot; + name + &quot;, request from consumer: &quot; + RpcContext.getContext().getRemoteAddress());
        return &quot;Hello &quot; + name + &quot;, response from provider: &quot; + RpcContext.getContext().getLocalAddress();
    }"><pre class="notranslate"><code class="notranslate"> @Override
    public String sayHello(String name) {
        System.out.println("[" + new SimpleDateFormat("HH:mm:ss").format(new Date()) + "] Hello " + name + ", request from consumer: " + RpcContext.getContext().getRemoteAddress());
        return "Hello " + name + ", response from provider: " + RpcContext.getContext().getLocalAddress();
    }
</code></pre></div>
<ol start="3" dir="auto">
<li></li>
</ol>
<p dir="auto">Pls. provide [GitHub address] to reproduce this issue.</p>
<h3 dir="auto">Expected Result</h3>
<p dir="auto">What do you expected from the above steps？<br>
i can update the dubbo version without change any code.</p>
<h3 dir="auto">Actual Result</h3>
<p dir="auto">an exception occurs, and the exception is bellow:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="26/03/19 06:27:23:023 CST] DubboServerHandler-10.99.62.72:20880-thread-200 ERROR filter.ExceptionFilter:  [DUBBO] Got unchecked and undeclared exception which called by 10.99.62.72. service: com.test.dubbo.DemoService, method: sayHello, exception: java.lang.NoSuchMethodError: com.alibaba.dubbo.rpc.RpcContext.getContext()Lcom/alibaba/dubbo/rpc/RpcContext;, dubbo version: 2.7.0, current host: 10.99.62.72
java.lang.NoSuchMethodError: com.alibaba.dubbo.rpc.RpcContext.getContext()Lcom/alibaba/dubbo/rpc/RpcContext;
	at com.test.dubbo.DemoServiceImpl.sayHello(DemoServiceImpl.java:31)
	at org.apache.dubbo.common.bytecode.Wrapper1.invokeMethod(Wrapper1.java)
	at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
	at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:86)
	at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56)
	at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
	at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:63)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:88)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:79)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:78)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:39)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:103)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:200)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)"><pre class="notranslate"><code class="notranslate">26/03/19 06:27:23:023 CST] DubboServerHandler-10.99.62.72:20880-thread-200 ERROR filter.ExceptionFilter:  [DUBBO] Got unchecked and undeclared exception which called by 10.99.62.72. service: com.test.dubbo.DemoService, method: sayHello, exception: java.lang.NoSuchMethodError: com.alibaba.dubbo.rpc.RpcContext.getContext()Lcom/alibaba/dubbo/rpc/RpcContext;, dubbo version: 2.7.0, current host: 10.99.62.72
java.lang.NoSuchMethodError: com.alibaba.dubbo.rpc.RpcContext.getContext()Lcom/alibaba/dubbo/rpc/RpcContext;
	at com.test.dubbo.DemoServiceImpl.sayHello(DemoServiceImpl.java:31)
	at org.apache.dubbo.common.bytecode.Wrapper1.invokeMethod(Wrapper1.java)
	at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
	at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:86)
	at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56)
	at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
	at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:63)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:88)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:79)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:78)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:39)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:73)
	at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:103)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:200)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
</code></pre></div>