mv /var/lib/etcd/restored/* /var/lib/etcd/ rmdir /var/lib/etcd/restored 

6. The current kubelet pods need to be deleted so that the cache will be consistent between the kubelet and etcd data. Do this on the master and management  nodes: 

mount | grep kubelet | awk '{ system("umount "$3)}' rm -rf /var/lib/kubelet/pods 

7. Start kubelet on the master and management nodes: 

systemctl start kubelet 

8. Enable the etcd pods on all master  nodes: 

mv /etc/cfc/podbackup/* /etc/cfc/pods/ 

9. Allow around 30 seconds for etcd to rebuild its cluster state and verify it is healthy, replacing 172.24.19.201 with the IP address of any master node. See Example 3-15. 

[root@icp-master ~]# docker run --entrypoint=etcdctl -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd ibmcom/etcd:3.2.24 --cert-file=/certs/client.pem --key-file=/certs/client-key.pem --ca-file=/certs/ca.pem --endpoints https://172.24.19.201:4001 cluster-health member 8a2d3ec6df19666f is healthy: got healthy result from https://172.24.19.201:4001 cluster is healthy 

Note that the ibmcom/etcd:3.2.24 image may need to be replaced with the private image registry name for the environment. For example, mycluster.icp:8500/ibmcom/etcd:3.2.24 . 

10.Clear your web browser cache. 

11.The IBM Cloud Private platform may take some time to start up all the pods on the master node and become available for use. Once the dashboard is accessible. verify that the cluster has started recreating the previous state by redeploying workloads from the backed up cluster. You may experience a few pod failures for some time until the system becomes stable again. Logging may be particularly slow as it deals with the sudden influx of log data from newly recreated pods all at once. 

## Restoring etcd on dedicated etcd installations 

To restore etcd on an installation with dedicated etcd nodes, you can use the same steps for single and multi-master installations, but using the dedicated etcd nodes in place of the masters. All etcd data resides on the dedicated etcd nodes so the only steps that apply to the masters in this case is the same steps that apply to the management nodes. 

## Failed etcd restores 

If the etcd snapshot was not restored, it may be possible to recover the initial state of the cluster. A common error on RHEL systems is that systemd becomes unresponsive when Docker is stopped and started. A usual symptom of this is when the docker restart command takes a long time to run, and simple commands that depend on systemd (such as reboot) just hang. In this situation, only a reboot from the hypervisor will fix a hung systemd. 

If the etcd snapshot restore was successful on some nodes and failed on others, you will need to either address the issue on the failed nodes and try again, or revert the successful nodes to the original state by restoring the etcd snapshot taken from the new installation. 

