## Restoring etcd on remote cluster 

To restore etcd in a remote cluster, repeat steps 1 to 9 in the section “Restoring etcd on local cluster” on page 106. Some further steps need to be performed to clean up the old nodes and data from the restored etcd snapshot and apply the working configuration. Perform the following steps: 

1. After restoring etcd, all the cluster nodes from the initial installation are not present in the new Kubernetes cluster, so for each node kubelet and docker should be restarted. You can check which nodes currently exist using kubectl get nodes . Restart kubelet and docker on all  cluster nodes using systemctl stop kubelet && systemctl restart docker && systemctl start kubelet . After this, the new cluster nodes should be visible using kubectl get nodes . 

2. Wait for about 90 seconds after restoring etcd for the restored nodes to change from Ready  to NotReady  state, then remove the NotReady  nodes. Be sure to only  remove the nodes that contain the old cluster IP addresses. If nodes with the current cluster IP addresses are in NotReady  state, you should investigate whether the node is online first. 

a. Get the current list of nodes in the NotReady state. 

kubectl get nodes | grep NotReady | awk '{print $1}' 

b. Remove the old nodes. 

kubectl delete node $(kubectl get nodes | grep NotReady | awk '{print $1}') 

3. Replace the ConfigMap and Secrets restored by etcd with the data exported in the section “Restoring MongoDB” on page 101. Run the script in Example 3-16. 

#!/bin/bash echo "Restoring..." # re-genereate default secret kubectl get secret -n kube-system -o wide | grep "\-token-" | awk '{system("kubectl delete secret "$1 " -n kube-system")}' kubectl get secret -n services -o wide | grep "\-token-" | awk '{system("kubectl delete secret "$1 " -n services")}' kubectl get secret -n istio-system -o wide | grep "\-token-"| awk '{system("kubectl delete secret "$1 " -n istio-system")}' kubectl get secret -n kube-public -o wide | grep "\-token-"| awk '{system("kubectl delete secret "$1 " -n kube-public")}' kubectl get secret -n ibmcom -o wide | grep "\-token-"| awk '{system("kubectl delete secret "$1 " -n ibmcom")}' kubectl get secret -n cert-manager -o wide | grep "\-token-"| awk '{system("kubectl delete secret "$1 " -n cert-manager")}' namespaces=(kube-system services istio-system kube-public ibmcom cert-manager) for ns in ${namespaces[@]} do #secret for s in $(ls $ns.secret/ | grep -v "\-token-") do kubectl delete -f $ns.secret/$s && kubectl create -f $ns.secret/$s done #configmap for s in $(ls $ns.configmap/) do 

