The recommended way to reduce the risk of application data loss is to use storage providers such as GlusterFS or Ceph on Virtual Machines external to the cluster. This way the dependency on the IBM Cloud Private platform itself is removed and the use of dynamic volume provisioning is still enabled. Regardless of the storage provider, ensure backups of the application data are taken regularly. 

## 3.4.4 Platform restore process 

No backup is good without first testing it can be restored, and this process should be thoroughly tested to ensure that each backup can be restored multiple times for any given environment. The information in this section assumes that a new IBM Cloud Private cluster needs to be installed on new or existing infrastructure and that the guidelines in “Platform backup process” on page 84 has been followed to for each of the necessary components. 

In the example commands provided, the same backup files created in the “Platform backup process” section will be used. It’s important to take a full infrastructure backup of the new environment before attempting to restore the cluster. It’s also recommended to use the steps in “Backup etcd using kubectl” on page 89 to create an etcd backup of the current environment, in case the restore process fails and you end up in a situation where some nodes are restored and others are not. There are several steps that involve replacing core data files and there is a high margin for error. 

The restore process will address the following use cases: 

GLYPH<SM590000> Restore an IBM Cloud Private configuration and data to an existing cluster 

GLYPH<SM590000> Restore an IBM Cloud Private configuration and data to a remote cluster 

The local cluster restore steps in this section were performed in a lab environment with LDAP configured and light workloads, tested on the following cluster configurations: 

GLYPH<SM590000> Single master, management, proxy, Vulnerability Advisor and 3 x worker nodes 

GLYPH<SM590000> 3 x masters, 2 x management, 2 x proxies, 3 x Vulnerability Advisor and 3 x worker nodes 

GLYPH<SM590000> 3 x etcd, 3 x masters, 2 x management, 2 x proxies, 3 x Vulnerability Advisor and 3 x worker nodes 

The remote cluster restore steps in this section were performed in a lab environment with LDAP configured and light workloads, tested on the following cluster configurations: Single master, management, proxy, Vulnerability Advisor, and 3 x worker nodes. 

The core components should be restored in the following order: 

1. MongoDB 

2. Private image registry 

3. etcd 

4. Persistent Volume data 

The other components can be restored in any order, although it is recommended that the logging data is restored before the Vulnerability Advisor. The Persistent Volume Data is restored last  so that when a cluster is restored, the backed up data is not overwritten with new cluster data, which would later corrupt the restored applications. 

## Local cluster restore prerequisites 

If IBM Cloud Private is reinstalled on the existing infrastructure, the content in the cfc-certs directory must be copied to the <installation_directory>/cluster directory before installation. This ensures the same certificates from the previous installation are used in the new installation and that any restored components are able to interact with the cluster. 

