content_image/985329.jpg
<img src='content_image/985329.jpg'>

Tip:  Some users experience high CPU utilization by the java processes on the host. This is due to no limit specified on the containers, allowing them to consume all the available host CPU, if necessary. This is intentional and setting limits may impact the ELK stack stability. It is worth noting that high CPU utilization may be an indication of memory pressure due to garbage collection and should be investigated. 

100 GB of storage via a LocalVolume PersistentVolume (PV) is allocated to the Elasticsearch data pods, which resides at /var/lib/icp/logging  on the management node filesystem. Each PV has affinity in place so that the PV is bound to one management node only, to ensure consistency across data nodes. Each Elasticsearch data node also has affinity rules in place so that only one data pod runs on one management node at any one time. 

The size of the Elasticsearch cluster deployed in an environment entirely depends on the number of management nodes in the cluster. The default number of Elasticsearch master and data pods are calculated based on the available management nodes and take on the following rules: 

GLYPH<SM590000> One Elasticsearch data pod per IBM Cloud Private management node 

GLYPH<SM590000> Number of Elasticsearch master pods is equal to number of management nodes 

GLYPH<SM590000> One Logstash pod per IBM Cloud Private management node 

GLYPH<SM590000> One Elasticsearch client pod per IBM Cloud Private management node 

Elasticsearch client and Logstash replicas can be temporarily scaled as required using the default Kubernetes scaling methods. If any scaling is permanent, it’s recommended to use the Helm commands to update the number of replicas. 

## Data retention 

The default retention period for logs stored in the platform ELK stack is 24 hours. A curator is deployed as a CronJob that will remove the logstash indices from Elasticsearch every day at 23:30 UTC. If Vulnerability Advisor is enabled, another CronJob runs at 23:59 UTC to remove the indices related to Vulnerability Advisor older than 7 days. 

Modifying the default retention period without proper capacity planning may be destructive to the ELK stack. Increasing the retention period will increase the resources required to search and store the data in Elasticsearch, so ensure the cluster has the required resources to be able to do so. For more information about resource allocation, see “Capacity planning” on page 164. 

## Configuring data retention during installation 

It’s possible to specify the defa ult data retention period before installing IBM Cloud Private by adding the curator configuration to the config.yaml . Use Example 5-3 to set a default index retention of 14 days. 

logging: curator: schedule: "30 23 * * *" 

