content_image/995875.jpg
<img src='content_image/995875.jpg'>

## Collecting the data 

On every IBM Cloud Private cluster node, an instance of Filebeat is running. The Filebeat pods are controlled by a Daemonset that is actively keeping at least one Filebeat pod running on each cluster node to ensure all node are collecting log data across the whole cluster. By default, ELK is configured to be cluster-wide, monitoring all namespaces on all nodes. 

All containers running on a host write out data to stdout and stderr, which is captured by Docker and stored on the host filesystem. In IBM Cloud Private, Docker is configured to use the json-file logging driver, which means that Docker captures the standard output (and standard error) of all the containers and writes them to the filesystem in files using the JSON format. The JSON format annotates each line with its origin (stdout or stderr) and its timestamp and each log file contains information about only one container. For each container, Docker stores the JSON file in a unique directory using the container ID. A typical format is /var/lib/docker/containers/<container-id>/<container-id>-json.log . 

The /var/lib/docker/containers/  directory has a symlink for each file to another location at /var/log/pods/<uid>/<container-name>/<number>.log . 

Filebeat then continuously monitors the JSON data for every container, but it does not know anything about the container IDs. It does not query Kubernetes for every container ID, so the kubelet service creates a series of symlinks pointing to the correct location (useful for centralized log collection in Kubernetes) and it retrieves the container logs from the host filesystem at /var/log/containers/<container-name>_<namespace>_<uid> .log. In the Filebeat configuration, this filepath is used to retrieve log data from all namespaces using a wildcard filepath /var/log/containers/*.log  to retrieve everything, but it’s also possible to configure more accurate filepaths for specific namespaces. 

