Answers: <p>I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7.  One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).</p>
  
  <p>I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:</p>
  
  <pre><code>cat /etc/sysconfig/kubelet
  KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external
  </code></pre>
  
  <p>My kubeadm init line is (I'm using an env variable to set the control plain endpoint):</p>
  
  <pre><code>kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs &gt;&gt; $HOME/log/init-cluster.log
  </code></pre>
  
  <p>I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.</p>
  
  <pre><code>cgroup_manager = "systemd"
  </code></pre>
  
  <p>Also make sure you are enabling the modules / sysconfig options specified on <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o" rel="nofollow noreferrer">this page</a>.</p>
  
  <pre><code>modprobe overlay
  modprobe br_netfilter
  
  # Setup required sysctl params, these persist across reboots.
  cat &gt; /etc/sysctl.d/99-kubernetes-cri.conf &lt;&lt;EOF
  net.bridge.bridge-nf-call-iptables  = 1
  net.ipv4.ip_forward                 = 1
  net.bridge.bridge-nf-call-ip6tables = 1
  EOF
  
  sysctl --system
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2020-02-11T11:46:39.013Z
Answers: Thanks for your suggestions. Please review the question as I've updated it with new info and also with the outputs of running kubeadm init with your parameters. Still no luck :/
Answers: carrotcakeslayer
Answers: CC BY-SA 4.0
Answers: 2020-02-11T12:34:40.283Z
Answers: What's your cri-o version?  Can you show output from `crictl version` and `crictl info`?
Answers: Luke
Answers: CC BY-SA 4.0
Answers: 2020-02-12T11:48:12.787Z
Answers: Hi LArntz. I've just added the info to the original question. Thank you!
Answers: carrotcakeslayer
Answers: CC BY-SA 4.0
Answers: 2020-02-12T12:05:32.777Z
Answers: OK. I think this might be your problem....
  
  `crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]`
  
  See the [compatibilty matrix on the cri-o github](https://github.com/cri-o/cri-o).  You need to be running a version of cri-o that matches the kubernetes version. 
  
  If there's not binary version available for your OS it's pretty easy [to build](https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md) (but I've heard of issues with redhat/centos 8).
Answers: Luke
Answers: CC BY-SA 4.0
Answers: 2020-02-12T12:08:38.910Z
Answers: Makes sense and matches your original answer. I overviewed this point as it is not mentioned in the "getting started" kubernetes doc. I'll try to build the package and comment here the output. Thank you!
Answers: carrotcakeslayer
Answers: CC BY-SA 4.0
Answers: 2020-02-12T12:11:23.810Z
Answers: That's a good point. If their docs are opensource a pull request might be helpful to others.
Answers: Luke
Answers: CC BY-SA 4.0
Answers: 2020-02-12T15:13:12.973Z
Answers: That was it!!! Also, in cri-o github page it says that kubelet needs to be same version as cri-o (not mentioned in kubernetes doc for kubeadm install where only kubeadm and kubelet match is mentioned). I was not able to compile v1.17, so I just shortcut to run all in v1.15 (latest available in repo). Thank you very much!
Answers: carrotcakeslayer
Answers: CC BY-SA 4.0
Answers: 2020-02-12T17:07:50.080Z
Answers: No problem! Happy your issue is resolved. 
  
  I submitted a pull request for the kubernetes docs also so hopefully that will be updated soon.
Answers: Luke
Answers: CC BY-SA 4.0
Answers: 2020-02-10T19:24:08.633Z
Answers: 2020-02-10T19:24:08.633Z
Answers: Luke
Body: <p>I've been following this doc to deploy kubernetes <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/</a></p>
  
  <p>After launching the "kubeadm init" command and during its excecution, this is the output:</p>
  
  <blockquote>
    <p>[kubelet-check] The HTTP call equal to 'curl -sSL <a href="http://localhost:10248/healthz" rel="nofollow noreferrer">http://localhost:10248/healthz</a>' failed with error: Get <a href="http://localhost:10248/healthz" rel="nofollow noreferrer">http://localhost:10248/healthz</a>: dial tcp 127.0.0.1:10248: connect: connection refused.
    Unfortunately, an error has occurred:
            timed out waiting for the condition
    This error is likely caused by:
            - The kubelet is not running
            - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
    If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
            - 'systemctl status kubelet'
            - 'journalctl -xeu kubelet'
    Additionally, a control plane component may have crashed or exited when started by the container runtime.
    To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
    Here is one example how you may list all Kubernetes containers running in docker:
            - 'docker ps -a | grep kube | grep -v pause'
            Once you have found the failing container, you can inspect its logs with:
            - 'docker logs CONTAINERID'
    error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
    To see the stack trace of this error execute with --v=5 or higher</p>
  </blockquote>
  
  <p>It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".</p>
  
  <p>I've tried several things, but this is the content of my last try:</p>
  
  <blockquote>
    <p>cat etc/sysconfig/kubelet</p>
    
    <p>KUBELET_ARGS="--container-runtime=remote
    --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster_domain=topota.madre"</p>
    
    <p>systemctl cat kubelet</p>
    
    <p># /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=<a href="https://kubernetes.io/docs/" rel="nofollow noreferrer">https://kubernetes.io/docs/</a></p>
    
    <p>[Service] ExecStart=/usr/bin/kubelet Restart=always
    StartLimitInterval=0 RestartSec=10</p>
    
    <p>[Install] WantedBy=multi-user.target</p>
    
    <p># /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
    # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service]
    Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
    --kubeconfig=/etc/ Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
    # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS var
    EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
    # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul
    # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sour
    EnvironmentFile=-/etc/sysconfig/kubelet ExecStart=
    ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS
    $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS</p>
    
    <p>cat /var/lib/kubelet/kubeadm-flags.env</p>
    
    <p>is empty right now</p>
  </blockquote>
  
  <p>Cri-O service ir running ok and Docker is not installed.</p>
  
  <ul>
  <li>kubelet version --short -> Client Version: v1.17.2</li>
  <li>kubeadm version -o short -> v1.17.2</li>
  <li>cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)</li>
  <li>crictl version -> ["Version:  0.1.0","RuntimeName: cri-o","RuntimeVersion:  1.15.1-2.el7","RuntimeApiVersion:  v1alpha1"]</li>
  <li>crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}</li>
  <li>grep -i cgroup_manager /etc/crio/crio.conf -> cgroup_manager = "systemd"</li>
  <li>lsmod | grep -i "overlay\|br_netfilter" -> ["br_netfilter","overlay"] (both ok).</li>
  <li>Other "extra args tested" (/etc/sysconfig/kubelet)</li>
  </ul>
  
  <blockquote>
    <p>KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
    --container-runtime-endpoint=unix:///var/run/crio/crio.sock</p>
  </blockquote>
  
  <ul>
  <li>Full output of some commands:
  -> "kube admin init --upload-certs" -> <a href="https://pastebin.com/isbAynEg" rel="nofollow noreferrer">https://pastebin.com/isbAynEg</a>
  -> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> <a href="https://pastebin.com/9pgtDMNu" rel="nofollow noreferrer">https://pastebin.com/9pgtDMNu</a>
  -> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"]
  -> "journalctl -xeu kubelet" -> <a href="https://pastebin.com/qreJyhsu" rel="nofollow noreferrer">https://pastebin.com/qreJyhsu</a> (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).</li>
  </ul>
  
  <p>Any help please?</p>
  
Comments: CC BY-SA 4.0
Comments: 2020-02-10T19:28:21.667Z
Comments: What's the output when you follow instructions bellow:
  
  The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet'
Comments: Vader
Comments: CC BY-SA 4.0
Comments: 2020-02-11T11:45:48.343Z
Comments: I just added the links to pastebin with those outputs. I did not see anything conclusive in them, but there are some interesting lines that I may be overseeing.
Comments: carrotcakeslayer
Comments: CC BY-SA 4.0
Comments: 2020-02-11T21:59:39.167Z
Comments: What distribution are you using?
Comments: Matt
Comments: CC BY-SA 4.0
Comments: 2020-02-12T11:47:45.023Z
Comments: I've just added the info to the original question. Thx for your help!
Comments: carrotcakeslayer
Tags: <kubernetes><kubeadm>
Title: kubeadm init fails when using cri-o
UserDisplayName: carrotcakeslayer
ThreadText: # kubeadm init fails when using cri-o
  
  (Asked by: username_0 on 2020-02-10)
  
  I've been following this doc to deploy kubernetes <https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/>
  
  After launching the "kubeadm init" command and during its excecution, this is the output:
  
  > [kubelet-check] The HTTP call equal to 'curl -sSL <http://localhost:10248/healthz>' failed with error: Get <http://localhost:10248/healthz>: dial tcp 127.0.0.1:10248: connect: connection refused.
  > Unfortunately, an error has occurred:
  > timed out waiting for the condition
  > This error is likely caused by:
  > - The kubelet is not running
  > - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
  > If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
  > - 'systemctl status kubelet'
  > - 'journalctl -xeu kubelet'
  > Additionally, a control plane component may have crashed or exited when started by the container runtime.
  > To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
  > Here is one example how you may list all Kubernetes containers running in docker:
  > - 'docker ps -a | grep kube | grep -v pause'
  > Once you have found the failing container, you can inspect its logs with:
  > - 'docker logs CONTAINERID'
  > error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
  > To see the stack trace of this error execute with --v=5 or higher
  
  It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".
  
  I've tried several things, but this is the content of my last try:
  
  > cat etc/sysconfig/kubelet
  >
  > KUBELET\_ARGS="--container-runtime=remote
  > --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster\_domain=topota.madre"
  >
  > systemctl cat kubelet
  >
  > # /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=<https://kubernetes.io/docs/>
  >
  > [Service] ExecStart=/usr/bin/kubelet Restart=always
  > StartLimitInterval=0 RestartSec=10
  >
  > [Install] WantedBy=multi-user.target
  >
  > # /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
  > # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service]
  > Environment="KUBELET\_KUBECONFIG\_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
  > --kubeconfig=/etc/ Environment="KUBELET\_CONFIG\_ARGS=--config=/var/lib/kubelet/config.yaml"
  > # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET\_KUBEADM\_ARGS var
  > EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
  > # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul
  > # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET\_EXTRA\_ARGS should be sour
  > EnvironmentFile=-/etc/sysconfig/kubelet ExecStart=
  > ExecStart=/usr/bin/kubelet $KUBELET\_KUBECONFIG\_ARGS
  > $KUBELET\_CONFIG\_ARGS $KUBELET\_KUBEADM\_ARGS $KUBELET\_EXTRA\_ARGS
  >
  > cat /var/lib/kubelet/kubeadm-flags.env
  >
  > is empty right now
  
  Cri-O service ir running ok and Docker is not installed.
  
  * kubelet version --short -> Client Version: v1.17.2
  * kubeadm version -o short -> v1.17.2
  * cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)
  * crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]
  * crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}
  * grep -i cgroup\_manager /etc/crio/crio.conf -> cgroup\_manager = "systemd"
  * lsmod | grep -i "overlay\|br\_netfilter" -> ["br\_netfilter","overlay"] (both ok).
  * Other "extra args tested" (/etc/sysconfig/kubelet)
  
  > KUBELET\_EXTRA\_ARGS=--cgroup-driver=systemd
  > --container-runtime-endpoint=unix:///var/run/crio/crio.sock
  
  * Full output of some commands:
    -> "kube admin init --upload-certs" -> <https://pastebin.com/isbAynEg>
    -> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> <https://pastebin.com/9pgtDMNu>
    -> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip\_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"]
    -> "journalctl -xeu kubelet" -> <https://pastebin.com/qreJyhsu> (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).
  
  Any help please?
  
  **Tags:** kubernetes, kubeadm
  
  > **Comments:**
  > **username_1**: What's the output when you follow instructions bellow:  The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet'  
  > **username_0**: I just added the links to pastebin with those outputs. I did not see anything conclusive in them, but there are some interesting lines that I may be overseeing.  
  > **username_2**: What distribution are you using?  
  > **username_0**: I've just added the info to the original question. Thx for your help!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-02-10. Score: 2 | Accepted answer
  
  I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7. One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).
  
  I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:
  
  ```
  cat /etc/sysconfig/kubelet
  KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external
  
  ```
  
  My kubeadm init line is (I'm using an env variable to set the control plain endpoint):
  
  ```
  kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs >> $HOME/log/init-cluster.log
  
  ```
  
  I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.
  
  ```
  cgroup_manager = "systemd"
  
  ```
  
  Also make sure you are enabling the modules / sysconfig options specified on [this page](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o).
  
  ```
  modprobe overlay
  modprobe br_netfilter
  
  # Setup required sysctl params, these persist across reboots.
  cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
  net.bridge.bridge-nf-call-iptables  = 1
  net.ipv4.ip_forward                 = 1
  net.bridge.bridge-nf-call-ip6tables = 1
  EOF
  
  sysctl --system
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks for your suggestions. Please review the question as I've updated it with new info and also with the outputs of running kubeadm init with your parameters. Still no luck :/  
  > **username_3**: What's your cri-o version?  Can you show output from `crictl version` and `crictl info`?  
  > **username_0**: Hi LArntz. I've just added the info to the original question. Thank you!  
  > **username_3**: OK. I think this might be your problem....  `crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]`  See the [compatibilty matrix on the cri-o github](https://github.com/cri-o/cri-o).  You need to be running a version of cri-o that matches the kubernetes version.   If there's not binary version available for your OS it's pretty easy [to build](https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md) (but I've heard of issues with redhat/centos 8).  
  > **username_0**: Makes sense and matches your original answer. I overviewed this point as it is not mentioned in the "getting started" kubernetes doc. I'll try to build the package and comment here the output. Thank you!  
  > **username_3**: That's a good point. If their docs are opensource a pull request might be helpful to others.  
  > **username_0**: That was it!!! Also, in cri-o github page it says that kubelet needs to be same version as cri-o (not mentioned in kubernetes doc for kubeadm install where only kubeadm and kubelet match is mentioned). I was not able to compile v1.17, so I just shortcut to run all in v1.15 (latest available in repo). Thank you very much!  
  > **username_3**: No problem! Happy your issue is resolved.   I submitted a pull request for the kubernetes docs also so hopefully that will be updated soon.  
  
  ---
  
  