<https://forum.proxmox.com/threads/cannot-start-any-kvm-lxc.58297/>
<http://schema.org/articleBody>
"Hi,\n\nhere I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).\n\nThe issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds.\nAll these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd\n\nThis pool hdd is relatively small compared to the largest pool: hdb_backup\nroot@ld3955:~# rados df\nPOOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY\nUNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR\nbackup 0 B 0 0 0\n0 0 0 0 0 B 0 0 B 0\nB 0 B\nhdb_backup 589 TiB 51262212 0 153786636\n0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0\nB 0 B\nhdd 3.2 TiB 281884 6568 845652\n0 0 1658 275277357 16 TiB 208213922 10 TiB 0\nB 0 B\npve_cephfs_data 955 GiB 91832 0 275496\n0 0 3038 2103 1021 MiB 102170 318 GiB 0\nB 0 B\npve_cephfs_metadata 486 MiB 62 0 186\n0 0 7 860 1.4 GiB 12393 166 MiB 0\nB 0 B\n\ntotal_objects 51635990\ntotal_used 597 TiB\ntotal_avail 522 TiB\ntotal_space 1.1 PiB\n\nThis is the current health status of the ceph cluster:\nroot@ld3955:~# ceph -s\n  cluster:\n    id:     6b1b5117-6e08-4843-93d6-2da3cf8a6bae\n    health: HEALTH_ERR\n            1 MDSs report slow metadata IOs\n            78 nearfull osd(s)\n            1 pool(s) nearfull\n            Reduced data availability: 2 pgs inactive, 2 pgs peering\n            Degraded data redundancy: 304136/153251136 objects degraded (0.198%), 57 pgs degraded, 57 pgs undersized\n            Degraded data redundancy (low space): 265 pgs backfill_toofull\n            3 pools have too many placement groups\n            75 slow requests are blocked > 32 sec\n            78 stuck requests are blocked > 4096 sec\n\n  services:\n    mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 94m)\n    mgr: ld5505(active, since 3d), standbys: ld5506, ld5507\n    mds: pve_cephfs:1 {0=ld3976=up:active} 1 up:standby\n    osd: 368 osds: 368 up, 367 in; 303 remapped pgs\n\n  data:\n    pools:   5 pools, 8868 pgs\n    objects: 51.08M objects, 195 TiB\n    usage:   590 TiB used, 563 TiB / 1.1 PiB avail\n    pgs:     0.023% pgs not active\n             304136/153251136 objects degraded (0.198%)\n             1673548/153251136 objects misplaced (1.092%)\n             8563 active+clean\n             195  active+remapped+backfill_toofull\n             57   active+undersized+degraded+remapped+backfill_toofull\n             36   active+remapped+backfill_wait\n             13   active+remapped+backfill_wait+backfill_toofull\n             2    active+remapped+backfilling\n             2    peering\n\n  io:\n    client:   264 KiB/s wr, 0 op/s rd, 0 op/s wr\n    recovery: 18 MiB/s, 4 objects/s\n\n\nI believe the cluster is busy with rebalancing pool hdb_backup.\nI set the balance mode upmap recently after the 589TB data was written.\nroot@ld3955:~# ceph balancer status\n{\n\"active\": true,\n\"plans\": [],\n\"mode\": \"upmap\"\n}\n\nIn order to resolve the issue with pool hdd I started some investigation.\nFirst step was to install drivers for the NIC provided Mellanox.\nThen I configured some kernel parameters recommended\n<https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.\n\nHowever this didn't fix the issue.\nIn my opinion I must get rid of all \"slow requests are blocked\".\n\nWhen I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd.\nThis means none of the disks belonging to pool hdb_backup is showing a comparable behavior.\n\nThen I checked the running processes on the different OSD nodes; I use tool \"glances\" here.\nHere I can see single processes that are running for hours and consuming much CPU, e.g.\n63.9   0.5   4.95G 3.81G   14894 ceph        6h14:22 58    0 S  16M 293K /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph\n\nSimilar processes are running on 4 OSD nodes.\nAll processes have in common that the relevant OSD belongs to pool hdd.\n\nWhat can / should I do now?\nKill the long running processes?\nStop the relevant OSDs?\n\nPlease advise?\n\nTHX\nThomas"
<https://forum.proxmox.com/threads/cannot-start-any-kvm-lxc.58297/>