title: harbor - internal server issue - code 500 - after disk space resizing
body: <p>Since a diskspace issue on Harbor node, we increase it and restart the process but the GUI is not working properly for actions like replications, retention policy, CVE scanning</p>
  
  <p>I checked the logs on server side :</p>
  
  <pre><code>[root@ip-10-0-25-133 ec2-user]# docker container ps
  CONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS                  PORTS                                                                 NAMES
  2abf5d05dab7        goharbor/nginx-photon:v1.9.3                             "nginx -g 'daemon of…"   6 months ago        Up 6 months (healthy)   0.0.0.0:4443-&gt;4443/tcp, 0.0.0.0:80-&gt;8080/tcp, 0.0.0.0:443-&gt;8443/tcp   nginx
  f33407aa2177        goharbor/harbor-jobservice:v1.9.3                        "/harbor/harbor_jobs…"   6 months ago        Up 7 weeks (healthy)                                                                          harbor-jobservice
  6420cc8f84ab        goharbor/notary-server-photon:v0.6.1-v1.9.3              "/bin/sh -c 'migrate…"   6 months ago        Up 6 months                                                                                   notary-server
  2546c8894905        goharbor/harbor-core:v1.9.3                              "/harbor/harbor_core"    6 months ago        Up 6 months (healthy)                                                                         harbor-core
  c04e08177850        goharbor/clair-photon:v2.1.0-v1.9.3                      "/docker-entrypoint.…"   6 months ago        Up 6 months (healthy)   6060-6061/tcp                                                         clair
  5e318f8c61d5        goharbor/notary-signer-photon:v0.6.1-v1.9.3              "/bin/sh -c 'migrate…"   6 months ago        Up 6 months                                                                                   notary-signer
  2555d6e8ff9e        goharbor/chartmuseum-photon:v0.9.0-v1.9.3                "/docker-entrypoint.…"   6 months ago        Up 6 months (healthy)   9999/tcp                                                              chartmuseum
  599417cd4413        goharbor/harbor-registryctl:v1.9.3                       "/harbor/start.sh"       6 months ago        Up 6 months (healthy)                                                                         registryctl
  8967774fa4f2        goharbor/redis-photon:v1.9.3                             "redis-server /etc/r…"   6 months ago        Up 6 months (healthy)   6379/tcp                                                              redis
  c4181096df06        goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3   "/entrypoint.sh /etc…"   6 months ago        Up 6 months (healthy)   5000/tcp                                                              registry
  28583420d59b        goharbor/harbor-db:v1.9.3                                "/docker-entrypoint.…"   6 months ago        Up 6 months (healthy)   5432/tcp                                                              harbor-db
  f06fa3ac558d        goharbor/harbor-portal:v1.9.3                            "nginx -g 'daemon of…"   6 months ago        Up 6 months (healthy)   8080/tcp                                                              harbor-portal
  a358bc227c1c        goharbor/harbor-log:v1.9.3                               "/bin/sh -c /usr/loc…"   6 months ago        Up 6 months (healthy)   127.0.0.1:1514-&gt;10514/tcp                                             harbor-log
  
  [root@ip-10-0-25-133 ec2-user]# docker logs harbor-log
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  </code></pre>
  
  <p>There is an rsyslogd error related to harbor which I suppose blocked previous action on the GUI.</p>
  
  <p>Trying to relaunch the containers via docker compose does not help, 2 containers stay on unhealthy status :</p>
  
  <pre><code>[root@ip-10-0-25-133 harbor]# docker-compose start
  Starting log           ... done
  Starting registry      ... done
  Starting registryctl   ... done
  Starting postgresql    ... done
  Starting portal        ... done
  Starting redis         ... done
  Starting core          ... done
  Starting jobservice    ... done
  Starting proxy         ... done
  Starting notary-signer ... done
  Starting notary-server ... done
  Starting clair         ... done
  Starting chartmuseum   ... done
  [root@ip-10-0-25-133 harbor]# docker container ps
  CONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS                            PORTS                                                                 NAMES
  2abf5d05dab7        goharbor/nginx-photon:v1.9.3                             "nginx -g 'daemon of…"   7 months ago        Up 3 seconds (health: starting)   0.0.0.0:4443-&gt;4443/tcp, 0.0.0.0:80-&gt;8080/tcp, 0.0.0.0:443-&gt;8443/tcp   nginx
  f33407aa2177        goharbor/harbor-jobservice:v1.9.3                        "/harbor/harbor_jobs…"   7 months ago        Up 3 seconds (health: starting)                                                                         harbor-jobservice
  6420cc8f84ab        goharbor/notary-server-photon:v0.6.1-v1.9.3              "/bin/sh -c 'migrate…"   7 months ago        Up 4 seconds                                                                                            notary-server
  2546c8894905        goharbor/harbor-core:v1.9.3                              "/harbor/harbor_core"    7 months ago        Up 2 seconds (health: starting)                                                                         harbor-core
  c04e08177850        goharbor/clair-photon:v2.1.0-v1.9.3                      "/docker-entrypoint.…"   7 months ago        Up 3 seconds (health: starting)   6060-6061/tcp                                                         clair
  5e318f8c61d5        goharbor/notary-signer-photon:v0.6.1-v1.9.3              "/bin/sh -c 'migrate…"   7 months ago        Up 5 seconds                                                                                            notary-signer
  2555d6e8ff9e        goharbor/chartmuseum-photon:v0.9.0-v1.9.3                "/docker-entrypoint.…"   7 months ago        Up 8 seconds (health: starting)   9999/tcp                                                              chartmuseum
  599417cd4413        goharbor/harbor-registryctl:v1.9.3                       "/harbor/start.sh"       7 months ago        Up 7 seconds (health: starting)                                                                         registryctl
  8967774fa4f2        goharbor/redis-photon:v1.9.3                             "redis-server /etc/r…"   7 months ago        Up 7 seconds (health: starting)   6379/tcp                                                              redis
  c4181096df06        goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3   "/entrypoint.sh /etc…"   7 months ago        Up 7 seconds (health: starting)   5000/tcp                                                              registry
  28583420d59b        goharbor/harbor-db:v1.9.3                                "/docker-entrypoint.…"   7 months ago        Up 7 seconds (health: starting)   5432/tcp                                                              harbor-db
  f06fa3ac558d        goharbor/harbor-portal:v1.9.3                            "nginx -g 'daemon of…"   7 months ago        Up 7 seconds (health: starting)   8080/tcp                                                              harbor-portal
  a358bc227c1c        goharbor/harbor-log:v1.9.3                               "/bin/sh -c /usr/loc…"   7 months ago        Up 9 seconds (health: starting)   127.0.0.1:1514-&gt;10514/tcp                                             harbor-log
  </code></pre>
  
  <p>The harbor logs are pointing again on a disk space issue :</p>
  
  <pre><code>[root@ip-10-0-25-133 harbor]# docker logs harbor-jobservice
  Error response from daemon: configured logging driver does not support reading
  [root@ip-10-0-25-133 harbor]# docker logs clair
  Error response from daemon: configured logging driver does not support reading
  [root@ip-10-0-25-133 harbor]# docker logs harbor-log
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
  </code></pre>
  
  <p>The disk space is not full on the docker host :</p>
  
  <pre><code>[root@ip-10-0-25-133 harbor]# df -ha
  Filesystem      Size  Used Avail Use% Mounted on
  sysfs              0     0     0    - /sys
  proc               0     0     0    - /proc
  devtmpfs        3.8G     0  3.8G   0% /dev
  securityfs         0     0     0    - /sys/kernel/security
  tmpfs           3.9G     0  3.9G   0% /dev/shm
  devpts             0     0     0    - /dev/pts
  tmpfs           3.9G  383M  3.5G  10% /run
  tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
  cgroup             0     0     0    - /sys/fs/cgroup/systemd
  pstore             0     0     0    - /sys/fs/pstore
  bpf                0     0     0    - /sys/fs/bpf
  cgroup             0     0     0    - /sys/fs/cgroup/devices
  cgroup             0     0     0    - /sys/fs/cgroup/freezer
  cgroup             0     0     0    - /sys/fs/cgroup/pids
  cgroup             0     0     0    - /sys/fs/cgroup/net_cls,net_prio
  cgroup             0     0     0    - /sys/fs/cgroup/hugetlb
  cgroup             0     0     0    - /sys/fs/cgroup/cpuset
  cgroup             0     0     0    - /sys/fs/cgroup/perf_event
  cgroup             0     0     0    - /sys/fs/cgroup/memory
  cgroup             0     0     0    - /sys/fs/cgroup/cpu,cpuacct
  cgroup             0     0     0    - /sys/fs/cgroup/blkio
  cgroup             0     0     0    - /sys/fs/cgroup/rdma
  configfs           0     0     0    - /sys/kernel/config
  /dev/xvda2      489G  383G  107G  79% /
  selinuxfs          0     0     0    - /sys/fs/selinux
  systemd-1          -     -     -    - /proc/sys/fs/binfmt_misc
  hugetlbfs          0     0     0    - /dev/hugepages
  debugfs            0     0     0    - /sys/kernel/debug
  mqueue             0     0     0    - /dev/mqueue
  binfmt_misc        0     0     0    - /proc/sys/fs/binfmt_misc
  tracefs            0     0     0    - /sys/kernel/debug/tracing
  tmpfs           782M     0  782M   0% /run/user/1000
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/cac923a265f3adc2ba8a8b3f14a0fb21330db4a73562a484475cc782deb000fb/merged
  shm              64M     0   64M   0% /var/lib/docker/containers/a358bc227c1cc3f986255adc03d354b773fb18f7a383182632b303e8977ddc0a/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/b0b97ca08f9a
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/b8f287f1437be444902922a600eb4ae1fb83c9de1be9d439f9d36a08995ff544/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/6db8129dde48fcf90260c06d287e2e02db335dc80d097d8166b92ef0c1105e4e/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/69d464cb11c9c12dd754190f734eaabbed2fe867e5cfa76d7812a068e0801602/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/b2177176bd4c93cf12ff717aabe563c8469cf6bec426ca17e8d82df0618a3b0e/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/11b8dd2688d07ad2a1b81fe11f235f7d3e648580c32c7757109d15fad8aa9030/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/8b0fe0a0b616f6d2343a919e62590b62b0bb714fbce4373936d5a3962d0cb12c/merged
  shm              64M     0   64M   0% /var/lib/docker/containers/599417cd44133ac65ed8f084e1c645bf04b41548fc71cf150aa3e2ef224df7a8/mounts/shm
  shm              64M     0   64M   0% /var/lib/docker/containers/2555d6e8ff9e3586a41f6f37fc66244d0b3496187da724787cfac175c7f999c4/mounts/shm
  shm              64M     0   64M   0% /var/lib/docker/containers/f06fa3ac558da59cf8be1387b811e6d4e622d1ea49a4d97277eb68edf05e6388/mounts/shm
  shm              64M     0   64M   0% /var/lib/docker/containers/8967774fa4f28c3df91ca40722dad81949b1d6c2fe60e3bbcf083cf325ebd7f2/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/a6281418be81
  nsfs               0     0     0    - /run/docker/netns/053f162aea2a
  nsfs               0     0     0    - /run/docker/netns/2c01de1d2aa5
  shm              64M  4.0K   64M   1% /var/lib/docker/containers/28583420d59b3e4316b72f7a4d56d5ddf5fdf8c5932860f1f017d7b47ce2b0cf/mounts/shm
  shm              64M     0   64M   0% /var/lib/docker/containers/c4181096df06edcddf3b2406556523ef26a47c7f441382bcd42446802cb708ec/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/19276a843cd7
  nsfs               0     0     0    - /run/docker/netns/8d1949d48460
  nsfs               0     0     0    - /run/docker/netns/85ba684c4adf
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/7faa4fa7c2fa7247d4f7a889f7d47ac97aa5002d83a03fc28bfa48e951e860ad/merged
  shm              64M     0   64M   0% /var/lib/docker/containers/5e318f8c61d509346c688b4a3514597f571654620e4bd04dac536caaaa41ddf5/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/f8e546c2d757
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/d93e3d145924701e719d7a5fd9e93531c36b46407052e14ec578ca887678f94d/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/92736d5ae32243282bb26ebf9410f645388e5b8b1c1d498c369ad56282211dc6/merged
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/19653cd43524e1094144b7daaeb55b3305f688c1d64bd6eacdd899f2b7c57c38/merged
  shm              64M     0   64M   0% /var/lib/docker/containers/6420cc8f84ab45b145e6f142154c3ad3c09a3aa69455ded75fb33aab3415183c/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/c31bdc8081b5
  shm              64M     0   64M   0% /var/lib/docker/containers/f33407aa21776052609f2edaec1f0c637069992279cc9d19968f2e2d77b2d965/mounts/shm
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/6f53a0994736d049eebd058f2f188e606acfa753e428e605a22943074eca7428/merged
  nsfs               0     0     0    - /run/docker/netns/02436d61c648
  shm              64M     0   64M   0% /var/lib/docker/containers/2abf5d05dab7d47f8fc528f78d112f7a0c3404d87b60649e897f349439e86c58/mounts/shm
  shm              64M     0   64M   0% /var/lib/docker/containers/c04e081778502e8b8e0137512f6d9275daefa2d3641ce220f6d1f7e29a40b378/mounts/shm
  overlay         489G  383G  107G  79% /var/lib/docker/overlay2/63e4dbe4d6213f6dcc1322f3691c7ff3244fb71e6562888b955d6bc4f93c74a4/merged
  shm              64M     0   64M   0% /var/lib/docker/containers/2546c88949055855eef6967c63365b97e06ca38f46c781a336e2e6d7e67226bf/mounts/shm
  nsfs               0     0     0    - /run/docker/netns/9f232ba1bf77
  nsfs               0     0     0    - /run/docker/netns/1f50ca91cfa6
  nsfs               0     0     0    - /run/docker/netns/2a0c5ac700e3
  [root@ip-10-0-25-133 harbor]#
  </code></pre>
  
  <p>This does not help : <a href="https://www.rsyslog.com/solving-rsyslog-write-errors/" rel="nofollow noreferrer">https://www.rsyslog.com/solving-rsyslog-write-errors/</a></p>