Answers: <p>My saviour was</p>
  <p><code>kubectl edit deploy portainer -n portainer -o yaml</code></p>
  <p>changed the node name, saved the file and watched everything work</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-10-29T11:28:27.307Z
Answers: 2023-10-29T11:28:27.307Z
Answers: JoSSte
Body: <p>I have a home lab running k3s.</p>
  <blockquote>
  <pre><code>NAME         STATUS   ROLES                         AGE    VERSION
  k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
  k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1
  </code></pre>
  </blockquote>
  <p>I recently removed a node forcefully, since the disk stopped working.
  (k8s-node-003 was the failure - now deleted from the cluster)
  Unfortunately that node was running the pvc for my portainer setup.</p>
  <p>I looked for a way to &quot;move&quot; the pvc to another node without much luck. So I deleted it <code>kubectl delete pvc portainer -n portainer</code></p>
  <p>I scaled down the portainer deployment with <code>kubectl scale deployment portainer -n portainer --replicas=0</code> and tried to scale it back up <code>kubectl scale deployment portainer -n portainer --replicas=1</code></p>
  <p>Inspecting the pod under deployment <code>kubectl describe pod portainer-858644df85-sjdbk -n portainer</code> I see that</p>
  <blockquote>
  <pre><code>...
  
  Events:
    Type     Reason            Age   From               Message
    ----     ------            ----  ----               -------
    Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim &quot;portainer&quot; not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
    Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim &quot;portainer&quot; not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  </code></pre>
  </blockquote>
  <p>I verified that the claim is gone</p>
  <p><code>kubectl get pv -n portainer portainer</code></p>
  <blockquote>
  <pre><code>Error from server (NotFound): persistentvolumes &quot;portainer&quot; not found
  </code></pre>
  </blockquote>
  <p><code>kubectl get pvc -n portainer portainer</code></p>
  <blockquote>
  <pre><code>Error from server (NotFound): persistentvolumeclaims &quot;portainer&quot; not found
  </code></pre>
  </blockquote>
  <p>I guess I should have saved the pvc and then reinstated that using a command like <code>kubectl get pvc --namespace portainer -o json &gt; portainer-pvc.yaml</code> and then reinstated it with something like <code>kubectl apply -f portainer-pvc.yaml -n portainer</code> but 20/20 hindsight is not useful now...</p>
  <p>Is it sufficient to find the portainer pvc definition and applying that?</p>
  <h2>UPDATE</h2>
  <p>I found the command i used to update portainer and ran that</p>
  <pre class="lang-sh prettyprint-override"><code>helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
       --set enterpriseEdition.enabled=true \
       --set tls.force=true \
       --set tls.existingSecret=portainer-tls-secret
  </code></pre>
  <p>which appears to have reinitialised the pvc</p>
  <p><code>kubectl get pvc -n portainer portainer</code></p>
  <blockquote>
  <pre><code>NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  portainer   Pending                                      local-path     4m12s
  </code></pre>
  </blockquote>
  <p>...but it seems that there is some other reference to the defunct node:</p>
  <blockquote>
  <pre><code>Events:
    Type     Reason            Age    From               Message
    ----     ------            ----   ----               -------
    Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  </code></pre>
  </blockquote>
  <p>Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?</p>
  
Tags: <kubernetes><kubectl><k3s><persistent-volumes>
Title: Redefine deployment (Portainer) pvc after node failure
UserDisplayName: JoSSte
ThreadText: # Redefine deployment (Portainer) pvc after node failure
  
  (Asked by: username_0 on 2023-10-28)
  
  I have a home lab running k3s.
  
  > ```
  > NAME         STATUS   ROLES                         AGE    VERSION
  > k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
  > k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1
  >
  > ```
  
  I recently removed a node forcefully, since the disk stopped working.
  (k8s-node-003 was the failure - now deleted from the cluster)
  Unfortunately that node was running the pvc for my portainer setup.
  
  I looked for a way to "move" the pvc to another node without much luck. So I deleted it `kubectl delete pvc portainer -n portainer`
  
  I scaled down the portainer deployment with `kubectl scale deployment portainer -n portainer --replicas=0` and tried to scale it back up `kubectl scale deployment portainer -n portainer --replicas=1`
  
  Inspecting the pod under deployment `kubectl describe pod portainer-858644df85-sjdbk -n portainer` I see that
  
  > ```
  > ...
  >
  > Events:
  >   Type     Reason            Age   From               Message
  >   ----     ------            ----  ----               -------
  >   Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  >   Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  >
  > ```
  
  I verified that the claim is gone
  
  `kubectl get pv -n portainer portainer`
  
  > ```
  > Error from server (NotFound): persistentvolumes "portainer" not found
  >
  > ```
  
  `kubectl get pvc -n portainer portainer`
  
  > ```
  > Error from server (NotFound): persistentvolumeclaims "portainer" not found
  >
  > ```
  
  I guess I should have saved the pvc and then reinstated that using a command like `kubectl get pvc --namespace portainer -o json > portainer-pvc.yaml` and then reinstated it with something like `kubectl apply -f portainer-pvc.yaml -n portainer` but 20/20 hindsight is not useful now...
  
  Is it sufficient to find the portainer pvc definition and applying that?
  
  ## UPDATE
  
  I found the command i used to update portainer and ran that
  
  ```sh
  helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
       --set enterpriseEdition.enabled=true \
       --set tls.force=true \
       --set tls.existingSecret=portainer-tls-secret
  
  ```
  
  which appears to have reinitialised the pvc
  
  `kubectl get pvc -n portainer portainer`
  
  > ```
  > NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  > portainer   Pending                                      local-path     4m12s
  >
  > ```
  
  ...but it seems that there is some other reference to the defunct node:
  
  > ```
  > Events:
  >   Type     Reason            Age    From               Message
  >   ----     ------            ----   ----               -------
  >   Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  >
  > ```
  
  Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?
  
  **Tags:** kubernetes, kubectl, k3s, persistent-volumes
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-10-29. Score: 0 | Accepted answer
  
  My saviour was
  
  `kubectl edit deploy portainer -n portainer -o yaml`
  
  changed the node name, saved the file and watched everything work
  
  
  ---
  
  