Answers: <p>We were <a href="https://github.com/freelawproject/courtlistener/issues/2079#issuecomment-1145998549" rel="nofollow noreferrer">eventually able to answer this</a> by following the rules for when a node is terminated. According to the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">FAQ</a>, there are a number of ways that pods can prevent the cluster autoscaler from removing a node. One type of pod is:</p>
  <blockquote>
  <p>Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc).</p>
  </blockquote>
  <p>So our solution is to create a pod in that way via a manifest file. This lets us have a pod named <code>maintenance</code> that sticks around and isn't killed by the cluster autoscaler:</p>
  <pre class="lang-yaml prettyprint-override"><code>---
  apiVersion: v1
  kind: Pod
  metadata:
    name: maintenance
    namespace: blah
    labels:
      type: maintenance
  spec:
    containers:
      - name: web
        image: whatever
        imagePullPolicy: IfNotPresent
        command: [bash]
        stdin: true
        tty: true
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2022-06-03T19:19:39.820Z
Answers: 2022-06-03T19:19:39.820Z
Answers: mlissner
Body: <p>We just moved to kubernetes, but the engineer that helped launch it is on paternity leave before we had hoped (never trust a baby not to be eager!).</p>
  <p>Now we're trying to do maintenance tasks and one off work and having nodes get killed in the middle of things.</p>
  <p>I've looked at using kubernetes jobs for this, but that's overkill. We don't want to write manifest files for everything.</p>
  <p>We just need long lived shell access to do this and that.</p>
  <p>What's the pattern for this so your maintenance task doesn't get killed?</p>
  
Comments: CC BY-SA 4.0
Comments: 2022-05-27T15:35:05.863Z
Comments: Could you be a bit more explicit as to what you mean by "maintenance"? Are you talking about working on the node itself, or Kubernetes objects ?
Comments: Bruce Becker
Comments: CC BY-SA 4.0
Comments: 2022-05-27T21:22:50.720Z
Comments: Like, I want bash access to run an ETL script, say. Or to do some random maintenance task that might take a few hours or a few days to run. I'm not at the point where I could do maintenance on the cluster itself. Or, like, if I need to do a database migration script, where do I run that without having to make a complex job manifest?
Comments: mlissner
Comments: CC BY-SA 4.0
Comments: 2022-05-28T06:02:23.267Z
Comments: My 2c, so take it for what it's worth and consider it a comment, not an answer, but dealing with "complex job manifest" is going to be unavoidable. My opinion is that if you have made the decision to adopt K8s, you should also accept that you're going to have to write a lot of YAML. The tasks you mention sound like they should be implemented as sidecars or batch jobs.
Comments: Bruce Becker
Tags: <kubernetes><sysadmin>
Title: How to do maintenance work in kubernetes cluster
UserDisplayName: mlissner
ThreadText: # How to do maintenance work in kubernetes cluster
  
  (Asked by: username_0 on 2022-05-27)
  
  We just moved to kubernetes, but the engineer that helped launch it is on paternity leave before we had hoped (never trust a baby not to be eager!).
  
  Now we're trying to do maintenance tasks and one off work and having nodes get killed in the middle of things.
  
  I've looked at using kubernetes jobs for this, but that's overkill. We don't want to write manifest files for everything.
  
  We just need long lived shell access to do this and that.
  
  What's the pattern for this so your maintenance task doesn't get killed?
  
  **Tags:** kubernetes, sysadmin
  
  > **Comments:**
  > **username_1**: Could you be a bit more explicit as to what you mean by "maintenance"? Are you talking about working on the node itself, or Kubernetes objects ?  
  > **username_0**: Like, I want bash access to run an ETL script, say. Or to do some random maintenance task that might take a few hours or a few days to run. I'm not at the point where I could do maintenance on the cluster itself. Or, like, if I need to do a database migration script, where do I run that without having to make a complex job manifest?  
  > **username_1**: My 2c, so take it for what it's worth and consider it a comment, not an answer, but dealing with "complex job manifest" is going to be unavoidable. My opinion is that if you have made the decision to adopt K8s, you should also accept that you're going to have to write a lot of YAML. The tasks you mention sound like they should be implemented as sidecars or batch jobs.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-03. Score: 1
  
  We were [eventually able to answer this](https://github.com/freelawproject/courtlistener/issues/2079#issuecomment-1145998549) by following the rules for when a node is terminated. According to the [FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node), there are a number of ways that pods can prevent the cluster autoscaler from removing a node. One type of pod is:
  
  > Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc).
  
  So our solution is to create a pod in that way via a manifest file. This lets us have a pod named `maintenance` that sticks around and isn't killed by the cluster autoscaler:
  
  ```yaml
  ---
  apiVersion: v1
  kind: Pod
  metadata:
    name: maintenance
    namespace: blah
    labels:
      type: maintenance
  spec:
    containers:
      - name: web
        image: whatever
        imagePullPolicy: IfNotPresent
        command: [bash]
        stdin: true
        tty: true
  
  ```
  
  
  ---
  
  