<p dir="auto">I enable <code class="notranslate">ResourceQuota</code> in my cluster. I have one namespace <code class="notranslate">ns-xnxin</code>. I have 2 quotas created for that namespace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl describe quota quota --namespace=ns-xnxin
Name:           quota-xnxin
Namespace:      ns-xnxin
Resource        Used    Hard
--------        ----    ----
cpu         0   20
memory          0   1Gi
pods            3   3
services        0   5

Name:       quota-xnxin-test
Namespace:  ns-xnxin
Resource    Used    Hard
--------    ----    ----
pods        3   1
services    0   5"><pre class="notranslate"><code class="notranslate">(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl describe quota quota --namespace=ns-xnxin
Name:           quota-xnxin
Namespace:      ns-xnxin
Resource        Used    Hard
--------        ----    ----
cpu         0   20
memory          0   1Gi
pods            3   3
services        0   5

Name:       quota-xnxin-test
Namespace:  ns-xnxin
Resource    Used    Hard
--------    ----    ----
pods        3   1
services    0   5
</code></pre></div>
<p dir="auto">Before creating quotas, there are already 3 Pods running under <code class="notranslate">ns-xnxin</code>.</p>
<p dir="auto">Then, I tried to create new Pod two times, but got different response:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating &quot;busybox-limit.json&quot;: Pod &quot;busybox-limit&quot; is forbidden: limited to 3 pods"><pre class="notranslate"><code class="notranslate">(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating "busybox-limit.json": Pod "busybox-limit" is forbidden: limited to 3 pods
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating &quot;busybox-limit.json&quot;: Pod &quot;busybox-limit&quot; is forbidden: limited to 1 pods"><pre class="notranslate"><code class="notranslate">(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating "busybox-limit.json": Pod "busybox-limit" is forbidden: limited to 1 pods
</code></pre></div>
<p dir="auto">It seems that api-server do quota validation in a random behavior by the first get quota record.<br>
I think this is not a consistent behavior and need to be enhanced. Two proposed options:</p>
<ul dir="auto">
<li>If k8s accept duplicate resource definition in different quota (for example, in two quotas, both of them contains Pod quota), the quota validation should be done against the sum of them.</li>
<li>Another option is to have resource conflict validation during quota creation. Say, do not allow duplicated resource in different quota.</li>
</ul>
<p dir="auto">One should be able to upgrade a petset through the standard kubectl rolling-update syntax. While it's desirable to have a controlled upgrade strategy (eg: deployments), it may not be necessary for beta.</p>
<p dir="auto">The tricky part is making sure that <code class="notranslate">pet-v1-0</code> and <code class="notranslate">pet-v2-0</code> come up with the same data. For beta I propose we just make <code class="notranslate">kubectl rolling-update</code> on petset update the image and kill pets one by one (as opposed to creating a new petset like a classic rolling-update). If any pet doesn't become ready within a timeout, the rolling-update reverts.</p>
<p dir="auto">Node upgrade: mark node as unschedulable, delete pets one by one, wait till they show up on new node as ready, upgrade node. Can we get away with just documenting this (upgrading kubelet on a node shouldn't lead to container restarts anyway)?</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/smarterclayton/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/smarterclayton">@smarterclayton</a> @kubernetes/sig-apps Is this enough?</p>
<ul dir="auto">
<li>To improve the upgrade UX we could use 2 new events: <code class="notranslate">pre-delete</code> and <code class="notranslate">pre-update</code>. We can use <code class="notranslate">terminationGrace</code> simply to leave the cluster, <code class="notranslate">pre-delete</code> to do something drastic like backup all data, and <code class="notranslate">pre-update</code> to do something like give-up leadership (but retain data). When a member is removed/rejoined the rest of the cluster should receive <code class="notranslate">on-change</code>. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="151777021" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/24957" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/24957/hovercard" href="https://github.com/kubernetes/kubernetes/issues/24957">#24957</a> should help with the events.</li>
<li>To improve master/slave upgrade UX, we need to avoid "chasing" the master, i.e by picking the master to upgrade, while the upgrade is happening a new master is elected, the next pod we pick is this new master, and so on.</li>
</ul>
<p dir="auto">This might work as follows:</p>
<ul dir="auto">
<li>upgrade all slaves</li>
<li>send <code class="notranslate">on-upgrade</code> to master so it gives up mastership, and doesn't compete in re-election</li>
<li>wait for new master</li>
<li>upgrade old master</li>
</ul>
<p dir="auto">A leader election service might help here (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="164442169" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28658" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28658/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28658">#28658</a>).</p>
<ul dir="auto">
<li>To improve active-active upgrade UX, the <code class="notranslate">on-change</code> sent to all other members needs to exclude the upgrading member by re-writing the config, so writes aren't blocked on its ack. The upgraded member will have to play catchup when it rejoins.</li>
<li>Some databases like Cassandra have special roles, like seed provider. These need to be dealt with in a  similar way as leadership transfer (promote new seed, update peers, upgrade existing seed).</li>
<li>To improve node upgrade UX, we could teach <code class="notranslate">kubectl drain</code> to move pets off a node slowly. If people are running 1 pet per node (eg: pod anti-affinity), we have less of a problem.</li>
</ul>