title: Cannot rename nodeset in ECK
body: <p>I am trying to rename a nodeset in my ECK cluster. Below is my Elastisearch cluster yaml:</p>
  <pre><code>apiVersion: elasticsearch.k8s.elastic.co/v1 
  kind: Elasticsearch 
  metadata: 
    name: elastic-test
  spec: 
    version: 7.11.1
    auth:
      roles:
      - secretName: elastic-roles-secret
      fileRealm:
      - secretName: elastic-filerealm-secret
    nodeSets: 
    - name: default
      count: 1 
      config:
        node.store.allow_mmap: false 
      volumeClaimTemplates:
      - metadata:
          name: azure-pvc
        spec:
          storageClassName: &quot;&quot;
          accessModes:
          - ReadWriteMany
          resources:
            requests:
              storage: 25Gi
          volumeName: elasticsearch-azure-pv
      podTemplate:
        spec:
          initContainers:
          - name: install-plugins
            command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install --batch ingest-attachment
  </code></pre>
  <p>I want to change the nodeset name from <strong>default</strong> to <strong>default2</strong>.
  However, the new pod created is stuck on Pending.</p>
  <p>kubectl describe the new pod:</p>
  <pre><code>Events:
    Type     Reason            Age    From               Message
    ----     ------            ----   ----               -------
    Warning  FailedScheduling  4m12s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
    Warning  FailedScheduling  4m12s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  </code></pre>
  <p>Because both the old PVC was not deleted, the new PVC cannot bind to the same PV. AFAIK, for the intended behaviour the old PVC and pod should be deleted and the new pod and PVC can bind to the PV.</p>
  <p>To provide some context, my deployment environment only allows me to apply yaml files (no running kubectl delete), and the goal is to add the ingest-attachment plugin. So I am trying to restart the existing pod by renaming it.</p>