title: Detach/Attach of persistent volumes in EKS
body: <p>I'm working with EKS and am creating persistent volumes dynamically from a storage class and a persistent volume claim. Here's my demo yaml: -</p>
  <pre><code>kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: test-storage
  provisioner: kubernetes.io/aws-ebs
  parameters:
    type: gp2
    zones: us-east-1b
  reclaimPolicy: Retain
  allowVolumeExpansion: true
  mountOptions:
    - debug
  </code></pre>
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: test-pvc
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: test-storage
    resources:
      requests:
        storage: 1Gi
  </code></pre>
  <p>I'm testing what happens when the node that the persistent volumes are attached to &quot;dies&quot; (I'm shutting down the EC2 instance in the AWS console).</p>
  <p>What I've noticed is that the persistent volumes won't be detached from the dead node and reattached to a live node in the cluster until the node is removed. Either with <code>kubectl delete node</code> or automatically by EKS (after around 20 minutes).</p>
  <p>What I'm wondering is, is there a way to change that default behaviour? Either reduce the amount of time it takes EKS to automatically remove the node from the cluster or change when the detach-attach controller kicks in to move the persistent volumes to a live node.</p>
  <p>Thanks in advance.</p>
  <p>-- EDIT --</p>
  <p>Initially I realised that the pod wouldn't come online as both nodes (2 node cluster) were in different availability zones so the volumes couldn't be attached to the other node in the cluster and would have to wait for EKS to spin up a new node in the same availability zone.</p>
  <p>I spun up another node in the same availability group as another node and ran the test again.</p>
  <p>The pod still tries to come up on the new pod but stays pending as the volumes are still attached to the old pod</p>
  <pre><code>Multi-Attach error for volume &quot;pvc-xxxxxxxx&quot; Volume is already used by pod(s) podname-xxxxx&quot;
  </code></pre>
  <p>Is there anyway to alter the default time it takes EKS to terminate the pod on the dead node so that the volumes are released and can attach to the new node where the new pod is pending?</p>
  <p>By default, EKS takes 20 mins to remove the dead node from the cluster and then the volumes are released.</p>