Answers: <p>Seems that the problem was with the <code>replicas: {{ .Values.replicaCount }}</code> definition. It seems that, if you are using HPA, <code>replicas</code> can't be used. I removed this line and the HPA started scaling.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-07-04T17:54:05.773Z
Answers: 2022-07-04T17:54:05.773Z
Answers: Alexandru Constantin Arvinte
Body: <p>So I have a <code>HorizontalPodAutoscaler</code> set up for my backend (an fpm-server and an Nginx server for a Laravel application).</p>
  <p>The problem is that when the <code>HPA</code> is under load, it scales up the pods but it terminates them instantly, not even letting them get into the <strong>Running</strong> state.</p>
  <p>The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.</p>
  <p>What could be the problem?</p>
  <p><strong>Edit:</strong> The same <code>HPA</code> is used on the frontend and it's working as expected, the problem seems to be only on the backend.</p>
  <p><strong>Edit 2:</strong> I have <code>Cluster Autoscaler</code> enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.</p>
  <p><strong>deployment.yaml</strong></p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: fpm-server
    labels:
      tier: backend
      layer: fpm
  spec:
    replicas: {{ .Values.replicaCount }}
    selector:
      matchLabels:
        tier: backend
        layer: fpm
    template:
      metadata:
        labels:
          tier: backend
          layer: fpm
      spec:
        {{- with .Values.imagePullSecrets }}
        imagePullSecrets:
          {{- toYaml . | nindent 8 }}
        {{- end }}           
        containers:
          - name: fpm
            image: &quot;{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}&quot;
            ports:
              - name: http
                containerPort: 9000
                protocol: TCP
            env:
            {{- range $name, $value := .Values.env }}
            - name: {{ $name }}
              value: &quot;{{ $value }}&quot; 
            {{- end }}             
            envFrom:
              - secretRef:
                  name: backend-secrets
            resources:
              {{- toYaml .Values.resources | nindent 12 }}
  </code></pre>
  <p><strong>hpa.yaml</strong></p>
  <pre><code>apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: fpm-server-hpa
    labels:
      tier: backend
      layer: fpm
  spec:
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: fpm-server
    minReplicas: {{ .Values.autoscaling.minReplicas }}
    maxReplicas: {{ .Values.autoscaling.maxReplicas }}
    metrics:
      {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
      - type: Resource
        resource:
          name: cpu
          targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
      {{- end }}
      {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
      - type: Resource
        resource:
          name: memory
          targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
      {{- end }}
  </code></pre>
  
Tags: <kubernetes>
Title: HorizontalPodAutoscaler scales up pods but then terminates them instantly
UserDisplayName: Alexandru Constantin Arvinte
ThreadText: # HorizontalPodAutoscaler scales up pods but then terminates them instantly
  
  (Asked by: username_0 on 2022-07-02)
  
  So I have a `HorizontalPodAutoscaler` set up for my backend (an fpm-server and an Nginx server for a Laravel application).
  
  The problem is that when the `HPA` is under load, it scales up the pods but it terminates them instantly, not even letting them get into the **Running** state.
  
  The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.
  
  What could be the problem?
  
  **Edit:** The same `HPA` is used on the frontend and it's working as expected, the problem seems to be only on the backend.
  
  **Edit 2:** I have `Cluster Autoscaler` enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.
  
  **deployment.yaml**
  
  ```
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: fpm-server
    labels:
      tier: backend
      layer: fpm
  spec:
    replicas: {{ .Values.replicaCount }}
    selector:
      matchLabels:
        tier: backend
        layer: fpm
    template:
      metadata:
        labels:
          tier: backend
          layer: fpm
      spec:
        {{- with .Values.imagePullSecrets }}
        imagePullSecrets:
          {{- toYaml . | nindent 8 }}
        {{- end }}           
        containers:
          - name: fpm
            image: "{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}"
            ports:
              - name: http
                containerPort: 9000
                protocol: TCP
            env:
            {{- range $name, $value := .Values.env }}
            - name: {{ $name }}
              value: "{{ $value }}" 
            {{- end }}             
            envFrom:
              - secretRef:
                  name: backend-secrets
            resources:
              {{- toYaml .Values.resources | nindent 12 }}
  
  ```
  
  **hpa.yaml**
  
  ```
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: fpm-server-hpa
    labels:
      tier: backend
      layer: fpm
  spec:
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: fpm-server
    minReplicas: {{ .Values.autoscaling.minReplicas }}
    maxReplicas: {{ .Values.autoscaling.maxReplicas }}
    metrics:
      {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
      - type: Resource
        resource:
          name: cpu
          targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
      {{- end }}
      {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
      - type: Resource
        resource:
          name: memory
          targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
      {{- end }}
  
  ```
  
  **Tags:** kubernetes
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-07-04. Score: 1 | Accepted answer
  
  Seems that the problem was with the `replicas: {{ .Values.replicaCount }}` definition. It seems that, if you are using HPA, `replicas` can't be used. I removed this line and the HPA started scaling.
  
  
  ---
  
  