text: Title: Experiment stuck due to hyperparameter suggestion pod gettin OOM Killed
  Body: /kind bug
  
  **What steps did you take and what happened:**
  
  Every time I try to run an experiment (in this case using Bayesian Optimization) after 18-25 trials, the pod that schedules the trials with suggested hyperparameters gets OOM Killed and the experiment does not continue.
  
  If you manually increase the memory resources and limits of the deployment, it works like a charm.
  
  **What did you expect to happen:**
  
  It should either take less memory and work with this kind of numbers of trials without any problem, or more easily just increase the default limit of the suggestion pod memory to a more realistic number.
  
  **Anything else you would like to add:**
  The experiment is the following:
  ```yaml
  metadata:
    name: [REDACTED]
    namespace:  [REDACTED]
    uid: ee155028-e668-4ce3-a737-87e376268b43
    resourceVersion: '68982612'
    generation: 2
    creationTimestamp: '2024-02-09T09:17:02Z'
    finalizers:
      - update-prometheus-metrics
    managedFields:
      - manager: Go-http-client
        operation: Update
        apiVersion: kubeflow.org/v1beta1
        time: '2024-02-09T09:17:02Z'
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
           ...
      - manager: kubectl-edit
        operation: Update
        apiVersion: kubeflow.org/v1beta1
        time: '2024-02-19T08:13:38Z'
        fieldsType: FieldsV1
        fieldsV1:
          f:spec:
            f:parallelTrialCount: {}
      - manager: Go-http-client
        operation: Update
        apiVersion: kubeflow.org/v1beta1
        time: '2024-02-19T09:06:05Z'
        fieldsType: FieldsV1
        fieldsV1:
         ...
        subresource: status
  spec:
    parameters:
      - name: batch_size
        parameterType: discrete
        feasibleSpace:
          list:
            - '32'
            - '64'
            - '128'
            - '256'
      - name: fnok_weight
        parameterType: discrete
        feasibleSpace:
          list:
            - '2.0'
            - '2.8'
            - '3.2'
      - name: nok_weight
        parameterType: discrete
        feasibleSpace:
          list:
            - '2.0'
            - '2.8'
            - '3.2'
      - name: epochs
        parameterType: discrete
        feasibleSpace:
          list:
            - '30'
            - '50'
            - '100'
            - '150'
            - '200'
            - '250'
      - name: model
        parameterType: categorical
        feasibleSpace:
          list:
            - [REDACTED]
            -  [REDACTED]
      - name: base_lr
        parameterType: discrete
        feasibleSpace:
          list:
            - '0.001'
            - '0.0001'
      - name: fraction
        parameterType: discrete
        feasibleSpace:
          list:
            - '0.05'
            - '0.1'
            - '0.25'
            - '0.5'
      - name: trainable_layers
        parameterType: categorical
        feasibleSpace:
          list:
            - dense_12
            - dense_12;dense_11
            - dense_12;dense_11;dense_10;dense_9
            - dense_12;dense_11;dense_10;dense_9;conv2d_12;conv2d_11
            - all
    objective:
      type: maximize
      objectiveMetricName: test_f2
      additionalMetricNames:
        - test_recall
        - test_specificity
        - test_accuracy
        - test_precision
        - model_runid
      metricStrategies:
        - name: test_f2
          value: max
        - name: test_recall
          value: max
        - name: test_specificity
          value: max
        - name: test_accuracy
          value: max
        - name: test_precision
          value: max
        - name: model_runid
          value: latest
    algorithm:
      algorithmName: bayesianoptimization
      algorithmSettings:
        - name: base_estimator
          value: GP
        - name: n_initial_points
          value: '10'
        - name: acq_func
          value: gp_hedge
        - name: acq_optimizer
          value: auto
        - name: random_state
          value: '12'
    trialTemplate:
      retain: true
      trialSpec:
        apiVersion: batch/v1
        kind: Job
        spec:
          backoffLimit: 1
          template:
            metadata:
              annotations:
                sidecar.istio.io/inject: 'false'
            spec:
              containers:
                - command:
                    - python3
                    - '-u'
                    - scriptKubetrain.py
                    - '--batch_size=${trialParameters.batch_size}'
                    - '--epochs=${trialParameters.epochs}'
                    - '--model=${trialParameters.model}'
                    - '--base_lr=${trialParameters.base_lr}'
                    - '--fnok_weight=${trialParameters.fnok_weight}'
                    - '--nok_weight=${trialParameters.nok_weight}'
                    - '--fraction=${trialParameters.fraction}'
                    - '--referencias=" [REDACTED]"'
                    - '--images_folder=/rx/L4'
                    - '--patches_folders=/data/patches/ALL/ [REDACTED]/'
                    - '--in_memory=False'
                    - '--save_folder=/data/models/ [REDACTED]/'
                    - '--model_name= [REDACTED]'
                    - '--trainable_layers="${trialParameters.trainable_layers}"'
                  image: docker-registry:5000/ [REDACTED]
                  imagePullPolicy: Always
                  name: training-container
                  resources:
                    limits:
                      nvidia.com/mig-2g.10gb: 1
                  volumeMounts:
                    - mountPath: /data
                      name: mlops
                    - mountPath: /rx
                      name: rx-mount
              imagePullSecrets:
                - name: registry-access-secret
              restartPolicy: Never
              volumes:
                - name: mlops
                  nfs:
                    path: /mlops
                    server: nfs-srv
                - flexVolume:
                    driver: fstab/cifs
                    fsType: cifs
                    options:
                      mountOptions: iocharset=utf8,file_mode=0777,dir_mode=0777,noperm
                      networkPath:  [REDACTED]
                    secretRef:
                      name: cifs-creds
                  name: rx-mount
      trialParameters:
        - name: batch_size
          reference: batch_size
        - name: epochs
          reference: epochs
        - name: model
          reference: model
        - name: base_lr
          reference: base_lr
        - name: fraction
          reference: fraction
        - name: fnok_weight
          reference: fnok_weight
        - name: nok_weight
          reference: nok_weight
        - name: trainable_layers
          reference: trainable_layers
      primaryContainerName: training-container
      successCondition: status.conditions.#(type=="Complete")#|#(status=="True")#
      failureCondition: status.conditions.#(type=="Failed")#|#(status=="True")#
    parallelTrialCount: 4
    maxTrialCount: 1000
    maxFailedTrialCount: 10
    metricsCollectorSpec:
      collector:
        kind: StdOut
    resumePolicy: LongRunning
  status:
    startTime: '2024-02-09T09:16:41Z'
    conditions:
      - type: Created
        status: 'True'
        reason: ExperimentCreated
        message: Experiment is created
        lastUpdateTime: '2024-02-09T09:16:41Z'
        lastTransitionTime: '2024-02-09T09:16:41Z'
      - type: Running
        status: 'True'
        reason: ExperimentRunning
        message: Experiment is running
        lastUpdateTime: '2024-02-09T09:17:24Z'
        lastTransitionTime: '2024-02-09T09:17:24Z'
    currentOptimalTrial:
      ...
  ```
  
  **Environment:**
  
  - Katib version (check the Katib controller image version): v0.16.0-rc.1
  - Kubernetes version: (`kubectl version`): Server Version: v1.25.9
  - OS (`uname -a`): Linux 6.5.0-15-generic
  
  ---
  
  <!-- Don't delete this message to encourage users to support your issue! -->
  
  Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍
  