text: Title: ComfyUI seems to ignore the --reserve-vram and/or  --disable-smart-memory ? Is there anything going wrong ? 
  Body: ### Your question
  
  So I am using everything to reduce the vram usage amount. 
  `ComfyUI_windows_portable>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --lowvram --force-fp16 --reserve-vram 2.4 --disable-smart-memory`
  
  However I keep seeing that the memory usage of the loaded model is increased -> 
  ![image](https://github.com/user-attachments/assets/229049c3-6695-4df4-9545-2bf92340ef41)
  and then I am getting memory errors in later generations as shown in the logs and the generation slows down significantly. Sometimes I get an OOM error and the generation dies. 
  
  I am basically using a XY generation of pictures with different guidance values and base_shifts. I am using the flux-q4-ks.gguf model (with clip models t5_v1.1_q8 and vit-l-text-detail )  and this [lora](https://civitai.com/models/730373/hyper-realism-lora-by-aidma-flux).
  
  I can provide the workflow if needed for reproduction. 
  
  Seems like the issue is very close to #5958 , #5385 and #4318.
  
  However I am using an SamplerCustomAdvanced and a nvidia card. So just wondering if this is a different problem. Or I am doing something wrong.
  
  
  ### Logs
  
  ```powershell
  [2025-01-01 20:36:33.946] 
                                                                                                               [ARequested to load Flux
  [2025-01-01 20:36:34.009] 0 models unloaded.
  [2025-01-01 20:36:34.118] loaded partially 6320.525634765625 6320.427978515625 0
  [2025-01-01 20:36:34.123] Attempting to release mmap (44)
  [2025-01-01 20:36:34.259] 
  [2025-01-01 20:36:34.260] [A
  [2025-01-01 20:36:34.261] [AERROR lora diffusion_model.single_blocks.0.linear2.weight Allocation on device 
  [2025-01-01 20:38:46.045] 
  [2025-01-01 20:38:46.045] [A
  [2025-01-01 20:40:26.966] [A
  [2025-01-01 20:42:00.008] [A
  [2025-01-01 20:43:26.563] [A
  [2025-01-01 20:45:06.902] [A
  [2025-01-01 20:46:33.755] [A
  [2025-01-01 20:48:04.298] [A
  [2025-01-01 20:49:35.220] [A
  [2025-01-01 20:52:40.229] [A
  [2025-01-01 20:55:56.443] [A
  [2025-01-01 20:58:42.092] [A
  [2025-01-01 21:01:50.056] [A
  [2025-01-01 21:02:17.609] [A
  bosh3: 100%|█████████████████████████████████████████████████████████████████| 1/1 [25:43<00:00, 1543.35s/it]
  bosh3: 100%|█████████████████████████████████████████████████████████████████| 1/1 [25:43<00:00, 1543.35s/it]
  [2025-01-01 21:02:17.612] 
  [2025-01-01 21:02:17.613] 
                                                                                                               [ARequested to load Flux
  [2025-01-01 21:02:17.679] 0 models unloaded.
  [2025-01-01 21:02:17.759] loaded partially 6384.427978515625 6383.381103515625 0
  [2025-01-01 21:02:17.765] Attempting to release mmap (40)
  [2025-01-01 21:02:17.869] 
  [2025-01-01 21:02:17.869] [A
  [2025-01-01 21:02:17.870] [AERROR lora diffusion_model.single_blocks.0.linear2.weight Allocation on device 
  [2025-01-01 21:04:24.271]
  ```
  
  
  ### Other
  
  Total VRAM 8192 MB, total RAM 40352 MB
  pytorch version: 2.5.1+cu124
  Forcing FP16.
  Set vram state to: LOW_VRAM
  Disabling smart memory management
  Device: cuda:0 NVIDIA GeForce RTX 3070 Laptop GPU : cudaMallocAsync
  Using pytorch cross attention
  
  ### Loading: ComfyUI-Manager (V2.55.5)
  ### ComfyUI Version: v0.3.7-13-g44db978 | Released on '2024-12-10'