text: Title: [Bug]: Executor performance degradation
  Body: ### Your current environment
  
  <details>
  <summary>The output of `python collect_env.py`</summary>
  
  ```text
  Versions of relevant libraries:
  [pip3] numpy==1.26.4
  [pip3] nvidia-cublas-cu12==12.1.3.1
  [pip3] nvidia-cuda-cupti-cu12==12.1.105
  [pip3] nvidia-cuda-nvrtc-cu12==12.1.105
  [pip3] nvidia-cuda-runtime-cu12==12.1.105
  [pip3] nvidia-cudnn-cu12==9.1.0.70
  [pip3] nvidia-cufft-cu12==11.0.2.54
  [pip3] nvidia-curand-cu12==10.3.2.106
  [pip3] nvidia-cusolver-cu12==11.4.5.107
  [pip3] nvidia-cusparse-cu12==12.1.0.106
  [pip3] nvidia-ml-py==12.560.30
  [pip3] nvidia-nccl-cu12==2.20.5
  [pip3] nvidia-nvjitlink-cu12==12.6.85
  [pip3] nvidia-nvtx-cu12==12.1.105
  [pip3] pyzmq==26.2.0
  [pip3] torch==2.4.0
  [pip3] torch_tensor_module==0.0.0
  [pip3] torchvision==0.19.0
  [pip3] transformers==4.47.0
  [pip3] triton==3.0.0
  [conda] numpy                     1.26.4                   pypi_0    pypi
  [conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
  [conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
  [conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
  [conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
  [conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
  [conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
  [conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
  [conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
  [conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
  [conda] nvidia-ml-py              12.560.30                pypi_0    pypi
  [conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
  [conda] nvidia-nvjitlink-cu12     12.6.85                  pypi_0    pypi
  [conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
  [conda] pyzmq                     26.2.0                   pypi_0    pypi
  [conda] torch                     2.4.0                    pypi_0    pypi
  [conda] torch-tensor-module       0.0.0                    pypi_0    pypi
  [conda] torchvision               0.19.0                   pypi_0    pypi
  [conda] transformers              4.47.0                   pypi_0    pypi
  [conda] triton                    3.0.0                    pypi_0    pypi
  ROCM Version: Could not collect
  Neuron SDK Version: N/A
  vLLM Version: 0.6.3.post1
  ```
  
  </details>
  
  
  ### 🐛 Describe the bug
  
  I am currently using `vllm serve` to create an online server and using this server as a Scheduler to send and execute `ExecuteModelRequest` to one `LLM` instances created with `self.llm = LLM(model=self.model, gpu_memory_utilization=self.gpu_memory_utilization, enforce_eager=self.enforce_eager, tensor_parallel_size=self.tp_size)`. The inference is then performed by calling `self.llm.llm_engine.model_executor.execute_model(req)`. However, when I test using `vllm/benchmarks/benchmark_serving.py`, I notice a significant drop in inference performance, with both TTFT and TPOT increasing substantially, while directly using `vllm serve` for inference does not exhibit this issue. Is it possible that `vllm` has implemented some caching or optimizations above the Executor layer in the `LLMEngine`?
  
  ### Before submitting a new issue...
  
  - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.