text: Title: [CG, Core] Illegal memory access with Ray 2.44 and vLLM v1 pipeline parallelism
  Body: ### What happened + What you expected to happen
  
  We got the following errors when running vLLM v1 PP>1 with Ray 2.44. It was working fine with Ray 2.43.
  
  ```
  ERROR 03-21 10:34:30 [core.py:343]   File "/home/ray/default/vllm/vllm/v1/worker/gpu_model_runner.py", line 1026, in execute_model
  ERROR 03-21 10:34:30 [core.py:343]     self.intermediate_tensors[k][:num_input_tokens].copy_(
  ERROR 03-21 10:34:30 [core.py:343] RuntimeError: CUDA error: an illegal memory access was encountered
  ERROR 03-21 10:34:30 [core.py:343] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
  ERROR 03-21 10:34:30 [core.py:343] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
  ERROR 03-21 10:34:30 [core.py:343] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
  ```
  
  ### Versions / Dependencies
  
  - Python 3.11
  - CUDA 12.4
  - NVIDIA L4 / L40S GPUs
  - Ray 2.44
  - vLLM 0.8.1 (or any newer commits)
  
  ### Reproduction script
  
  ```python
  from vllm import LLM, SamplingParams
  
  # Sample prompts.
  prompts = [
      "Hello, my name is",
      "The president of the United States is",
      "The capital of France is",
      "The future of AI is",
  ]
  # Create a sampling params object.
  sampling_params = SamplingParams(temperature=0.0, max_tokens=50)
  
  # Create an LLM.
  llm = LLM(
      model="Qwen/Qwen2.5-0.5B-Instruct",
      distributed_executor_backend="ray",
      pipeline_parallel_size=2,
      enforce_eager=False,
  )
  
  # Generate texts from the prompts. The output is a list of RequestOutput objects
  # that contain the prompt, generated text, and other information.
  outputs = llm.generate(prompts, sampling_params)
  
  # Print the outputs.
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  ```
  
  Run the script with
  ```
  VLLM_USE_V1=1 python run.py
  ```
  
  ### Issue Severity
  
  High: It blocks me from completing my task.