text: Title: [Bug]: Unit test `tests/models/embedding/vision_language/test_phi3v.py` failing
  Body: ### Your current environment
  
  <details>
  <summary>The output of `python collect_env.py`</summary>
  
  ```text
  Your output of `python collect_env.py` here
  Collecting environment information...                                                                                            
  PyTorch version: 2.5.1+cu124                                                                                                     
  Is debug build: False                                                                                                            
  CUDA used to build PyTorch: 12.4                                                                                                 
  ROCM used to build PyTorch: N/A                                                                                                  
                                                                                                                                   
  OS: Ubuntu 24.04.1 LTS (x86_64)                                                                                                  
  GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0                                                                               
  Clang version: 18.1.3 (1ubuntu1)                                                                                                 
  CMake version: version 3.28.3                                                                                                    
  Libc version: glibc-2.39                                                                                                         
                                                                                                                                   
  Python version: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] (64-bit runtime)                
  Python platform: Linux-6.8.0-47-generic-x86_64-with-glibc2.39                                                                    
  Is CUDA available: True                                                                                                          
  CUDA runtime version: 12.6.77                                                                                                    
  CUDA_MODULE_LOADING set to: LAZY                                                                      
  Nvidia driver version: 550.127.05                                                                                                
  cuDNN version: Could not collect                                                                                                 
  HIP runtime version: N/A                                                                                                         
  MIOpen runtime version: N/A                                                                                                      
  Is XNNPACK available: True    
  
  Versions of relevant libraries:                                                                                                  
  [pip3] mypy-extensions==1.0.0                                                                                                    
  [pip3] numpy==1.26.4                                                                                                             
  [pip3] nvidia-cublas-cu12==12.4.5.8                                                                                              
  [pip3] nvidia-cuda-cupti-cu12==12.4.127                                                                                          
  [pip3] nvidia-cuda-nvrtc-cu12==12.4.127                                                                                          
  [pip3] nvidia-cuda-runtime-cu12==12.4.127                                                                                        
  [pip3] nvidia-cudnn-cu12==9.1.0.70                                                                                               
  [pip3] nvidia-cufft-cu12==11.2.1.3                                                                                               
  [pip3] nvidia-curand-cu12==10.3.5.147                                                                                            
  [pip3] nvidia-cusolver-cu12==11.6.1.9                                                                                            
  [pip3] nvidia-cusparse-cu12==12.3.1.170                                                                                          
  [pip3] nvidia-nccl-cu12==2.21.5                                                                                                  
  [pip3] nvidia-nvjitlink-cu12==12.4.127                                                                                           
  [pip3] nvidia-nvtx-cu12==12.4.127                                                                                                
  [pip3] pyzmq==26.2.1                                                                                                             
  [pip3] sentence-transformers==3.2.1                                                                                              
  [pip3] torch==2.5.1                                                                                                              
  [pip3] torchaudio==2.5.1                                                                                                         
  [pip3] torchvision==0.20.1                                                                                                       
  [pip3] transformers==4.49.0                                                                                                      
  [pip3] transformers-stream-generator==0.0.5                                                                                      
  [pip3] triton==3.1.0                                                                                                             
  [pip3] tritonclient==2.51.0                                                                                                      
  [pip3] vector-quantize-pytorch==1.21.2 
  [conda] numpy                     1.26.4                   pypi_0    pypi                                                        
  [conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi                                                        
  [conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi                                                        
  [conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi                                                        
  [conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi                                                        
  [conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi                                                        
  [conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi                                                        
  [conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi                                                        
  [conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi                                                        
  [conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi                                                        
  [conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi                                                        
  [conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi                                                        
  [conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi                                                        
  [conda] pyzmq                     26.2.1                   pypi_0    pypi                                                        
  [conda] sentence-transformers     3.2.1                    pypi_0    pypi                                                        
  [conda] torch                     2.5.1                    pypi_0    pypi                                                        
  [conda] torchaudio                2.5.1                    pypi_0    pypi                                                        
  [conda] torchvision               0.20.1                   pypi_0    pypi                                                        
  [conda] transformers              4.49.0                   pypi_0    pypi                                                        
  [conda] transformers-stream-generator 0.0.5                    pypi_0    pypi                                                    
  [conda] triton                    3.1.0                    pypi_0    pypi
  [conda] tritonclient              2.51.0                   pypi_0    pypi
  [conda] vector-quantize-pytorch   1.21.2                   pypi_0    pypi
  ROCM Version: Could not collect
  Neuron SDK Version: N/A
  vLLM Version: 0.1.dev5072+g63d635d (git sha: 63d635d
  vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
  LD_LIBRARY_PATH=miniconda3/envs/vllm073/lib/python3.12/site-packages/cv2/../../lib64:/usr/lib/x86_64-linux-gnu/openmpi/lib/:/usr/local/cuda-12.6/lib64:
  NCCL_CUMEM_ENABLE=0
  TORCHINDUCTOR_COMPILE_THREADS=1
  CUDA_MODULE_LOADING=LAZY
  ```
  
  </details>
  
  
  ### 🐛 Describe the bug
  
  BUG: unittest `tests/models/embedding/vision_language/test_phi3v.py` is failiing
  
  After installing vllm, run
  
  `pytest -sv tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full]`
  
  A snippet of unittest failure log
  ```
  tests/models/embedding/vision_language/test_phi3v.py:119:                                                                        
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  tests/models/embedding/vision_language/test_phi3v.py:69: in _run_test                                                            
      check_embeddings_close(                                                                                                      
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                                                                                                                                   
      def check_embeddings_close(                                                                                                  
          *,                                                                                                                       
          embeddings_0_lst: Sequence[list[float]],                                                                                 
          embeddings_1_lst: Sequence[list[float]],                                                                                 
          name_0: str,                                                                                                             
          name_1: str,                                                                                                             
          tol: float = 1e-3,                                                                                                       
      ) -> None:                                                                                                                   
          assert len(embeddings_0_lst) == len(embeddings_1_lst)                                                                    
                                                                                                                                   
          for prompt_idx, (embeddings_0, embeddings_1) in enumerate(                                                               
                  zip(embeddings_0_lst, embeddings_1_lst)):                                                                        
              assert len(embeddings_0) == len(embeddings_1), (                                                                     
                  f"Length mismatch: {len(embeddings_0)} vs. {len(embeddings_1)}")                                                 
                                                                                                                                   
              sim = F.cosine_similarity(torch.tensor(embeddings_0),                                                                
                                        torch.tensor(embeddings_1),                                                                
                                        dim=0)                                                                                     
                                                                                                                                   
              fail_msg = (f"Test{prompt_idx}:"                                                                                     
                          f"\n{name_0}:\t{embeddings_0[:16]!r}"                                                                    
                          f"\n{name_1}:\t{embeddings_1[:16]!r}")   
  >           assert sim >= 1 - tol, fail_msg
  E           AssertionError: Test0:
  E           hf: [0.0005359649658203125, 0.0028438568115234375, -0.033050537109375, 0.009765625, 0.003566741943359375, 0.004444122314453125, -0.0088958740234375, -0.032867431640625, -0.006267547607421875, -0.021881103515625, 0.021087646484375, 0.0010557174682617188, 0.0147247314453125, -0.004642486572265625, 0.0032863616943359375, 0.006305694580078125]
  E           vllm:       [0.0029010772705078125, 0.005504608154296875, -0.030792236328125, 0.00962066650390625, 0.0016222000122070312, 0.0036258697509765625, -0.00925445556640625, -0.03411865234375, -0.005634307861328125, -0.0216522216796875, 0.022430419921875, 0.0002880096435546875, 0.016082763671875, -0.00467681884765625, 0.004161834716796875, 0.004535675048828125]
  
  tests/models/embedding/utils.py:32: AssertionError
  ------------------------------------------------------- Captured log call -------------------------------------------------------
  WARNING  transformers_modules.microsoft.Phi-3.5-vision-instruct.4a0d683eba9f1d0cbfb6151705d1ee73c25a80ca.modeling_phi3_v:logging.py:329 You are not running the flash-attention implementation, expect numerical differences.
  ======================================================= warnings summary ========================================================
  tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full]
    /home/tan/miniconda3/envs/vllm073/lib/python3.12/site-packages/transformers/models/auto/image_processing_auto.py:594: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
      warnings.warn(
  
  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  ==================================================== short test summary info ====================================================
  FAILED tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full] - AssertionError: Test0:
  =========================================== 1 failed, 1 warning in 311.72s (0:05:11) ============================================
  
  ### Before submitting a new issue...
  
  - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.