text: Title: Intermittent failing of ONNX model
  Body: # Bug Report
  
  ### Describe the bug
  I have a script from compiling a `pytorch` model to ONNX that runs inference with the ONNX model, and when running inference on the GPU, it intermittently fails with the error: 
  
  ```File "/home/ec2-user/anaconda3/envs/onnx/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
  return self._sess.run(output_names, input_feed, run_options)
  
  onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Expand node. Name:'/Expand_591' Status Message: /Expand_591: left operand cannot broadcast on dim 0 LeftShape: {243}, RightShape: {267}.
  ```
  
  Some additional notes: 
  
  1. In the script (see below), I'm running inference 10x (via a for loop). When it fails, it fails on the first iteration of the for loop and crashes the script. But, if I re-run the script, it sometimes doesn't fail on that first iteration and completes successfully. Thus, the intermittent nature here seems to be between iterations of the script, _not between iterations of the for loop_. 
  2. Each time it runs into the error, it does have the `Expand_591` node called out, and the `RightShape {267}` remains the same. However, the `LeftShape` (243 in the error example above) changes. 
  
  
  ### System information
  - OS Platform and Distribution (*e.g. Linux Ubuntu 20.04*): 
  <img width="317" alt="image" src="https://github.com/onnx/onnx/assets/124316637/20ccd564-5d0b-4515-a3e1-09fb27b5eb36">
  
  - ONNX version (*e.g. 1.13*): 
  <img width="235" alt="image" src="https://github.com/onnx/onnx/assets/124316637/c1680d8f-17f0-4239-910f-e84c69ac1a2d">
  
  - Python version: 3.10.6
  - Torch version 
  <img width="158" alt="image" src="https://github.com/onnx/onnx/assets/124316637/ef625eb7-6ff7-4487-bf61-5a93e3afcd1f">
  
  
  ### Reproduction instructions
  
  Script I'm using to test (with private details removed): 
  
  ```
  import onnx
  import onnxruntime
  import torch
  import numpy as np
  
  device = torch.device("cuda")
  input_tensor = torch.randn(1, 3, 1280, 896)
  input_tensor = input_tensor.to(device)
  
  def to_numpy(tensor):
        return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
  
  onnx_model = onnx.load("exp_06_aug_stacked_strong_v5_step_50_epoch_69.onnx")
  onnx.checker.check_model(onnx_model)
  
  ort_session = onnxruntime.InferenceSession(
      "exp_06_aug_stacked_strong_v5_step_50_epoch_69.onnx",
      providers=['CUDAExecutionProvider']
  )
  # compute ONNX Runtime output prediction
  ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(input_tensor)}
  for idx in range(10):
      ort_outs = ort_session.run(None, ort_inputs)
  ```
  
  ### Expected behavior
  I would expect the model to run successfully each time and not intermittently fail. 
  
  ### Notes
  We got several different flavors of warnings when compiling: 
  
  - TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  - TracerWarning: torch. tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. 
  - TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  - When we compiled the model on the GPU but ran on the CPU, it ran successfully each time. However, it did not produce the same results as the underlying pytorch model. 
  - When we compiled this same model on CPU and tested using the `CPUExecutionProvider`, we ran into this error 100% of the time:
  
  ``` 
  2024-02-08 22:53:05.966710901 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running Gather node. 
  Name:'/Gather_2452' Status Message: indices element out of data bounds, 
  idx=264 must be within the inclusive range [-264,263] 
  
  Traceback (most recent call last): File "/home/ec2-user/projects/onnx/test_onnx_model.py", line 60, 
  in <module> ort_outs = ort_session.run(None, ort_inputs) 
  File "/home/ec2-user/anaconda3/envs/onnx/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run return self._sess.run(output_names, input_feed, run_options)
  ``` 