`provider_options` for running on GPU
#3
by
cecheta
- opened
The GPU microsoft/Phi-3.5-mini-instruct-onnx model was recently updated, however provider_options in genai_config.json is now set to []. When trying to run the model using the package onnxruntime-genai-cuda, it now seems to run on CPU, even if the device has GPU.
The provider_options have been intentionally left blank since the GPU ONNX model can run on both CUDA and DirectML. You can add back the execution provider information in your downloadedgenai_config.json file or you can set that information at runtime (note thatargs.provider can be cpu, cuda, or dml).
kvaishnavi
changed discussion status to
closed