James Clarke

Ah we can work with that, then the issue is that the space is incomplete/misconfigured, (i would reccomend amending your original post to avoid confusion).

I just read your blog post: https://huggingface.co/blog/nroggendorff/train-with-llama-architecture

It provides some useful context, thanks.

From reading the dockerfile and image file, it appears that cuda was never included in the image.

You may find the following resources helpful for using docker with spaces:
https://huggingface.co/docs/hub/en/spaces-sdks-docker

If you are using cuda, this may also help inform on how to setup cuda, and also test if cuda works (with docker):
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

References
https://huggingface.co/spaces/nroggendorff/train-llama/blob/main/Dockerfile

https://hub.docker.com/layers/nroggendorff/train-llama/latest/images/sha256-8cd7859f8a7cc3b669b344e87fa342e3c464e449141e267fbb52cfb48c32310f

Hope you find this helpful,
Let me know if you have any more questions, let me know here or email me.

@hf-staff Do you have any insight into what might be the issue here?

I am not sure if that makes sense, I am under the impression that, if the space is not running(not started), no models can be actively loaded in the space.

Can you share your relevant workflow(docker-compose, app code, etc) so i can see more clearly whats happening?

I might be able to aid in a solution, its possible that there is an issue in the workflow itself.

EDIT: I looked at the spaces, Do you mean this space as an example? 'https://huggingface.co/spaces/nroggendorff/train-llama'
Because this space shows a missing "CUDA_HOME" env var, most your other spaces throwing errors about missing CUDA drivers or are paused. These are configuration errors.

Could you tell me the space and error message?
I might be able to help you fix it.

What policies? all my spaces that use docker work fine.
And i got no email about policy changes.

Please elaborate.