Spaces:

---

Dataset-Tools

/

Spark-Notebooks

like 9

Running

Hugging Face Spaces provide single node applications, therefore Spark Notebooks are limited to single node at the moment.
This is fine in most cases, since single node is generally capable of handling fine-tuning datasets.

This doesn't scale for bigger datasets though, for pretraining datasets for example.

There might be a way to solve this though. It is possible to run jobs on Hugging Face in a serverless fashion (e.g. using hfjobs). What do you think ?