Necessary resources for training a (small/tiny) LM from scratch?
This is mostly a follow up question in regards to this hugging face blog post on training a LM (and a tokenizer) from scratch :
https://huggingface.co/blog/how-to-train

I think this may be an ideal approach to try out in my situation, but I'm wondering about cost and how much data I really need to train a LM from scratch on my domain-specific dataset? 

I'm quite new to the field and haven't read many papers on this subject as of yet, so I was hoping someone might be able to provide some ballpark estimates about computing resources required for training some small LM(s) from scratch? I'd like to obtain a fine-tuned (or trained from scratch) domain-specific LM to serve as a backbone for various downstream NLP tasks on my domain-specific text data. I have been experimenting with the fine-tuning LM approach, (i.e. fine-tuning BERT based models on MLM before performing task-specific fine-tuning), but I'm curious about the training from scratch option if I can get a rough idea the required compute resources / cost.

Thanks very much in advance for any help / tips on unpacking this question.
Hi,

could you please ask this question on the [forum](https://discuss.huggingface.co/) rather than here? We like to keep Github issues for bugs/feature requests.

Thanks!<|||||>Sure, thanks @NielsRogge . Here is a [link to the post on the forum](https://discuss.huggingface.co/t/necessary-resources-for-training-a-small-tiny-lm-from-scratch/8139), cheers.