Train model from scratch with Tensorflow
First of all thank you for a great library. I am quite new to the library so sorry if my questions sound dumb. I see a tutorial to train a model from scratch (https://huggingface.co/blog/how-to-train) which is very useful. My questions are:

1/ It seems like that tutorial is for training using PyTorch. Is there any tutorial for Tensorflow?

2/ I am thinking about training an ALBERT model with just 6 layers. If possible I want to use the weights of the pre-trained ALBERT (since all layers share parameters) and also the structure but just change the number of layers from 12 to 6. Is it possible or I need to train entirely from scratch?
> 1/ It seems like that tutorial is for training using PyTorch. Is there any tutorial for Tensorflow?

[Here](https://github.com/stefan-it/turkish-bert/blob/master/CHEATSHEET.md) is a cheatsheet for pretraining using TF code. It is written for BERT but the training for ALBERT is similar. 

Regarding pretraining code for ALBERT in TensorFlow, you will find relevant resources in [the official ALBERT repository](https://github.com/google-research/ALBERT). You will need a corpus in txt format with one sentence per line. Ideally after splitting the corpus in smaller shards, you can use the `create_pretraining_data.py` file to transform the txt files in tfrecords. After that, you can run `run_pretraining.py` to pretrain your model. 

> 2/ I am thinking about training an ALBERT model with just 6 layers. If possible I want to use the weights of the pre-trained ALBERT (since all layers share parameters) and also the structure but just change the number of layers from 12 to 6. Is it possible or I need to train entirely from scratch?

This makes me think of what @VictorSanh et al. did in [DistilBERT](https://arxiv.org/abs/1910.01108), namely initializing with every other layer from a vanilla BERT base. The difference is they added a distillation loss for the student to learn from the teacher. I'm not sure whether you can modify the number of layers in the configuration (that could be) but pretraining with less layers might severely impact your model performance. One thing you could do to maximize performance is to pretrain a normal 12-layer ALBERT base and then distil it (more on how to do this [here](https://github.com/huggingface/transformers/tree/master/examples/distillation)). <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hello, Has there been anything done on this issue, if yes can you please share?