In pre-training a model how can I resume from last saved model?
Using flax example for training a T5 model from scratch, I am training a model using:

```
thon run_t5_mlm_flax.py \
	--output_dir="/content/drive/MyDrive/Pouramini/" \
	--cache_dir="/content/drive/MyDrive/cache/" \
	--preprocessing_num_workers="20" \
  --model_type="t5" \
	--config_name="/content/drive/MyDrive/Pouramini/" \
	--tokenizer_name="/content/drive/MyDrive/Pouramini/" \
	--dataset_name="oscar" \
	--dataset_config_name="unshuffled_deduplicated_fa" \
	--max_seq_length="256" \
	--per_device_train_batch_size="32" \
	--per_device_eval_batch_size="32" \
	--adafactor \
	--learning_rate="0.005" \
	--weight_decay="0.001" \
	--warmup_steps="2000" \
	--logging_steps="500" \
	--save_steps="5000" \
	--eval_steps="50000" \
  --resume_from_checkpoint=True
```

Sometimes, the training interrupts and I must run the command again. But while I have specified `resume_from_checkpoint` it seems it trains from scratch, because the loss it shows is higher than the loss of the last saved model. What did I miss in the code above?
I thought to add:

```
  --model_name_or_path="/content/drive/MyDrive/Pouramini/" \
```
Is it necessary?
cc @sgugger <|||||>It's Flax, so it's more up to @patil-suraj aisle than mine :-)<|||||>`resume_from_checkpoint` is not yet correctly implemented in flax examples.  You could set the `--model_name_or_path` to the last saved checkpoint, this will load the model from that checkpoint, but note that the optimizer state is not saved.
For a complete resume, we need to load both model and optimizer states from the last checkpoint. 

For now, you could find how to do this in [this t5 example ](https://github.com/gsarti/t5-flax-gcp/blob/34def61c98097224b4d9bca6c1e0832cd015f06a/run_t5_mlm_flax.py#L405)<|||||>@patil-suraj  Thank you, I followed the example and added save and restore functions. However, I resumed a model (with 30000 steps) by just adding `--model_name_or_path` and if I interrupt my current running it will again save a model (with another 30000) without saving the optimizer.  I would like to know if the training was okay? is it a serious stage? I mean should I train the model again from scratch or I can still rely on these saved models?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.