Hi guys,
I’m trying to train a model based on HuggingFace (Pytorch backend) and I have a problem and I was hoping that I might get some help here (there was no answer on HF unfortunately). After a certain number of steps (~8000) my checkpoints only contain a rng_state.pth file:

** Marker unrelated **
Why could this be the case? I sync my checkpoints automatically to resum training if Colab Crashes. Unfortually out of 22 Checkpoints (1-23k) only the first 6 (1k - 7k) are valid (all required data included).
TrainingArguments(
    output_dir=/share/datasets/output_run, 
    overwrite_output_dir=True, 
    do_train=True, 
    do_eval=True, 
    do_predict=False, 
    evaluation_strategy=IntervalStrategy.STEPS, 
    prediction_loss_only=False, 
    per_device_train_batch_size=20, 
    per_device_eval_batch_size=16, 
    gradient_accumulation_steps=1, 
    eval_accumulation_steps=None, 
    learning_rate=0.0001, 
    weight_decay=0.0, 
    adam_beta1=0.9, 
    adam_beta2=0.999, 
    adam_epsilon=1e-08, 
    max_grad_norm=1.0, 
    num_train_epochs=20.0, 
    max_steps=-1, 
    lr_scheduler_type=SchedulerType.LINEAR, 
    warmup_ratio=0.0, 
    warmup_steps=0, 
    logging_dir=runs/May12_05-06-46_a600ce861ff7, 
    logging_strategy=IntervalStrategy.STEPS, 
    logging_first_step=False, 
    logging_steps=1000, 
    save_strategy=IntervalStrategy.STEPS, 
    save_steps=1000, 
    save_total_limit=3, 
    no_cuda=False, 
    seed=42, 
    fp16=True, 
    fp16_opt_level=O1, 
    fp16_backend=auto, 
    fp16_full_eval=False, 
    local_rank=-1, 
    tpu_num_cores=None, 
    tpu_metrics_debug=False, 
    debug=[], 
    dataloader_drop_last=False, 
    eval_steps=1000, 
    dataloader_num_workers=2, 
    past_index=-1, 
    run_name=cv_sm_1, 
    disable_tqdm=False, 
    remove_unused_columns=True, 
    label_names=None, 
    load_best_model_at_end=True, 
    metric_for_best_model=loss, 
    greater_is_better=False, 
    ignore_data_skip=False, 
    sharded_ddp=[], 
    deepspeed=None, 
    label_smoothing_factor=0.0, 
    adafactor=False, 
    group_by_length=True, 
    length_column_name=length, 
    report_to=['wandb'], 
    ddp_find_unused_parameters=None, 
    dataloader_pin_memory=True, 
    skip_memory_metrics=False, 
    use_legacy_prediction_loop=False, 
    push_to_hub=False, 
    resume_from_checkpoint=None, 
    _n_gpu=1, 
    mp_parameters=
)


Ty in advcane
Edit://
According to the Trainer Output the File are created:
Saving model checkpoint to /share/datasets/output_run/checkpoint-24000
Configuration saved in /share/datasets/output_run/checkpoint-24000/config.json
Model weights saved in /share/datasets/output_run/checkpoint-24000/pytorch_model.bin
Configuration saved in /share/datasets/output_run/checkpoint-24000/preprocessor_config.json

But i could not validate that using File Explorer, ls /share/datasets/output_run/checkpoint-24000/.
I also tried to search on the entire Drive !find / -name optimizer.pt but i could only find pytorch_model.bin etc on the “valid” Checkpoints i mentioned above (1-7k), and there fore only for Checkpoints < 7000.