readme: # RotoBART
  
  ## Running the script
  
  ### Script arguemnts
  
  Available model config arguments from script:
  ```
  encoder_layers
  encoder_ffn_dim
  decoder_layers
  decoder_ffn_dim
  d_model
  vocab_size
  max_position_embeddings
  encoder_layerdrop
  decoder_layerdrop
  ```
  
  Training Arguments:
  
  `testing` : only uses 1 batch, for testing the script
  
  `adafactor`: will enable adafactor, removing the command will revert to Adam
  
  `grad_accum`: what value for gradient accumulation to use, default is 4
  
  `use_bf16`: convert the model to bf16
  
  `colab_tpu`: if running on a colab TPU
  
  `use_wandb`: log using Weights & Biases (via Tensorboard)
  
  `save_strategy`: whether or not to save model checkpoints based on steps or epoch
  
  
  ```
  python rotobart/run_dnlm_flax.py \
    --output_dir rotobart_output \
    --overwrite_output_dir \
    --dataset_path rotobart/pile.py \
    --model_name_or_path rotobart \
    --tokenizer_name ./rotobart/vocab-2/the_pile.model \
    --shuffle_buffer_size 1000 \
    --do_train --do_eval \
    --max_seq_length 1024 \
    --encoder_layers 2 \
    --decoder_layers 2 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --logging_steps 8 \
    --num_train_steps 1000 \
    --eval_steps 1000 \
    --save_steps 1000 \
    --save_strategy steps \
    --num_eval_samples 100 \
    --warmup_steps 30 \
    --learning_rate 1e-4 \
    --use_wandb \
    --testing \
    --use_bf16 \
    --adafactor
  ```
  
  alt
  
  ```
  python3 run_dnlm_flax.py   --output_dir rotobart_output   --overwrite_output_dir   --dataset_path pile.py   --model_name_or_path rotobart   --tokenizer_name vocab-2/the_pile.model   --shuffle_buffer_size 1000   --do_train --do_eval --max_position_embeddings 2048  --max_seq_length 2048   --encoder_layers 6   --decoder_layers 6   --per_device_train_batch_size 1   --per_device_eval_batch_size 1   --logging_steps 100   --num_train_steps 50000   --eval_steps 2500   --save_steps 2500   --save_strategy steps   --num_eval_samples 5000   --warmup_steps 5000   --learning_rate 1e-4   --use_wandb   --use_bf16   --adafactor
  ```
  