I tried your commandline (with teacher)–
--student_decoder_layers 3 --student_encoder_layers 6 --tokenizer_name t5-small \
--learning_rate=3e-4 --freeze_encoder --no_teacher --freeze_embeds \
--do_train --train_batch_size 32 \
--do_predict --n_train 100 \
--model_name_or_path t5-small --eval_beams 2 --eval_max_gen_length 142 \
--val_check_interval 0.25 --n_val 1000 \
--output_dir distilt5 --gpus 1 --logger_name wandb

I get–
Traceback (most recent call last):
  File "/home/sumithrab/transformers/src/transformers/configuration_utils.py", line 349, in get_config_dict
    resolved_config_file = cached_path(
  File "/home/sumithrab/transformers/src/transformers/file_utils.py", line 832, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file t5-small/config.json not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "distillation.py", line 306, in <module>
    distill_main(args)
  File "distillation.py", line 296, in distill_main
    model = create_module(args)
  File "distillation.py", line 263, in create_module
    model = module_cls(args)
  File "/home/sumithrab/transformers/examples/seq2seq/finetune.py", line 63, in __init__
    super().__init__(hparams, num_labels=None, mode=self.mode, **kwargs)
  File "/home/sumithrab/transformers/examples/lightning_base.py", line 83, in __init__
    self.config = AutoConfig.from_pretrained(
  File "/home/sumithrab/transformers/src/transformers/configuration_auto.py", line 318, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/sumithrab/transformers/src/transformers/configuration_utils.py", line 368, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 't5-small'. Make sure that:

- 't5-small' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-small' is the correct path to a directory containing a config.json file