Hi @AlanFeder,
I’m not familiar with imbalanced datasets, but if I were you, I would try using examples/text-classification/run_glue.py 10.
In this example, instead of assigning GLUE benchmarks task names, we can use our own train_file, validation_file, and test_file.
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --train_file train_file_name \
  --validation_file validation_file_name \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /path/to/output/

You can use CSV or JSON.
datasets.load_dataset is used in the script, so the document of datasets.load_dataset for local files 1 may help you.
I remember that columns of the following format were easier to handle, but even if the names or order of the columns are different, it may work if the code is rewritten appropriately.
(Please see transformers/run_glue.py at 5ed5a54684ef059fa4c9710858b8e03c61295914 · huggingface/transformers · GitHub 3 for the detail.)
sentence1, sentence2, label

or
sentence, label

Again, I’m not familiar with imbalanced datasets, so I don’t think I’ve answered your question “do I have to fine-tune?” Sorry.
I am just one of the users who is learning about this library, so I hope you can get more useful advice from someone who knows more than I do.