I’m completely new to Transformers and BERT, and I’m struggling to pull together a functional workflow.
Like the OP, I’m trying to use convert_pytorch_checkpoint_to_tf2.py to convert a pre-trained model from Google Tensorflow BERT format (via TPU) into the pytorch version. I want to go on to fine-tune this model.
But, crucially, my pre-trained model is based on a custom vocabulary - I trained BERT from scratch on my own data. It isn’t like any of the Huggingface pre-trained models.
But I’m trying the Huggingface conversion script as below:
python convert_pytorch_checkpoint_to_tf2.py --model_type=bert --pytorch_checkpoint_path=model.ckpt-20 --config_file=bert_config.json --tf_dump_path=pytorch_model.bin
(The checkpoint is a ‘hello world’ example to help me understand the training workflow).
But I get an error from the conversion script:
File “convert_pytorch_checkpoint_to_tf2.py”, line 402, in convert_all_pt_checkpoints_to_tf
config_class, model_class, pt_model_class, aws_model_maps, aws_config_map = MODEL_CLASSES[model_type]
ValueError: not enough values to unpack (expected 5, got 4)
When I put a diagnostic print(MODEL_CLASSES[model_type]) in the script, I see that the model_type I gave (‘bert’) is being matched against many existing Huggingface pre-trained models. The simple text match finds the string ‘bert’ in the MODEL_CLASSES list but it’s not fully matched against any Huggingface pre-trained model. So I think now that I must specify a Huggingface pre-trained model definition - but how is this used? As some sort of template for further training?
I am obviously missing the point totally here. What I want is to be able to pre-train a model from scratch with Tensorflow on Google TPU, or the equivalent with pytorch, then fine-tune it. Is that possible in the Huggingface ecosystem? Is there a HF script to do the conversion to pytorch format, without the constraint of specifying a pre-trained model? I think maybe I’m at saturation point, continually hitting dependency mismatches etc. as I try to learn to work with BERT.
I’d be very grateful for your advice.