QUESTION: How do I know what type of positional encoding to input during fine-tuning or pretrained BERT?
Hello HuggingFace team,

I am familiarizing myself with the HuggingFace tutorials and understand the functionality of the various methods. However, I have a general question for example when using models like BERT.

Considering that I am doing sentiment classification and I want to fine-tune the whole BERT based on pre-trained weights, how do I know what should be the positional encoding as input during the `forward ( )` method? 

I know it has a default value of None, but doesn't it mean that during fine-tuning I need to input the same value that it was originally trained on in the first place? If so, how do I know what it was trained on during its original training from scratch? Is there a documentation for that somewhere?

Following to this, if I am freezing the weights of BERT for sentiment classification, I again have the same question on what should be my positional encoding input to the `forward ( )` method.

Please clarify this. Thanks for your time!


Hi! If you're new to the library, I heavily recommend taking a look at the [glossary (position IDs in this case)](https://huggingface.co/transformers/glossary.html#position-ids), which explains how to use such inputs.

If you ignore the `position_ids`, then they're always automatically generated to be the same as the model's pre-training scheme. If you're fine-tuning and wish to keep the same position embeddings, then you don't need to pass them to the model.<|||||>THanks @LysandreJik , your second statement pretty much answered it completely! Thanks<|||||>@LysandreJik Hi, does that mean the position embeddings won't get updated if `position_ids` are not passed to the model (or with the default value of `None`)? Would you point me to the related lines in the code that implements this logic? Thanks!<|||||>This means that the position IDs will be generated on the fly, and the position embeddings will be exactly the same than during the pre-training. You can check the code [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L192-L214).