GPT2 - support data type torch.DoubleTensor for Position embedding
Hi,

Could the team add support for floating data types for the position embedding? Currently, it only allows torch.LongTensor between 0 - config.n_positions - 1 and must be of the same shape as the input. I see this a restriction as one may use floating values between e.g. 0-2 to represent the sequence.

What would be even better, if the team could add build-in position encoding techniques like the one used in Attention Is All You Need paper, where they use cos and sin: 
PE(pos,2i) = sin(pos/10000**(2i/dmodel))
PE(pos,2i+1) = cos(pos/10000**(2i/dmodel))

@thomwolf Could you please share your thoughts with me on that?

Regards,
Adrian.

Hi, we won't be able to add these feature as they would render the PyTorch model not compatible with the pretrained model open-sourced by OpenAI.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
