Hi ptrblck,
I tried to implement what you suggests, and I get the following error:
File "/home/qianlim/dl_signal/transformer/modules/position_embedding.py", line 85, in forward
    self.positions = make_positions.range_buf[:input.size(1)].expand_as(input)
RuntimeError: The expanded size of the tensor (512) must match the existing size (10) at non-singleton dimension 0.  Target sizes: [512, 10].  Tensor sizes: [10, 10]