Thanks so far!
Seems to work now, but I am getting an error while slicing my indices tensor after the model was calibrated and quantized. The slicing worked every time before the quantization itself.
emb(Xi[:, i - self.num, :])

Error:
RuntimeError: Expect weight, indices, and offsets to be contiguous.

Using torch.LongTensor(128, 1).random_(0, 10) which leads to the same tensor shape as input works, but the tensor slicing seems to make problems.
Any suggenstions?