Solved by supriyar in post #15 
              
              
                It is possible to set the qconfig of Embeddings to None if you wish to skip quantizing them. For example 
class EmbeddingWithLinear(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.emb = torch.nn.Embedding(num_embeddings=10, embedding_dim=12)
…