[512, 46, 300] doesn’t work, as the first conv expects the input to have a single input channel.
If I use a single input channel, the activation is too small after a certain layer:
Given input size: (1x1x1). Calculated output size: (1x1x0). Output size is too small

Using an input of x = torch.randint(0, 100, (512, 46)) and initializing the embedding via nn.Embedding(100, 300), yields:
size mismatch, m1: [512 x 12], m2: [9 x 100] at /opt/conda/conda-bld/pytorch_1581149526594/work/aten/src/TH/generic/THTensorMath.cpp:41

Could you post an executable code snippet, please?