I actually figured what is happening.
I set the weights to 1 and pass layers through ELU. This generates very high values for embeddings resulting in very high values for logits.
The different between Linear and Conv implementations become negligible if you consider the magnitude of the logits.
Simply change ELU to Sigmod (a squashing activation) and the intial weight to 0.001. The difference between Linear and Conv implementations is going to be small.