Ok. Got it working, seems like I messed up a git merge and re inserted the code that
loads the pre-trained embeddings after initializing the model 
For some reason after struglling with this for some days writing a post here made
me find the bug   Now the difference is within the float accuracy.