Overall it looks fine. In this part:
E_global_context = torch.cat([E_output[:,-1,0,:], E_output[:,0,1,:]],-1)

I was wondering should it be instead:
E_global_context = torch.cat([E_output[:,-1,0,:], E_output[:,-1,1,:]],-1)