OMG. I noticed that view's behavior was not that I thought.
I needed to use concat for this.
# h = h.view(batch_size, -1) # (bs, 2H)
h = torch.cat([hh for hh in h], -1) # (bs, 2H)

This is a simple example.
In [93]: h=torch.randn(2,3,4)

In [94]: h
Out[94]:

(0 ,.,.) =
 -0.3609 -1.3455 -0.5885  0.2888
  0.5073  1.5901  0.5655 -0.0937
  0.4892  0.7472 -0.4066 -0.8092

(1 ,.,.) =
 -1.0095  0.1905 -0.1127  2.2638
 -0.7856 -0.0417 -0.0152  1.2649
  0.6107  1.5546  0.5480  0.6572
[torch.FloatTensor of size 2x3x4]

In [96]: torch.cat([hh for hh in h], -1)  // Correct
Out[96]:

-0.3609 -1.3455 -0.5885  0.2888 -1.0095  0.1905 -0.1127  2.2638
 0.5073  1.5901  0.5655 -0.0937 -0.7856 -0.0417 -0.0152  1.2649
 0.4892  0.7472 -0.4066 -0.8092  0.6107  1.5546  0.5480  0.6572
[torch.FloatTensor of size 3x8]

In [97]: h.view(3, -1) // Wrong
Out[97]:

-0.3609 -1.3455 -0.5885  0.2888  0.5073  1.5901  0.5655 -0.0937
 0.4892  0.7472 -0.4066 -0.8092 -1.0095  0.1905 -0.1127  2.2638
-0.7856 -0.0417 -0.0152  1.2649  0.6107  1.5546  0.5480  0.6572
[torch.FloatTensor of size 3x8]