Solved by ptrblck in post #2 
              
              
                This should work: 
B, H, V = 2, 3, 4

A1 = torch.randn([B, H, V])
A2 = torch.randn([H])
A2 = A2[None, :, None].expand(B, -1, -1)
print(A2.shape)
out = torch.cat((A1, A2), dim=2)
print(out.shape)
torch.Size([2, 3, 5])