Hi Tomas, thanks very much for your suggestions!
I can imagine that matmul involves many operations for this special case, and after trying einsum the memory cost is comparable to Batch strategy 2 now. It is really surprising!
For people who encounter a similar problem, here is what I did
# Batch strategy 1' ===> optimizes Batch strategy 1                                                                                                                                                                   
x1 = x.view(8192, 8, 64)  # 512 = 8 * 64                                                                                                                                           
W1 = torch.FloatTensor(8, 64, 64).normal_().cuda()                                                                                                                                   
out1 = torch.einsum('bij,abj->abi', (W1, x1))