Both tensors will allocate 2MB of memory (8 * 8192 * 8 * 4 / 1024**2 = 2.0MB) and the result will use 2.0GB, which would fit your last error message. You could run this code snippet to verify it:
a = torch.randn(8, 8192, 8, device='cuda')
b = torch.randn(8, 8, 8192, device='cuda')
print('{:.3f}MB'.format(torch.cuda.memory_allocated()/1024**2))
print(torch.cuda.memory_summary())

c = torch.matmul(a, b)
print(c.shape)
print('{:.3f}MB'.format(torch.cuda.memory_allocated()/1024**2))
print(torch.cuda.memory_summary())

However, this doesn’t fit the initial description of the error which claims you are trying to allocate 256GB of memory, so you would have to narrow down the operation further, as it’s not caused by this matrix multiplication with these tensor shapes.