Since the result of torch.matmul(a, c) will have the shape [1, 49920, 49920], it will use approx. 9.5GB of memory.
Given that the CUDA context uses a bit of memory as well as a and b, the usage of 10GB seems reasonable.