Yes, if you are able to reduce the matrix size used in the matmul, you would save computations and could thus speed it up. Depending on the actual layout, the logic to remove parts of the matrix etc. you might be able to see a performance gain.