If the matrices are square with same dimensions, yes the time complexity of matrix manipulation’s are similar.
Coming to GPU or CPU is faster. From my personal experience if you are using very very small networks and you have a higher core processor which can fit your data, you can go for CPU, in this case the device to host and host to device data transfer overhead seems higher compared to calculations. But when you have a larger model like 2D/3D CNNs etc,. You choose to GPU for parallel tensor computations.