I found out why it was so unstable: My input-tensors were just to small (e.g. 1xCx12x12).
Seems like, that measuring very small tensors is just way to noisy as the latency is somewhere in the micro-seconds range.
I ended up multiplying the spatial-size with a constant factor of 8 for “measuring” the latency of my modules. In this way it will of course not reflect the real latency, but at least I can compare different modules for my NAS algorithm now.