Oh, I see. Well, that could be explainable by the time it takes to copy data to and from the gpu.  But I have trouble visualizing how you’re rewriting the layers to cause such a copy?  Perhaps you can paste a short representative working example of how such code looks?