Ah! I found something: https://medium.com/@awildtaber/building-a-rendering-engine-in-tensorflow-262438b2e062 37
The first part should help with 2D, and the second with 3D. Not sure if the writer of that article was using a GPU, but it should help a lot.