Hi Thomas, thanks again for the input. In reply to your two points:

Thanks, this was left over from a previous version of the function and is an unecessary loop, I will remove it
This would be the ideal way forward, I agree, and I have been search for a solution in this direction. Unfortunately I haven’t been able to rewrite the expectation of a loss function (such as the squared error between terms in the expression in my original post above) as a loss function applied to an expectation. In the most general case, Jensen’s inequality of course tells us that (when using a convex loss function) the latter term will be less than or equal to the first, and so bounding the latter gives no guarantee on bounding the former, although it maybe the case that there is a trick to get round this here?

I realise this is diving a little more into the maths than the Pytorch implementation at this point, but I’m not so experienced with translating the first into the second, so I could definitely be missing some obvious things here!