For corrections, perhaps enforce an order:
y0 = globalCnn(x; pars)
y1 = f(y0, localCnn(x_patch[0]; lpars)) + y0
y2 = f(y1, localCnn(x_patch[1]; lpars)) + y1
…
loss = criterion(yN, target_y)
there are many possibilities for f(), the key is that this computation is no longer permutation invariant
more generally, patch specific losses don’t feel right to me, idk…