Why do you say that the rtansformation matrix is a pixel-wise operation? If there’s an affine transformation on the whole image, it has the form x' = M*x + b, where M and b are uniquely defined over the whole image. Each pixel will have a different result, but that’s only because we feed different inputs…
The way I see your problem, is you want to predict the affine transformation that was applied to an original image, so that you can correct it. Training is easy enough, you take some input images and apply random transforms on it, then predict the various components of the transforms (specifically the M and b from above). However, I don’t know if it makes much sense in general. How do you know what is the original image? Why should the rotated version, for example, be a transformation of the original and not the original itself?
One use case I can think of is that you have affine distortions or deformations (as described in the Matlab page for affine transforms 7) for a specific lens / camera setup and your dataset is only images taken with that lens / camera setup. But then you only have to do calibration once and apply the resulting correction on every image.