First decoder has weaker (bayesian) prior, i.e. potentially discardable starting context as sequence length grows. For models in later pictures, that’s achievable (much easier in gated rnns), but has to be learned. From other perspective, they have a shortcut connection to time zero, that may be beneficial if initial context is strongly informative.
Re: integral. I think you may face some issues with gradient flow, if you go that route and do things like cumsum(). Look into neural ODEs if you feel that learning changes is more suitable for your task, but they’re more complex and slower.