Assuming dim0 represents the batch dimension, the activation shapes seem wrong, since the number of samples would change at these points.
Could you post an executable code snippet, which would reproduce this behavior?