@apaszke Thanks for the batchnorm info!
For the ReLUs/Max-pool, I agree that in this case they are equivalent, however in the more general case of an arbitrary activation function they will not necessarily be - consider a “relu” activation function that is flipped and exists on the second and forth quadrants instead of just the first and border between third and second, in this case the two operation orders would matter. (Granted they are not that popular, I just wanted to point out the subtlety  )