Preprocessing utilities have always bugged me 
So, back in the days, there was nothing like tf.keras.layersexperimental.preprocessing.Rescaling. In order to better streamline the preprocessing steps, I was wrapping my utilities inside a Lambda layer and then was interesting that layer into my final model. As expected, the model’s performance was terrible when it was compiled as a graph. So, we decided to do the preprocessing externally in a separate job.
During the staging, I had mistakenly altered a single digit in our preprocessing utility and this led our model to predict everything to all the same. It took us ~4 days to figure out what was going wrong (yes, I acknowledge our codebase wasn’t organized that well). But I learned my lesson.
Fast-forwarding to later 2020 when TensorFlow released the preprocessing layers it really came as a relief