operations has been a big thing for a lot of my clients lately; there are a number of things i think about when deciding on retraining…
how stationary is the input? sometimes it’s clear that a problem is stationary in the input; e.g. a vision model trained on a large diverse natural set images from a phone is usually pretty stable whereas a time series problem often is not. i’ve found rapid retraining of a time series problem on a small recent window can give better results than a larger model trained on a longer window across all data.
how stationary is the output? the big thing i’ve found here to consider is feedback loops. when building a model that has outputs very close to the user experience; e.g. P(click|impression) then retraining can be critical to ensure the feedback loop doesn’t push things out of distribution too quickly.
monitoring in any case, expectations about feedback loops and stationarity are often wrong so the main thing is to be able to monitor drift. it’s not just about knowing when to retrain but, more importantly in the operational sense, when to occasionally turn off and fall back to whatever graceful degradation plan is in place ( you do have a graceful degradation plan don’t you?  )
[related]  active learning loops more and more i’ve seen clients wanting to make use of unlabelled data; a key thing being to direct labelling effort. things like uncertainty and diversity sampling, which are needed for active learning loops, end up being super useful for monitoring too.
challenger/champion sometimes the question isn’t “when do i retrain?” but “when do i expose users to my latest trained model?” in which case it’s more about retraining as frequently as you can and then focusing instead on how you want to slowly expose your model through techniques like shadow releases. the big question here is whether the added complexity is justified to get empirical info about a new model’s performance.
so many interesting problems in this space!