Tarsnap performance issues in late March, most of April
In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried.  How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case.  This didn't trip our monitoring at the time because they all completed successfully.<p>n.b. Our backups run outside of the hotspot times for Tarsnap, so we may have had less performance impact than many customers.  I have an old habit of "Schedule all cron jobs to start predictably but at a random offset from the hour to avoid stampeding any previously undiscovered SPOFs."  That's one of the Old Wizened Graybeard habits that I picked up from one of the senior engineers at my last real job, which I impart onto y'all for the same reason he imparted it onto me: it costs you nothing and <i>will</i> save you grief some day far in the future.