@stas - many thanks for this. I’m going to implement your suggestion of attempting to allocate some known large tensor right at the start of the job, and report & rerun upon failure.
Very much appreciate your help. Thank you.