My rest times are a separate activity that doesnt require me to sit there like a zombie staring into the void. I give myself the mental freedom to do whatever I want during my rest time. Seems like you consider your rest as a part of your exercise so for you, you have to remain focused throughout your rest or else you consider yourself distracted during your exercise as a whole. 