Solved by ptrblck in post #5 
              
              
                You would calculate 5 different losses, which you could average into a single one to call .backward() on. 
Actually both approaches might be quite similar. 
Either you split your last layer(s) into 5 different heads, each with an own loss, or you split the large layer (25 outputs) into 5 x 5 snippet…