It depends a bit on your use case. Are you planning on using multiple stages?
I.e.:
stage0 -> class0 vs. rest
stage1 -> class1 vs. rest
stage2 -> class2 vs. rest
...

If so, than the “rest” class prediction would matter, since the following stages won’t be able to see this sample anymore as it was already wrongly classified.
Could you also explain a bit, what the current outputs (51% and 81%) mean?