Yes, setting a heuristic is indeed a way to tackle this. But it could turn out to be a little tricky cause

How would we decide a threshold and how will we ensure that we have a data driven way to calculate it say for a new dataset or a slightly different output setting?
As you know, NN even learn patterns when random labels are passed. So it could turn out that despite the model wanting to tell “I don’t know”, the results will be skewed towards 1 class or the other

These are some issues I guess might occur, but yes as a quick fix and for something which has decent coverage this will work.