Yea as @eqy mentioned, there could be an instance where you really are using your gpu for a decent amount of it’s capacity, but you don’t “see it” being used because most of the time there’s other things running (non-gpu).
In addition to checking for that, I’d also double check that things aren’t staying on the gpu longer then they should be! (maybe the batch size 12 case said the gpu was too much because as things keep running, more gpu space was getting utilized and not freed? not sure if you’re saying batch size 12 instantly errored out of space, but just an open-end thought