Unfortunately you can’t increase you GPU vRAM except buying a new one. GPU vRAM and computer RAM are different things. Error you described tells about your overusage PC RAM not vRAM.
What do you want to do exactly, please share more detail including terminal log maybe.
Greetings.