The error message seems to point to your RAM, not the GPU memory.
Could you check it with free -h and see how much memory is left?
This particular error message can be triggered, if new processes would like to spawn / fork and there is not enough memory left.