Solved by ptrblck in post #2 
              
              
                The error message seems to point to your RAM, not the GPU memory. 
Could you check it with free -h and see how much memory is left? 
This particular error message can be triggered, if new processes would like to spawn / fork and there is not enough memory left.