Update with more information - since posting the above, I’ve installed more ram (now 16 GB). I’ve also run free -m and nvidia-smi, and I can see that significant amounts of memory are being used by TF - both in the GPU and RAM.
I’m not sure why it’s using such memory. I’m using the following command to help ensure that it at least doesn’t allocate all the memory at once:

gpus = tf.config.experimental.list_physical_devices(‘GPU’)
print(gpus)
tf.config.experimental.set_memory_growth(gpus[0], True)

Do the NUMA errors on my original post have any bearing on TF’s ability to use memory efficiently?

erick@erickusb:~$ free -m
total    	used    	free  	shared  buff/cache   available
Mem:      	16033   	14287     	177      	41    	1568    	1418
Swap:      	7381     	898    	6483
erick@erickusb:~$ nvidia-smi
Sat Jul 31 14:13:48 2021  	
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03	Driver Version: 460.91.03	CUDA Version: 11.2 	|
|-------------------------------±---------------------±---------------------+
| GPU  Name    	Persistence-M| Bus-Id    	Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|     	Memory-Usage | GPU-Util  Compute M. |
|                           	|                  	|           	MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070	Off  | 00000000:01:00.0  On |              	N/A |
| 36%   53C	P2	34W / 151W |   5056MiB /  8116MiB |  	5%  	Default |
|                           	|                  	|              	N/A |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes:                                                              	|
|  GPU   GI   CI    	PID   Type   Process name              	GPU Memory |
|    	ID   ID                                               	Usage  	|
|=============================================================================|
|	0   N/A  N/A  	1139  	G   /usr/lib/xorg/Xorg            	101MiB |
|	0   N/A  N/A 	12157  	G   /usr/lib/xorg/Xorg            	187MiB |
|	0   N/A  N/A 	12274  	G   /usr/bin/gnome-shell           	74MiB |
|	0   N/A  N/A 	13778  	G   /usr/lib/firefox/firefox      	113MiB |
|	0   N/A  N/A 	17034  	C   /usr/bin/python3             	4541MiB |
|	0   N/A  N/A 	18032  	G   …mviewer/tv_bin/TeamViewer   	16MiB |
±----------------------------------------------------------------------------+

A couple of additional details… the TF kernel is now crashing instead of the GUI. Also, what I’m trying to implement is more or less Wang’s algorithm 2 in this paper: https://arxiv.org/pdf/1809.03428.pdf 5
I’ve also been playing around with batch size and it doesn’t seem to help much.