Hi Ashley,
What you are trying to do is to use batch_size properly.
If you have your pipeline of data using tf.data.Dataset (tf.data.Dataset  |  TensorFlow Core v2.6.0 10) it will load the data from disk for you and provide it for the model in chunks that fit the memory. Of course the size of these chunks it’s up to you to define.
This is a great tutorial to give more insights: Better performance with the tf.data API  |  TensorFlow Core 19