Hi.
I’m training a model with two parts: part A randomly generates data, and part B consumes data. Part B reuses the data generated for several times, and the data will then be regenerated by part A. The pseudo code is shown as following:
for epoch in range(n_epochs):
    if epoch % 10 == 0:
        dataset = partA()
    for i, (data, label) in enumerate(dataset):
        partB(data)  # training partB

The dataset generated by part A is pretty large (2,000,000x16x16x16 in total), so I think I need to save it somewhere. I try to use json/ujson. Is there any way to do this faster?
Thanks!