can I create in a model and custom data iterator inside the main_method?

Given the above example, you created a generator to produce input data? If that is the case, yes, sure you can do that.

will there be 4 data sets loaded into the RAM / CPU memory?

I am assumingg pass # some stuff statement will be replaced by actual forward-backward-step functions? If that is the case, then the 4 date sets won’t be loaded into memory at the same time. Instead, each data set will no longer be needed and can be gc-ed at the end of every iteration.

will each “for batch_data in…” iterate independently

Yes. It will have it’s own forward pass (building autograd graph), backward pass (generating grads and sync them if necessary), and step function (updating params)

will the model be updated e.g. every independed batch operation. Obviously I don’t want to have four independed models. What’s the process flow in this case… when gradients are updated etc?

When you call backward, the gradient will be accumulated into Tensor.grad, and it is up to you regarding when to call Optimizer.step() to apply those grads into the parameter.