There are different way of handling this use case.
The simplest would be to load the complete data into memory and just use some conditions on the index to select the right numpy array where the current sample is located to load it.
The other approach would be to create a separate Dataset for each numpy array.
Based on the length of the underlying numpy arrays you could then create the indices and shuffle them (if shuffling is needed).
Once this is done you could then use Subset to pass these indices and enforce the shuffling, and pass all datasets into ConcatDataset.
Finally wrap the ConcatDataset into a DataLoader and it should make sure to load the data sequentially (but shuffle it inside each dataset).
Here is an example code snippet showing this approach, as the explanation might be a bit vague:
from torch.utils.data import TensorDataset, DataLoader, Subset, ConcatDataset

dataset1 = TensorDataset(torch.arange(10).view(10, 1))
dataset2 = TensorDataset(torch.arange(10, 20).view(10, 1))

idx1 = torch.randperm(len(dataset1))
idx2 = torch.randperm(len(dataset1))

dataset1 = Subset(dataset1, idx1)
dataset2 = Subset(dataset2, idx2)

datasets = ConcatDataset((dataset1, dataset2))

loader = DataLoader(datasets, batch_size=5)
for data in loader:
    print(data)

Note that this approach would also load the complete dataset into memory.
If you want to avoid it, you could create the “next” dataset once the old dataset was used completely.
Let me know, if any approach would work for you.