Solved by SimonW in post #2 
              
              
                The dataset api is very very general. You can do what ever you want with it. If N is large, then each data can be a file, or an entry in a database, or a file in a remote HDD.