For a research project, I need to load a dataset of images, and numerical labels for drive and motor data. Each image is also labeled with a flag, and I need to be able to set which image flags to ignore and keep during runtime. Currently the data loading system was written and optimized for Caffe and uses the hdf5 format for extracting these files. During training, extracting and converting the hdf5 data is the main bottleneck. Because of the ignore flag feature, I was not able to use the default PyTorch Dataset to dynamically load the data, since a single call to the get item in the dataset must return an item at each call. I am looking to convert our current dataset into a format that would be optimized for loading in to PyTorch, that also supports loading a frame and skipping it if it has an ignore flag set. My current code for importing the dataset and using it for training are available here:
###Loading hdf5 Dataset w/ Ignore List

  
      github.com
  
  
    sauhaardac/pytorch-neural-z2color/blob/master/libs/import_utils.py 14
import pickle

import h5py

from utils import *

Segment_Data = {}
hdf5_runs_path = desktop_dir('bair_car_data/hdf5/runs')
hdf5_segment_metadata_path = desktop_dir('bair_car_data/hdf5/segment_metadata')


def load_hdf5(path):
    F = h5py.File(path)
    labels = {}
    Lb = F['labels']
    for k in Lb.keys():
        if Lb[k][0]:
            labels[k] = True
        else:
            labels[k] = False


  This file has been truncated. show original

  
  
    
    
  
  


###Use of Dataset in Traning Code

  
      github.com
  
  
    sauhaardac/pytorch-neural-z2color/blob/master/train.py 9
import argparse
import datetime
import random

import torch
import torch.nn as nn
import torch.nn.utils as nnutils
from torch.autograd import Variable
from libs.import_utils import *
from nets.z2_color import Z2Color

# Define Arguments and Default Values
parser = argparse.ArgumentParser(description='PyTorch z2_color Training',formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--validate', type=str, metavar='PATH',
                    help='path to model for validation')
# parser.add_argument('--skipfirstval', type=str, metavar='PATH',
#                     help='Skip the first validation (if restoring an end of epoch save file)')
parser.add_argument('--resume', type=str, metavar='PATH',
                    help='path to model for training resume')
parser.add_argument('--ignore', default=['reject_run', 'left', 'out1_in2', 'racing', 'Smyth'], type=str, nargs='+',


  This file has been truncated. show original

  
  
    
    
  
  


What would be the best format to use for my particular use case? Are there any examples of setting up a similar type of dataset?