Quickly loading raw binary files
I need to load raw int16 binary files on-the-fly during training.  In Lua Torch I could just do the following: <SCODE>img = torch.ShortStorage(filename)
<ECODE> It appears that that functionality does not exist in PyTorch.  So instead I am loading into a numpy array as follows: <SCODE>img = np.fromfile(filename, 'int16')
<ECODE> The problem is that np.fromfile is extremely slow.  For my large data files (>100MB), np.fromfile loads four orders of magnitude slower than the old torch.ShortStorage method.  How can I get the fast load speeds in PyTorch?
abweiss
apaszke
Apparently numpy has (fast) memory mapping: <SCODE>img = np.memmap(filename, dtype='int16', mode='r').__array__()
<ECODE>
abweiss
apaszke
RuntimeError:  can’t convert a given np.ndarray to a tensor - it has an invalid type.  The only supported types are: double, float, int64, int32, and uint8.
abweiss
Huh, we should add int16 to. I’ll open an issue.
apaszke