I have an idea for a sparse training library for pytorch.
A lot of the work will involve speed/performance optimization, particularly parallelizing IO memmap operations so that it doesn’t slow down training.
The library will then be used to train embeddings for ~40 million research papers.
If you haven’t heard of the Facebook Pytorch Hackathon, Google ‘Facebook Pytorch Hackathon’