turian:

What sort of performance issues will I encounter versus bare-metal? My main experience has been loading from disk is slower.


I’m not sure, if I understand the question correctly. What would you mean from disk? Do you mean the performance difference between the Kaggle kernels with their (free) GPU runtime vs. a bare metal machine? If so, I cannot comment on the performance of the free Kaggle kernels and am not sure what is being used currently.



 turian:

Right now we are leaning towards keeping the evaluation tasks secret, thus we don’t want to share the evaluation data until the competition is over. Curious about what code you have for that.


Kaggle is using an approach where the users submit the predictions and will get a score based on a secret part of the submission samples. E.g. the score of only 10% of the submission will be reported while internally the score of the entire submission will be stored. This is known as the “public” and “private” leaderboard. Since number of submissions are limited, users (usually) cannot brute force the private leaderboard samples. However, as so often, users might find a data leak which could allow them to somehow get more information about the public/private split, but issues with data leaks are sometimes hard to predict. 



 turian:

Docker is also useful because I can disable networking, which mitigates the risk anyone tries to exfiltrate the blinded evaluation data. Not really familiar with Singularity, what are the pros and cons versus using Docker for a competition like this.


I’m not deeply familiar with Singularity either, but have heard that it can be “more secure”, so you would need to check some comparisons conline or blog posts etc.