I want to detect outlier present in my dataset including 205854 loci in 5 populations.
I have tried using LOSITAN on windows but unfortunately it was not working and showing "First simulation pass to determine initial neutral set" since several hours. 

can anyone please suggest me how to detect outlier of this big dataset???

Thank you in advance!!
Maybe try GraphPad Prism.  Alternately, you might be able to use scikit-learn in Python.