unlike LuaTorch, we dont dispatch DataParallel via python threads in PyTorch.

If this is not supported now, do you have any plan to support this ?

No, we plan to (and do) dispatch multi-GPU differently at high performance, without the user needing to do anything.

Also, when I employed DataParallel, it seemed to me that pytorch use only one thread.

This is irrelevant to the user, we are working on improving the internals and multi-gpu perf.

First GPU consumed almost all memory, while the others consumed only half of their memories.
Is this normal ?

Depending on how many parameters you have in your model, this is possible.