ModuleList (http://pytorch.org/docs/master/nn.html#modulelist 54) is designed to be a list-like container that also supports .cuda() and other methods. There isn’t a similar one for dict yet. Some discussions on ModuleDict are at https://github.com/pytorch/pytorch/issues/4048 14. There is supposed to be a PR soon according to the thread.