title: HyperNetworks
abstract: This work explores hypernetworks: an approach of using one network, also known as a hypernetwork, to generate the weights for another network.  We apply hypernetworks to generate adaptive weights for recurrent networks. In this case, hypernetworks can be viewed as a relaxed form of weight-sharing across layers. In our implementation, hypernetworks are are trained jointly with the main network in an end-to-end fashion.  Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks.
authors: David Ha Andrew M. Dai Quoc V. Le
hIndex: Quoc V. Le 105