Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network. But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively. Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression. We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weightsharing, and use it to study the role of weight-sharing in neural networks. We show that common neural networks can be expressed as ArbNets with different hash functions. We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.
The manuscript advocates to study the weight sharing in a more systematic way by proposing ArbNets which defines the weight sharing function as a hash function. In this framework, any existing neural network architectures, including CNN and RNN, could be incorporated into ArbNets.
The manuscript is not well written. There are multiple grammar errors and typos. Content-wise, it is already well known that CNN and RNN can be expressed as general MLP with weight sharing. The introduction of ArbNets does not bring much value or insight to this area. So it seems that most content before experimental section is common sense.
In the experimental section, it is interesting to see how different hash function with different level of entropy can affect the performance of neural nets. However, this single observation cannot enrich the whole manuscript. Two questions:
(1) What is the definition of sparsity here, and how is it controlled?
(2) There seems to be a step change in Figure 3. All the results are either between 10 to 20, or near 50. And the blue line goes up and down. Is this expected?