for the P1NN to learn the features of Pentomino shapes perfectly on the 40000 training examples. The P2NN has 2048 hidden units. L1 and L2 penalty coecients for the P2NN are
1e-6, and the learning rate is 0.1. These were selected by trial and error based on validation
set error. Both P1NN (for each patch) and P2NN are fully-connected neural networks, even
though P1NN globally is a special kind of convolutional neural network.
Filters of the rst layer of SMLP are shown in Figure 6. These are the examples of the
lters obtained with the SLMP-hints trained with 40k examples, whose results are given in
Table 1. Those lters look very noisy but they work perfectly on the Pentomino task.
3.2.3 Deep and Structured Supervised MLP without Hints (SMLP-nohints)
SMLP-nohints uses the same connectivity pattern (and deep architecture) that is also used
in the SMLP-hints architecture, but without using the intermediate targets ( Y). It directly
predicts the nal outcome of the task ( Z), using the same number of hidden units, the same
10
We explore the effect of introducing prior information into the intermediate
level of neural networks for a learning task on which all the state-of-the-art
machine learning algorithms tested failed to learn. We motivate our work from
the hypothesis that humans learn such intermediate concepts from other
individuals via a form of supervision or guidance using a curriculum. The
experiments we have conducted provide positive evidence in favor of this
hypothesis. In our experiments, a two-tiered MLP architecture is trained on a
dataset with 64x64 binary inputs images, each image with three sprites. The
final task is to decide whether all the sprites are the same or one of them is
different. Sprites are pentomino tetris shapes and they are placed in an image
with different locations using scaling and rotation transformations. The first
part of the two-tiered MLP is pre-trained with intermediate-level targets being
the presence of sprites at each location, while the second part takes the
output of the first part as input and predicts the final task's target binary
event. The two-tiered MLP architecture, with a few tens of thousand examples,
was able to learn the task perfectly, whereas all other algorithms (include
unsupervised pre-training, but also traditional algorithms like SVMs, decision
trees and boosting) all perform no better than chance. We hypothesize that the
optimization difficulty involved when the intermediate pre-training is not
performed is due to the {\em composition} of two highly non-linear tasks. Our
findings are also consistent with hypotheses on cultural learning inspired by
the observations of optimization problems with deep learning, presumably
because of effective local minima.
Çağlar Gülçehre Yoshua Bengio