without hints architecture for which results are given in Table 1. In those graphs only the results
for the training on the randomly generated 545400 Pentomino samples have been presented. As
shown in the plots SMLP-nohints was not able to generalize without standardization. Although
without standardization the training loss seems to decrease initially, it eventually gets stuck in
a plateau where training loss doesn't change much.
Training of SMLP-nohints online minibatch SGD is performed using standardization in the
intermediate layer and Adadelta learning rate adaptation, on 1046000 training examples from
the randomly generated Pentomino stream. At the end of the training, test error is down to
27.5%, which is much better than chance but from from the score obtained with SMLP-hints
of near 0 error.
In another SMLP-nohints experiment without standardization the model is trained with the
1580000 Pentomino examples using online minibatch SGD. P1NN has 2048 hidden units and 16
sigmoidal outputs per patch. for the P1NN hidden layer. P2NN has 1024 hidden units for the
hidden layer. Adadelta is used to adapt the learning rate. At the end of training this SMLP,
the test error remained stuck, at 50.1%.
21
0 100 200 300 400 500 600 700 800
We explore the effect of introducing prior information into the intermediate
level of neural networks for a learning task on which all the state-of-the-art
machine learning algorithms tested failed to learn. We motivate our work from
the hypothesis that humans learn such intermediate concepts from other
individuals via a form of supervision or guidance using a curriculum. The
experiments we have conducted provide positive evidence in favor of this
hypothesis. In our experiments, a two-tiered MLP architecture is trained on a
dataset with 64x64 binary inputs images, each image with three sprites. The
final task is to decide whether all the sprites are the same or one of them is
different. Sprites are pentomino tetris shapes and they are placed in an image
with different locations using scaling and rotation transformations. The first
part of the two-tiered MLP is pre-trained with intermediate-level targets being
the presence of sprites at each location, while the second part takes the
output of the first part as input and predicts the final task's target binary
event. The two-tiered MLP architecture, with a few tens of thousand examples,
was able to learn the task perfectly, whereas all other algorithms (include
unsupervised pre-training, but also traditional algorithms like SVMs, decision
trees and boosting) all perform no better than chance. We hypothesize that the
optimization difficulty involved when the intermediate pre-training is not
performed is due to the {\em composition} of two highly non-linear tasks. Our
findings are also consistent with hypotheses on cultural learning inspired by
the observations of optimization problems with deep learning, presumably
because of effective local minima.
Çağlar Gülçehre Yoshua Bengio