<file_sep>import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np


class Net(nn.Module):
    def __init__(self,cf):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1, cf['conv1'], kernel_size=5)
        self.conv2 = nn.Conv2d(cf['conv1'], cf['conv2'], kernel_size=5)
        self.conv2_drop = nn.Dropout2d(cf['keep_prob'])
        L = np.floor((np.floor((28-5)/2+1)-5)/2+1)**2*cf['conv2']
        self.fc1 = nn.Linear(int(L), cf['fc1'])
        self.fc2 = nn.Linear(cf['fc1'], 10)

        self.init_weights()

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

    def init_weights(self):
        # follow http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf
        for m in self.children():
            if isinstance(m,nn.Linear):
                size = m.weight.size()
                fan_in = size[1]  # number of columns
                scale = np.sqrt(2.0 / (fan_in))
                m.weight.data.uniform_(-scale, scale)
                m.bias.data.uniform_(-scale, scale)
            elif isinstance(m,nn.Conv2d):
                size = m.weight.size()
                fan_in = size[2]*size[3]
                scale = np.sqrt(2.0 / (fan_in))
                m.weight.data.uniform_(-scale, scale)
                m.bias.data.uniform_(-scale, scale)
        return
<file_sep># Semi supervised learning with self ensembling
This repo implements the self ensembling technique in PyTorch. The code spans only 80 lines and about 50 lines of helper code. I wouldn't use the code in production, but it illustrates the concept.

[Here](https://arxiv.org/abs/1610.02242) is the paper.

# Quick summary
Self emsembling trains neural networks on partially labelled data. For some training batch, we run the network twice with different [dropouts](https://arxiv.org/pdf/1207.0580.pdf). On all samples, we penalize the squared difference between the predictions from both networks. On the labelled data, we also penalize the negative log likelihood of the labels. This technique beats other semi-supervised learners on the [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html) and [SVHN](http://ufldl.stanford.edu/housenumbers/) datasets.

# What is semi-supervised learning (SSL)?
In semi supervised learning we use both labelled and unlabelled data. In many problems, we have data in abundance. Yet, it cost much time/effort/money to obtain labels. For example:

  * __Chest xrays__ We can have lots of scans in a database, but is cost much money to pay a radiologists labelling them.
  * __Computer vision__ We can scrape Google Images and get a huge dataset of images, but it costs effort to set up AMT and have people label them
  * __NLP__ We have many records of open books, emails and Twitter feed, but it cost mush effort to have people label the sentiment

# What links SSL and neural networks?
Neural networks learn feature representations. (Also named [distributed representations](http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html), latent representations or hidden representations) This representations could be learned on both labelled and unlabelled data. With labelled data, we could think of the hidden neurons in a convolutional neural network. With unlabelled data, we could think of, for example, word vectors.

# Examples of SSL techniques
Self ensembling is not the first SSL technique. Other notable attempts:

  * [__Auto encoders__](https://arxiv.org/abs/1406.5298) Auto encoders learn to encode and decode data onto itself. Usually we obtain feature representations from a bottleneck between the encoder and decoder. In SSL, we train the auto encoder on all data. Secondly, we train a small classifier for the labelled data, using the feature representations as input. (You can use a variational auto encoder too)
  * [__Ladder networks__](https://arxiv.org/abs/1507.02672) Ladder networks extend the auto encoder to multiple levels. People visualize auto encoders as an up-side-down diagram. Ladder networks stack many such auto encoders. You can visualize it as a ladder.
  * [__Generative adversarial networks__](https://arxiv.org/abs/1606.03498) GAN's play a mini-max game. The generator learns to generate fake images; the discriminator learns to discriminate between fake and real images. In SSL, we view the discriminator as an extended classifier. Say you classify the 10 [MNIST](http://yann.lecun.com/exdb/mnist/) digits. The GAN discriminator would classify 11 classes: the 10 MNIST classes and the 11th class representing a fake. 

Personally, I like the self ensembler for its simplicity. In auto encoders, we train a decoder without using it; in ladder networks, we train multiple decoders without using them; and for GAN's, we train a generator without using it. Moreover, the reseach community still works on stable algorithms to train GAN's. In that perspective, self ensembling trains only a single network. So in the engineering, we need to monitor only one piece of code.

# The code
The code consists of three scripts 

  * ```main.py``` initializes data and the network
  * ```model.py``` contains the PyTorch module for a convnet
  * ```dataloader.py``` provides some sample functions for the MNIST data


As always, I am curious to any comments and questions. Reach me at <EMAIL><file_sep>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import numpy as np

from model import Net
import dataloader

mnist = dataloader.read_data_sets("MNIST_data",norm=True)

cuda = True and torch.cuda.is_available()
bsz = 64
bsz_test = 100
epochs = 10
lr= 0.01
momentum = 0.5
seed = 1111
log_interval = 100
keep_prob = 0.8
cf = {'keep_prob'           :0.7,
      'conv1'               :10,
      'conv2'               :20,
      'fc1'                 :50}

torch.manual_seed(seed)
if cuda:
    torch.cuda.manual_seed(seed)

mnist.cuda = cuda
mnist.bsz = bsz
data, targets = mnist.sample()

model = Net(cf)
if cuda:
    model.cuda()

optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)
self_ensemble_penalizer = nn.MSELoss()
batches = 0

def ramp_up():
    """Decay/ramp up function for the scaling of self_ensemble loss."""
    global batches
    batches += 1
    final = 1500.
    if batches < final:
      p = batches/final
      return float(np.exp(-5.*(p - 1) ** 2))
    else:
        if batches == final: print('Finished schedule for ramping up the lambda')
        return 1.0



def loss_calc(*,z1, z2, target):
    """Calculates the combined loss for semi-supervised learning. Note that on the validation
    set we calcalate only the nll_loss"""
    loss = F.nll_loss(z1[:5], target[:5])
    z2 = z2.detach()
    self_ensemble = self_ensemble_penalizer(z1,z2)
    loss += 0.05*ramp_up()*self_ensemble
    return loss


def train(epoch):
    """Train one epoch, which consists of a fixed number of batches"""
    model.train()
    for batch_idx in range(800):
        data, target = mnist.sample()
        optimizer.zero_grad()
        output1 = model(data)
        output2 = model(data)
        loss = loss_calc(z1=output1,z2=output2,target=target)
        loss.backward()
        optimizer.step()
        if batch_idx % log_interval == 0:
            print('Train Epoch: %3i [%6i/%6i (%.0f)] \tLoss: %.6f'%(epoch, batch_idx , 100, 100. * batch_idx / 100, loss.data[0]))

def test(epoch):
    """Test bsz_test number of batches from the validation set. Note that we calculate the normal nll_loss.
    The self_ensemble cost is not taken into account here"""
    model.eval()
    test_loss = 0
    correct = 0
    for k in range(bsz_test):
        data, target = mnist.sample(dataset='val')
        data.volatile = True
        output = model(data)
        test_loss += F.nll_loss(output, target).data[0]
        pred = output.data.max(1)[1] # get the index of the max log-probability
        correct += pred.eq(target.data).cpu().sum()

    acc = correct/(bsz_test*bsz)
    test_loss /= bsz_test # loss function already averages over batch size
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, bsz_test*bsz,
        100. * acc))

if __name__ == '__main__':
    try:
        for epoch in range(1, epochs + 1):
            train(epoch)
            test(epoch)
    except KeyboardInterrupt:
        #Except on keyboard interrupt so we can end training prematurely
        print('Finish training')
