Hi,
first of all thanks to everyone working on this awesome library!
I’ve started looking into pytorch for a couple of months now, but I’ve reached a point where I cannot explain a specific behavior. I am sorry for not being able to get a smaller minimal example up but in the smaller versions the problems do not appear:
"""
@author: Utku Ozbulak - github.com/utkuozbulak
Available at: https://github.com/utkuozbulak/pytorch-cnn-visualizations/blob/master/src/cnn_layer_visualization.py
Small modifications:
"""

import numpy as np
import torch
from torch.optim import Adam
from torchvision import models


class CNNLayerVisualization:
    """
        Produces an image that minimizes the loss of a convolution
        operation for a specific layer and filter
    """

    def __init__(self, model, selected_layer, selected_filter):
        self.model = model
        self.model.eval()
        self.selected_layer = selected_layer
        self.selected_filter = selected_filter
        self._conv_output = None

    def visualise_layer(self):
        # Generate a pseudo-random image
        np.random.seed(123)
        random_image = np.random.uniform(0, 1, (3, 224, 224))
        # Process image => For simplicitely to not transform here:
        # processed_image = preprocess_image(random_image, False)
        processed_image = torch.from_numpy(random_image).float().unsqueeze_(0)
        processed_image.requires_grad = True
        # Add one more channel to the beginning. Tensor shape = 1,3,224,224

        # Define optimizer for the image
        optimizer = Adam([processed_image], lr=0.1)
        for i in range(1, 5):
            optimizer.zero_grad()
            # Assign create image to a variable to move forward in the model
            x = processed_image
            for index, layer in enumerate(self.model):
                # Forward pass layer by layer
                # x is not used after this point because it is only needed to trigger
                # the forward hook function
                x = layer(x)
                # Only need to forward until the selected layer is reached
                if index == self.selected_layer:
                    self._conv_output = x[0, self.selected_filter]
                    # TODO: Explain why it makes a difference when the following
                    # line is removed!
                    break
            loss = -torch.mean(self._conv_output)
            loss.backward()
            # Loss function is the mean of the output of the selected layer/filter
            # We try to minimize the mean of the output of that specific filter
            print("Loss:", "{0:.2f}".format(loss.data.numpy()))
            optimizer.step()


if __name__ == "__main__":
    cnn_layer = 0
    filter_pos = 2
    pretrained_model = models.vgg16(pretrained=True).features
    layer_vis = CNNLayerVisualization(pretrained_model, cnn_layer, filter_pos)
    layer_vis.visualise_layer()

The problem lies in the TODO line. I wanted to visualize different filter by utilizing gradient ascend and I’ve used this awesome github project as a reference.
Now is our model not iterable like the vgg16 model used in the repository. In my opinion, it shouldn’t make a difference if the input image, which is being maximized, goes through the full model or stops at the desired layer, as the backward function is only applied to operations leading up to the desired layer/filter. Smaller tests confirm this. But if you run the given script once with the break line and then without the break line, the loss differs significantly. I cannot explain why and hope that somebody could demystify this for me. 
Thanks!