Hi Beilei!



 111220:

But unfortunately it doesn’t seem to have solved the problem


Well, at least you got a different error.


/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py



Note, you’re using a different version of python than in your first post
(and perhaps a different version of pytorch, as well).  This makes it
harder to compare your two runs.


Again, once act1 is removed, the error disappears！
So the error should be in the code of Dynamic_relu


Good point!
From your Dynamic_relu_b code:


class Dynamic_relu_b(nn.Module):
    def __init__(self, inchannel, R=8, k=2):
        ...
        self.init_alpha = torch.zeros(self.k)
        self.init_beta = torch.zeros(self.k)

...

    def forward(self, input):
        alpha = torch.zeros((self.k, bs, in_channel))
        beta = torch.zeros((self.k, bs, in_channel))
        ...



Those torch.zeros() are also cpu tensors.  So you will still have a
cpu-gpu mismatch.
I don’t know what the standard way is to write an activation so that it
“automatically” moves to the gpu when you call model.cuda(), but
I don’t think you get this behavior for free.
Things you could try:
First, run your whole model on the cpu, just to make sure it works.  So
leave Dynamic_relu_b as it is, and don’t move your model, images,
or labels to the gpu.
If that works, you might go back to the gpu.  The easiest way might
be to write a cuda version of Dynamic_relu_b, where you move its
internal tensors to the gpu.
And if that works, try to figure out how to make Dynamic_relu_b work
“automatically” on both the cpu and gpu.  After all, the various built-in
pytorch Modules know how to do this, so there must be some standard
way.  (But I don’t know what it is.)
Good luck.
K. Frank