Weight initilzation
How I can initialize variables, say Xavier initialization?
Hamid
Atcold
fmassa
michael_k
Kalamaya
Does it make any sense what I am saying? If it does not, I can try better and with an example.
Atcold
Kalamaya
You first define your name check function, which applies selectively the initialisation. <SCODE>def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        xavier(m.weight.data)
        xavier(m.bias.data)
<ECODE> <SCODE>net = Net() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
<ECODE>
Atcold
A less Lua way of doing that would be to check if some module is an instance of a class. This is the recommended way: <SCODE>def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)
<ECODE>
apaszke
Hamid
<SCODE>size = m.weight.size() # returns a tuple
fan_out = size[0] # number of rows
fan_in = size[1] # number of columns
<ECODE>
Atcold
apaszke
Atcold
my code: <SCODE>def weight_init(m): 
	if isinstance(m, nn.Linear):
		size = m.weight.size()
		fan_out = size[0] # number of rows
		fan_in = size[1] # number of columns
		variance = np.sqrt(2.0/(fan_in + fan_out))
		m.weight.data.normal_(0.0, variance)


class Residual(nn.Module):
	def __init__(self,dropout, shape, negative_slope, BNflag = False):
		super(Residual, self).__init__()
		self.dropout = dropout
		self.linear1 = nn.Linear(shape[0],shape[1])
		self.linear2 = nn.Linear(shape[1],shape[0])
		self.dropout = nn.Dropout(self.dropout)
		self.BNflag = BNflag
		self.batch_normlization = nn.BatchNorm1d(shape[0])
		self.leakyRelu = nn.LeakyReLU(negative_slope = negative_slope , inplace=False)

	def forward(self, X):
		x = X
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear1(x)
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear2(x)
		x = torch.add(x,X)
		return x
		
		
class FullyCN(nn.Module):
	def __init__(self, args):
		super(FullyCN, self).__init__()
		self.numlayers = arg.sm-num-hidden-layers
		self.learning-rate= args.sm-learning-rate
		self.dropout = arg.sm-dropout-prob
		self.BNflag = args.sm-bn
		self.shape = [args.sm-input-size,args.sm-num-hidden-units]		
		self.res =  Residual(self.dropout,self.shape,args.sm_act_param,self.self.BNflag)
		self.res(weight_init)
		self.res-outpus = []

	def forward(self,X):
		self.res-outpus.append(self.res(X))
		for i in range(self.numlayers):
			self.res-outpus.append(self.res(self.res-outpus[-1]))
		return self.res-outpus[-1]<ECODE>
Hamid
sorry about confusion
Hamid
Correct.
Atcold
But I have called weight_init once for the class while I call linear layers in a for loop (i.e., there are multiple sets of variables).
Hamid
<SCODE>net = Residual() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
<ECODE>
Atcold
apaszke
Hamid