Body: <p>I would like to build a neural network that takes a <strong>natural number</strong> and generates a <strong>one-hot encoding</strong> vector corresponding to that number.</p>
  <blockquote>
  <p>Example: <span class="math-container">$2 \rightarrow (0,0,1,0,\dots)$</span></p>
  </blockquote>
  <p>More formally, I want it to take an input <span class="math-container">$i \in [0, \dots, K]$</span> and produce an output vector <span class="math-container">$(o_0, \dots, o_j, \dots, o_k)$</span>, where all <span class="math-container">$o_j$</span> are 0, except <span class="math-container">$o_i$</span>, which is 1 (i.e. one-hot encoding).</p>
  <p>However, I am not sure on what the specific architecture should be. I believe it should have more than one layer, since there is no linear relationship between the input and the output.</p>
  <p>I have tried several architectures, but the best performance I have been able to get is 33%. I am using the following code:</p>
  <pre><code>import pandas as pd
  import torch
  import torch.nn as nn
  import torch.optim as optim
  from sklearn.model_selection import train_test_split
  import random
  random.seed(0)
  torch.manual_seed(0)
  
  class Net(nn.Module):
      def __init__(self):
          super(Net, self).__init__()
          self.fc1 = nn.Linear(1, 55)
          self.fc2 = nn.Linear(55, 30)
          self.fc3 = nn.Linear(30, 25)
          self.relu = nn.ReLU()
          self.dropout = nn.Dropout(p=0.2)
          self.softmax = nn.Softmax(dim=0)
  
      def forward(self, x):
          x = self.fc1(x)
          x = self.relu(x)
          x = self.dropout(x)
          x = self.fc2(x)
          x = self.relu(x)
          x = self.dropout(x)
          x = self.fc3(x)
          x = self.softmax(x)
          return x
      
  net = Net()
  
  # A list of 10000 random numbers between 0 and 24
  dataset = pd.DataFrame([random.randint(0, 24) for _ in range(10000)])
  dataset['label'] = dataset[0] # The label is the same as the input
  
  train, test = train_test_split(dataset, test_size=0.2)
  
  criterion = nn.CrossEntropyLoss()
  optimizer = optim.Adam(net.parameters(), lr=0.001)
  
  for epoch in range(5):
      # Train
      running_loss = 0.0
      for i in range(len(train)):
          inputs = train.iloc[i, 1:].tolist()
          inputs = torch.tensor(inputs, dtype=torch.float)
          labels = train.iloc[i, 0]
          labels = torch.tensor(labels, dtype=torch.long)
  
          optimizer.zero_grad()
          outputs = net(inputs)
          loss = criterion(outputs, labels)
          loss.backward()
          optimizer.step()
  
      # Evaluate
      correct = 0
      total = 0
      with torch.no_grad():
          for i in range(len(test)):
              inputs = test.iloc[i, 1:].tolist()
              inputs = torch.tensor(inputs, dtype=torch.float)
              labels = test.iloc[i, 0]
              labels = torch.tensor(labels, dtype=torch.long)
              outputs = net(inputs)
              predicted = outputs.argmax()
              total += 1
              correct += (predicted == labels)
      
      print('Accuracy of the network on the test set: {:.2f}%'.format(100 * correct / total))
  </code></pre>
  <p>I believe my task is pretty simple, but I just can't think of the right architecture to do it. Do you have any ideas? :)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-13T20:23:59.860Z
Comments.Text: I'm not familiar with pandas but how does this dataframe layout work? Is this training with only 1 number per batch?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-13T21:00:52.513Z
Comments.Text: Yup, I checked that and it's working :)
Comments.UserDisplayName: Aldan Creo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-13T21:05:05.457Z
Comments.Text: It is training with only 1 number per batch?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-14T12:17:04.073Z
Comments.Text: There's no batching here, to be more specific...
Comments.UserDisplayName: Aldan Creo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-15T17:10:10.190Z
Comments.Text: I think this generally leads to suboptimal training unless the learning rate is set very low. In each training step the network converges towards whichever number was presented in that step, with nothing making sure the other numbers also keep working.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <neural-networks><classification><training><pytorch><one-hot-encoding>
Title: Training a neural network to produce a one-hot encoding vector out of a single feature
UserDisplayName: Aldan Creo
ThreadText: # Training a neural network to produce a one-hot encoding vector out of a single feature
  
  (Asked by: username_0 on 2023-03-13)
  
  I would like to build a neural network that takes a natural number and generates a one-hot encoding vector corresponding to that number.
  
  Example: $2 \rightarrow (0,0,1,0,\dots)$
  
  More formally, I want it to take an input $i \in [0, \dots, K]$ and produce an output vector $(o_0, \dots, o_j, \dots, o_k)$, where all $o_j$ are 0, except $o_i$, which is 1 (i.e. one-hot encoding).
  However, I am not sure on what the specific architecture should be. I believe it should have more than one layer, since there is no linear relationship between the input and the output.
  I have tried several architectures, but the best performance I have been able to get is 33%. I am using the following code:
  import pandas as pd
  import torch
  import torch.nn as nn
  import torch.optim as optim
  from sklearn.model_selection import train_test_split
  import random
  random.seed(0)
  torch.manual_seed(0)
  
  class Net(nn.Module):
      def __init__(self):
          super(Net, self).__init__()
          self.fc1 = nn.Linear(1, 55)
          self.fc2 = nn.Linear(55, 30)
          self.fc3 = nn.Linear(30, 25)
          self.relu = nn.ReLU()
          self.dropout = nn.Dropout(p=0.2)
          self.softmax = nn.Softmax(dim=0)
  
      def forward(self, x):
          x = self.fc1(x)
          x = self.relu(x)
          x = self.dropout(x)
          x = self.fc2(x)
          x = self.relu(x)
          x = self.dropout(x)
          x = self.fc3(x)
          x = self.softmax(x)
          return x
      
  net = Net()
  
  # A list of 10000 random numbers between 0 and 24
  dataset = pd.DataFrame([random.randint(0, 24) for _ in range(10000)])
  dataset['label'] = dataset[0] # The label is the same as the input
  
  train, test = train_test_split(dataset, test_size=0.2)
  
  criterion = nn.CrossEntropyLoss()
  optimizer = optim.Adam(net.parameters(), lr=0.001)
  
  for epoch in range(5):
      # Train
      running_loss = 0.0
      for i in range(len(train)):
          inputs = train.iloc[i, 1:].tolist()
          inputs = torch.tensor(inputs, dtype=torch.float)
          labels = train.iloc[i, 0]
          labels = torch.tensor(labels, dtype=torch.long)
  
          optimizer.zero_grad()
          outputs = net(inputs)
          loss = criterion(outputs, labels)
          loss.backward()
          optimizer.step()
  
      # Evaluate
      correct = 0
      total = 0
      with torch.no_grad():
          for i in range(len(test)):
              inputs = test.iloc[i, 1:].tolist()
              inputs = torch.tensor(inputs, dtype=torch.float)
              labels = test.iloc[i, 0]
              labels = torch.tensor(labels, dtype=torch.long)
              outputs = net(inputs)
              predicted = outputs.argmax()
              total += 1
              correct += (predicted == labels)
      
      print('Accuracy of the network on the test set: {:.2f}%'.format(100 * correct / total))
  
  I believe my task is pretty simple, but I just can't think of the right architecture to do it. Do you have any ideas? :)
  
  **Tags:** neural-networks, classification, training, pytorch, one-hot-encoding
  > **Comments:**
  > **username_1**: I'm not familiar with pandas but how does this dataframe layout work? Is this training with only 1 number per batch?  
  > **username_0**: Yup, I checked that and it's working :)  
  > **username_1**: It is training with only 1 number per batch?  
  > **username_0**: There's no batching here, to be more specific...  
  > **username_1**: I think this generally leads to suboptimal training unless the learning rate is set very low. In each training step the network converges towards whichever number was presented in that step, with nothing making sure the other numbers also keep working.  
  
  ---
  
  ## 0 answers
  
  