I am trying to model a real-valued function with real-valued input and outputs.
however, before that I thought it’s a good idea to understand how to model the simplest function, the “constant” function. The architecture is simply an input y, some non-linearity, then back to a single output y_pred. hence the title “autoencoding a single real number”
here is what I came up with so far

  
      gist.github.com
  
  
    https://gist.github.com/evanthebouncy/acf1bacd277ac98fdd588ee9d8ee9a57
auto_enc_real.py
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
import torch.nn.functional as F
import random
from tqdm import tqdm

if torch.cuda.is_available():
    def to_torch(x, dtype, req = False):
This file has been truncated. show original




  
  
    
    
  
  


I could not get it to converge. My value of y is between 0 and 1, so my model happily learned that it should output 0.5 all the time (kind of a common issue). I tried changing the loss function to L1 loss, L2, nothing worked.
anything helps. I implemented a transformer network for some fancy task and had to tear it all down to realise this was the core problem of my model not converging . . .
edit: solved.
solution: David Qiu torch.sum((y - y_pred) ** 2): dimensions don’t match and it is doing some broadcasting to mess you up