Stack Exchange
Large number of trainable parameters and small data but the model is working well
Large model, small training data but good result?
I have text auto-encoder model where I have used few layers of Transformer and trained it using my own data. I have a vocab size of 18664.
Here is my model:
Encoder:
class TransformerEncoder(nn.Module):
  def __init__(self, input_size, num_head, hidden_size, num_layers):

    super(TransformerEncoder, self).__init__()

    self.embd = nn.Embedding(word_count,input_size)
    encoder_layer = nn.TransformerEncoderLayer(input_size, num_head, hidden_size)
    self.transformer_enc = nn.TransformerEncoder(encoder_layer, num_layers)
    self.linear1 = nn.Linear(input_size,270)

def forward(self, x):
    x = x.long()
    emb = self.embd(x)
    mem = self.transformer_enc(emb)
    out = self.linear1(mem)
    return out, mem

Decoder:
class TransformerDecoder(nn.Module):
  def __init__(self, input_size, num_head, output_size, hidden_size, num_layers):
    super(TransformerDecoder, self).__init__()

    decoder_layer = nn.TransformerDecoderLayer(input_size, num_head, hidden_size)
    self.transformer_dec = nn.TransformerDecoder(decoder_layer, num_layers)
    self.linear1 = nn.Linear(input_size, output_size)

  def forward(self, x, mem):
    out = self.transformer_dec(x, mem)
    out = self.linear1(out)
    return out

The summary of the model:



As you can see from the model summary, there are huge number of parameters 71 and 81 millions. And I only have 7K text sentences where I have truncated each sentences into 10 words. So I have 7K sentences each with 10 words to train the model.
Is that enough to train these amount of parameters? Or am I missing something?
Though I am getting good result, I feel I am missing something.
Am I getting good result due to some leakage?

check if you are giving same data for training and validation

@prashanth i have different data for training and validation. From review of the code what I found is, I haven't used positional encoding and masking in the above code. Will that impact? Is it mandatory to use positional encoding and masking when using transformer for any task?

Can you confirm that the task is really autoencoding? That is, given an input $x$, the goal is to predict $x$?
In general, it’s impossible to know if (a) there’s data leakage and (b) masking is needed if we don’t see (a) the data processing code and (b) what the inputs are and how they’re batched. Also, without positional encodings a Transformer reduces to just a fancy bag-of-words model, which may or may not matter for your task. It’s hard to say more without knowing what the task really is.
