Structuring Dataset for Fine Tuning GPT-2 with Song Lyrics?
I'm looking to fine tune a GPT-2 model with several songs from the same artist. However, I'm not exactly sure how I should be feeding nearly 300 songs into the model to learn from.

Currently, I'm breaking the lyrics into blocks of 100 and feeding them into my model:

```
lyrics = open('/JayZ_lyrics.txt', "r", encoding="utf-8").read()
lyrics = lyrics.split('\n')
lyrics = ''.join(lyrics)

tokenizer = GPT2Tokenizer.from_pretrained('distilgpt2')
tokenized_text = tokenizer.encode(lyrics)

examples = []
block_size = 100
for i in range(0, len(tokenized_text)-block_size+1, block_size):
    examples.append(tokenized_text[i:i+block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])
```


Then, I feed my dataset into the model and train on it.

```
dataset= tf.data.Dataset.from_tensor_slices((inputs,labels))
BATCH_SIZE = 16
dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(epsilon=1e-08, clipnorm=1.0) 
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model = TFGPT2LMHeadModel.from_pretrained('distilgpt2')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=4, callbacks=[LRFinder()])
```

Is this the right approach? Or, should I turn every song into a vector, find the len of longest song, pad the other songs to keep uniformity amongst vector lengths, feed in dataset.

Also, should I include a BOS and/or EOS token `<|endoftext|>` for each block that I create?
By the way, did this approach work for you? I tried a similar one and I couldn't stop getting an error about incompatible output shapes when calculating sparse categorical accuracy.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Any solution or suggestions yet? @1337-Pete 