finetune.py
The way to do it with seq2seq/finetune.py is to put the docs in a directory with the following format:
train.source
train.target
val.source
val.target
test.source
test.target

line i of train.source should be “corrupted”, line i of train.target should have puncation.
same with val, test.
Then, depending on the language of the docs, run:
export PYTHONPATH="../":"${PYTHONPATH}"
python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 --data_dir $DATA_DIR \
    --model_name_or_path bart-base

From scratch
You can also use finetune.py to train from scratch by calling, for example,
config = BartConfig(...whatever you want..)
model = BartForConditionalGeneration.from_pretrained(config)
model.save_pretrained('rand_bart')

But I would not do that in your position.
(If the docs are not in english you could try starting from mbart.)
From your code
If you want to do it from scratch as a learning exercise, make sure your decoder has a causal mask so it can’t attend to the next word.
I don’t totally understand the tensor you printed, but it looks concerningly redundant.
Let me know if that helps, sorry for the slow response.