thx for the ping
for 12-6 it’s  [0, 2, 4, 7, 9, 11] bc I arbitrarily decided I wanted to keep the first and last layer in.
for 12-3 it’s [0,6,11] for 12-1 it’s [0]. 12-4 [ 0, 4, 8, 11] working way better than 12-3 for mbart-large-enro in early experiments.
Note: A full copy of the teacher would be 12-12.
Code: https://github.com/huggingface/transformers/blob/09a2f40684f77e62d0fd8485fe9d2d610390453f/examples/seq2seq/distillation.py#L403 5
            layers_to_copy = {  # maps  num layers in student -> which teacher layers to copy
                1: [0],
                2: [0, 6],
                3: [0, 6, 11],
                4: [0, 4, 8, 11],
                6: [0, 2, 4, 7, 9, 11],
                9: [0, 1, 2, 4, 5, 7, 9, 10, 11],
                12: all_layers,
            }