Ok, so the mask is obviously NOT a part of the gradient, but how is the segment embedding and it’s respective bias not differentiable? according to XLNet paper

Architecturally, different from BERT that adds an absolute segment embedding to the word embedding at each position, we extend the idea of relative encodings from Transformer-XL to also encode the segments. Given a pair of positions i and j in the sequence, if i and j are from the same segment, we use a segment encoding sij = s+ or otherwise sij = s−, where s+ and s− are learnable model parameters for each attention head. In other words, we only consider whether the two positions are within the same segment, as opposed to considering which specific segments they are from. This is consistent with the core idea of relative encodings; i.e., only modeling the relationships between positions. When i attends to j, the segment encoding sij is used to compute an attention weight aij = (qi + b)⊤sij, where qi is the query vector as in a standard attention operation and b is a learnable head-specific bias vector.

It clearly states the relative segement embeddings and the relative segment bias are learnable parameters. Are they, by default, only learnable during pre training, and the learning is “turned off” during fine tuning?