Stack Exchange
Does order matter in this causal language model?
Say you've implemented a causal language model like so:
def get_causal_xy(data, max_len=8):
    xs=[]
    ys=[]
    index = 0
    for i in range(len(data) - max_len):
      xs.append(data[index: min(len(data), index + max_len)])
      ys.append(data[index+1: min(len(data), index + max_len + 1)])
      index = index + 1
    return xs, ys

would then order matter. Say you have A,B,C,D words and youre trying to find P(y)
would P(y | A, B, C, D) be the same as P(y | C , B , A , D)?
My hunch is yes, because you still have access to the same words, and casual language model like this simple one doesnt use positional encoding

Probably you mean "causal", not "casual"?

yes! do you know the answer @MightyCurious
