I am trying to train a transformerXL model from scratch, but I am struggling to understand the meaning of the d_head and d_inner in the config.
I understand d_head as being the dimension of the value vector after attetion has been applied, but I have no clue what d_inner should be.
The doc only states that:
d_inner (int, optional, defaults to 4096) — Inner dimension in FF

What does FF mean here?