Solved by tom in post #2 
              
              
                You could apply the first step separately.
Most times I have seen this, people used non-overlapping samples an cached the states (see e.g. fast.ai’s lecture on language models).

Best regards 
Thomas