Since the entire batch seems to use the first 6 layers, you could directly pass it to them and split the batch afterwards e.g. by indexing. Once you have the two chunks you could then pass them to the corresponding head and if needed concatenate them afterwards (in the same order).