YuvalA:

however, if I use 2*N it won’t be good by the problem’s logical.


If you have some additional input structure, it is best to learn some fixed size 1d “summaries” before RNN, as none of standard implementations supports input groups or structures beyond vectors.