you can keep your “neurons” in a ParameterList, and keep adding new Variable or nn.Parameter of neurons to that list, whether that be each individual neurons (not super efficient), or say a block of 4096 neurons whenever you want a new set.
You can also take the “some are silenced” approach, and that’s not a bad idea either.