My perhaps inefficient but general (and correct by construction:) ) way of doing it is to start with an example input, and when building up the model in __init__() feed that input through each layer in turn, get the size of the output, use that to initialize the next layer, and repeat until done