Are you sure the right image shows the graph from top to bottom?
If so, how would e.g. the module with weight (320x320x3x3x3) be connected to weight (256x512x3x3x3), which should yield a shape mismatch error?