I’m not sure how exactly your code works, but as far as I understand you are checking if there is a size mismatch and if so you use factor to downsample the larger tensor to finally concat both.
This should generally work. However just by looking at your code snippets there seems to be a mismatch between both codes in the conditions.
While you are using merge_layer = {..., 'input': [layer1_id, new_id_pool]} in the first case, the arguments are switched for the second case merge_layer = {..., 'input': [new_id_pool, layer2_id]}.
Could this be the issue? If not, could you provide a small executable code snippet so that we could try to reproduce this problem?