In a feedforward model, it might be difficult to describe the relative importance of layers if they are in sequence because, the learned representations can be different (e.g., at different levels of abstraction such as simple patterns vs. high level concepts). It’s more common to compare the relevance of two layers in parallel because one can directly weigh the contribution of each to the output—but it sounds like you want to compare two layers in sequence.
In this case, a hack might be to use a ResNet-style model with identity or skip connections between certain layers, and to do an ablation study. For example, if you already know your desired metric (e.g., accuracy) you can simply ablate layers with this style of architecture and observe how the accuracy is affected (possibly with some finetuning after truncating the model).