I solved it by using multiple output, with one of them being the attribute I desired. It works, although a bit ugly and is not the elegant solution I expected. I use dictionary for output to indicate which is the original output tensor, and which is the attribute. I am not sure yet if the gradient will work correctly but I did not find reason why not. Using tuple or list might not be possible as I only use these for those layers that I need to get the attribute for, such as Conv, and some other interface layer, such as Sequential, and for other layers I need to use the traditional behavior (otherwise I can reimplement all layer types I used and all have output with the desired attribute using some default value). Anyway, this is one possible solution, but probably not the best.