It seems to me that you use the self.binary twice in the forward pass. (Although I don’t see that in your code)
To avoid all the possible conflicts, I would suggest not to save the Function instance as a class attribute(especially if the Function has buffer).
Another suggestion is to replace your binary_activate with other simple Function(but also has save_for_backward), and see if it’s the problem of your implementation of the binary_activate, or because of the way you use the Function.