Thanks for the description as it fits my understanding. The peak memory would be increased, since you would have to store U, B, W, and Y at one point during the forward pass. Since Autograd needs to backpropagate through Y, the intermediates will be kept.
If you have a way to delete W and recalculate it in the backward pass, you try to could implement a custom autograd.Function with this logic.