Here is what I would try

implement x’ W y using nn.Bilinear(p, q, k)

implement V[x, y] using nn.Linear(p+q, k) and torch.cat((x, y), -1) as [x, y]

both of the above provide a bias term by default so that takes care of b

implement u' * output_of_f using nn.Linear(k, 1, bias=False)