I’m not sure I’m following. I went into more detail of my issue at hand. I wonder if your approach works? If so, I’m not sure how to apply your suggestions here…
Specifically - I’d like to do the following:
For every sample, I am interested in the topk activated units,
but each topk activated unit for a particular sample should be avoided in the next samples in the batch, so that all the topk activations are exclusive for each sample.
This is why I am inhibiting units with a mask before passing through topk, and the inhibitions are based on the topk activationis of the previous sample. So within a forward pass, it ends up like
x = activation

# run this chunk on each sample, not on the whole batch.
x = x * inhibition_based_on_prev_top_k   # initialized as ones_like(x)
x = get_topk(x)
inhibition_based_on_prev_top_k.update(x)


return x