Interesting, Write with Transformer isn’t based on the generate method, but is simply sequential decoding with topk/topp filtering. We should update it to take full advantage of that method, which is stronger.
We won’t have the bandwidth to do the upgrade just yet, but I’ll keep you posted here.