Thanks for the responses, @Narsil and @osanseviero!
I’m integrating the HF text-generation API into a word-processor, and it would be a nice user experience if the tokens could be streamed directly into the client.
Right now, if I’m generating 100 tokens, it might take 10 seconds to complete the task. And the user has to sit and wait the full duration before they can see any results at all. But it probably takes less than 1 second to generate the first token, and each subsequent token probably takes only a few milliseconds to generate… There’s no need to wait for the last token to arrive before showing the first one, so it would be great if the tokens could stream.
Here’s how OpenAI does it…

I understand the underlying model class has a generate method, that returns one or more output sequences. But how difficult would it be to have a generateAndStream method, that takes an output stream as an argument, and writes tokens into that stream?