Using hidden states from BERT (Similar to using precomputed hidden states in GPT2 model "past" argument)
Hi,

In question and answer model using BERT, I will be querying context and around 50 questions on the same context. To reduce latency for obtaining results, I would like to cache hidden states of context before prediction.
For each question answer prediction on the same context, can the model use precomputed hidden states of context and get answer predictions?
Similar to GPT2, past can be used to reuse precomputed hidden state in a subsequent predictions.

Thank you,


Hi, no, Bert doesn't have a cached hidden-states option.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
