 Evaluation  See the notebook `evaluate_ct5.ipynb` for an example of how to evaluate cT5 in terms of accuracy and perplexity. The notebook `profile.ipynb` shows how to profile the model to get runtimes.  Here is a comparison between cT5-small and T5-small on a subset of the WikiText-103 dataset using deterministic greedy search:  | Model | Exact match ↑ | Edit distance ratio ↑ | Perplexity ↓ | Time (seconds) ↓ | |-------|---------------|----------------------|--------------|-----------------| | T5-small | 0.11          | 0.60                 | 2.22         | 44.71           | | cT5-small | 0.09          | 0.58                 | 1.48         | 10.63           |  On this toy dataset, cT5-small has a lower perplexity while being faster than T5-small. However, more experiments are needed for a rigorous evaluation.  If you are interested in applying cT5 to real data, please contact me. 