Even as pre-trained language encoders such as BERT are shared across many tasks, the output layers of question answering, text classification, and regression models are significantly different. Span decoders are frequently used for question answering, fixed-class, classification layers for text classification, and similarityscoring layers for regression tasks, We show that this distinction is not necessary and that all three can be unified as span extraction. A unified, span-extraction approach leads to superior or comparable performance in supplementary supervised pre-trained, low-data, and multi-task learning experiments on several question answering, text classification, and regression benchmarks.
This paper asks whether it works to remove task-specific heads and treat classification and regression problems as span extraction, by formatting problems in such a way that a single span extraction model can be used.  This is a reasonable question to ask, and the authors performed a very large number of experiments attempting to answer this question.  The authors claim that using span extractive models instead of task-specific heads yields improved performance over separate heads.
My main concern with this work is actually with something that would otherwise be a strength - the very large number of experiments.  Looking at the results tables, I come to a different conclusion from the authors: there does not appear to be a significant difference between using a single head or using multiple heads (this is still an interesting conclusion).  The numbers presented all appear to be within the bounds of random fluctuations, which are not controlled for at all with standard statistical testing methodologies.  And with the very large number of experiments, there are bound to be some that stand out.  This is especially true given the methodology used for small datasets - even if it was used by prior work, it is still not statistically sound to take the max of 20 samples from a training distribution and report the difference without any other statistical testing.
As an example, look at table 3b.  This is claimed as a big win for SpExBERT when comparing the MNLI fine-tuned version.  But if you look at the other rows in the table, SpExBERT is worse than BERT by a *larger margin* than it is better in the MNLI case (contradicting the claimed result from table 2).  And this general trend is seen across the tables - the differences are small and inconsistent, making it look very much like we are just seeing random fluctuations.  This must be controlled for statistically in order to make any valid conclusions.  The one possible exception here seems to be SST.  Those results on the dev set do indeed seem to be more consistent, which is interesting, and hints at the utility of using "positive" and "negative" as natural language descriptions, as the authors claim.  It's not very convincing, however, as the test set difference is very small, and SpExBERT had more opportunities to find a good dev set number, as it had more experiments.
My second major concern is with the experimental set up.  The authors want to claim that using a unified span extraction format yields superior performance to having separate heads.  But there are baselines missing to really demonstrate this claim.  The multiple head setup isn't really evaluated as a baseline in most of the experiments (e.g., using SQuAD / other QA datasets as an intermediate task in table 2, using BERT for any of the QA datasets in table 3).  So, even if the above issue of statistical testing were solved, it would still be very hard to evaluate the claim, as the proper comparisons are not present in the majority of cases.
My main conclusion from reading this paper is that it does not appear to matter what head you use for these particular datasets.  This is an interesting result, though it is not the one that is claimed in the paper.  I think it would be very challenging to extend this approach to a broader set of tasks, however, as the authors suggest towards the end of the paper.  How do the authors propose handling cases where it is not feasible to put all possible outputs in the context?  This includes any generative task (including generative QA) and any kind of structured output (like parsing, tagging, etc.), or a regression task that does not lend itself well to bucketing.
Minor issues:
I would be careful about claiming that you've successfully captured regression in a span extraction format.  You have one regression task, and it's one where the original labels were already bucketed, so the bucketing makes sense.  I am very skeptical that this would actually work for more general regression.
Re the paragraph titled "SpEx-BERT improves on STILTs": Note that SpExBERT requires additional data preprocessing and hand-written templates when switching between tasks, which is not necessary in the method of Phang et al.  There are always tradeoffs.  Neither using separate heads nor doing your additional processing are very hard, so I wouldn't make as big a deal out of this as you do.  If you want to talk about one of the methods using more "human inductive bias", or whatever, the hand-written templates in yours might actually be _more_ "incidental supervision" than a very small difference in model architecture.  But again, the difference here is very small, not worth noting in the paper.