 Dear Fellow Scholars, this is two-minute papers with Karojona Ifehir. We are here to answer a simple question. What is peer review? Well, in science, making sure that the validity of published results is beyond doubt, is of utmost importance. To this end, many scientific journals and conferences exist, where researchers can submit their findings in the form of a science paper. As a condition of acceptance, these papers shall undergo extensive scrutiny by typically two to five other scientists. This referring process we call peer review. Single-blind reviewing means that the names of the reviewers are shrouded in mystery, but the authors of the paper are known to them. In double-blind reviews, however, the papers are anonymized and none of the parties know the names of each other. These different kinds of blind reviews were made to eliminate possible people-related biases. There's a lot of discussion whether they do a good job at that or not, but this is what they are for. After the review, if the results are found to be correct and the reviews are favorable enough, the paper is accepted and subsequently published in a journal and slash or presented at a conference. Usually the higher the prestige of the publication venue is, the higher the likelihood of rejection, which inevitably raises a big question. How to choose the papers that are to be accepted? As we are scientists, we have to try to ensure that the peer review is a fair and consistent process. To measure if this is the case, the NIPS experiment was born. NIPS is one of the highest quality conferences and machine learning with a remarkably low acceptance ratio, which typically hovers below 25%. This is indeed remarkably low considering the fact that many of the best research groups in the world submit their finest works here. So here's the astute idea behind the NIPS experiment. A large amount of papers would be secretly disseminated to multiple committees, they would review it without knowing about each other, and we would have a look whether they would accept or reject the same papers. Re-reviewing papers and see if the results are the same, if you will. At a given prescribed acceptance ratio, there was a disagreement for 57% of the papers. This means that one of the committees would accept the paper and the other wouldn't and vice versa. Now to put this number into perspective, the mathematical model of a random committee was put together. This means that the members of this committee have no idea what they are doing, and as a review, they basically toss up a coin and accept or reject the paper based on the result. The calculations conclude that this random committee would have this disagreement ratio of about 77%. This is hardly something to be proud of. The consistency of expert reviewers is significantly closer to a coin flip than to a hypothetical perfect review process. So experts, 57% disagreement, coin flip committee, 77% disagreement. It is not as bad as the coin flip committee, so the question naturally arises, where are the differences? Well it seems that the top 10% of the papers are clearly accepted by both committees, the bottom 25% of the papers are clearly rejected. This is the good news. And the bad news is that anything in between might as well be decided with a coin toss. If the consistency of peer review is subject to maximization, we clearly have to do something different. Huge respect for the NIPs organizers for doing this laborious experiment, for the reviewers who did a ton of extra work, and kudos for the fact that the organizers were willing to release such uncomfortable results. This is very important and is the only way of improving our processes. Hopefully someday we shall have our revenge over the coin flip committee. Can we do something about this? What is a possible solution? Well, of course this is a large and difficult problem for which I don't pretend to have any perfect solutions. But there is a really interesting idea by a renowned professor about crowdsourcing reviews that I found to be spectacular. I leave the blog post in the comment section both for this and the NIPs experiment and we shall have an entire episode about this soon. Stay tuned. Thanks for watching and for your generous support and I'll see you next time.