It’s infuriatingly hard to understand how closed models train on their input
There&#x27;s been an existential issue I&#x27;ve seen with publishing, concerning this (same&#x2F;related issues to the article). There&#x27;s models and datasets that are closed and being published in works. Here&#x27;s an example[0]. ICLR states<p>&gt; Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names. This means that the submission must not contain acknowledgements or any link (e.g., github) that would reveal authors’ identity.<p>While Chinchila certainly is not a github link it is an acknowledgement that reveals the lab identity. Which leads us to one of two situations<p>1. The reviewers don&#x27;t know the model&#x2F;dataset is proprietary: In this case, we likely have a reviewer that is unqualified to even be reviewing.<p>2. The reviewers know: In this case, there&#x27;s an ethics violation.<p>== Why this is a problem ==<p>There&#x27;s several issues here: bias, evaluation, reproduction, ethics&#x2F;scientific goals.<p>Bias: Most of these models and datasets are well known within the community and thus we know which labs they are from. We&#x27;re biased to believe that works from big labs are higher quality and more accurate. This may be true to some extent, but your purpose as a reviewer is to judge the merit of the work. We try to do so in as much isolation as possible to prevent ourselves from being influenced and just focus on the merits[note0]. For example, any work with JFT is going to be known as a work from Google.<p>Evaluation: We&#x27;ll use this work[2] as an example (it uses JFT and a lot of TPU compute: i.e. Google). Since the work only uses JFT we don&#x27;t know if this scaling only works for JFT trained models, especially since it depends on the pretraining not overfitting (I&#x27;m unconvinced tbh)[note1]. At least their table compares mostly to other JFT models, but it is mostly meaningless because JFT changes size frequently and we can&#x27;t assume consistent data. All we know is that this result is valid for JFT pre-trained models. Their claims are much stronger than this.<p>Reproduction: This is rather obvious. But we also have good reason to believe that these models change as well as the datasets. So even if there was a leak we have issues with works quickly losing validity as the next work is using different settings than the previous and intimate knowledge would be required to know if this affects results or not. NLP has this issue with OpenAI changes. It is very noisy research doing things like this.<p>Ethics and Scientific goals: Our job is to advance human knowledge. This can be done in a noisy process, but we do have a responsibility to make this as clear as possible. Our job is NOT to be ads for big companies, but closed work does so. In fact, this is a big reason big companies publish. They are great publicity. This is fine, as long as they provide utility to the community. We have other ethical issues[note2]<p>== Proposed Solution ==<p>Submissions must utilize open models and datasets. The work is evaluated based on that. After acceptance the works can add. This way there is no deanonymizing information, we ensure reproducibility, but we also are able to gleam the potential results from the proprietary efforts. This allows for the big labs to get their publicity while ensuring that anonymity is not broken and we&#x27;re not biased to big labs. This doesn&#x27;t solve the arxiv issue and the dumb social media policies[note4] but it is a a simple step we can implement right away. It is also a step that YOU the reviewer, a single person, can take action against. We need to take a serious look at ourselves and ask what is the point of our science. Is it to make money or to progress human knowledge? You can do both, but what is the main goal. I&#x27;d argue that our goal is knowledge. It is fine to make profits off this knowledge, but I&#x27;m afraid we&#x27;ve incentivized the profits too much and our system is not aligned with the original goals.<p>Goodhart&#x27;s Law is a bitch.<p>== Notes&#x2F;Bib ==<p>[note0] There are some biases we should concern ourselves with, such as compute power, but this requires more nuanced understanding. A work that can spend months tuning hyperparameters will always have better benchmarks than works that spend a month of compute for their entire work. These can&#x27;t be evaluated on benchmarks alone and doing so will cause us to miss extremely important works (see Lottery Ticket Hypothesis[1]). Reviewing should be hard and require nuance, so there should be certain roadblocks to make it difficult to be lazy.<p>[note1] They do near de-dupe but we can&#x27;t know how effective the techniques are so we can&#x27;t know how valid these claims are.<p>[note2] Conferences are a bad way to do science. Conferences are filters, journals are QC. ML&#x2F;CS uses conferences. This is a zero sum environment where your competitors are given the opportunity to filter your work. You don&#x27;t need active collusion to form bad actors. It pits the people evaluating the work against the work. A journal system instead has reviewers focusing on adding value to the work. This is because journaling has rounds and discussions. Conferencing is nearly zero shot with a single chance for rebuttal, often limiting to a single page to respond to 4 reviewers with vastly different concerns. We have other issues like reviewer quality (remember ACs oversee reviewers[3]), qualifications, collusion[4,5], fraud, and more. But these are for a longer discussion.<p>[note3] Social media rules prevent authors from evangelizing their works in public, such as on Twitter. It does not prevent doing so in private or others from doing so for you. This not only leads to (actively encourages) collusion rings but worsens the problem. Big labs will always get extra publicity. People watch for their works, they have more followers, and so on. Small labs&#x27; only chance to get heard is to actively evangelize their work, and we just cut off their tools while not affecting big labs. A scratch to the giant is the loss of a limb to an ant. We shouldn&#x27;t need evangelization but networks exist so it is necessary. I don&#x27;t think we should stop pre-prints (they&#x27;re too valuable), but we need to recognize the problems that they result in. We also shouldn&#x27;t have people at the top of the ivory tower making decisions about how the handle the plights of those at the bottom without actively working with those at the bottom.<p>[0] (Notice Carmen&#x27;s review) <a href="https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=OpzV3lp3IMC" rel="nofollow">https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=OpzV3lp3IMC</a><p>[1] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1803.03635" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1803.03635</a><p>[2] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2305.13035" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2305.13035</a><p>[3] <a href="https:&#x2F;&#x2F;qipeng.me&#x2F;blog&#x2F;what-does-an-area-chair-do&#x2F;" rel="nofollow">https:&#x2F;&#x2F;qipeng.me&#x2F;blog&#x2F;what-does-an-area-chair-do&#x2F;</a><p>[4] <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;MachineLearning&#x2F;comments&#x2F;nxbcl9&#x2F;d_collusion_rings_noncommittal_weak_rejects_and&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;MachineLearning&#x2F;comments&#x2F;nxbcl9&#x2F;d_c...</a><p>[5] <a href="https:&#x2F;&#x2F;analyticsindiamag.com&#x2F;whats-wrong-with-the-paper-review-process-of-reputed-ml-conferences&#x2F;" rel="nofollow">https:&#x2F;&#x2F;analyticsindiamag.com&#x2F;whats-wrong-with-the-paper-rev...</a>