Hey everyone. I'm a graduate student currently studying machine learning. I've had a decent amount of exposure to the field; I've already seen many students publish and many students graduate. This is just to say that I have some experience so I hope I won't be discounted when I say with my whole chest: I hate machine learning conferences.
Everybody puts the conferences on a pedestal The most popular machine learning conferences are a massive lottery, and everyone knows this and complains about this, right? But for most students, your standing in this field is built off this random system. Professors acknowledge the randomness but (many) still hold up the students who get publications. Internships and jobs depend on your publication count. Who remembers that job posting from NVIDIA that asked for a minimum of 8 publications at top conferences?
Yet the reviewing system is completely broken Reviewers have no incentive to give coherent reviews. If they post an incoherent review, reviewers still have no incentive to respond to a rebuttal of that review. Reviewers have no incentive to update their score. Reviewers often have incentive to give negative reviews, since many reviewers are submitting papers in the same area they are reviewing. Reviewrs have incentive to collude, because this can actually help their own papers.
The same goes for ACs: they have no incentive to do anything beyond simply thresholding scores.
I have had decent reviewers, both positive and negative, but (in my experience) they are the minority. Over and over again I see a paper that is more or less as good as many papers before it, but whether it squeaks in, or gets an oral, or gets rejected, all seem to depend on luck. I have seen bad papers get in with faked data or other real faults because the reviewers were positive and inattentive. I have seen good papers get rejected for poor or even straight up incorrect reasons that bad, negative reviewers put forth and ACs follow blindly.
Can we keep talking about it? We have all seen these complaints many times. I'm sure to the vast majority of users in this sub, nothing I said here is new. But I keep seeing the same things happen year after year, and complaints are always scattered across online spaces and soon forgotten. Can we keep complaining and talking about potential solutions? For example:
- Should reviewers have public statistics tied to their (anonymous) reviewer identity?
- Should reviewers have their identities be made public after reviewing?
- Should institutions reward reviewer awards more? After all, being able to review a project well should be a useful skill.
- Should institutions focus less on a small handful of top conferences?
A quick qualification This is not to discount people who have done well in this system. Certainly it is possible that good work met good reviewers and was rewarded accordingly. This is a great thing when it happens. My complaint is that whether this happens or not, seems completely random. I'm getting repetitive, but we've all seen good work meet bad reviewers and bad work meet good reviewers....
All my gratitude for people who have been successful with machine learning conferences but are still willing to entertain the notion that the system is broken. Unfortunately, some people take complaints like this as if they were attacks on their own success. This NeurIPS cycle, I remember reading an area chair complain unceasingly about reviewer complaints. Reviews are almost always fair, rebuttals are practically useless, authors are always whining...they are reasonably active on academic Twitter so there wasn't too much pushback. I searched their Twitter history and found plenty of author-side complaints about reviewers being dishonest or lazy...go figure.
On the one hand, everything you said is very true. On the other hand, I think the CS community is unusually open about changing up their publishing system and trying things like double blind at a scale I don't usually see in other fields. Maybe with enough momentum, some of the current faults in how we publish can change (despite the misaligned incentives?)
Oh for sure, there are a lot of people who feel the way you do and who are open to trying new things to mitigate these problems.
I think it's important to be clear about the core problem, though, because otherwise you might be tempted to do a lot of work on solutions that are ultimately mostly cosmetic. Like, why is reviewing such a problem to begin with? It's ultimately because, for authors, there's a lot of incentive to prioritize publishing volume rather than publishing quality, because that's what gets you a job at NVIDIA.
Thus the publishing incentives are fundamentally set up such that you need a large amount of labor to do reviewing, because there's just such a large number of submissions. Double blind reviewing etc. can help to adjust the incentives a bit in favor of fairness but it ultimately does nothing to stem the firehose of frivolous garbage research that people try to get published in the first place.
So a real solution would do at least one of two things:
This problem exists throughout academia, but I think it's especially acute in CS and ML because of the weirdly constrained channels for publishing research. For example I think that using conferences as the primary method of communicating results has unnecessarily hamstrung the entire field of research. Other fields of study primarily use journals, which is inherently less expensive and more scalable.
I appreciate your laying things out clearly. I think breaking up the field into smaller, more journal-like venues sounds like a step in the right direction, and I'm sure some of that thought went into creating TMLR. I do wonder though if that becomes too popular, whether the same problems would reappear...people/companies would uphold a select number of venues, and everyone will end up submitting to those venues, etc...