ARR review quality
Over the past year, with 8+ papers submitted to ARR, I can confirm that the quality of reviews has dropped significantly, and this is reflected in discussions with colleagues from many universities and labs who share the same experience.
As an NLP community, what do you think we can do to avoid such low-quality reviews further, while also reducing randomness in paper review assignments? There are several reasons: first, inexperienced authors review the paper and do not clearly understand the task or the evaluation criteria; next, experienced authors are assigned to a new topic; and finally, there are problems with the review rubrics. I think ARR currently lacks explicit criteria for paper evaluation, such as TACL/TMLR journals, like: "Does the paper introduce a new Method? benchmark? evaluation framework/tool? Is the related work properly discussed, and are the baselines properly selected? "
I would be interested to hear what others think. What changes could improve the quality of ARR reviews?