Peer Review

Peter Henne posted a series of questions about peer review. Dan Nexon noted that while the post had generated lots of traffic, no one had offered answers to Peter's questions. Here are mine.

1. Is there an objective standard for “so what?” No, there is not. Yet, this doesn't make it fully subjective. Any good paper will explain why what it reports matters, and few papers under-sell their findings. A reviewer's job is to evaluate the degree to which those assertions are warranted. A good rule of thumb here might be, if you are struggling to decide whether a paper you are reviewing is important, it isn't.

2. When is a methodological flaw a disqualifier? Always. Whether such flaws warrant a reject or an R&R depends upon the severity of the flaw and the potential significance of the results once the flaw is corrected. 

3. Should I take into account the prestige of the journal? Yes. But this consideration matters only in regard to the "so what" question. Thus, disallow flawed research everywhere. Competent work (even cutting edge methods) that makes smaller contributions belongs in less prestigious outlets. Thus, many reviews I write begin with, "this is an excellent piece of research, but it is not important enough to be published in (APSR/IO/AJPS)."

4. Is it fair to impose my sub-field's standards on interdisciplinary journals? Yes, interdisciplinary does not mean absence of discipline. If you do not impose the standards that you know (those in your discipline), what standards will you apply?

5. Should theory-free inductive studies be published in scholarly journals? Isn't this a variant of the so what question?

6. Can I tell them to cite me? Sometimes. See below.

7. How extensive should the literature review be? As extensive as it needs to be, and not a bit more so. The purpose of the lit review is to answer the "so what" question. Presumably, the research a paper reports offers a novel contribution to some research program. To do so the lit review must accurately characterize the current state of knowledge. If important contributions are omitted, the author must be encouraged to incorporate them. If your work is an important and omitted contribution, then yes, you can tell them to cite you. If your work isn't an important contribution, then find some other way to get people to read your work.

More broadly, Peter asks, "what constitutes a publishable paper, what necessitates revise and resubmit, and when is a rejection fair?" There are no rules, but here are my general guidelines. Well-written papers that report well-designed and well-executed research that speaks to some problem of public or scholarly concern deserve to be published (but not necessarily in the APSR). Rejection is fair when a paper is profoundly flawed or is trying to punch above its weight. Everything else warrants a revise and resubmit on the first iteration. Be unforgiving on the second iteration.