When you peer-review a paper, you can make one of a few basic recommendations to the editor. One option is this: do not publish the paper.
So what criteria should you use to make such a recommendation? In this post, I argue that some criteria are better than others.
1. Is the paper convincing?
A friend of mine mentioned this criterion the other day: “…[philosophy] papers ought to be convincing.” Call this the Convince Me standard or CM.
Maybe you think that CM sounds like a reasonable standard for peer-review. I don’t. I think CM let’s in too much noise. There are just too many unimportant factors that can have an impact on our being convinced (or not being convinced).† We can be convinced by bad arguments. Further, we can fail to be convinced by good arguments. This suggests that the quality of an argument is not always the only determining factor in our being convinced (or not being convinced) by an argument. So what other factors might influence us? Perhaps we are disposed to accept or reject certain arguments. Or perhaps we are less disposed to reflect on the quality of arguments, in general. For example, religious believers seem to be less reflective than atheists and agnostics (Pennycook, Ross, Koehler, and Fugelsang 2016).
2. Are the arguments logically valid?
If CM won’t work, then what other standard should we use? Analytic philosophers will probably agree about one thing: arguments should be logically valid. (Not sure what logical validity is? Check out the video below).
If logical validity (V) is the right starting point, then maybe we can take a cue from Stephen Clarke. We can “recast [philosophers’ arguments] as hypothetical rather than categorical arguments” (2013). If we do this, then maybe a philosophy paper need not offer more than “valid arguments that might possibly be sound, if evidence could be found to support currently unsubstantiated premises” (ibid., italics added).
Fair enough. So what counts as “evidence”?
3. Do the premises seem intuitively correct?
If a claim is intuitive, is that claim thereby supported by evidence? Consider premise P:
P: gratuitously torturing human babies is bad.
You might think this premise is obviously true. Philosophers sometimes count on you having such intuitions. Consider this line: “[my claim] is so intuitive that most will need no more proof than its statement” (Wenar 2008). So, when a philosopher points out that you intuitively accept P, has she thereby presented evidence for P?
To answer that question, we must first answer a prior question: Is there a well-confirmed developmental theory about intuition that would explain why we should treat our intuitions as evidence? Answer: no (Clarke 2013).
Some philosophers disagree. They don’t think that we need a good developmental theory. Rather, they offer a priori arguments that intuitions should count as evidence (e.g., Huemer 2007; Williamson 2011). If we accept these arguments, then we might think that, for the purposes of peer-review, the premises of arguments should at least be intuitive and they should not be counterintuitive. Call this the intuitionist criteria, or I.
Philosophers disagree about I, but they usually agree about one thing: having an intuition that P just isn’t the same thing as P being well-supported by evidence. After all, we can have an intuition that is demonstrably false — even friends of intutions like Michael Huemer will admit that much. So even if we grant that intuitions count as evidence, they don’t necessarily count as good evidence.
But if intuitions won’t do, then what will?
4. Are the premises well-supported by evidence?
Let’s say that a couple studies support a premise. Does that mean that the premise is well-supported by evidence? Probably not. After all, science itself often undermines such findings (Open Science Collaboration 2015). What we want are findings that have replicated many times. Further, we want findings that replicated in the vast majority of replication attempts. In other words, we want well-confirmed empirical generalizations. Sometimes these kinds of findings manifest in meta-analyses, meta-syntheses, and review articles. (And, importantly, sometimes they don’t.) The demand to support premises with well-confirmed empirical generalizations can be called… uhh… the well-confirmed generalization criterion? WCG for short.
5. What if the evidence isn’t clear?
There’s a catch: philosophers often use premises about which there aren’t any well-confirmed generalizations, one way or the other. The reason might be that the relevant science is too amniotic and undeveloped to have discovered well-confirmed empirical generalizations or that, in some domains, empirical generalizations just aren’t a thing (e.g., math). This suggests that there might be arguments to which WCG does not apply.
So how do we evaluate claims about a domain in which there are no well-confirmed generalizations? Perhaps we have to tentatively fall back on assumptions. But not any assumption will do. I will propose that such assumptions should be (i) widely shared and/or (ii) pragmatically useful. Call this the practical speculation criteria, or PS.
Conclusion
There are too many irrelevant factors involved in our being convinced (or not convinced) by an argument, so we cannot include the Convince Me, or CM, in our criteria for publication. We probably want to include logical validity in our criteria, but we will need more than that if we want to prevent bad (albeit valid) arguments from being published. This is why we need the well-confirmed generalization criterion, or WCG. And when that criterion doesn’t apply, we might look to intuitions. But even if we grant that intuitions count as evidence, they might not be widely shared and/or pragmatically useful, so I would prefer the practical speculation, or PS criterion to the intuitionist, or I, criterion.
This leaves us with a combination of criteria: the logical validity (V) criterion and either the well-confirmed generalization (WCG) criterion or (if that doesn’t apply, then) the practical speculation criterion (PS). Call this the Validity plus Evidence or else Practical Speculation criteria, or V⋅WCG∨PS for short.
On this criteria, reviewers can reject a paper if the paper lacks a logically valid argument and if the premises are not supported by widely accepted and/or pragmatic assumptions (or by well-confirmed empirical generalizations, where applicable). But reviewers should not reject a paper merely because the paper didn’t convince them or because certain premises were counterintuitive.
Disclaimer: There might be additional standards and criteria that we wish to impose as peer-reviewers — feel free to suggest some in the comments. I only mean to argue that we need not impose CM and I and that, at the very least, we should impose V⋅WCG∨PS.
Notes
† I am not claiming that the quality of an argument has no effect on whether one is convinced by the argument; I am only saying that the quality of an argument is only one among a suite of factors that can have an effect on whether one is convinced by the argument.
Featured image: “Urval av de bocker…” by Johannes Jansson, CC BY 2.5 Denmark
.jpg){kind=link}