Philosophers and cognitive scientists tend to think that reflective reasoning will improve our judgments and decisions. The idea reflection will lead us to test our judgments by “looking for their coherence with our beliefs about similar cases and our beliefs about a broader range of …issues” a la reflective equilibrium. This sounds intuitively plausible. But is it true? In this post I briefly present some research suggesting that reflective reasoning often, but does not always improve our judgments and decisions.
1. Background
Cognitive scientists have developed so-called reflection tests. They’re designed to see whether you reflectively considered your answer to various questions. Want to try one?
The Cognitive Reflection Test
A bat and a ball cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?
Most people are lured into answering questions like this incorrectly (Frederick 2005). However, some people get it right.
The Test’s Assumptions
Philosophers and cognitive scientists tend to assume that falling for the lured incorrect answer to reflection test questions (e.g., that the ball costs 10 cents) indicates a lack of reflection from research participants. Further, cognitive scientists tend to assume that correct answers involved reflection.
Surprisingly, however, the lured responses to reflection test questions can apparently involve substantial reflection. Here are some examples.
Wrong. [You think the answer is 5 cents but] you haven’t thought this through correctly.
There are NINE different configurations that work.
Ball 9 cents + (bat 1 cent + 100 cents) = 110
Ball 8 cents + (bat 2 cents + 100 cents) = 110
Ball 7 cents + (bat 3 cents + 100 cents) = 110
Ball 6 cents + (bat 4 cents + 100 cents) = 110
>>Ball 5 cents + (bat 5 cents + 100 cents) = 110<<
Ball 4 cents + (bat 6 cents + 100 cents) = 110
Ball 3 cents + (bat 7 cents + 100 cents) = 110
Ball 2 cents + (bat 8 cents + 100 cents) = 110
Ball 1 cent + (bat 9 cents + 100 cents) = 110 (Source)
I swear I looked at this for awhile and still don;t get how the bat cost $1.05. How in the world did they come up with this number. (Source)
Also, those are examples from the comments section of the internet. So maybe we shouldn’t take them too seriously. Unfortunately, there may not be great evidence about whether reflection helps most people solve trick questions like this …until now.
2. New Research
My colleagues and I decided to test the idea that reflection usually helps people solve reflection test questions similar to the one above—except that we didn’t use math mathematical reflection test questions like the one above (Szaszi et al., 2017) so that we could rule out a confound between reflection and numeracy.
Method
We had people think aloud as they took a new validated, non-mathematical, 10-item reflection test (Sirota et al., 2020). Importantly, thinking aloud didn’t seem to impact reflection test performance.
Then we listened to every recording of every answer to determine whether participants engaged in reflection by “backing up” to (re)consider their initial impulse (Kahneman & Frederick, 2002; Korsgaard, 1996). Specifically, we coded whether they verbally (a) reconsidered their initial answer and (b) considered reasons for or against any particular answer. So what did we find?
Result
We found that reflection usually helped reflection test performance but that in a substantial minority of cases (e.g., 10-30%), it didn’t. That is reflection test performance was usually (but not always) better among those who verbally reconsidered their initial response and/or verbalized a reason for or against any response.
Here’s a link to the project on ResearchGate in case you want to follow along to see the final published version when the paper is accepted: researchgate.net/publication/341131477
Implications
- Cognitive scientists may need to rethink our interpretation of reflection test questions. That or they need to use think aloud protocols to verify their interpretation of reflection test responses.
- Philosophers may want to rethink their claims about the alleged benefits of reflection. Reflection may usually help our reasoning, but it seems to be unhelpful in a substantial number of cases—the subject of another paper of mine titled “Bounded Reflectivism” (under review).
Follow along for more about my own research and other research that I share (at least daily) on social media.
