The data quality on Amazon Mechanical Turk (mTurk) has suffered for years now (Byrd, 2023; Chandler & Paolacci, 2017; Moss & Litman, 2018; Chmielewski & Kucker, 2019; Ahler et al., 2020; Kennedy et al., 2020; MacInnis et al., 2020). There are a few ways to protect online survey data quality. In this post, I will briefly cover five strategies for weeding out junk data in online research (not just via mTurk), from easiest to hardest.
Continue reading 5 Ways To Overcome Junk Data From mTurk (and online surveys more generally)Tag: Research
Two Years In The Life Of A Grad Student: Time Logging Data
I have had some side gigs in graduate school that involved creating invoices for hourly work—web development, copyediting, research assistance, etc. I used Toggl to log my time. At some point, I realized that I could log all of my work time—not just the billable time. So in 2018 and 2019, I logged all of my work time. In this post, I will summarize the 2018 and 2019 data and mention some take-aways for 2020.
Continue reading Two Years In The Life Of A Grad Student: Time Logging Data
Why Make an Academic Social Network Profile (and a Website)?
Reality check: if I am not automatically notified of your research, I’ll almost certainly never know about it. And if I can’t find you online, you might as well not exist beyond your classroom, office, or lab. So if you’re an academic who wants people to actually read your work or even know that you exist, then read the following 300 words. They explain how to make your research followable and visible. It’s really, really easy. Don’t believe me? Check out the two videos to watch me do it in less than 15 minutes. So stop making excuses. In the words of the great scholar, Shia Lebouf:
Continue reading Why Make an Academic Social Network Profile (and a Website)?
The Moral Argument Against Footnotes and PDF
[Update: the latest version of this argument is now on the APA’s Blog.]
Once upon a time, I loved footnotes and PDF documents. Now I don’t. I prefer eBook format and endnotes. I admit that footnotes are handy sometimes. For example, when I read visually, it’s nice to have the notes on the same page as the body text. However, footnotes are not so handy for auditory reading. Neither are PDF documents. For instance, footnotes wreak havoc on auditory reading. They interrupt the audio stream of the main body of text — sometimes mid-sentence. And since many people have to rely on auditory reading to consume academic research, this means that PDF documents and footnotes decrease the accessibility of research. That’s bad. If we can avoid this bad, we should. And we can avoid it. So we should.
1. Books vs. Articles
Sometimes academic books are available in an eBook version that is amenable to auditory reading — e.g., Amazon’s Kindle format and Apple’s iBook format. And some academic books have a proper audiobook version — e..g, Amazon’s audiobooks. This is great, but… Continue reading The Moral Argument Against Footnotes and PDF
Research Questions & Mental Shortcuts: A Warning
Daniel Kahneman talks extensively about how we make reasoning errors because we tend to use mental shortcuts. One mental shortcut is ‘substitution‘. Substitution is what we do when we (often unconsciously) answer an easier question than the one being asked. I find that I sometimes do this in my own research. For instance, when I set out to answer the question, “How can X be rational?” I sometimes end up answering easier questions like, “How does X work?”. In an effort to avoid such mistakes, I will (1) explain the question substitution error, (2) give an example of how we can distinguish between questions, (3) give a personal example of the substitution error, and (4) say what we can do about it.
1. Substitution
In case you’re not familiar with Kahnemen’s notion of ‘substitution’, here is some clarification. In short, substitution is this: responding to a difficult question by (often unintentionally) answering a different, easier question. People use this mental shortcut all the time. Here are some everyday instances:
| Difficult Question | Easier Question |
|---|---|
| How satisfied are you with your life? | What is my mood right now? |
| Should I believe what my parents believe? | Can I believe what my parents believe? |
| What are the merits/demerits of that woman who is running for president? | What do I remember people in my community saying about that woman? |
For further discussion of mental shortcuts and substitution, see Part 1 of Kahneman’s Thinking Fast and Slow (2012).
Now, how does this mental shortcut apply to research? Continue reading Research Questions & Mental Shortcuts: A Warning
Visual Brainstorming on Whiteboards for Posters, Slides, and More
A whiteboard is pretty versatile. It can be used many times for many purposes. I use it during meetings and while working alone. In this post, I’ll explain how I use a whiteboard for creating visual aids.
1. Visual Brainstorming
I am very committed to the digital workspace. My library, papers, notes, handouts, etc. are in the cloud (more about that in this post). I do all of my reading and writing on a computer or a smartphone. But very occasionally a physical workspace trumps my digital workspace.
Visual brainstorming is one task for which a physical workspace outshines the digital counterpart. By ‘visual brainstorming’ I just mean Continue reading Visual Brainstorming on Whiteboards for Posters, Slides, and More
Peer-review: should we get rid of it?
There are way more manuscripts than opportunities for respected peer-reviewed publications (Sinhababu 2016). So many good manuscripts might never be properly reviewed (or published). This would be bad. In this post, I’ll mention a few potential solutions. Then I’ll briefly evaluate one: eliminating compulsory peer-review altogether.
1. Peer Review Is New
I learned from Kate Norlock that peer-review is a relatively recent thing.†
… the surprisingly short history of what we now think of as peer-review [Times Higher Ed.] … the Google ngram on peer-review: [Google ngram article] …. suggests that academics have only been so fixated on it as the measure of our worth since the 1970s.
2. The Current Form of Peer Review Isn’t Obviously Optimal
One reason for peer-review might be that it inhibits bias. And there is some evidence that anonymous peer-review reduces bias (Budden et al 2008). However, a review of 17 studies Continue reading Peer-review: should we get rid of it?
Peer-review: on what basis should we reject papers?
When you peer-review a paper, you can make one of a few basic recommendations to the editor. One option is this: do not publish the paper.
So what criteria should you use to make such a recommendation? In this post, I argue that some criteria are better than others.
1. Is the paper convincing?
A friend of mine mentioned this criterion the other day: “…[philosophy] papers ought to be convincing.” Call this the Convince Me standard or CM.
Maybe you think that CM sounds like a reasonable standard for peer-review. I don’t. Continue reading Peer-review: on what basis should we reject papers?
Podcasts …for research?
In a recent APA blog post historian of philosophy and pun-loving podcaster, Peter Adamson, floated the idea of using podcasts for teaching. Sounds like a good idea, sure. In this post, I’d like to focus on the idea of using podcasts for research. As I see it, podcasts could be AMAZING for research! Yeah, like, all-caps amazing! Continue reading Podcasts …for research?
Text-To-Speech for Speed Reading & More
My job requires lots of reading. But sometimes I read very slowly. Other times my body is occupied doing something that precludes the ability to read from a book or an electronic display. So I have been looking for ways to fit in more reading and to read faster. Text-to-speech technology provides the means to do this. So I use text-to-speech for speed reading, for multi-task reading, for and a few other things. In this post, I will (a) talk you about the best PDF-to-speech app that I have found and (b) talk about how I use text-to-speech more generally.
1. PDF-to-speech
Most computers, tablets, and smartphones can read text aloud in one way or another. However, until recently, I have not found text-to-speech software that can do both of the following:
- Speak the whole document start-to-finish. Every new page seems to trip up the software, so I have to restart the speech playback at the beginning of every new page.
- Ignore header and footer text. If the software can do 1, then it gets sidetracked by the text in the headers and footers every time it advances to the next page (e.g., copyright notices and page numbers; see figure 1 below).