Muddling the Dunning-Kruger Effect

The Dunning-Kruger Effect states that people with the lowest competence tend to overrate their competence, but people with the highest competence tend to underrate themselves.  This was shown in 1999 paper by Dunning and Kruger which won an Ig Nobel Prize[1].  Here’s one of the figures from the paper:

This figure shows scores from a humor test (where a “high” score means good agreement with professional comedians).  There are similar figures for tests on grammar and logic.

The Dunning-Kruger effect has entered popular wisdom, and is frequently brought up when people feel like they’re dealing with someone too stupid to know how stupid they are.

But I have to admit that the popular wisdom led me wrong. I had a misconception: I thought that people’s self-assessment was actually anti-correlated with their competence.  As you can see from the above figure, this is plainly not the original finding.  People with lower competence tend to overrate themselves, but their self-assessments don’t quite surpass the self-assessments of people with higher competence.

I find this discovery slightly disappointing, because, under the misconception, there is an amusing Newcomb’s paradox embedded within Dunning-Kruger effect.  I would like to think highly of my own competence, but I’d rather actually be competent than think myself as such.  So do I choose to rate myself lower (which increases the posterior probability that I have high competence), or rate myself higher (because obviously rating myself lower won’t cause me to be more competent)?  Alas, this particular dilemma is resolved by the fact that self-assessment is in fact positively correlated with competence.  Therefore, the correct choice is simply to rate myself as being the very best.  I recommend this strategy to everyone.

On that note, I wish to inject a little bit of skepticism into the Dunning-Kruger effect.

First, consider a scenario where people have zero knowledge about their ability.  You’d expect that people’s self-assessment would be totally uncorrelated with their ability, thus people with less ability overestimate themselves, and people with more ability underestimate themselves.

Next, consider a scenario where people have some knowledge, but not perfect knowledge about their ability.  This would lead to results just like the figure above.[2]

But the interpretation would be different!  It is not the case that people in the bottom quartile are the worst at assessing their own ability.  Rather everyone is bad at assessing their ability, and people in the third quartile are simply lucky that their ability is similar to the average self-assessment of people who are completely ignorant of their own ability.

What remains is the question, why is the average self-rating at 66%?  Another interesting fact I discovered in research is that this is culturally dependent.  East Asians systematically underrate themselves.  The researchers claim that this actually has some utility: Americans tend to work harder when they think they’re good at something, whereas East Asians tend to see failure as an invitation to try harder.  Therefore, I would like to retract my earlier recommendation that everyone give themselves highest marks.  I only recommend it to Westerners.

Dunning and Kruger continue to oppose this picture[3].  They argue, with further data, that people with low competence actually have less capability to self-evaluate than people with high competence.  After reading their argument, I remain unconvinced.

In statistical analysis, there’s a statistical metric called “reliability”.  A more reliable measurement means more self-consistency among results.  For instance, if the bottom quartile has a very wide spread of self-ratings, then we would say that this rating is unreliable.  In their studies, Dunning and Kruger simply correct for unreliability and leave it at that.  However, I don’t think this is enough.  My contention is that everyone is mostly ignorant of their own ability, and when ignorant, they default to an “above average” assessment.  Perhaps they do so reliably, rendering invalid the correction method used by Dunning and Kruger.

Anyway, this doesn’t necessarily mean Dunning and Kruger are wrong.  I don’t have any data to back up my arguments, and I just skimmed a few papers.  But as you may know, pop psychology is often based on research that is more muddled and complicated than people think, and this one is no exception.

1. Kruger, J. and Dunning, D. “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”. Journal of Personality and Social Psychology 77 (6), 1121–34 (1999). (return)

2. Ackerman, P. L. et al. “What We Really Know About Our Abilities and Our Knowledge”. Personality and Individual Differences 33, 587-605 (2002). (return)

3. Ehrlinger, J. et al. “Why the Unskilled Are Unaware: Further Explorations of (Absent) Self-Insight Among the Incompetent”. Organizational Behavior and Human Decision Processes 105 (1), 98-121 (2008). (return)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s