Often Wrong - Never In Doubt

Sometimes you come across some writings or a research paper that perfectly crystallizes (and in many cases, validates) some thoughts that you have always internally harbored, but could never quite tie together in a coherent fashion. 

I have always had a suspicion that there are a large set of folks whose confidence level has way outstripped their chops. It always amazes me how these folks could be so damn sure about something, when they really don't have any right to be. 

It may be anecdotal, but in my personal experience in working with and interviewing engineers, I have noticed a correlation where the best engineers almost never exhibit over-the-top cock-sureness and arrogance. They understand the known unknowns and appreciate the unknown unknowns and have no trouble using phrases like "It depends" or "That might work, but here are the potential downsides" or "Right now, I don't know".

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know. - Donald Rumsfeld 

I always found it interesting that Rumsfeld took so much crap for that quote. That was one of the most intelligent things I have ever heard uttered by a public official about the War on Terror. I realize it doesn't have the sound-bite quality of "Bring It On". 

But why is there all of this misguided confidence? It probably takes folks far smarter than me to answer that. All I could come up with was the following: 

• The marketplace appears to love confidence. We have a whole gaggle of political partisan hacks who have entire websites devoted to tracking their daily lies, but yet who have turned their confidence and shamelessness into big bucks. I suspect the most effective of these are just playing games - they would do or say anything for a buck. The cult of personality comes into play here as well. We've all encountered folks that were "No cattle - all hat" but yet had no problem moving up the success ladder. Think about clueless middle managers interviewing other clueless middle managers. 
• Willful ignorance. Many confident folks simply ignore alternative information that doesn't support their world view. They don't want to have their ideas challenged, so simply stick their heads in the sand. From their perspective, since there is no possible alternative, why wouldn't they be confident.

"Ignorance more frequently begets confidence than does knowledge" - Charles Darwin

I guess I have to now acknowledge that there is another factor at play. Two researchers at Cornell University (Dunning and Kruger) have proven that for many incompetent folks, their confidence is derived from the fact that they are too incompetent to realize they are incompetent. 

Check out the full paper at "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments". 

Chew on this graph for a bit. 

So the bottom quartile, which achieved an actual test score in 12th percentile, perceived their overall ability and test score to be near the 70th percentile. The top quartile, on the other hand, underestimated their ability and test score.

"A man's got to know his limitations." - Harry Callahan in Magnum Force

The wikipedia writeup linked above introduced me to the term illusory superiority. I think that is just about the coolest term I have ever heard to describe someone that is full of shit but doesn't realize it.