Via Karlie: No one knows what they're doing. (Warning: light profanity. Also wryly hilarious.)

When we were interviewing students for co-op positions on Tuesday morning, because it told us not how good they were, nor how good they thought they were, but how good they were going to tell us they thought they were. In some ways, it's a cockiness filter; you watch for how they back up their answers with other things they say and do in their interview. In some ways, it's a gauge of how much you know-you-don't-know - how conscious are you of what you need to learn?

When I thought of how I'd answer these questions myself, I figured that the full answer would come in three parts.

  1. Here's what I know exists in the domain you just described. (For instance, "ASL ability implies handshapes, spatial grammar, regional variants, vocabulary, idioms....")
  2. Of that set, here is what I do know and what I don't, and in what context. ("I can fingerpell, I only know a few idioms, I've mostly signed with a small group of people in my high school from a book so I know there are regional variants and idioms but don't know what they are...")
  3. Here is how confident I am that the set I described in #1 actually describes the domain - how much I think I don't-know-I-don't-know. ("There may be other things I don't know about yet, but I think this should mostly span the space.")

Now, that's long. And complicated. And you want a 1-10 number right now. So I would clarify my scale ("0 is 'never heard of it,' 5 is 'fuzzy, but at the point where I can learn by doing,' 10 is 'would be comfortable teaching you this right now,' based on the amount of skill I think we'd need to do the project that you just described) and then give the quick numbers runoff.

But I also prefer letting people just show me what they can do. Instead of asking "how well do you learn new programming languages," I plonk down code in a language they're unfamiliar with and tell them to look at it, think out loud, and ask questions about how it works. Tell me a story about how you taught yourself something you were interested in - and not for any school assignment. Link me to your blog or the archives of a forum or a mailing list you're active on so I can see how you talk with other people in normal online conversation.

There are no perfect ways of measuring. I'm still thinking through this - the quality of writing in this post reflects the scatteredness of my thoughts - but it's a set of ideas that I'm thinkering with (I love that portmanteau) because I think there are better ways to catch better people for precisely what you need, and better ways to figure out what you are looking for, and to express that.

Figuring this out will mean I get to go through some old memories and notes of really painful interviews (of me - I have flamed out in interviews before, and I have also nailed them). What worked, what didn't work, what do I think they measured, what do I think they missed? (Sometimes interviews go badly because they should - because it's not a fit. Other times, the measurement tool is in error.) I'll also be comparing notes with other people. Stories welcome.