How to search for great programmers, Part 2

Aline Lerner of Trialpay recently posted statistics on an experiment about resume review. The conclusion was that recruiters, engineers, and just about everyone score resumes all over the place and therefore resumes have weak signal value. The claim was that the strongest signal in the resume was the number of typos. Although the study seems extensive, I think there are a number of weaknesses in the experimental design. One weakness was acknowledged: the ground truth is the author's own subjective evaluation of the candidates. Another weakness was how the survey questions were somewhat misleading in the first place. The questionnaire asks "would you interview this candidate" and yet this was compared with the ground truth of "will the candidate perform well on the job or technical interview". As I alluded to in an earlier post in this series, the role of a resume is to help filter for red flags and to guide the formal interview, not to determine whether a candidate is a star performer by itself. The fact of the matter is, a resume is a self-reported synopsis of a candidate's track record. To evaluate a candidate, I would think track records are important, as is potential.

Lerner's post claims that coding exercises can evaluate candidates more effectively as quickly as resume review. This claim seems dubious. Resume review by Lerner's own admission takes less than a couple of minutes. I do not think there is a reasonable coding exercise that takes that amount of time or less. Fizzbuzz might be accomplished in less time but producing test cases and running run the test cases can easily swallow up a lot more time. Any involved exercise involving dynamic programming or graph algorithms can easily take up > 10 minutes. Most coding interviews give 45 minutes for the coding portion. This is an investment of engineer time. I do not see how this is comparable to resume review.

One thing I find about all the recent discourse about hiring: there is a sense of software engineering exceptionalism in the air. Even in Aline's post, the claim is the software engineering is different from all other fields. Every field uses resumes to filter down to a set of interviewees, but with software engineers we can somehow do better. For the short while that software is an ascendant vocation, this might pass, but I question whether software engineering is truly that much different from electrical engineering or even journalism. The hiring process varies somewhat from vocation to vocation, but I don't see journalists throwing away resumes in favor of just a writing sample. Some people truly favor coding exercises as the end all be all, but what kind of workforce does that truly breed? It seems to me that it strongly favors the programming competition crowd, those who are exceptional at TopCoder and CodeChef. How many top programmers were TopCoder and CodeChef winners?