Friday, November 28, 2014

How to search for great programmers, Part 1

photo by super-structure via PhotoRee

A running theme these days is that Github is the new resume and a great source for learning. I am skeptical of these claims. If Github is the new resume, then it is one that is potentially even more difficult to evaluate than before and will greatly limit the candidate pool and not necessarily to a high quality one. The bigger question is what role should Github play in the hiring process. After all, many if not most of the famous hero programmers out there (e.g., Brian Kernighan, Herb Sutter, Alex Stepanov) do not have Github profiles. Linus Torvalds does, but he seems to be an exception. A Github profile is also difficult to evaluate. Do we consider only superficial metrics such as commit/contribution frequency or dive deep into a random project? Either of those options do not seem particularly attractive. At most, they will provide an imperfect measure for enthusiasm. It seems to me if code samples is what we are after, then we should ask for code samples. A real resume provides a highly condensed means to evaluate for relevant experience and education that can be evaluated in seconds.

photo by mathowie via PhotoRee

What role should a Github or a resume play in the hiring process? In hiring, there are two major phases with different aims. There is the initial screening phase consisting of resume review and phone screens including initial coding exercise. This phase is meant to filter for red flags, candidates who are clearly not viable. This phase ought to be relatively objective and fairly straightforward. It isn't about finding the gems as much as eliminating the clearly unqualified. At this stage, the goal is to avoid keeping a candidate in the process who really should not be. That is to say, minimize false negatives (where a negative result would be saying a candidate with red flags is alright). The second phase is the more extensive on-site interview. Here, the goal is to find the best candidate by minimizing false positives. In this framework, Github would not play a role in the initial screening because there would be a lot of false negatives otherwise. As I noted above, not everyone has a Github profile and not everyone contributes actively. Github is great for finding subjective pluses (i.e., reasons that a candidate is impressive) but not red flags.

How does one truly evaluate a Github profile? On several occasions I have had candidates with Github profiles. Admittedly I solicited none of them, but some candidates do offer links to their Github profiles in resumes. What a Github profile reflects is participation in more recently started open source projects usually with a bent towards JavaScript libraries and occasionally Go-based microservices. Many of the big open source projects from the Apache Software Foundation (which is generally known for some projects with high quality software) and the Linux kernel now mirror their repos on Github, but generally speaking, contributions to the projects being mirror won't necessarily be reflected in a contributor's Github profile nor does Github include such contributions in profile metrics. Regardless, the problem I faced was that the Github profiles provided a lot of data of limited relevance. The profile provides commit histories, a list of projects contributed to, one of projects owned, commit frequency, day of last commit including numbers of commits, and items such as number of starred repositories. How can one use this information to evaluate a candidate? Do I look at the last couple of commits and use that to gauge a candidate's competence? The downside there is last couple of commits is quite arbitrary and not all commits are equal. Some projects prefer larger, more self-contained rebased commits while others small, staccato snippets for commits. Moreover, there is no reason to believe the last few commits are a person's grand opus. One view that is somewhat more interesting is what magnitude of contributor is a candidate to a project, but one can't really get to this by the profile. Looking at an open source project's contributor graph can show how big of a contributor is a person. If one produced a large proportion of the commits, then there is something to be said for the diligence and work ethic. However, such candidates are fairly rare as I am discovering in a related study of open source contribution patterns.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.