Who tests the testers?

On the face of it, using an external IT skills assessment company for vetting technical staff is a win-win. The hiring company can concentrate on the interview process, and for their part, the assessment company offer a neutral view of each candidate - each is tested independently and the results are passed on. However, dig a little deeper and there are obvious problems.

First, is that you can't be sure the questions the candidates face have been sufficiently peer reviewed and quality controlled - more on this later. Yes, the assessment company can provide highly polished and vetted sample tests, but the actual test will draw on a vast repository of questions that the hiring company won't have had sight of - this is, after all, supposed to be an independent process.

Secondly, is the question of whether an assessment company even has staff with sufficient expertise at their disposal to write the tests. IT skills are constantly changing so the number of experts required to cover every skill in the marketplace could easily go into the hundreds. Certainly far more than the average assessment company has on their payroll.

But the real elephant in the room is the lack of an effective feedback loop. If a failed candidate raises issues with the tests, this smacks of sour grapes. Successful candidates who make it to interview, or better still, secure the job aren't likely to raise a formal complaint.

So much for the theory, but how about some real life examples. N.B. in each case, the assessment company admitted fault and in two cases, the reputation of the hiring company was damaged as a result.

Having tried to recruit for a very specialist role for some time, a well known company selected two candidates to make it through to the final stage although they hadn't quite passed the test. Each candidate insisted that it was highly unlikely that they had failed the test and on inspection, the wrong answer set had been loaded. The problem wasn't identified earlier as the company had limited the number of candidates from each recruitment agency to two. An investigation revealed the company had rejected well over a hundred candidates and had to restart the process from scratch at considerable expense.

A control candidate in an online test identified serious problems with two questions and took screen grabs. The questions were then independently peer reviewed afterwards and it was found that the offending questions were complete non sequiturs - the possible answers bore no relation at all to the question. The assessment company in question admitted fault and was subsequently removed from the process.

A multi-platform software function wanted to recruit a number programmers proficient in an ANSI standard programming language. Some time later, a number of the candidates (now employed) happened to mention that a number of the questions were vendor specific and not ANSI standard. The hiring company in question now use a different assessment company.

Pros

• Cheap
• Good for high volumes of candidates

Cons

• Rarely a perfect fit for the role
• Very narrow scope

Conclusion

• Know the pitfalls
• Supplement the process with other methods like phone vetting
• Consider developing your own - after all, you know what you are looking for better than anyone
• Use a third party agency so your reputation isn't damaged

- 19 February 2017

An open letter to recruiters of developers

Clearly a sensitive topic but this is something that will be high on the list of priorities for any candidate. You may not want this sort of information to be freely available to your competitors but you should at least be able to indicate the salary range a role can command. The worst possible case is that you like a candidate and the candidate likes you but you fall at the final hurdle due to salary discrepancies. If you must be cloak and dagger about the exact figure, do at least ask the candidate their currently salary and/or salary expectations.

So your office is a featureless little place in the middle of a business park - there is no reason you can't be effusive about the area and any additional perks. If it has lovely countryside nearby then say so. Local facilities also go a long way to selling a vacancy. Do you provide any freebies such as the odd lunch or endless tea and coffee? These sort of things are generally welcomed by potential candidates.

Please (I'm begging you) don't just list a huge set of technical skills with no thought as to the relative weighting. If you're listing more than about 5-10 core technologies, alarm bells immediately start ringing.

Not all skills are created equal. For smaller technologies, any dev worth their salt can easily pick up the odd skill or two given time if it is something they are light on. If it is something they absolutely must have from day one, factor it into the salary negotiations or require them to learn it as part of the code test (see below).

You will of course want to make sure the developer is as proficient as they claim. Make sure the tests are relevant and are reviewed regularly. If the test is on a piece of paper that has been photocopied so often it is becoming hard to read, that is generally a good sign that it could do with refreshing. Likewise, test the core skills that will be required. I remember sitting a test a few years back that read more like a GCSE maths paper. Am I really going to be adding fractions by hand in this job?

If you are outsourcing your testing, make sure it does what it says on the tin. A recruiter once mentioned to me a testing suite used by a high street bank. One failed candidate questioned the test having been rejected when he thought he should clearly have passed. It turned out the testing software had failed every single one of more than 100 candidates across multiple agencies as the wrong answer set had been loaded. The bank in question had to restart their recruitment from scratch at considerable cost.

Skills a candidate must have are considered deal breakers if they don't have them. Consider screening these in advance rather than picking thru CVs and covering letters to make sure they have them. A quick phone call will often suffice.

As bad as not asking for deal breakers is getting a candidate to interview and then subtly dropping in a key skill that appears nowhere in the job description. You are wasting the candidate's and more importantly, your company's time by overlooking key skills.

Having candidates write code is a great way to prove their worth. Having them do it in the interview isn't so great. Devs will of course have to code to a deadline sometimes but they'll be doing it on their own workstation with their own configuration, code snippets and tools. Give the candidate a small project to work on at home and ask them to complete it in a day or two.

- 20 July 2016