Feds: Brightest not Diverse Enough for Bravest

The US DOJ has filed suit against the FDNY charging that the written exam for screening out candidates is not job related and disqualifies too many minority applicants. As far as whether the test, which was given in 2002, is job relevant they may have a point in that 97.2% of white applicants passed! You've got to wonder what kind of skills such an exam could possibly test for if so many pass. But not so many minorities passed - 92.8% of Hispanic test takers passed while 85.6% of blacks passed. Those are pretty high numbers too, and so the FDNY probably thought they'd hit diversity pay dirt - a written test that minority candidates overwhelmingly pass. In the previous test given in 1999, 90% of white applicants passed while 61.2 % of blacks and 77% of Hispanics passed. Sounds like a big improvement, but statistically the results are only a small improvement. As a result, there wasn't much change in the ethnic make-up of the force. Only about 7.5% of firefighters are black or Hispanic.

Provided it requires some cognitive skills, it's basically impossible to design a written exam that won't show these kinds of disparities in test results. By making the test easier and easier, and eliminating questions that test logical or reasoning skills, as was obviously attempted here, you can close the gap somewhat, but the statistically significant differential, which the DOJ points to here, will remain. The DOJ argues that the test is not relevant to the job, which is about all they could argue at this point, since the test has been designed to be so easy that almost every white applicant passes.

Compare these test results to this analysis of the July 2004 Texas Bar Exam (which I chose because it showed up first in a Google search). Among first time test takers, 85% of whites, 69% of Hispanics and 53% of blacks passed. These results are very nearly statistically identical (measured in terms of z-score differentials) to the 1999 NYC firefighters exam. They also are similar to what we find with the SATs, NAEP assessments* - just about any written test requiring cerebral energy. As la Griffe du Lion has pointed out, it's one of the few things you can count on in the social sciences, but count on it you can.

The best way to get around this (i.e., if an organization wants to avoid getting sued while trying to measure smarts) is via credentialism. CPA's, Actuaries, stock dealers, and many others (including obviously doctors and lawyers) belong to professional societies whose membership requirements involve passing one or more relatively difficult written examinations. Organizations can then post jobs which require membership in these organizations, or a certain amount of progress in attaining it. This leaves the company off the hook for having to justify that the exam these professionals have passed is job related - they merely have to demonstrate that the profession itself is important. No such recourse is available in the case of the FDNY - so they'll have to fight it out in court. Hopefully, if the feds win, they'll prove to be right, or people's lives will be at stake.

* The gaps in standardized tests are usually a bit wider than we see with the Texas Bar exam or the firefighters exams, presumably because they require studying rather than reliance on purely cognitive skills, which levels the field a tad.