Wednesday, October 23, 2013

The Hidden Abundance of IT Professionals

I'm not quite sure why the headlines I've been reading irk me so much.  Is it the fact that the conclusions drawn defy basic economics or the realization that ignorance is being spread quickly?  I've seen more than a few articles over the past few months stating an alleged shortage of computer security specialists, data analysts, and software developers in general.  It's the same idea that circulates from time to time in many tech fields as a means of justifying H1B Visas, but is it real?

To say there is a shortage of computer specialists, cyber warriors, or machine learning experts is like saying there is a shortage of Lamborghinis.  If everyone is willing to pay a higher price, then the shortage disappears.  It is true that there is a shortage of new Lamborghinis that sell for the price of a Toyota Camry, and this is the same strikingly foolish line of thinking that leads to news articles about the dearth of workers for technology ____ (fill in the blank).

Are you saying companies should just pay more and the problem will be solved?
Yes, this is precisely correct.  In the Western world we live in a mostly free market economy where supply and demand are the primary economic forces.  Saying that you want a Machine Learning expert for $90K may be a lot like saying you want a Lamborghini for $30K.  On the salary website Glassdoor.com, the national median for Radiologists is right at $250K.  According to the NY Times, demand for imaging has slowed since 2006 and recent radiology graduates are having difficulty finding jobs.  Nevertheless, hospitals have paid and continue to pay incredibly high salaries for this profession.  If we turn our attention to salaries for machine learning experts or even for directors of IT security, it should be clear to companies why there seems to be a shortage.  Even the maximum salaries listed across all companies do not contain a single entry above $200K.

Whether you are the Director of Machine Learning at Amazon or the Director of IT Security at UBS, whether you have a PhD in Computer Science or a dozen years mastering your enterprise's infrastructure, you will still earn less than the average radiologist.  This contrast remains even while there is an alleged shortage of these IT professions and a surplus of radiologists.

What then is holding companies back from paying more?  I believe the answer to this varies based on the particular job, but the root cause is that people stray from rational choices.  Let's consider some plausible scenarios. 

Russian Roulette Scenario
Upper management of Company XYZ has a hunch that security is lacking around its services, but so far they have avoided any major catastrophes.  They lost one computer security specialist to a better opportunity last month, but they are reluctant to increase the pay of the remaining specialists and post a new job for a replacement.  In short, they've chosen to increase their risk to save some on human capital costs.  When a serious and costly security breech does occur, they will likely claim to be a victim of a small pool of talented workers and the media will run with that headline.

Irrationally Too Expensive Scenario
Let's face it, we are emotional creatures.  A quick read of Dan Areily's work will convince you that humans frequently make harmful, irrational choices, especially when it comes to pricing.  This is certainly the case when it comes to salaries.  Rarely does an executive ask, "What is this position (or person) worth to our company?" and pay that person accordingly.  Instead, they try to conjure up a figure based on their own beliefs (e.g., doctors deserve more than PhD's), or they look at what everyone else seems to be paying.  The latter approach will often work, but not if most employers are also paying less than they should.  The bottom line is that you need to increase the pay (and perhaps the exposure of the job listing) if you are not able to hire for the position.  If a worker can save (or gain) $500K per year for a company, then it would be foolish to throw your hands up in the air and say that there is a shortage when you can't find a senior developer for $120K.  Simply increase the salary to $150 and see what happens.  The difference between the salary and expected gain from the employee is still much greater than zero.  In short, employers need to be rational.  They should use basic math and economics rather than emotions and preconceived notions.

I'm sure there are other causes as well -- feel free to post some in the comments.  Thanks for reading!