The 4 Most Meaningless IT Metrics

As we approach the end of another NFL season, I continue to be amazed at the broadcast time devoted to the discussion of meaningless statistics. Statistics regarding the touchdown-to-interception passing ratios of individual quarterbacks, time of possession, ‘red zone’ points and team penalties are referenced repeatedly as key arbiters of a team’s success or failure. If you reflect on any one of these metrics for a few minutes, you have to ask yourself if they have any real significance whatsoever.

Consider, for example, the number of interceptions attributed to your favorite quarterback. If the quarterback throws the ball directly to an open receiver, but the receiver tips the ball and it’s ultimately caught by a defender on the opposing team, was that really the quarterback’s fault? If the quarterback isn’t being properly protected and is forced to throw prematurely to avoid being hit, can he really be blamed for the interception?

Don’t get me wrong – interceptions are bad, but they may have far more to do with the inadequacies of the receivers or offensive linesmen than the athletic capabilities of the quarterback. These nuances aren’t captured in a gross metric like the quarterback’s touchdown/interception passing ratio but they can be significant, creating misleading impressions about a quarterback’s passing ability.

Similar nuances apply to the interpretation of commonly used IT metrics, but are so rarely discussed that the metrics have been enshrined as key predictors of the efficiency or success of individual IT organizations. Here are four of the most commonly referenced IT metrics that I personally find to be quite useless.

1. IT Spending as a Percentage of Revenue

There are so many variables that impact this percentage that it’s impossible to draw meaningful conclusions by comparing spending percentages across companies or industries. IT spending levels can be significantly impacted by:

• the percentage of staff members based in North America versus other parts of the world
• the extent to which subscription cloud services such as SaaS applications and AWS/Azure have displaced on-premise systems built and maintained through multi-year capital expenditures
• the extent to which IT capabilities are being procured and paid for outside the corporate IT budget by groups such as marketing, sales, R&D and manufacturing
• reliance on business partners for outsourced services such as warehouse management or customer support, in which associated IT expenses are embedded in the overall service cost
• the manner in which capital costs – labor, hardware and software – are being depreciated over time and whose budget is carrying those costs

There are many other factors that could be added to this list. Nevertheless, research groups go to great lengths to report average IT spending percentages in specific industries on an annual basis. Business executives frequently use this metric as a barometer of IT cost efficiency within their company simply because they have no other basis for comparison.

2. IT Staff Attrition Rate

Staff attrition is generally perceived to be a bad thing and successful IT organizations are presumed to have low attrition levels. Some degree of voluntary staff turnover is actually desirable and should be differentiated from the loss of individuals with needed skills and expertise. All IT managers inevitably have to deal with ‘bad hires’ – individuals who don’t have the technical knowledge or interpersonal skills needed to work productively with their peers.

Other individuals have career aspirations that will never be met by their current employer and are openly resentful about their annual performance ratings and pay increases. Staff members who leave under either of these circumstances have made decisions for their own good but these decisions also benefit the company. I would refer to this as ‘good attrition’ and as evidence that the performance feedback processes within the IT organization are working effectively. ‘Bad attrition’, in my judgment, refers to the loss of highly rated individuals or staff members with hard-to-replace skills.

There can also be significant regional differences in attrition levels that render any type of average percentage calculation meaningless. For example, annual North American attrition levels could be at 5% while turnover in AsiaPac might exceed 10%. Any average calculation that blends attrition in these two regions will land somewhere between these two extremes and provide very little insight into an organization’s ability to recruit and retain the critical skills it needs to be successful in the future.  

3. Percent of IT Spending Dedicated to Running/Growing/Transforming the Business

This is a well-intentioned metric designed to differentiate spending required to support routine operations (aka ‘keep the lights on’) from more discretionary spending devoted to enhancing existing IT capabilities or implementing wholly new ones. As a practical matter, however, it’s very difficult to reach agreement on the definition of run/grow/transform spending within one company, much less across multiple companies or industries. Furthermore, there’s no consensus within the industry about which run/grow/transform spending percentages are ‘best-in-class’.

For example, if you are one of the last companies within your competitor group to implement an ERP system or provide your customers with a mobile purchasing app, are you transforming your business when you invest in these capabilities or are you simply implementing industry-standard capabilities required to run the business? If you replace on-premise storage systems with cloud-based storage, have you simply found a more cost effective way of running the current business or do the unique capabilities associated with cloud storage (e.g. global replication, inherent disaster recovery and encryption) constitute a business transformation? Different IT organizations could easily arrive at very different answers to these questions.

The best use of the run/grow/transform percentages is to define them internally within your company and strive to employ them on a consistent basis during consecutive planning cycles. However, any hopes of drawing meaningful conclusions from a comparison of your percentages to those of other companies should be abandoned altogether.

4. Data Center Virtualization Percentage

Server virtualization is widely regarded as a means of reducing cost and improving the agility of IT infrastructure organizations. CIOs like to boast about the percentage of servers in their data centers that are virtual and not physical.

Part of the appeal of virtual servers is the ease with which they can be built and destroyed. They can also be turned off until needed. Setting aside the measurement difficulties involved in monitoring the demographics of this constantly changing environment, the sheer number of virtual servers that are active or implemented is not a direct indicator of data center efficiency. The primary goal of server virtualization is to increase the utilization of the underlying physical servers hosting the virtual machines. In practice, physical server utilization is rarely monitored on a comprehensive basis and even more rarely discussed in public. So, while any given data center may host thousands or tens of thousands of virtual machines, a significant portion of the compute capacity of their underlying physical servers may go unused every day, totally undermining the correlation between this metric and data center efficiency.

The challenge for all IT leaders is to spend as little time as possible measuring and monitoring these metrics since they will have almost no impact on the efficiency or credibility of their organizations. If IT leaders work with business partners who find any of these metrics useful, the IT leader should attempt to disabuse his or her business counterpart of that notion as quickly as possible!

I invite readers to chime in and add any other IT metrics that they love to hate in the comments section!

Read Mark Settle's follow up blog, Grazing at the IT Metrics Smorgasbord.