Grazing at the IT Metrics Smorgasbord

In an earlier blog I discussed what I consider to be the four most meaningless metrics that have gained wide currency within the IT industry. Not all readers agreed with my assertions and, quite naturally, the article triggered a healthy debate about the metrics that should be used to monitor the performance of an IT organization.

In this follow-up post, I present some suggested guidelines for constructing an effective metrics dashboard for your organization. 

Over a long career in IT, including six CIO positions, I have observed that metrics serve just two purposes:

1. they establish objective reality, and
2. they modify human behavior.

Objective reality can be hard to come by in large, geographically dispersed IT organizations. In the absence of metrics, performance information is largely based on anecdotes and subjective experience which can be very misleading.

The behavioral impact of metrics is best summed up by Peter Drucker’s famous quote: “what gets measured gets managed.” Metrics send a message to IT staff members about what’s important to their managers and customers.

IT organizations have an affinity for metrics that can sometimes undermine their utility. Most IT professionals have received formal training in some form of math, science or engineering, so they have been conditioned to employ quantitative information in prioritizing their activities and making work-related decisions. IT organizations manage diverse portfolios of business systems, infrastructure assets and service vendors that generate a tidal wave of metrics on a regular basis. This profusion of metrics can actually make it difficult to determine whether an organization’s overall performance is actually improving or declining.

Performance measurement choices abound

There’s a huge smorgasbord of metrics available to IT leaders and their teams. Key performance indicators are generally drawn from one or more of the following categories:

Financial – All IT organizations are expected to meet or beat specific spending targets.  Financial performance is routinely monitored by the Corporate Finance teams within most commercial enterprises.

Operational – Metrics regarding the efficiency or effectiveness of internal practices are always appealing because individual team members can have a direct impact on the execution and outcomes of routine processes.

Customer Satisfaction – Notoriously difficult to measure accurately, most IT shops nevertheless try to obtain feedback from their end users through various types of surveys.

Business Performance – Investments in new IT capabilities are frequently justified in terms of business benefits such as revenue growth, employee productivity, customer retention, etc.  In many cases IT groups attempt to measure such benefits following the completion of major projects or initiatives.

Organizational Health – Many organizations conduct engagement surveys to assess employee satisfaction with the nature of the work they are performing, their relationships with managers and coworkers, the availability of training programs, advancement opportunities, etc. Attrition is also considered to be a barometer of organizational morale, even though no one really understands what the ‘right level’ of attrition should actually be!

Information Security/Regulatory Compliance – The integrity of security safeguards is an area of growing concern in most companies. However there are no widely accepted metrics for gauging the completeness or effectiveness of such safeguards. Compliance metrics are typically defined and monitored by a company’s Internal Audit team. 

Selecting the right IT performance metrics

All too often, IT managers avoid contentious discussions about the ‘right metrics’ for their organization by selecting key performance indicators from each of the categories listed above. This is a grave mistake.

Building an exhaustive laundry list of metrics fails to inform the staff about what’s really important to IT management. If twelve or more things are really, really important, that’s simply a fancy way of saying everything is important! Furthermore, metrics are not free.  Significant effort is required to collect, normalize and interpret metric information on a routine basis. Many metric initiatives flounder when managers realize just how much work is involved in issuing and interpreting metric reports every week, month or quarter.

Metrics programs are loaded weapons. Properly aimed and deliberately managed, they can do much good. When they are haphazardly designed and casually managed, they fail to produce any practical benefits and can actually undermine the credibility of the IT management team. 

So we’re back to the fundamental question: what are the right metrics for your organization? Here are some practical suggestions for grazing among the smorgasbord of options.

Dashboard suggestions

Corporate Finance will track and report financial metrics such as spending variances versus budget on a routine basis. Similarly, the Internal Audit team will monitor and report controls compliance to the Board of Directors at each Board meeting. IT doesn’t need to dream up additional metrics or reporting mechanisms to help these two teams do their jobs. Most IT managers are acutely aware of the financial targets they are expected to meet and don’t need additional motivation to achieve such targets.

Information security metrics are hugely important but there’s little agreement about the specific metrics that should be used to gauge the effectiveness of security safeguards. Furthermore, there’s a great deal of sensitivity about discussing security statistics outside the infosec team.  Very few large companies are comfortable publishing security metrics on a wholesale basis throughout their IT organizations. Consequently they rarely appear on organization-wide dashboards. 

IT managers are constantly encouraged to present their achievements in terms of business performance, with specific reference to the improvements in revenue, margin, productivity and customer satisfaction that they have enabled. Although business metrics are alluring, IT managers should be extremely cautious in collecting and reporting business information. It’s always difficult to isolate IT’s contribution to the outcome of any business process or activity. Claiming credit for business benefits may actually end up ruffling the feathers of the same business executives that IT is desperately trying to impress! 

Furthermore, IT managers rarely possess the domain knowledge needed to defend the analysis and interpretation of business metrics. Consequently, they run the risk of appearing ignorant or naïve when engaging in such discussions. Although business metrics are highly desirable in principle, they can actually undermine IT’s credibility when they are collected and reported by the IT team. Producing business metrics for business partners is ultimately the responsibility of the business leaders themselves, not IT.

Feedback regarding customer satisfaction and organizational health is typically obtained through surveys. Surveys can be useful in detecting the causes of customer dissatisfaction or low employee morale. However, negative feedback in either of these dimensions can only be rectified by addressing root cause problems, not by simply resolving to ‘do a better job’ in the future. Chronic end user issues or employee job concerns can only be resolved by making changes to the current operational practices that are giving rise to customer or employee dissatisfaction.

And the winner is: Operational

By process of elimination, the most effective performance metrics within an IT group are typically operational in nature. Staff members have a more direct line-of-sight to operational metrics and a more direct impact on operational outcomes. Targeted performance levels can be established by IT managers without the endorsement or concurrence of external stakeholders. Operational metrics can serve as leading indicators of improvements in customer satisfaction or organizational health if they are properly targeted to rectify chronic customer and employee concerns. 

Metrics initiatives have produced some of the best results when used to enforce internal process discipline around service management (ITIL), software engineering (Agile) or operational (DevOps) practices. They can also play a key role in maintaining organizational focus on strategic initiatives such as cloud adoption, software quality improvement or mobile app deployment.

In summary, financial and regulatory compliance metrics will largely take care of themselves since they are defined and monitored by external organizations. Security metrics are likely too sensitive to be discussed outside the confines of the infosec team. Business metrics should be approached cautiously to ensure that IT doesn’t overstep the bounds of its responsibilities or knowledge. And finally, customer satisfaction and organizational health metrics can only be used to discover problems. They can’t be used to solve customer dissatisfaction or low employee morale. 

All roads ultimately lead back to operational metrics as the most effective means of improving organizational performance. The specific operational metrics that should be incorporated in individual dashboards should be customized to reflect the tactical needs and transformational strategies of each IT organization. IT management teams that embark on metric initiatives need to pause and first determine what problems they are trying to solve. Are they trying to address internal efficiency or morale issues? Are they trying to enhance the credibility of the organization in the eyes of its customers? Are they trying to support new business strategies? The answers to these questions should ultimately determine the key operational metrics incorporated in any metrics dashboard.