How to measure quality and anticipate product defects? In this guide, we will discuss specific quality metrics and show how they contribute to the overall strategy.
The article starts with a list of quality metrics sorted out into groups. We’ll discuss some popular quality KPIs and the ways to use them. Learn more >
In the second part, we’ll talk about the business context behind quality and build an example of the Quality Balanced Scorecard. Learn more >
In the third part, we’ll discuss ten ideas that one can use to convert a scorecard into an effective tool for ongoing quality control. Learn more >
Part 1. Quality KPIs
Before discussing specific KPIs, let’s answer two questions:
If you prefer to go ahead with the KPIs, here are the topics that we’ll discuss below:
- Quality KPIs for Stakeholders
- Quality KPIs to Estimate Financial Impact
- KPIs to Prevent Quality Problems
- Quality Response Metrics
- Quality Culture
Stakeholders of Quality
Quality is not an absolute value. When talking about quality, we always talk about the perception of quality. Who perceives the quality of the product or service?
We can start with:
- End customers
- Internal customers (our team and partners)
Thinking about end customers, we can be even more specific and define, for example:
- Prospective clients
- Paying customers
- Returning customers
- High-end customers
Definition of Quality
What is quality?
Oxford dictionary defines quality as:
The standard of something as measured against other things of a similar kind; the degree of excellence of something.
Different stakeholders perceive quality in different ways. For example:
- Prospective clients might be fascinated by your marketing and buy on a very basic perception of quality (quality of package and marketing materials).
- Customers who used your product for a longer time will have a much better idea about its quality (as applied to their needs of course); those who will find the match will become returning customers; those who won´t – will switch to the competitors.
- High-end customers will most likely have different needs and different ideas about the product’s quality.
- Your team will deal with the internal mechanics of your business, and their perception of quality will be more related to the product architecture and complexity.
Quality KPIs for Stakeholders
Below, we will discuss some popular KPIs for quality. For the best implementation results, make sure that these metrics fit your business context perfectly. We discuss how to achieve this in the second part of the article.
Metrics that we review below:
NPS (Net Promoter Score) as a Quality Indicator
In its classical form (a question “Would you recommend us?”), this metric indicates a general perception of the product or service.
- The low NPS might be a result of poor product or service quality
- Ask some follow-up questions to find out what is actually broken
A metric, in this case, is as simple as:
- NPS, %
Number of Complaints per Period
Any customer feedback is important. In some cases, we only know about the problem after getting an email from an angry client.
Even if you do your best to anticipate all possible problems, such complaints will still exist. Actually, they show that:
- Your customers still care (otherwise, they wouldn’t spend their time giving you feedback).
- There is some problem that needs to be fixed.
- There is something that you didn’t know about your product before, and you need to improve your quality control system.
Indicator to use:
- Number of complaints per period of time
A general approach to complaint solving is to understand and fix problems in the business systems rather than blame specific people.
Social Proof of Quality
In many cases, prospective clients cannot really evaluate the product before buying. Still, they can make a good choice based on social proof metrics.
- Average rating score. The aggregated value of the clients’ rating of the product and their comments form social proof metrics.
Think about booking a hotel or buying something on Amazon. You can get a rough idea about the product by reviewing the photos and the description, but too many products look the same:
In this case, we look at the hotel rating or starts on Amazon and form our expectation of product quality accordingly.
This quality metric is similar to NPS, but in this case, buyers are more likely to share more detailed unbiased options.
The more effort your team invests in the quality control and assurance, the less problems you will see reported daily.
But it doesn’t mean that your product is ideal; on the contrary, you’ll face an even more challenging situation now:
Instead of fixing obvious the problem, now you need to read between the lines and understand what issues (less obvious ones) prevent users from unlocking the true potential of your product.
I’m talking about complexity.
Before, we discussed how to measure complexity, and here are some key points to look at:
- Excessive timing
- Excessive choices
- Excessive explanations
The choice of specific metrics depends on your product; we can group them under the umbrella of:
- Complexity index, %
Fighting complexity is tough, but it’s worth the time you spend on it. Here is an example of how eliminating unnecessary complexities can help organizations to improve.
Quality KPIs to Estimate Financial Impact
Sometimes, quality specialists have to justify their budgets and strategies in board meetings. Here are some finance-related quality metrics to use.
Metrics that were analyzed below:
Cost of Poor Quality
Use this metric to understand the impact of poor quality. Estimate all costs associated with problem detecting and solving:
- Customer support specialist costs
- Solution costs
- Brand impact costs
In many cases, the goal is not to find the exact dollar value, but to understand how big the impact is on the stakeholders´ perception of quality.
For this article, I´ve created an indicator:
- Cost of Poor Quality
This indicator summarizes the values of the indicators inside, giving us the total cost of quality. In my example, the calculations are for the average cost per accident.
Cost of High Quality
Similar to the “Cost of Poor Quality,” but in this case, we are estimating the costs of trying to achieve excellence.
In this case, our indicator is:
- Cost of high quality, $
Its formula depends on the specific situation. For this example, we can calculate the budget needed to decrease the defects rate from 0.01% to 0.001%.
Opportunity or Risk Metrics for Quality
The resources of any organization are limited. We cannot treat all quality issues equally; some need to be solved immediately, and some will stay in your backlog for a longer time.
How to justify decision making in this case? A simple opportunity (or risk) estimation will help:
- Estimate the impact of the quality issue
- Estimate the probability of this impact
Now we have:
- Opportunity estimation metric, % = Impact, % x Probability, %
How to estimate the impact of the quality issue? Look at the:
- The number of customers who are affected by the problem
- Customers’ financial potential
- Customers’ involvement in the issue (the number of emails customer sent)
How to estimate probability? Try to map it on a scale from 0% to 100% where:
- 0% is something that potentially influences your customers, but you have no proof of it.
- 100% is for the critical factors that, for example, will prevent the customer from buying or will result in important costs.
Let’s use a case with BSC Designer software. A few years ago, we got two feature requests from the customers (both functions are implemented now):
- (A) Add a feature to allow/deny users to edit historical data in their scorecards
- (B) Change the way initiatives are displayed on the strategy map
(A) was a critical issue for a big potential client, and three other clients had previously asked about this.
(B) was a less critical issue for the existing client and only one more client had requested a similar feature.
Our efforts to “fix the problem” (actually, develop new functions) were about the same.
Here is an example of opportunity estimation for these two cases:
|A||70% (was critical for buying decision of high-end customer)||80% (3 more clients asked for a similar feature)||= (0.7 * 0.8) * 100% = 56%|
|B||50% (would make life of existing customer something easier)||60% (1 more client asked for a similar feature, and our developers liked the idea)||= (0.5 * 0.6) * 100% = 30%|
In this example, the value of the opportunity metric was higher for “A,” so we gave it a higher priority during the implementation.
KPIs to Prevent Quality Problems
For now, we were quantifying and measuring the impact of (poor) quality. Let’s try a different focus:
- Instead of fixing quality issues, let’s be proactive. Let’s work to decrease the possibility of future quality problems.
Here are the metrics that we can track to achieve this goal:
- Near Miss Metric
- Requirements Ambiguity Metric
- Product Age and the Number of Modifications
- Expert Review Metric (WTF/Minute)
- Aleatory Test Metric
- Quality Self-Test Metric
- Automated Quality Control Metrics
- Stress Test Metric
- Manual Test Metrics
- Change Risk Analysis
Near Miss Metric
How can we predict something that has not happened yet?
To predict some important quality problems, we need to look at the preceding, less important problems.
It’s that simple.
How to build a near miss metric?
- Check out general thoughts about near miss
- Find near miss events relevant for your business processes
- Quantify, measure and track these findings
Implementing Near Miss metric is about changing the culture of the organization.
You need to teach your team to report and discuss minor quality issues that might lead to more important problems.
The metrics can be formulated as simply as:
- Leading metric: Near miss awareness training penetration, %
- Lagging metric: The number of near miss issues reported
Requirements Ambiguity Metric
Similar to the Near miss metric, but in this case, we want to be proactive.
Instead of waiting for some quality issues to appear, we ask our best experts to review the product/service requirement documentation and spot possible problematic points.
The metric in this case might be:
- Ambiguities per requirements page, %
In software development, the experts review the software requirements for ambiguity.
The requirements, in this case, are the preceding documentation, so if we make the requirements better, we can prevent future quality problems proactively.
Product Age and the Number of Modifications
We are in the software business, and I’m always curious about the best ways to quantify the software quality and specifically the quality of software code.
A classical “number of license of code” doesn’t make a lot of sense. Intuitively, software developers understand that:
- Frequently changed code will contain more defects than code that remains unchanged for a long time
In other words, we expect code quality to be the function of code age and the number of code modifications. That what was confirmed empirically in the “Predicting Fault Incidence Using Software Change History”.
Replace “code” word with “product,” and I’m sure the same findings will make sense for any business with a complex product. Thinking about this in the context of quality strategy, we can predict defects more effectively and update testing strategy, respectively.
Expert Review Metric (WTF/Minute)
Similar to the requirement ambiguity metric, but in this case, we can ask experts to review anything – our product, user manual, customer service.
How to quantify such reviews? A humoristic (still practical) response to this question was given by Thom Holwerda in his comics for osnews:
- WTF/minute (What’s This For / Minute).
The original metric was for software code quality reviews, but as you can imagine, it works in any domain.
Aleatory Test Metric
Another metric is useful to estimate the quality of large samples.
This is applicable mostly to physical products.
- Mix the samples
- Take a few random items
- Check out their quality
- If some defects were found, estimate proportionally the real scale of the problem
The metric, in this case, might be:
- % of defects in aleatory sample
Obviously, this approach works on early product stages where there are often quality issues. With the development of quality control systems, the size of the aleatory sample should be increased.
Quality Self-Test Metric
Sometimes, we have to check the “quality of quality,” e.g., how the quality control system works itself.
- Do quality reviews actually work?
- Do automated tests actually detect the problems?
The idea of self-test is to introduce the input with known errors and see if those pre-engineered problems were detected.
Medical analysis laboratories have this routine implemented. The laboratory equipment is a combination of complex optic devices and chemical reagents. The factors of quality are the way reagents are stored, the quality of distilled water, the stability of the electric current, human factors, and random errors off course.
The easiest quality control, in this case, is to load a known set of samples and compare actual results with expected ones.
Regulators use the same approach doing blind tests when the properties of the test samples are not known beforehand.
The metrics in this case are:
- The number of self-tests per period
- Defects detection efficiency, % = Defects detected / Defects injected
This approach works as a starting point before doing more expensive and time-consuming tests.
Automated Quality Control Metrics
Similar to the self-test, but in this case, we are not looking at the system as a black box, we can do more specific automated tests.
The benefits of automated tests compared to manual tests are obvious:
- No human factor involved
- Less costs
Still, we need to balance automated test coverage with the cost to develop and maintain such tests.
The possible metric in this case:
- Automated test coverage, %
Stress Test Metric
The quality of the system can change when the environment changes significantly.
The metric in this case:
- Stress changes readiness, %
The idea is to foresee the possible conditions and try to be prepared. Imagine a best- and worst-case scenario and try to come up with emergency plans.
- Have a smartphone? Try mountain hiking during the winter season and see how long your battery will last.
- Do you run a website? Will it survive a DDoS attack?
- Designing consumer products? Will it work properly if used in India?
Manual Test Metrics
Old good manual tests are a must. Here are two recommendations:
- Be the user of your product
- Take into account different stakeholders of quality
The metrics in this case:
- Test plan coverage, %
- Test plan execution time
Less formal, but still useful metric:
- Time being end-user of the product, hours
For example, to illustrate this article and create a scorecard for quality, I was using our BSC Designer software. This always helps in generating some improvement ideas.
Change Risk Analysis
When the complexity of the product is high, changes in the product might produce many quality issues. The solution to this challenge lies in the level of product architecture. For example, software engineers build their solutions using microservices to avoid bad dependencies.
From the viewpoint of quality, the starting point is to estimate risks of change. Ask your engineers:
- If we change function “A” of the product, how likely is that something will be broken?
Following this logic, we can use:
- The number of recent changes metric
As a leading factor for future quality issues.
Quality Response Metrics
We tried to fix the problem on the design stage. We tried to fix the problem before the customer (or other stakeholders) saw it. But sometimes, a problem passes all these filters, and customers face a quality issue. We discuss below the metrics that we can track in this case.
Here are the metrics that we discuss below:
- Time to Solve Issue
- Time to Respond
- Quality Backlog
- Quality Team Metrics
- Findings Closure Ratio
- Quality of Communication
- Solution Effectiveness
Time to Solve Issue
Yes, there was a problem, but it was fixed overnight. That’s a wow experience for any customer.
- Average time needed to solve the critical problem
- Average time needed to solve a minor problem
Track separately timing for different stakeholders. See the risk paragraph for the detail.
Another important idea is that normally, time to solve the problem doesn’t include:
- Time needed to develop prevention plan, or
- Time needed to implement prevention plan
Time to Respond
In many cases, the problem is not that critical for the client, and what is more important is that your team care. A quick email, “Hey, thank you for reporting. We are on it, expect a fix in 2 days” will show a customer that you care.
- Time to respond to quality issue
Similar to the sales backlog, a quality backlog is an indicator of how quality flows throughout your organization.
No backlog of quality issues most likely indicates that you:
- Have not changed your product or service for a long time (and this fact itself is a problem), or
- You are not listening to your clients well enough
Overloaded backlog shows that your response efforts are not sufficient, and something needs to be changed.
My recommendation is to track in dynamic this metric:
- Quality issues backlog
Quality Team Metrics
A popular metric for quality teams is:
- Quality personnel / Normal personnel, %
If analyzed together with issue fix time and returning problems rate, % it might give some information about the team’s performance.
Findings Closure Ratio
Imagine that a company did a quality audit. For example, using the requirements ambiguity analysis described above. We need to make sure that those findings are analyzed and closed.
Metric to track in this case:
- Findings closure ratio, %
We are not always talking about implementing the recommendations, but at least they need to be reworked into some tangible form (for example, updated requirements).
Quality of Communication
The complexity is a known leading factor of quality.
Track the communication redundancy metric in this case:
- Communication redundancy, % = (Average message size – Shortest message size) / Average message size.
Did the customer need to make just one call to fix the problem, or it was an exchange of five emails to solve a simple issue?
For example, in the case emails exchange, we can calculate the metric:
- Communication redundancy, % = (5 – 1) / 5 = 80% redundancy.
- First contact resolution rate, %
We fixed the problem, and we communicated efficiently and effectively along the process, but was the problem actually solved?
Metric to track:
- % of returning problems
- Right first time, %
- OTIF, % – On Time in Full in logistics
These metrics work best if bundled with the time to solve metric. The organization needs to find a balance between the speed of providing solutions and their effectiveness.
Quality culture is about establishing new behaviour habits that will help to avoid critical problems and maximize the effectiveness of solving less critical issues.
Does your team have a habit of tracking and improving quality?
KPIs that we review here:
- Rejected Defects
- Root-Cause Analysis Metric
- Systematic Error
- Quality Awareness Metric
- Improvement Acceptance Ratio
- Retention Rate
- Your Unique Quality Metrics
- Quality Measurement by Design
Imagine a scenario when a user reports a problem in your product, but it appears not to be a problem but normal behaviour. The developers “reject” this report as a defect.
As long as quality is perceived by the clients, this situation will be considered by a customer as a defect. What does a rejected defect mean? It means that some other system did not work well.Metric to track:
- % of rejected defects
Here is an example:
- A small fridge is making too much ice and is causing a problem to its owners. They call customer service, a repairman comes, and tells them that this small fridge is not suitable for continuous use; it is supposed to be used in motels where people come and go. This is a quality issue, and it already has its costs (brand impact, guaranty service fees), but this defect will be rejected by the engineers. Where is the problem then? I guess with marketing…
A similar situation is when a tester of the product reports a problem that is rejected by the engineers. Where is the problem in this case? I believe it is in the ambiguity of the initial requirements.
Root-Cause Analysis Metric
Fixing quality issue on time is a good starting point, but an organization also needs to learn from its mistakes (see the quality assurance article on Wikipedia). Do a root cause analysis for quality issues:
- What was the reason for the problem?
- Why didn’t we detect it before? What can we change in our quality control system to detect similar problems?
- What can we change in our systems to eliminate the possibility of such problems?
Off course, the degree of root-cause analysis should correspond to the importance of the product.
Metric to track:
- Leading: Root-cause analysis training penetration, %
- Lagging: % of the problems where root-cause was analyzed
Part of the quality assurance challenge is to detect and find systematic errors. Let’s say your SaaS is hosted on a slow server, and it results in many other problems with system usability. It doesn’t make sense to review all those usability problems when they are caused by a bigger systematic error.
Quality Awareness Metric
Quality is everyone’s job, so it makes sense to explain the necessary quality procedures to all involved members of your team:
- How should people report a “near miss”?
- What is root-cause analysis and how to conduct it?
- What is a response plan to the critical issues?
- What are our quality standards?
The metrics to track:
- Leading: Quality training penetration, %
- Leading: Outdated training materials, %
- Lagging: Quality program awareness, %
- Lagging: Quality standards awareness, %
- Lagging: % of policies understood by employees
As you can see, the lagging awareness metrics are balanced with the leading training metric. Professionals responsible for the training might focus their measurement efforts even more by tracking:
- Immediate reaction
- Change in actions
- Business impact
We discussed this approach in the training scorecard article.
Improvement Acceptance Ratio
Your team and your customers might suggest a lot of improvement ideas. The question is, how many of them do your engineers actually analyze and implement? If ideas go directly to the rubbish bin, then we cannot expect high morale from the employees.
A better approach would be to establish a procedure to deal with ideas, similar to what we have discussed for the innovations scorecard.
- Improvement Acceptance Ratio, % = Suggested ideas / Implemented ideas
The poor quality affects the stakeholders of quality.
We might see that customers are no longer recommending your product, or valuable members of a team are looking for a new employer. Track these two metrics to have an early warning signal:
- Customer retention rate, %
- Top performers turnover, %
Your Unique Quality Metrics
I believe most of the metrics discussed in this article can be applied to your domain. Still, there are always some unique metrics that need to be tailor-made for a specific business. How to find these metrics? I do recommend having a look at the KPI System that we have for you.
My best guess is that:
Your unique quality metrics will be related to bottleneck process in your organization
Did my guess work for you? It’s easy to explain; if something happens to a process that limits the system, it will affect the rest of the system.
Quality Measurement by Design
Another topic that we discussed in the KPI System is measurement implemented “by design.” Instead of trying to measure something afterwards, we need to build a product with future measurement needs in mind.
Applying this idea to quality, we need to find out where the quality measurements can be implemented by design.
Ask your quality team these questions to get started:
- When our end customer reports some issue, how will we know what is causing the problem?
- What can we change in the product or service to make quality measurement easier?
The answer can be a simple checklist or a kind of black box with operation log recording.
And here is a trick:
If you find ways to better measure quality, it automatically results in quality improvement.
For example, in the software development domain, adepts of DevOps practices first create tests for software functionality, then build the functionality itself. On the one hand, it allows measuring quality better; on the other hand, it improves the quality out of the box.
Implementing Quality Culture
We discussed some ideas related to the quality measurement culture. A more important topic is how to focus a company’s culture in general on quality. There are many books written on the topic, and in part 3 of this article, you will find my short list of top 10 tips.
Part 2. Quality Balanced Scorecard
The Balanced Scorecard is a strategy execution framework. What are the goals of quality? How do those goals fit into the overall strategy of the organization? How to track the execution of those goals? Below, we’ll create an example of the Balanced Scorecard for quality.
Here is our plan:
- Define the mission statement
- Map financial goals in the Finance perspective
- Define stakeholders’ needs in the Customer perspective
- Explain response plan in the Internal perspective
- Identify enablers for the internal improvements in the Learning and Growth perspective
What’s the purpose of focusing on quality? Let´s formulate some inspiring mission statement for a quality team:
Solve quality problems effectively, think about prevention plan
That´s a simple message that anyone in the quality team should understand. We are here to:
- Solve quality issues fast
- Investigate and implement a prevention plan
Let’s continue with the perspectives of the Balanced Scorecard.
Financial Impact of Quality
What is the financial impact of quality? In other words, what will happen if stakeholders of quality are not satisfied with the quality of the product?
The answer depends on the stakeholders:
- If clients are not satisfied – they won´t buy or won´t do repeat purchase.
- If your team is not happy about product quality – they may leave the company to find a better place to apply their talents.
- If you don’t comply with the standards set by the regulator – your organization might be fined or closed.
I suggest mapping these ideas under the goal:
- Sustainable growth
This is a very vague goal. How can we make it something more specific? We can align some performance metrics with it:
- Cost of Poor Quality – will show the financial impact of the quality
To ensure the sustainable growth of the organization, we need to satisfy the needs of our clients (stakeholders of quality like end users, regulators and team).
Goal: Avoid critical defects, fix non-critical defects quickly. That’s what all stakeholders want.How can we estimate the degree of achievement of this goal? Let’s use some of the indicators discussed above. These will be lagging indicators in the context of the goal:
What about leading indicators? The leading performance will come from the goals in the internal perspective that we will discuss below.
Another customer goal is more related to the internal stakeholders of quality:
- Goal: Maintainable product. A product that is relatively easy to maintain, change, test, etc. That’s what your team want.
What lagging metrics can we use to track the execution of this goal? For example, these two:
That’s a simple model of how product complexity and team engagement is connected:
- If a product gets too complex, the members of our team will complain about this, and if we do nothing – they will leave.
What about leading metrics? What can we do to prevent such cases? It depends on the specific situation, but I would say that we need to understand the reasons for quality/complexity problems. For that purpose, we’ll have “Root-Cause Analysis and Quality Assurance” goal in the Internal perspective that drives leading performance to the “Maintainable product” goal.
In this perspective, we need to answer two questions:
- How are we going to “Avoid critical defects” and “fix non-critical defects quickly”?
- How can we ensure that we have a “Maintainable product”?
I will formulate these three goals, and the links on the screenshot will show how they contribute to the goals in the Customer perspective:
- Prevent Critical Quality Problems
- Solve Quality Issues Quickly
- Root-Cause Analysis and Quality Assurance
Again, these goals are too vague for now. To make them more specific, we need to align some indicators with them.
Let’s start with “Prevent Critical Quality Problems” goal before we discuss different approaches to preventing quality problems, respectively lagging indicators can be:
- Near miss issues reported
- Ambiguities per requirements page, %
- Expert Review Metric (WTF/Minute)
How can we ensure that our team mastered, for example, “Near miss reporting”? For this purpose, we’ll map the “Near miss reporting training” goal in the Learning and Growth perspective.
What about the “Solve Quality Issues Quickly” goal? The basic measure will be Time to solve the critical problem.
It is also important to train all members of the team to report quality issues properly. We can estimate the achievement of this goal by using the Improvement Acceptance Ratio metric.
Finally, the “Root-Cause Analysis and Quality Assurance” goal contributes to the goal of having a “Maintainable product”. How can we make sure that our team is actually doing good root-cause analysis and quality assurance?
Let’s start with the basics. We can track that our team is thinking in the right direction by measuring:
- Time to develop prevention plan
- Time to implement prevention plan
You might ask:
How do we know that this time budget is spent effectively?
Well, with these metrics, we cannot know for sure, but if things go wrong, you will see that indicators aligned with “Maintainable product” goal from the Customer perspective will appear in the red zone:
- Internal stakeholders will start complaining about product quality/complexity (see the “Quality/Complexity Complaints from the Team” indicator).
OK, we know how the quality team should behave. What should we do to equip them with the right skills and tools? The answer is explained below in the Learning and Growth perspective.
Learning and Growth Perspective
For the Prevent Critical Quality Problems goal, we have two supporting goals in the Learning and Growth perspective:
- Near miss reporting training
- Learn to Prevent Quality Issues
I’ve separated them because I believe near miss reporting deserves special attention.
Let’s start with the Near miss reporting training goal.
The leading metrics will be:
Obviously, we want all involved stakeholders to be properly trained.
What about lagging metrics? If we want to do it seriously (see training scorecard approach for the detail), we need to understand how we are going to measure the:
- Results of the training
- Change in actions
- Impact on business
The results of the training is an exam score. Change in actions is what we measure with Near miss issues reported aligned with the Prevent Critical Quality Problems goal.
How to measure the impact of “near miss” training on business? I’m looking at the Avoid critical defects, fix non-critical defects quickly goal, and I believe the Number of Complaints per Period is an excellent indicator for this purpose.
Let’s move up to the next goal – Learn to Prevent Quality Issues. Here, we have two lagging metrics related to the initiatives aligned with this goal. The metrics are:
The initiatives are respectively those quality assurance ideas that we discussed before:
The last goal in the Learning and Growth perspective is Quality Awareness. Before, we discussed that it’s important to keep quality standards and respective training material up to date. For that purpose, we have:
- Outdated training materials, % leading metric.
Again, we need to make sure that all stakeholders were trained, and we use this metric:
- Quality training penetration, %
To track the results of the Quality Awareness program, we will look at these lagging metrics:
How can we quantify Quality program awareness, %? For example, by doing some simple surveys or tests.
How can we make sure that the Quality Awareness program changed the way stakeholders behave? We need to look at the specific behaviour metrics. I’ve selected these two:
Let’s use this goal as an example and track how the logic of the strategy map works.
Logic on Strategy Map in Action
If our Quality Awareness program is a success, we can expect that, for example, customer support specialists will be responding quickly (Time to respond to quality issue metric metric) and efficiently (Communication redundancy, % metric).I’m moving to the “Internal” perspective.
The Quality Awareness goal is a leading factor for the Solve Quality Issues Quickly goal. Basically, our hypothesis is that by improving quality awareness, we expect to solve quality issues better. And we will check this hypothesis by looking at the Improvement Acceptance Ratio, % and the Time to solve the critical problem indicators.
We are now looking at the “Customer” perspective.
Solve Quality Issues Quickly goal contributes to the Avoid critical defects, fix non-critical defects quickly, that finally contributes to the Sustainable growth goal in Finance perspective.
Relevant Importance of the Contribution
From the strategy map, we see that Avoid critical defects, fix non-critical defects quickly is influenced by the leading performance of two goals:
- Prevent Critical Quality Problems
- Solve Quality Issues Quickly
Do these goals contribute to the Avoid critical defects, fix non-critical defects quickly goal with the same weight?
Let’s suppose that preventing problems is more important than fixing them.
To reflect this logic, I can change the weight of leading indicators respectively:
- Prevent Critical Quality Problems – weight 70%
- Solve Quality Issues Quickly – weight 30%
BSC Designer software that I used to create a scorecard for this article will take these weights into account when calculating the leading performance of Avoid critical defects, fix non-critical defects quickly goal.
If you want to learn more about the way the scorecard values are calculated, check out the Scorecard 101 article.
Here is an explanatory video that summarizes some important findings of the article.
Part 3. Quality In Practice
In the previous parts, we discussed how to build a Balanced Scorecard for quality and execute a quality strategy effectively. What about quality on the operational level? In this part, we will discuss ten ideas about using quality KPIs for ongoing quality control.
1. Business Context Comes First, then Come Indicators
Always start with business context! The Balanced Scorecard framework is one of the possible approaches to describe and execute your strategy. If you do quality metrics on the operational level, then BSC might not be an appropriate tool; still I would recommend having clear answers to these questions:
- Why do you need to measure the quality?
- What aspects of quality are important to your customers?
- How exactly is the company planning to achieve current business goals?
- What quality-related problems need more attention?
- What are the business challenges of your company right now?
In BSC Designer: go to the “KPIs” tab. There, you can formalize your business goals and align KPIs with these goals.
2. Visualize Your Quality Challenges and Their Reasons
Quality control or speaking more generally, quality assurance, is not only about having a set of indicators and reporting them. The idea is to have an understanding of the processes in the organization, their strong and weak points and trying to prevent possible quality issues.
One needs to do a regular analysis of:
- Typical processes
- Quality challenges
- Their reasons
- Possible solutions
The result of this analysis can be formalized in the form of a flow chart.
In BSC Designer: go to the “Strategy Map” tab, create a new map and use it to visualize your process. Later, you will be able to visualize some KPIs directly on this map.
3. Balance Between Lagging and Leading Indicators
Have a look at the flow charts created in the previous step. On each step, there is some input used and some result produced according to the defined standards. You can have quality indicators associated with those control points.
Those might be leading or lagging indicators.
- Lagging indicators tell you a story of what has happened. This information is useful, but you are just observing what happened, and you cannot change anything.
- In contrast, leading indicators reflect inputs that influence the process.
On your quality scorecard, you need to have both – leading and lagging indicators.
In BSC Designer: select the type of indicator on the “Context” tab, leading and lagging indicators are calculated in different ways.
4. What are the Most Important Indicators?
- “All of them are important!”
The question is:
Are they equally important?
Reducing the defects number from 0,01% to 0,001% might slow down the whole production system, and while your clients were happy with a 0,01% defect rate, for them, other product parameters are much more important.
You need to find a balance and reflect the importance of each of the indicators on your scorecard.
In BSC Designer: you have a weight property for an indicator on the “Performance” tab. Learn here what weight means and how it is calculated.
5. What is the Performance of an Indicator?
Let’s say you have a 1% defect rate, then you optimized the process and achieved a 0.1% defect rate; finally, after an introduction of additional quality measures, you got a 0,01% defect rate on average.
My question is:
How was the performance of the respective quality indicator changing? Was it improved 10 times and then again improved 10 more times?
Not necessarily! From a business viewpoint, clients might be unhappy with a 1% defect rate, and really happy when you have a 0.1% defect rate and achieving a 0.01% defect rate may not have made any visible difference to them.
In real life:
- Some indicators have a linear performance function (you improve the value of an indicator 10 times, and the performance improves 10 times).
- Some indicators have non-linear performance, e.g., the performance of an indicator might grow slowly in the beginning and then the function converts into an almost flat line, meaning that further improvements don’t improve the end performance.
When designing your indicators, choose an appropriate performance function for them.
For example, on the quality scorecard discussed above, we have Time to develop prevention plan indicator measured in days that have a customized performance function:
- If we stay below the 6 days’ time frame, the performance is in the green zone.
- If we develop prevention plan for a longer time, then performance drops quickly to the red zone and then to 0%.
In BSC Designer: there is an “Optimization” drop list on the “Context” tab for indicators. Change there the linear optimization function to a more relevant one or specify your own function.
6. The Total Score of the Quality Scorecard
When you have 50+ indicators on your quality scorecard, each with its own performance function, benchmarks, values, and weights, then theoretically you can calculate the performance of each category and then the performance of the whole scorecard.As a result, you will have a single number (some call it a “quality score”).
I know some top managers ask for a single quality score, but in most cases, it won’t make any sense as this number aggregates too much data. It looks nice in the reports, but it won’t give any meaningful information for a top manager.The better option is to group indicators into categories and report the performance of each category.
For example, from the scorecard discussed above, I can see how the organization performs in terms of learning efforts or internal improvements.
In BSC Designer: the program can calculate the performance figures for indicators, goals, perspectives and for the whole scorecard, but the end usage is up to you.
7. Plan Your Actions
Be sure that you have (preferably in writing) some plan aligned to each quality indicator.
- What will you do if this indicator gets into the red zone?
- Are there any routine procedures associated with this indicator?
- What’s the proposed budget?
- How long will it take to execute the action?
- How often do you need to revise this indicator?
- Who is responsible for an indicator?
For example, in the quality scorecard created above, we have a Company-wide training initiative, aligned with a Near miss reporting training goal where we have specified all the necessary details:
In BSC Designer: select an indicator or goal, click the “Initiatives” button, specify relevant details, budget, link to some documents, and choose a person responsible.
8. Build a Quality Dashboard
Dashboards help to visualize quality indicators. For some of them, you will need to use a gauge-style chart, for others the best option is to display a time chart with a trend line. A well-designed dashboard will give you a top-level view. Use it together with indicators visualized on the map. For more dashboard-related ideas, check out the Dashboards Insight website.
In BSC Designer: you can create a number of dashboards to visualize your indicators.
9. Track the Trend
Even if you check your indicators regularly, you overlook a negative trend. For that purpose, track the indicators’ value for a longer period of time. For example, taking a 3-month period as a base for calculations, you might see that an indicator is trending into a red zone. This will serve as an excellent early-warning signal for your team.
In BSC Designer: Use “Forecast Analysis” on “Analysis” tab to forecast indicators’ value.
10. Get Bad News Early
If some quality indicator has gotten into the red zone, make sure that all of the quality specialists involved get a notification about the problem immediately.
Also, make sure that your quality scorecard is updated with fresh data, and you are looking at up-to-date quality information.
In BSC Designer: setup notifications for the indicators on the “Alerts” tab. The software can send out a notification if an indicator has got into the red zone, or if an indicator was not updated on time with fresh data.
There are many stakeholders of quality that perceive it in different ways – prospective clients, customers, partners, our team, regulators.
- Good news: we can still quantify, measure, and even predict quality.
In this article, we discussed:
- Part 1. KPIs that help to measure, predict, and report quality
- Part 2. A Balanced Scorecard example that helps to align quality initiatives with overall business strategy
- Part 3. 10 improvement ideas for ongoing quality control
Feel free to share in the comments some interesting findings about measuring quality.
- Access templates. Sign-up with a free plan at BSC Designer for immediate access to 23 scorecard templates, including Quality Scorecard discussed in this article.
- Master skills. Check out free video tutorial for the Balanced Scorecard. Master your strategy planning and execution skills with Strategy Execution training.
- Automate. Learn what Balanced Scorecard software is and how it can make your life easier by automating strategy execution, KPIs, and strategy maps.
- ^ Predicting Fault Incidence Using Software Change History, Todd L. Graves, Tood L. Graves, Alan F. Karr, J.S. Marron, Harvey Siy, IEEE Transactions on Software Engineering 26(7):653 – 661 · August 2000