Experimenting with a Strategy: How to Confirm Your Theory Using KPIs

An example of how to formulate strategic hypothesis, make the controlled experiments, and track the results with a set of leading and lagging KPIs.

Experimenting with a Strategy: discuss business goals, prepare experiments, track the results.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

We have a community website where people can ask for advice about KPIs. It uses a question & answer model, so the idea is that the content is partially generated by users who need help and partially by the professionals that answer their questions. We have some control over the answers, but the questions always depend on the visitors and, as it appeared, they were the biggest challenge in the way of the project’s growth.

However, the purpose of this article is not to share the best practices for the webmasters, but to:

  • Show the ways to organize a discussion around the strategy,
  • Prepare controlled experiments, and
  • Track the results with KPIs.

We started with a mission statement

Probably defining a mission was not that important for such a small project, but we decided to follow the standard approach. The purpose of the mission statement is to inspire and motivate the team. What was motivating our team? I sent emails to everyone involved in the project and asked them to share their thoughts.

My two questions were:

  1. Have you ever mentioned this project to your friend or colleague?
  2. If yes, how did you describe the project to them?

I was glad to know that most people mentioned that they were involved in the project and described their involvement as “helping others with KPIs” or “brainstorming the best ways to measure the performance.”

We asked two questions in order to find out what inspired our team and formulated mission statement.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online
That’s what was important for our experts. That is what they were proud of. As a result, we formulated the mission statement as:

“Help business professionals to find the best KPIs for their business.”

Good, short, and motivating (at least for our experts).

Formulating business objectives

In the next step we needed to think about the strategy taking into account the current market conditions, the problems that we had faced so far (the prototype of the project had already been running for several months), and the business model that we were using.

Strategy map for the project with two main objectives on it.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online
We found two success factors that were the keystones for the whole project:

  • Visitors. As long as the content is partially generated by the visitors, we needed to make sure that we have a steady traffic to the website and that this traffic is converting into the questions. For that purpose we had an umbrella goal – “Website that is easy to use” on the strategy map.
  • Engaging content. We needed to create engaging content. Our experts were happy to help when questions contained enough details and the person who asked the question did his “homework” before asking. That was reflected under the “Provide Engaging Content” goal on the strategy map.

Tracking failure and/or success

Before going ahead with the experiments we had to find the metrics that would tell us if we were doing good or not. We chose two for the content quality:

  • Leading indicator: the number of controlled experiments (nothing special, but at least we would know that we tried hard enough).
  • Lagging indicator: % of qualified questions. A “qualified question” was a subjective idea.

Leading and lagging indicators for the "provide engaging content" goal.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online
For example:

  • “We need a KPI for logistics” was not a good question, while
  • “We need a KPI for …, our business goal is …, we already tried …, but the problem was …” sounded like someone invested time in describing their problem and that is what was working for the creation of good content.

There were two indicators for the “Website that is easy to use” goal:

  • Leading indicator: Website Traffic, # of visitors.
  • Lagging indicator: Traffic/questions ratio, %

"Easyto use website" goal aligned with two indicators.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

Probably we could go with just a “Traffic/questions” ratio, but we had to be sure that we were getting some statistically significant traffic in order to make some conclusions based on the data.

Controlled business experiments

We were ready to start controlled experiments. The first one showed that there was no problem with traffic. We tried AdWords, we gave visitors of other websites an opportunity to ask their questions and there were people coming to the new project and posting their comments.

  • The “traffic/questions ratio” was comparable with the benchmarks that some of our colleagues shared with us.

The goal of creating engaging content was more challenging. We had no data about what would work the best, so we planned to try at least 3 models.

I’d like to make one important note:

In the beginning we didn’t know what exactly we were going to try. That’s the nature of making an educated hypothesis (rather than planning). We did the first experiment, got the data, and only then understood what we should try next.

Here are the hypothesis that we tested with our experiments.

“Template” hypothesis

Our first hypothesis was to use a “Template” model. The idea was to suggest for users to fill in a template with some predefined fields (like “the purpose of KPI,” “the business context,” etc.) According to our idea visitors would give us more details about their problem if they had a template that they could use. We also added the requirement for a minimum number of words that one needed to use.

When testing "Template" hypothesis, the “% of qualified questions” indicator was on the 20-30% level.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

The project was just starting and we did not have many visitors, so we tried this hypothesis for 5 months to get some representative results. The “% of qualified questions” was on the 20-30% level (red zone in our case) and never hit the target of 45%.

Introducing benchmark data

In June we had enough traffic to get some meaningful data from the experiments. Moreover, we found some benchmark data from other Q&A websites. According to our estimations their figures for “% of qualified questions” were about 60-70%, so we:

  • Updated our target for “% of qualified questions” indicator to 70%, and
  • Decided to experiment more with the question form.

“Make it easy” hypothesis

The next hypothesis to be tested was “Make it easy.” Give visitors a way to post their questions without hassle (without registration), they will spend less time on the registration and more time on the question details, and their questions will be more interesting as a result.

From July to November we tried this hypothesis:

  • For July “% of qualified questions” dropped to 15%.

We thought that the summer period might be causing this.

  • We waiting until November, but “% of qualified questions” was still around 20%.

When testing "Make it easy" hypothesis "% of qualified questions" dropped to 20%

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

People were asking short questions that did not make any sense either for our experts or for visitors (we could tell this by tracking the “time on page” metric). We analyzed the traffic once more, hoping to see that some source generated those “bad” visitors, but the problem was not with the traffic. The value of the “Website traffic” indicator was growing, and traffic structure (the distribution of the traffic sources) remained more or less stable.

Online wizard will help even inexperienced users to create a professional strategy map with business goals, KPIs, and initiatives.

Strategy Map Wizard – Balanced Scorecard in 6 Minutes

Answer simple questions - build a professional strategy map in 6 minutes.

Strategy formulation and description might be a time-consuming project even for an experienced strategist. This Strategy Scorecard Wizard will make the whole process fast and intuitive. Learn more!

Start now!

Re-analysis of the situation

Before I was writing about XY motivation theory by Herzberg. What we were doing so far looked like following the “X” theory. We used “carrot and stick” motivation first (the “templates” hypothesis), and then just started encouraging short anonymous questions (“make it easy” hypothesis). We needed to make a shift from the “X” theory to the “Y” theory, where people are “motivated by self-esteem, creativity and personal development.”

“Y” theory hypothesis

We wanted people to behave according to the “Y” theory, and be more responsible for what they asked and actually do some basic research before asking. The idea was not straight: instead of making the process easier we decided to complicate it a little bit.

The registration was now obligatory (but still not too complicated), and people had to enter their full names (not just logins) + some other changes were also introduced.

The results were impressive. In November “% of qualified questions” jumped up to the 75-80% level.

"Y" theory hypothesis implemented - "% of qualified questions" jumped up to the 75-80% level.

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

But this could also mean that the total number of questions asked will decrease. We checked the indicator “Traffic/questions ratio” and the dashboard showed that it stayed on the level of 1.5 – 1.7% which was similar to the level that we had when it was possible to post the questions anonymously.

The indicator "Traffic/questions ratio" stayed on the level of 1.5 - 1.7%

View View Scorecard for Testing Strategic Hypothesis online in BSC Designer Scorecard for Testing Strategic Hypothesis online

Main take-aways

I hope this is not yet the end of the story and we’ll come up with some new ideas regarding this project. Here are some conclusions that we can already make in the context of the strategy execution and Balanced Scorecard:

  • Strategy is not a plan, but a number of controlled experiments. In the beginning we had no idea how to achieve a good “qualified questions” ratio (no strategic planning was possible). The best practice was found as a result of controlled experiments.
  • Benchmarking data is important. Without this data it is hard to say if you are doing good or bad. Use benchmarking to motivate your team, especially if your figures are lower than ones from the competitors.
  • Experiments are controlled by the indicators. We were clear that we needed to look at “% of qualified questions,” and “visitor/question ratio” to be sure that we were moving into the right direction.
  • Visual picture is must. We visualized business goals with KPIs and initiatives on the strategy map. We visualized the KPI data on the dashboard. This is helping not only to tell the story in this article, but to focus the discussion around what matters.

I’m sure there were similar experiments in your business. Feel free to share the best practices that worked for you and ask your questions in the comment box below.

Use Scorecard for Testing Strategic Hypothesis discussed in this article as a starting template for your own scorecard! Access this example scorecard online or download .BSC project file for BSC Designer PRO.

View Scorecard for Testing Strategic Hypothesis online

Would you like to create your own scorecard? Try our free Strategy Map Wizard.

Related Articles

Strategy and KPIs Scorecard Expert | Speaker | CEO. Aleksey Savkin (LinkedIn, @bscdesigner) is helping companies to better formulate their strategies and make the process of strategy execution more tangible with KPIs. New book by Aleksey: 10 Step KPI System

If you prefer to receive posts like these by email sign up to our newsletter.

Posted in BSC Implementation

Thank you for sharing!

Whether you are looking for a professional Balanced Scorecard software, or just researching information about Balanced Scorecard and business strategies, we recommend you to download and try our BSC Designer software (no credit card is required).

We will follow up with you with lessons about the Balanced Scorecard and will keep you informed about the trending articles on bscdesigner.com

Follow us in Social Media

Send this to a friend

More in BSC Implementation
5 Steps for Effortless Balanced Scorecard Implementation

Who else wants an easy to follow Balanced Scorecard implementation guide? Here is a simple 5 step formula that will...

Balanced Scorecard Cascading: Make Your Strategy Resonate with Company

Good strategy implies the coherence between objectives and the way people interpret them on each level of the organization. In the business...