What matters more for AI: more KPI data or a short comment explaining what changed? Three controlled experiments show how rationale can change AI-powered management decisions.
How the Experiments Worked
Each experiment used the same management question:
Based only on the information available in this case, what is the single highest-priority action management should take next?
The AI had to return one action and one short explanation.
Each experiment contained the same four information conditions:
- Case 1: primary KPI only (no rationale)
- Case 2: primary KPI + additional KPI
- Case 3: primary KPI + additional KPI + rationale
- Case 4: primary KPI + rationale
Importantly, the cases vary not only in the amount of information available but also in how the same underlying management facts are represented. In some cases, information shown as a separate KPI is instead expressed as rationale linking the fact directly to its management meaning.
Each case was run 100 times using GPT-6 Sol with the same model settings. Braintrust was used to automate the trials and scoring.
Responses were evaluated only on the highest-priority management action and the reasoning directly supporting it.
- 100%: the action reflects the critical contextual issue.
- 50%: the action is reasonable but misses the critical contextual implication.
- 0%: the interpretation could lead management toward an unjustified action or conclusion.
The scoring criteria were defined separately for each experiment so that the evaluation reflected the specific management problem being tested.
How to Read the Scores: scores shown in the results tables are averages across 100 trials. Each individual response received 100%, 50%, or 0% according to the experiment-specific rubric. For example, an average score of 50% can result from different combinations of individual scores; it should therefore be interpreted as the average performance across the 100 trials rather than as the percentage of responses that were correct.
Experiment 1 — Misleading KPI Trend
This experiment tested whether AI would trust an improving KPI when the measurement process changed during the same period.
Primary KPI: Defect Rate
Target: ≤2.0%
| Date | Actual |
|---|---|
| Jan 2026 | 4.8% |
| Feb 2026 | 4.5% |
| Mar 2026 | 4.1% |
| Apr 2026 | 3.2% |
| May 2026 | 2.6% |
| Jun 2026 | 2.1% |
Viewed alone, the KPI shows a strong improvement toward the target.
At the same time, inspection sampling changed.
Additional KPI: Inspection Sample Size
Target: 100%
| Date | Actual |
|---|---|
| Jan 2026 | 100% |
| Feb 2026 | 100% |
| Mar 2026 | 100% |
| Apr 2026 | 50% |
| May 2026 | 50% |
| Jun 2026 | 50% |
The rationale added one short explanation:
April 2026 Comment: Inspection sampling reduced from 100% to 50%; fewer units are now inspected for defects.
Representing the Case in BSC Designer
The experiment was modeled as an operational scorecard with historical KPI values and a date-specific comment attached to the relevant reporting period.
Visual Representation of Cases in BSC Designer
The Case 1 included only the primary KPI, without any additional KPI or rationale:
The Case 2 included the primary KPI and an additional KPI, but no rationale:
The Case 3 included both the primary and additional KPIs, as well as a date-specific comment for the primary KPI:
The Case 4 included the primary KPI and a short date-specific comment:
Results
| Case | Information Available | Average Score |
|---|---|---|
| Case 1 | Defect Rate only | 50% |
| Case 2 | Defect Rate + Inspection Sample Size | 97% |
| Case 3 | Both KPIs + rationale | 53% |
| Case 4 | Defect Rate + rationale | 100% |
Case 3 produced the most unexpected result: with both KPIs visible, AI frequently treated Inspection Sample Size as a second underperforming KPI and prioritized restoring sampling to 100%.
The second KPI helped AI detect that something had changed, but it also created a competing management signal that deprioritised primary expected action.
Case 4 avoided that ambiguity. The rationale linked the measurement change directly to the interpretation of Defect Rate, so validation of the trend became the highest-priority action in all trials.
More information does not automatically create better context. The way information is structured changes what AI treats as the management problem.
Experiment 2 — Learning From Previous Interventions
This experiment tested whether AI would use evidence from a previous intervention when deciding what management should do next.
Primary KPI: Customer Churn Rate
Target: ≤5.0%
| Date | Actual |
|---|---|
| Jan 2026 | 6.0% |
| Feb 2026 | 6.4% |
| Mar 2026 | 6.9% |
| Apr 2026 | 7.5% |
| May 2026 | 8.0% |
| Jun 2026 | 8.4% |
A second indicator recorded an intervention that had already been tried.
Additional KPI: Average Retention Discount Offered
| Date | Actual |
|---|---|
| Jan 2026 | 0% |
| Feb 2026 | 0% |
| Mar 2026 | 15% |
| Apr 2026 | 15% |
| May 2026 | 15% |
| Jun 2026 | 0% |
The rationale added the result of that intervention:
June 2026 Comment: A 15% retention discount was tested from March through May. Churn continued increasing during the test, while revenue per retained customer declined. The discount initiative was stopped.
Results
| Case | Information Available | Average Score |
|---|---|---|
| Case 1 | Churn Rate only | 69% |
| Case 2 | Churn Rate + Retention Discount history | 100% |
| Case 3 | Both indicators + rationale | 100% |
| Case 4 | Churn Rate + rationale | 100% |
The intervention history alone gave AI enough evidence to recognize that churn continued rising while the discount was active.
The rationale made the learning explicit:
The action was tried. It did not solve the problem. It produced a negative side effect. It was stopped.
AI can know that retention discounts are a possible response to churn. The historical record tells what this organization already tried and what happened as a result.
Experiment 3 — Strategic Trade-Off
This experiment tested whether AI would try to improve an underperforming KPI without recognizing that management had deliberately accepted the deterioration to improve another outcome.
Primary KPI: Average Order Processing Time
Target: ≤24 hours
| Date | Actual |
|---|---|
| Jan 2026 | 20 h |
| Feb 2026 | 21 h |
| Mar 2026 | 22 h |
| Apr 2026 | 30 h |
| May 2026 | 34 h |
| Jun 2026 | 36 h |
Viewed alone, processing performance deteriorated sharply.
A second KPI shows what changed at the same time.
Additional KPI: Fraud Loss Rate
Target: ≤1.0%
| Date | Actual |
|---|---|
| Jan 2026 | 2.4% |
| Feb 2026 | 2.3% |
| Mar 2026 | 2.2% |
| Apr 2026 | 1.6% |
| May 2026 | 1.1% |
| Jun 2026 | 0.8% |
The rationale explained the relationship:
April 2026 Comment: Enhanced fraud checks were introduced. Management accepted longer processing times as a trade-off to reduce fraud losses.
Results
| Case | Information Available | Average Score |
|---|---|---|
| Case 1 | Processing Time only | 50% |
| Case 2 | Processing Time + Fraud Loss Rate | 95.5% |
| Case 3 | Both indicators + rationale | 100% |
| Case 4 | Processing Time + rationale | 100% |
The second KPI gave AI enough information to infer the relationship in nearly all trials. The explicit rationale made the management intent unambiguous.
Combined Results
| Experiment | Case 1 | Case 2 | Case 3 | Case 4 |
|---|---|---|---|---|
| Misleading KPI Trend | 50% | 97% | 53% | 100% |
| Learning From Previous Interventions | 69% | 100% | 100% | 100% |
| Strategic Trade-Off | 50% | 95.5% | 100% | 100% |
The results do not suggest replacing KPIs with narrative comments. The two provide different types of information.
Structured data adds signals. Rationale explains how those signals should be interpreted.
Finding 1. KPI Data Alone Can Lead to a Plausible but Incomplete Action
In all three experiments, the primary KPI gave AI enough information to produce a plausible management response.
Because the AI was required to choose one highest-priority action, the experiments tested not merely whether AI could identify a relevant action, but whether it would prioritize the action that reflected the critical contextual issue.
Finding 2. Rationale Improved the Management Decision
A short rationale was enough to tell AI:
- why a trend required validation,
- what had already been tried,
- what management had deliberately accepted.
In each experiment, Case 4 — the primary KPI plus rationale — achieved a 100% average score.
Finding 3. Context Structure Matters
Additional KPI produced major improvements in Experiments 2 and 3, but it did not always remove ambiguity.
In Experiment 3, the two KPI histories were enough for AI to infer the strategic trade-off in most trials, reaching 95.5%. Adding the rationale made the relationship explicit and increased the score to 100%.
Experiment 1 showed an even stronger effect. When Inspection Sample Size appeared as a separate KPI with its own target, it became another management problem to solve. AI could prioritize fixing that KPI instead of using the measurement change to question the validity of the Defect Rate trend.
The same underlying information can therefore influence an AI recommendation differently depending on how it is represented. The objective is to preserve the relationships and reasoning that make performance information interpretable.
What Organizations Should Start Preserving Now
The experiments point to the need to preserve context to improve the quality of AI-assisted performance management. Examples of useful context include:
- explanations of unusual performance,
- changes in measurement methods,
- corrective actions,
- results of previous interventions,
- assumptions behind decisions,
- reasons why initiatives were stopped or continued,
- intentional trade-offs.
Preserving This Context in BSC Designer
BSC Designer supports date-specific comments on KPI values, keeping each explanation connected to the reporting period it describes. This adds a second layer of context to the existing performance data.
A simplified data model can look like this:
Layer 1: What Happened?
Targets, actuals, trends, and status.
Layer 2: Why Did It Happen, and What Did We Learn?
Comments, rationale, actions, decisions, and trade-offs.
The experiments show why this second layer matters when AI is later asked to interpret the performance record.
Limitations and Reproducibility
This study used three constructed management experiments, GPT-6 Sol, and a fixed evaluation rubric.
It tests a specific question:
Does preserved management context change the highest-priority action recommended by AI?
It does not measure the general quality of AI management advice across industries, models, or management situations.
- Data sets used: Data-Set-1, Data-Set-2, Data-Set-3
- Scorers used: Scorer 1, Scorer 2, Scorer 3.
The same experimental design can then be repeated with other AI models and additional management situations.
Alexis Savkín is the founder of BSC Designer, strategy execution and performance management software that connects objectives, KPIs, initiatives, risks, and performance data. With more than 20 years of experience, his work focuses on Strategy Execution Architecture — how organizations bring together strategy, measurement, ownership, risks, and review processes into a coherent execution system. He created the Strategy Execution Canvas and the 10-Step KPI System and has authored two books and more than 100 articles on strategy and performance management. Connect with Alexis on LinkedIn.