[Skip to content](https://academy.innguma.com/en/method-governance-and-une-166006/evaluating-and-improving-without-falling-for-vanity-metrics/#contenido) Lesson 4.4 · Learning path 04 # Evaluating and improving without falling for vanity metrics How do we know whether the intelligence system works and what we should change? **You take away** monthly review dashboard + log of corrective actions. ≈ 30 min · Intermediate [Do the practice](https://academy.innguma.com/en/method-governance-and-une-166006/evaluating-and-improving-without-falling-for-vanity-metrics/#hazlo-tu-revision-de-rendimiento-20-minutos)[Exercise](https://academy.innguma.com/en/method-governance-and-une-166006/evaluating-and-improving-without-falling-for-vanity-metrics/#compruebalo)[Read the lesson](https://academy.innguma.com/en/method-governance-and-une-166006/evaluating-and-improving-without-falling-for-vanity-metrics/#medir-actividad-no-demuestra-valor) In 1 minute if an indicator does not help decide what to keep or change, it is probably a statistic, not a governance tool. ## Measuring activity does not prove value “We published 300 alerts” is a measure of output, not necessarily of intelligence. A well-designed function needs a combination of **operational, quality, usage and learning** indicators. None of them should be presented as absolute proof of strategic impact. | Dimension | Question | Indicative indicator | Caution | | --- | --- | --- | --- | | Operation | Are we monitoring what was planned? | % of checks carried out out of those planned | Does not measure the relevance of sources | | Coverage | Do we know our blind spots? | Critical sources with reviewed status and declared gaps | Does not equal total coverage | | Quality | Are we classifying correctly? | Errors detected in a reviewed sample, including false negatives | Depends on the sample and criteria | | Timeliness | Do we arrive while it is still useful? | Time from detection to relevant dissemination | Speed without verification is also a risk | | Usage | Does the recipient make use of it? | Products consulted and documented feedback | Reading is not the same as deciding | | Learning | Do we change the system after finding failures? | Corrective actions verified and closed | Closing tasks does not prove effectiveness | We do not recommend turning these metrics into a universal “intelligence quality” index: monitoring objectives, the criticality of risks and the available capabilities vary considerably. Interactive · Match ### Which dimension does each indicator measure? Match each indicator to its dimension. Activity: match each indicator to its dimension (operation, coverage, quality, timeliness, use or learning). ## The logic of continuous improvement A useful review follows this sequence: **performance observation → specific problem → probable cause → proposed change → owner and deadline → follow-up check**. For example, if a relevant news item was left out by a filter, it is not enough to widen the keywords: you have to review the criterion, test equivalent cases, measure the increase in noise and check whether it also affects other sources. AI can help group incidents, propose hypotheses about causes and draft actions; it should not assess the reliability of its own conclusions without outside checking. Interactive · Explore step by step ### The improvement sequence, with a case Go through the six steps with the example of the filter that left out a relevant news item. The continuous improvement sequence in six steps with an example. Company A case · simulation Fictional case ## Company A case: the wrong metric The team celebrates having doubled the number of news items received. However, the briefing still arrives after the investment meetings and omits official information about a pilot programme. The review reveals two distinct problems: the product is poorly synchronised with management, and official coverage is insufficient. The delivery date and the source map are changed before any new alerts are added. Educational simulation: fictional information and organisations. Do it yourself ## Do it yourself · Performance review (20 minutes) Create a dashboard with one metric per dimension, an owner and a review frequency. Then write an improvement decision you would take if the indicator worsens for two cycles. State explicitly what you can NOT conclude from that data point. **Deliverable:** monthly review dashboard + log of corrective actions. **Common mistakes:** choosing metrics only because the tool displays them; rewarding volume; not analysing omitted information; measuring satisfaction instead of use; declaring success without a baseline. **Key takeaway:** if an indicator does not help decide what to keep or change, it is probably a statistic, not a governance tool. R16 · Improvement ### Maturity assessment and 90-day plan Identify verifiable improvements and assign them a short cycle. [See the template →](https://academy.innguma.com/en/library/maturity-assessment-and-90-day-plan/) R04 · Quality ### Query and filter test Check for omissions and false positives before activating rules. [See the template →](https://academy.innguma.com/en/library/query-and-filter-test/) Check it · closed answer ## Interpret the metrics Describe the observed change without turning it into causal impact. 1. Checks: from 18/20 to 19/20. **See the answer** **Answer: Observed change, no causal proof.** It is an observed change. Attributing it to an intervention would require a test with stable definitions. 2. Alerts considered useful: from 2/4 to 3/5. **See the answer** **Answer: Observed change, no causal proof.** The samples are small and different: describe the change and which cases each sample contained. 3. Errors detected: from 3/10 to 1/10. **See the answer** **Answer: Observed change, no causal proof.** Identify which sources were omitted; do not conclude an overall improvement in precision. Propose a test for the next cycle with stable definitions, a pre-set sample and feedback from the recipient. ### And now, your own answer This part needs professional judgement: there is no automatic marking. Complete your work in the practice, rate it with the lesson's rubric and compare it with the indicative solution. What your answer should include - One metric per dimension, with an owner and a frequency. - An improvement decision if the indicator worsens for two cycles. - What you can NOT conclude from each data point. Self-assessment rubric (0–2 per criterion) It is not a professional certification grade or a UNE normative criterion. Link to a decision Provenance and limits Continuity and authority Product and recipient Improvement check **Indicative solution** Describe the observed changes without turning them into causal impact; identify which sources were omitted and which cases each sample contained. Propose a test for the next cycle with stable definitions, a pre-set sample and feedback from the recipient. Error that fails the exercise: claiming that a business improvement or an overall precision improvement has been proven with this simulation data. Illustrative solution for a fictional case: reasoned alternatives are valid. **Optional extension Extension · Measuring with denominators and decisions** Each indicator should be declared as a sheet with **definition, numerator, denominator, period, data source, owner, indicative threshold and associated decision**. Without a denominator, “14 incidents” can indicate deterioration or simply an increase in checks. A percentage is not solid evidence if we do not know its comparison base. ### Example · Company A · Entirely fictional data | Indicator | Period 1 | Period 2 | Prudent reading | | --- | --- | --- | --- | | Critical checks carried out / planned | 18/20 | 19/20 | Small operational improvement; find out which checks were omitted (two, then one) | | Alerts considered useful / alerts commented on by the recipient | 2/4 | 3/5 | Do not extrapolate: different samples, few comments | | Relevant errors found in a sample of 10 records | 3/10 | 1/10 | Positive signal, but a minimal sample and possible differences in difficulty | | Actions whose effectiveness was verified / actions closed | 0/2 | 1/3 | Closing actions is not the same as correcting the cause | How the conclusion is worded matters. It is legitimate to say “the team has recorded more checks carried out and fewer errors in this sample”. It is illegitimate to say “intelligence improved by 67%” or to attribute a successful investment to these figures without evidence about other causes. ### Distinguishing three levels of contribution 1. **Activity:** the briefing was produced on time. 2. **Usage:** the recipient read it, asked for clarification and brought it into a deliberation. 3. **Plausible influence:** the decision left a record of a hypothesis or risk coming from the product and of what changed. Do not turn “plausible influence” into demonstrated causality. Document limitations, disagreements and cases where the recommendation was not followed: they can offer more learning than a collection of positive testimonials. ### Optional extension · Two cycles and a decision Write a sheet for a coverage indicator and another for a usage indicator. Record the denominator and build two alternative explanations for an apparent improvement. Then design a test that would let you tell them apart. In the fictional case of 19/20 checks, was the one decisive source the one left out? If so, the aggregate percentage hides a critical risk. **Deliverable:** metrics sheet, rival hypotheses and an improvement decision with a test date. Take the findings to R16, without turning the system into a ranking of people. ## References and limits - UNE · Asociación Española de Normalización. *UNE 166006:2018. Gestión de la I+D+i: Sistema de vigilancia e inteligencia* (2018). [Document](https://www.une.org/encuentra-tu-norma/busca-tu-norma/norma?c=N0059973) Reference framework for watch and intelligence systems. Cited only as context, never as a source of requirements. Reference consulted on 2026-10-18. - Company A is a fictional case: its data, companies and figures do not describe real markets or organisations. - The five dimensions and the 0–3 scale are an educational rubric from Academy: they do not reproduce clauses of UNE 166006, do not replace an audit and do not allow a declaration of conformity. Library · R16 ## Maturity assessment and 90-day plan Putting it into practice? Download template R16 · Maturity assessment and 90-day plan and fill it in with your case. - [Download R16 (Excel)](https://academy.innguma.com/en/library/descargas/R16-maturity-assessment-and-90-day-plan.xlsx) - [Back to the learning path](https://academy.innguma.com/en/method-governance-and-une-166006/) **Download** - [R16 · Maturity assessment and 90-day plan](https://academy.innguma.com/en/library/maturity-assessment-and-90-day-plan/) - [R04 · Query and filter test](https://academy.innguma.com/en/library/query-and-filter-test/) Open reading, no registration.