The need for an Insight Extraction Framework
We often see that analysts do not know what to do with a dataset other than present an endless stream of analysis results and the corresponding facts. Reports and presentations covering endless facts, without discussing any implication or prompting any action end up in the drawer. Worst case, the report or presentation follows the questionnaire flow, or the happenstance order of analytic steps followed by the analyst. It’s pointless, because the audience will not see the wood for the trees, lose interest and fall asleep, and eventually, may lose faith in the work and the analyst.
The Insight Extraction Framework
This is why we suggest implementing a rigorous insight extraction framework. The framework consists of four questions we ask ourselves for every fact that we extract from our dataset:
- What? What do we observe with the fact? Do we have evidence of the validity of the fact?
- Why? Why do we think it is? What has caused the observation to be what it is?
- So what? What is the implication of this fact? Is it needed to know or nice to know?
- Now what? What do we do next or different now that we know this? What do we advise our stakeholders? What is the next thing to analyze?
In our framework, it is only an insight if all four questions are answered. Let’s look at two hypothetical examples from NudgeLab to see how we can answer the four questions.
| What? The fact & evidence | Why? The reason | So what? The implication | Now what? What to do next |
| We have as many males as females in our sample. It is the same in the various legs representing test and control conditions. 1) | The sample allocation procedure was successful to attract both genders, and our data cleaning has not ruined the distributions | We don’t need to worry that effects that we find later in the analysis, are caused by a misbalance in gender across legs. | Proceed with the analysis without worrying about gender balance across legs. |
| We have more females than females in our sample, but the balance is the same across the legs representing test and control conditions. 1) | Although the sample procedure attracted more females than males, the sampling and data cleaning did not cause a wrongful balance across the legs. | We need to make sure that effects that we find later in the analysis are not caused by more females in the sample. But we don’t need to worry about balance in gender across legs. | Keep checking for the effect of gender of overall results. Confirm the meaning and relevance of the gender balance with stakeholders. |
| We have a disbalance in the proportion of females and males in the legs or conditions of the study. In some legs, the proportion of females is significantly higher or lower than in others. | Either the allocation procedure of genders to the various legs or conditions of the study has failed, or the data cleaning procedure has caused a disbalance. | Any effects we may find in future analyses may be caused by the misbalance of genders across the legs and conditions. | Adapt the analysis plan to keep controlling for the balance of genders across conditions and legs. |
This is why I find it important to analyze the sample composition before engaging in the “real” analysis of study results: to rule out or control for any anomalies because of a lack of balance in the sample composition.
We can look at the same if it comes to a change in choice behavior, a typical dependent variable in NudgeLab. Let’s say that we have a dependent variable which equals 1 if the respondent displays the desired behavior, and 0 if not.
| What? The fact & evidence | Why? The reason | So what? The implication | Now what? What to do next |
| There is a meaningful difference in consumer behavior between the control and test leg(s), the null hypothesis gets rejected 2) | The theory holds and / or the operational execution of the nudge is effective | The nudge can be used to influence consumer behavior, at least in the situation as tested. The generalizability is a question. | Analyze if outcome holds if controlled for interaction effects or the effect of moderator or mediator variables. Suggest use of the nudge and investigate generalizability. |
| There isn’t a meaningful difference in consumer behavior between the control and test leg(s), the null hypothesis gets rejected 2) | If we’re willing to assume that the theory holds, the operational execution of the nudge is not effective | There is no evidence that the nudge as tested can be used to influence consumer behavior. More analysis or new work is needed. | Analyze if outcome holds if tested for interaction effects or the effects of moderator or mediator variables. Decide if it makes sense to keep going down this path. |
We suggest following this framework for every major step in your analysis. It means that you will end up with a set or series of completed frameworks for multiple sets of facts and answers to the four questions. This seems like a lot of work. But the systematic and time-consuming nature of this work will cause you to think twice about what to test and stop you from doing random tests and create chaos. Of course, your test cycle will primarily be informed by your (a priori) hypotheses, but it also applies to (post hoc) analysis steps and explorations.
Follow this link for a template for insight extraction that you may find helpful. Use one template form for every major step in your analysis.