Your latest survey shows that satisfaction has increased. That sounds encouraging—but before you describe it as an improvement, it’s worth checking what you’re comparing.
Have you asked the same questions? Heard from a similar mix of people? Collected feedback at a comparable point in their experience?
Tracking results over time can help organisations understand whether services are changing. A useful comparison starts with knowing how each set of results was produced.
Decide What You Want to Track
Start with the decision the research needs to support. You might want to understand whether customers find it easier to contact you, whether tenants are happier with repairs or whether employees feel better informed.
Choose a focused set of measures that relate to those objectives. Keep a record of exactly what each measure means and how it is calculated.
Our guide to writing a good research question explains why clear wording and purpose matter from the outset.
Check That the Questions Still Match
A small change in wording can change what you’re asking people to assess.
“How satisfied were you with your most recent enquiry?” asks about a specific experience. “How satisfied are you with our customer service?” asks for a broader judgement. The results shouldn’t be treated as interchangeable.
Check the answer scales too. A five-point satisfaction scale is different from a score out of ten. Changes to the order of questions, instructions or which respondents see a question may also affect the comparison.
If a question needs improving, make the change deliberately and document it. Sometimes the clearest approach is to treat the revised question as the start of a new series.
Look at Who Responded
Two surveys can receive the same number of responses and still represent different experiences.
For example, one survey might include more long-standing customers, while the next includes more people who recently joined. Or a particular location may account for a larger share of responses than before.
Review the mix of respondents against the audience you intended to hear from. Where suitable data is available, compare relevant groups as well as the overall result.
Keep subgroup findings proportionate to the number of responses behind them. Small groups can be useful for identifying questions to investigate, but shouldn’t automatically be treated as a firm basis for broad conclusions. Read more in our guide to reliable survey results.
Consider the Method and Timing
Switching from an online questionnaire to telephone interviews may change who participates and how people answer. That doesn’t make the new approach wrong, but it does mean you need to consider its effect when interpreting a trend.
Our comparison of online and telephone surveys explores the strengths and limitations of each method.
Timing also matters. Feedback collected immediately after an appointment may differ from feedback requested several weeks later. Seasonal demand, a recent service disruption or a change in the services being used can provide important context.
Describe the Change Accurately
Imagine that 70% of respondents were satisfied in one survey and 75% were satisfied in the next. That is an increase of five percentage points.
Calling it simply a “5% increase” is ambiguous because a relative percentage change is a different calculation. Use clear labels and show the starting and ending figures.
Include the number of responses and explain how the percentage was calculated, including how “don’t know” or “not applicable” answers were handled.
A movement in the headline figure also needs to be considered alongside the uncertainty in the research. Don’t describe every small change as meaningful without checking whether the evidence supports that interpretation.
Use Comments to Explore the Reasons
A score can indicate that something has changed. Written feedback or follow-up conversations can help you investigate why.
If satisfaction has fallen, look for recurring themes in respondents’ descriptions of their experiences. Compare those themes with service records and other relevant evidence.
A change following a new process doesn’t, on its own, prove that the process caused it. Other things may have changed at the same time. Our article on the difference between data and insight explores how context helps turn figures into understanding.
Keep a Simple Comparison Record
Before presenting a trend, check:
The question wording and answer options.
Who was invited, who responded and the response numbers.
The collection method and fieldwork dates.
The timing of the survey relative to the customer experience.
The calculation used for each reported measure.
Any changes or limitations that readers need to understand.
This record makes it easier for colleagues to interpret the findings and helps the next research project build on a clear starting point.
Use Trends to Guide the Next Step
A useful comparison should help you decide what to maintain, improve or investigate further.
At Viewpoint, our research services help organisations collect feedback and understand what it means. If you’re planning a repeat survey or reviewing a change in your results, get in touch to discuss your project.
