How to measure whether a design change helped

Say what “better” means in plain words

Before choosing a metric, describe the change you hope people will notice. For example: “Reviewers can spot missing information without opening several records.” Or: “Customers can fix an upload error without starting again.” Those are far more useful than a general wish to “improve engagement”.

Then ask what you could observe that would support the statement. Task completion, fewer steps, less repeated data entry, fewer people asking what happens next. Let the measure follow the goal. Don’t pick a familiar dashboard number and work backwards.

Use a few signals that check each other

One number rarely tells the whole story. Faster completion is good, unless people make more mistakes. More time in the product might mean more value, or more struggle. Pick a main signal and a couple of checks that would catch a misleading result.

Google’s HEART framework is a good reference for connecting user goals to metrics. For a single workflow change, you don’t need all five dimensions. Borrow the principle (goal first, then signal, then metric) and keep it proportionate.

Get a baseline you can trust

Record how the current experience performs before you change it, where you can. Define the task, the audience, the time window and the data source. And check that the tracking captures what you think it does. A click on “Submit” isn’t the same as a successful submission.

If you don’t have historical data, say so. You can still learn a lot after release, but you won’t be able to make a precise before-and-after claim. Don’t fill the gap with an assumed baseline, and don’t present stakeholder impressions as measured improvement.

Remember what else changed

A release can coincide with a new type of customer, a busy season, a change in support hours or a marketing campaign. All of these move numbers. A before-and-after comparison can suggest a link without proving the design caused it.

Take a self-service journey. Fewer support calls could mean clearer recovery. It could also mean fewer people using the product, or people contacting you a different way. Check the context before you celebrate.

Add qualitative evidence

Watch people try the task, read the relevant support tickets and ask targeted questions about the moments of doubt. Qualitative evidence explains why a number moved, or why the improvement you expected didn’t appear.

Keep the findings tied to the people and situations you studied. A small usability study can uncover a serious problem without telling you how common it is.

A good example of honest reporting: on a dashboard project for a UK pensions and investments provider, research showed 95 of 115 participants were happy to see complementary products, and between 50% and 60% said they’d select the example recommendation. That’s real evidence of acceptance. It isn’t a conversion rate, and the case study says so.

Decide what the evidence means

Before you look at results, agree what would make you keep, revise or investigate the design. A simple record covers:

• The original user problem.
• The expected change and the signals you chose.
• The baseline, or the fact that there wasn’t one.
• The observation period and relevant context.
• The decision, and what’s still uncertain.

All the case studies on this site separate what was delivered from what was measured. It’s a good discipline for your own product reporting too. If you’d like help setting this up for a specific initiative, Design Leadership can help.