Measuring the User Experience
Thomas Tullis and William Albert
“What cannot be measured cannot be managed
What I like
It’s academic which was expected. What stood out was the inclusion of stories and examples to make concepts easier to understand
There are some common in design research (e.g. confidence intervals, data visualization, and research planning best practices) that this book helps disambiguate
It’s practical, like a cook-book, while still respecting the contextual nuance of experience
What’s missing
Took a long time to read this book because it’s so dense. This book has a wide scope, which means jumping around between equally complex ideas takes a toll
Being an academic book, some sections were glanced over when they became so complicated. It felt hard to keep up with what the authors were saying sometimes
About 50% of this book felt impactful, but that could be because this was the second time reading it
Key Topics
Metrics, User Experience, Usability, Research, Communications
Review
Measuring the User Experience is a comprehensive guide on UX metrics and how to collect them. This was my second time reading this book, the first time was while in school.
Research methods can be formulaic yet contextual. Finding data can be just as, if not harder than, synthesizing it. It’s some means of capturing experience, purifying, flavoring it, and bottling it for distribution with an easy to understand label. Methods, like a set of an ingredients, are sometimes replicable. Algebra was not a strong suit of mine when I was in school, and neither were research methods. Being in the lab was fun for me, but remembering all of the rules was not. That’s where Measuring the User Experience shines and I think is helpful to read to understand what’s in the book, and store for later.
It’s not light reading, the chapters are dense, and the subject mater is academic. It’s a textbook, what do you expect? With all of that density and information, the authors include a lot of examples to help illustrate complex topics, and do a great job making complex things like confidence intervals easier to understand. The contents would probably seem redundant to someone familiar with research methods, but one might still find it interesting in the contexts of user experience.
Anyone that’s interested in collecting UX metrics, or deploying a UX metrics initiative, could benefit from this book. Each chapter provides a helpful overview of a specific facet of UX and UX metrics. The contents feel very practical and includes clear instruction on how to perform each task. There’s even a handy set of 10 principles at the end, along with a series of case studies from companies that have deployed UX research methods, lessons learned, decisions made.
For designers, this book alone provides a nice overview of collecting user metrics. The other side of this coin for someone in product would be collecting business metrics as well. I think the combination of those two things would be a good skillset for product designers to have. I’d also be curious how AI might make collecting and distilling UX metrics easier and perhaps more reliable.
Learnings
User experience is an interaction between a person and a product, system, or interface. Practically anything that people interact with involves an experience. As things become more complex, as does a user experience
Metrics are anything that help us as researchers measure a behavior or phenomena. In design, like UX, that behavior or phenomena includes what happens when a user interacts with a product, or in the case of CX, when a customer interacts with a brand
Independent variables are what we expose so we can collect data from dependent variables. Without the independent variable, the dependent variable would not produce data that interests us
There are different ways we can turn data into information, specifically by calculating the mean, median, mode, standard deviation, and confidence intervals. Generally we want to present the mean of a collection of data to report the average
When studying data for information to help us answer research questions, it is often helpful to look for correlations to understand the relationships between them, and to plot our data onto visualizations to help them communicate
When starting on any new project it’s good to identify what the project goals, UX goals, and business goals are
A great user experience exceeds user expectations- whether it’s easier, more fun, or better than expected
Usability testing can be done any time in any place, and there a collection of methods that can be used to make it possible
In user research we often need to report on how efficiently users can complete tasks
Errors are outcomes prompted by problems like usability issues
Task completion time and task successful are useful parts of the UX to measure
Learnability is how well users are able to learn how to do something, and how well they retain the information necessary to do something
One of the only things we care about is if users are happy with an experience, and to understand this we need to collect self-reported metrics
Rating scales like Likert scales help us turn subjective into objective data. They should have a neutral point and a 7 point scale
There are a number of scales that can be used to understand system usability, such as SUS and QUIS or NPS
Self-reported metrics are helpful for sourcing data about a product while it’s being used
Usability issues are issues based on user behavior while using a product, the opposite of a usability issues is a usability finding
The number of participants you have on a usability test should adapt based on your goals (i.e. formative vs summative evaluation)
Eye Tracking is a nice way to measure how effectively a product captures user attention
In UX there are seven unique aspects of the emotional experience: engagement, trust, joy, stress, frustration, confidence, and surprise
Usability studies can be completes, as there are often many metrics that need to be collected for things like task completion time, satisfaction, and number of errors
Netflix is a good example of a company that serves content depending on user needs. Based on numerous eye tracking and qualitative studies, it was found that Netflix uses bold imagery, stars, and “evidence” to help browsers make a decision
There are a number of ways to evaluate how a products features compare to a competitors, which is helpful for setting baselines
What can not be measured can not me managed, and this saying holds true for UX as well
when collecting UX metrics we need to remember to speak the language of businesses when reporting back results