Focus groups, user surveys and interviews, and opinion polls are all great tools, but they are not a substitute for user testing. Focus groups can provide critical information so that your marketing organization can learn how to better position your product in the marketplace and focus groups are also a great way to brainstorm ideas for new product or features. But it’s user testing that will help you hit your CX metrics out of the ballpark. Watch customers interact with your product. Testing provides data that helps you diagnose problems and gives you the context to help you fix them.
Watching what people do and listening to what they say they will do are dramatically different activities that yield dramatically different types of information.
Focus groups seem like they could be just as helpful, at least on the surface. You gather customers in a room and ask them questions about your product: Do you like how that feature works? Do you find it easy to login? Can you find what you’re looking for when you use the ‘Tools’ section? They provide great insights and you hit the ground running, armed with data. Right? Not so fast.
You’ve checked all the boxes: These people are customers or potential customers. They responded to the call for participants and showed up at the appointed time with smiles on their faces. They’re responding to your questions and even taking it easy on the cookies. So they seem like good folks, right? But they’re lying to you. And they can’t help it.
People are unreliable narrators. As Jakob Nielsen says, “The critical failing of user interviews is that you’re asking people to either remember past use or speculate on future use of a system. Both types of responses are extraordinarily weak and often misleading.”
Focus groups are particularly fallible in this regard. Put a group of people in a room a room together and ask them questions and an entire layer of obstacles bubbles up to the surface. And these obstacles will get in the way of these fine folks providing the information you need to improve your product.
First, there’s the group dynamic. And there is always a group dynamic, no matter how effective the facilitator, no matter how polite, respectful and well-behaved the group. There will be egos in every focus group. And there will be wallflowers and people who defer to authority and people who exert authority. There is always some level of group so don’t think you can avoid it.
Then there is the effect of memory and optimism. Memory is tricky. Studies have shown that eyewitness are unreliable and that dynamic is at play even in benign situations when people tell you what they have done (and not done) and liked (or not liked) when interacting with your product. Don’t count on people’s recollections about the past or optimism about their future endeavors with your service. Put them to the test, literally: test early and often.
Focus groups aren’t useless. They have their place in the product development toolbox. Focus groups are good for uncovering feeling and opinions. But those feelings and opinions don’t translate well against the reality of what users will do when they are interacting with a screen.
Testing uncovers patterns of use, roadblocks, and the most important for solving problems: nuance. Testing provides nuance and context that will help you figure out where and why things fail. Context gives you clues as to how to fix what’s broken.
When a team hears a finding that sounds something like this: “9 of the 10 test participants failed to find the ‘resource library link’” it’s common in a room full of stakeholders and developers to hear some gasps or even sighs followed by at least one exasperated declaration: “It’s right there! How can they not see it?”
“How” is the critical word in that moment of frustration. Testing reveals the how. Testing reveals the things that distract the user, the things that are missing from the interface. Testing uncovers the paths that people take based on the logic of the layperson rather than the developer. Product team members have been studying requirements documents and use case scenarios and wireframes and lines of code for months, maybe years. But users see the product through very different (and fresh) eyes.
Opinions are great, but use is different. Focus on use. Don’t pretend a focus group will suffice when testing is required. And it should always be required.