You say “potato,” I say “focus group”

August 12th, 2011 17 comments

There’s one phenomenon you really should be prepared for when you introduce the idea of usability tests in your organization…

Seriously. This really happens. All the time.

The problem is that a lot more people are familiar with focus groups than with usability tests. So sometimes no matter how often you correct them (politely and patiently, of course), they’ll still refer to your upcoming usability tests as…focus groups.

The good news is that as soon as you get them to actually come and observe a test, the difference becomes clear and the problem goes away. But until then, it can be disconcerting. And sometimes amusing.

Make sure you have your 30-second elevator pitch explanation of the difference down pat, something like

Usability tests are about watching people actually try to use what we’re building, so we can detect and fix the parts that confuse or frustrate them.

Focus groups are about having people talk about things, like their opinions about our products, their past experiences with them, or their reactions to new ideas that we show them.

So the main difference is that in usability tests, you watch people actually use things, instead of just talk about them.”

Categories: Usability
  1. August 16th, 2011 at 12:03 | #1

    Love that video. It’s funny ’cause it’s SO true.

    And it’s not just the clients who don’t get the difference. It’s the participants too. When recruiting, I explain what usability testing is all about and how it works. But when the particpants show up, guess what they’re expecting…

  2. August 22nd, 2011 at 17:01 | #2

    How TIMELY! I just gave a talk on Friday to a group of C-level executives who meet weekly for networking, education, and so forth. My talk was on the UX 5 W’s and H. I hope I don’t have to translate that. In the networking phase of the meeting before my talk, someone asked me if I was going to talk about focus groups. I was prepared, as I get this question so often! So, I asked him to please wait for the “next slide please,” which was in the WHAT

  3. August 22nd, 2011 at 17:04 | #3

    Something weird just happened and my post went up before I could finish . . . so, as I was saying, my response to the focus group question was in the WHAT part of my 5 W’s. My text slide was “It’s not your mother’s (or father’s) focus group.

    Thanks, Steve, for creating such a great video of the problem!

  4. Deaf
    August 23rd, 2011 at 15:58 | #4

    The video is not captioned – there are 37 millions of people with hearing loss in USA alone that cannot access audio. As part of usability/accessibility, would you please make any of your video/audio captioned/transcribed? Thank you.

  5. Steve Krug
    August 23rd, 2011 at 18:30 | #5

    Honestly–and I know this is very impolitic–it’s not likely to happen. It’s simply too much work. because the tools currently available are not good enough, particularly for someone like me who doesn’t use them routinely and has to figure out how to use them each time. As it is, I barely have enough time to write any blog posts.

    That said, if someone actually needs it (i.e., out of the relatively small number of people who access my site, someone who cannot hear the audio), I’d be more than happy to create a transcript for them if they email me at skrug at sensible dot com. (I know that it’s unfair to make someone ask, but all of our lives are full of tradeoffs, many less than pleasant.)

    BTW, the odds of this happening aren’t as high as you suggest. While there probably are that many people with some degree of hearing loss, the number who couldn’t access this video (by turning up the volume, for instance) is probably much lower. The Gallaudet Research Institute ( at Gallaudet University (presumably fairly objective) gathers this conclusion from the available government research:

    “Across all age groups, in the United States, approximately 1,000,000 people (0.38% of the population, or 3.8 per 1,000) over 5 years of age are “functionally deaf;” more than half are over 65 years of age. About 8,000,000 people (3.7%) over 5 years of age are hard of hearing (that is, have some difficulty hearing normal conversation even with the use of a hearing aid).”

    As near as I can tell, a term like “some hearing loss” (and the 35+ million Americans figure that gets mentioned a lot) includes everyone who has any trouble hearing (e.g., in a noisy room). I think it’s a disservice to overstate the case for accessibility, since it doesn’t need overstating. Making things accessible is doing the right thing, and everyone should be doing it on that basis. And the real key is that good tools should be made available that allow average folks to do the right thing with a reasonable amount of effort.

    Please don’t get me wrong; I don’t mean to take you to task personally at all. You’re clearly just trying to get people to do the right thing. But I fear that overstating the case only serves to make people skeptical, and give them an excuse (in their minds) not to do the right thing.

    There. I think I’ve gotten myself in more than enough trouble for one day.

  6. October 9th, 2011 at 15:38 | #6

    @Steve Krug

    I’m agree with your pragmatic view on accessibility Steve, but I feel I should pick you up on one thing…

    “That said, if someone actually needs it … I’d be more than happy to create a transcript for them if they email me…”

    That’s great and a perfectly valid approach to inclusive web publishing given the resources you have available, but until someone raised this with you there was no indication anywhere on your site that this was the case. OK, for anyone who has read your books or listened to you speak you seem like a perfectly open and approachable guy who I wouldn’t expect would turn away any reasonable request. But that’s just my interpretation of who you are and what you do.

    Perhaps a page on your site addressing your approach to accessibility, or a footer on any relevant article offering what you have here would plug a gap?

    From personal experience, the huuuge university website I work on has had a footer link on every page for 5 years essentially saying – “Let us know if you’ve had a problem; we’ll look to put it right and in the meantime get the info you need sent to you in a format that suits”. I’ve not been asked to do this once. And it’s not that the page doesn’t get viewed either – it has a few hundred page impressions a month…

  7. Whitney Quesenbery
    November 13th, 2011 at 14:52 | #7

    I’m going to jump in here and point out that, as of last night, there are captions on this video. I hadn’t seen these comments, but I did nag Steve to do it. (If you don’t see the CC button on the embedded video, click on the YouTube button, where captions will appear.)

    Steve said that it took 90 minutes, including the time to transcribe the dialog, figure out exactly what to do and check it. (I bet that’s more time than it took to write the reply above). That may seem like a lot, but consider this:

    * Once you know how to do something, it gets progressively easier.

    * Even if this video didn’t start with a script, there were probably notes to work from back when it was created.

    * It’ a lot easier than it used to be, thanks to new tools

    * If it was a routine part of posting a video, it wouldn’t seem like such a big deal

    I feel pretty strongly about both the ethical requirement for accessibility and that the things we do have all sorts of corollary benefits. But we will never reach the vision of an inclusive world as long as we think of this as an add-on instead of part of the normal process. After all, we’re usability folks who spend a lot of time and effort trying to convince product development folks that adding a morning a month (or sprint) is time worth spending on usability testing. It you think of accessibility as usability for a broader number of people (as the ISO standards do), then it becomes a natural extension of our work.

    I also feel for the “no one has asked for this” argument, until you remember a few things:

    * It’s work to ask for something to be put right. Unless you want to spend your whole life asking, it’s sometimes easier to shrug and move on. How often do you REALLY take the time to complain, meaningfully complain, about usability problems.

    * It’s easy to give up. That might mean going somewhere else. Or it might mean finding a different way to do something. Either way, when we do make things accessible, it can be disappointing not to find a crowd of people waiting to rush in. But changing habits takes time. If most videos are not captioned, maybe you don’t even bother to try it.

    * Tools keep getting better. Steve tells me that a lot of court reporting is now done with speech-to-text. And YouTube will try to automatically create both the transcription and the captions. Those technical processes will only get better with time, making it easier to do.

    * More and more, it’s the law. As I’ve followed the progress of all the lawsuits, what’s surprised me is how easily many of the problems were fixed, once everyone knuckled down to just do it. I bet all the technical costs would have fit into the coffee budgets for the armies of lawyers.

    Anyway, I’ve told Steve that I will write up user friendly instructions for how to caption a YouTube video, and maybe he’ll post them here to share with everyone. They won’t cover every single case, but hopefully they will make doing simple captions simple.

  8. LazerWonder
    November 28th, 2011 at 21:20 | #8

    Oh good grief!! Were you on my phone conversation with my manager? :) It was my FIRST very very FIRST Usability Test, and when discussing it with my manager, she insisted on (1) having more than 1 participant so it’ll cut down the number of meetings I would need and (2) that I would have a cross-representation of users. She definitely had “Focus Groups” in mind.

    I did ask users for their opinions of the new design after the main usability testing was done. So it had some “focus group” flavour because, after all, when your testers are managers, you have to ask their opinions.

    My only other problem is to get everyone on the phone to be quiet until the testing is over.

  9. June 27th, 2016 at 06:08 | #9

    I’m impressed, I have to admit. Seldom do I come across a blog that’s equally educative and engaging,
    and without a doubt, you’ve hit the nail on the head. The problem is something that too few
    people are speaking intelligently about. I’m very happy that I found this in my hunt for something concerning this.

  10. August 1st, 2016 at 21:43 | #10

    Great article.

  11. Billy
    August 30th, 2016 at 00:16 | #11


  12. 林豪
    October 17th, 2016 at 00:40 | #12


  13. reader
    January 1st, 2017 at 18:19 | #13

    @Steve Krug
    I know that the comment regarding captioning for videos is from five years ago, and the technology has likely improved since then, but I’d just like to point out that a common issue for people with hearing loss is not whether they can HEAR the audio, it’s whether they can UNDERSTAND it. Turning up the volume does not make a person’s voice CLEARER, and in fact, it often has the opposite effect. That is one reason why captioning online videos is a crucial part of making information accessible for all.

  1. May 20th, 2014 at 13:48 | #1
  2. May 12th, 2015 at 13:22 | #2
  3. October 20th, 2016 at 21:16 | #3
  4. February 17th, 2017 at 02:30 | #4

Add a comment