Down and Dirty Usability Testing

By Cathleen Zapata

Unless you’re testing the experience of mud wrestling or monster trucking, a down and dirty test isn’t really physically dirty at all. I’m going to walk you through an easy and very cheap way to get feedback on your latest and greatest design. It’s so easy that if you’re reading this at 8 a.m., it’s possible to have your feedback by lunchtime for the cost of your lunch – I know because I’ve done it. Now that’s fast and cheap.

Is this the best way? Probably not, but if you’re new to usability or are looking for some really quick feedback or your first plunge into this whole usability arena, this is a sure way to get you started. Bottom line, “down and dirty” usability testing is better than no usability testing at all. Now let’s get started.

1. Decide on exactly what you want to test

Maybe you’re looking for some feedback on the latest website you’ve created, a banner ad, a print ad or a painting you created for a very important client. Regardless of what your creative endeavor is, you need to figure out what you want to find out about the design.

Do you want to test the navigation? The appearance? The message? Write down a list of what you were trying to achieve with this project. This will serve as a guideline for what you need to test.

For example, let’s say you designed a website for ABC Company that sells widgets. ABC Company wanted you to design a site for the primary purpose of allowing their consumers to buy these widgets online at their site. Secondly, they want their customers to know about their other products at the time of purchase too to improve their cross-selling efforts.

So, you designed the site with these goals in mind. Based on these goals, you will want to test 1) the experience of users purchasing widgets at this site and 2) the awareness and reaction of the cross-sell products at the point of sale.

2. Develop a questionnaire

After you decide what you want to test, you’ll need to put together a questionnaire. You can either have participants complete this questionnaire on their own, or you can read them the questions and write down their responses yourself.

Your questionnaire should set expectations for the user. Using the above example, you could start out with, “You are going to experience making an online purchase at a website today” to set the expectations without giving too much away.

Your questionnaire should state specific tasks that you want the participants to do. For example:

a. Go to ABC Company’s website and find widget A.

b. After you find widget A, purchase this widget online at this site.

Besides the tasks, you will also have specific questions based on what you’re trying to measure. For example, you probably wouldn’t have a specific task for viewing the cross-sell product, but rather ask a question about it in your questionnaire like, “During the purchasing process, did you notice information about any other products that ABC Company sells?”, “Where?”, “Which products?”, “How did you feel about that?”

If having cross-sell products during check-out is a primary goal for this website for ABC Company in order to generate more revenue for the company, by asking these questions you’ll be making sure that users noticed it in the location where you put it and that you put the best message out there.

When you develop your questionnaire, keep in mind the following key concepts:

c. Make sure your questions are not leading. Ask questions like, “How do you feel about that?”, “Why do you feel that way?”, “What do you like the best about (that)?”, “The least?” Do not lead the participant into the answer you want them to give. Questions like “Do you think the message is too salesy?” may get the user thinking about the message being salesy even if they didn’t think of it that way before. You want the user to tell YOU that they think it’s salesy, without leading them to that thought.

d. It’s okay to have multiple choice answers or have the user rate a response on a scale (usually 1-5). If you’re asking yes or no questions, it’s a good idea to include a “why?” after it.

e. At the end, make sure to include an “anything else?” question to gather additional feedback about the experience. Sometimes people will say new things that will add value to the study here.

3. Run your questionnaire by someone – Anyone

The purpose of running your questionnaire by “anyone” is to check for a couple of things. One, it will give you an idea of how long it will take your participants to complete it so you can set expectations when recruiting. Two, it will allow you to see what questions cause confusion so you can revise them accordingly. And three, it will give you a dry run so you’re familiar with the questionnaire yourself.

4. Find some people

Okay, depending on who’s available to you, this can’t always be done in a half days worth of work. You’re going to want to find people that meet your target demographic as closely as possibly, and who have the experience you need. But since this is “down and dirty” you may not find the perfect mix. Just make sure your basics are covered – that you have a somewhat mix of people that would possibly do what your target audience would do and who have the experience (e.g. Web experience) needed to perform the tasks.

Find 5-8 people to participate in your test. These people may be family, friends, acquaintances, co-workers from outside departments, people who know people you know – anyone in your network that you feel comfortable contacting. In a formal test, you’d want to be more careful on the screening process and perhaps recruit an outside agency to find participants for you. But for fast results without an existing participant database, utilizing your personal network is your best bet.

Once you have a list of contacts, start calling and recruiting. Let them know what will be happening and what the compensation is (if any). Compensation doesn’t always have to be money – it could be a gift certificate, free product, or some type of give-away.

Set a time and place to meet. If you don’t have a designated usability lab, you can perform the test in your office, a meeting room, or at their home/your home if a computer is available if even needed. Make sure you give directions and if you’re not synching up that day or the next, call to remind/confirm to avoid cancellations and no-shows which are very common.

5. Run through it with them, or let them go through it themselves

Right off the bat, let them know that this is not a test. Make sure they’re comfortable. Offer them something to drink or some snacks. Ask them to “talk out loud” as much as possible so you understand what they’re thinking as they go through each task. Tell them again that this is not a test – you’re just looking for their honest opinions and feedback. Do NOT tell them that you designed the site or they’re less likely to be honest with you about their feelings; not to purposely be dishonest, but this is just basic human behavior. We don’t like to insult people to their face (just behind their backs ?).

Write down or type their responses and video tape them if possible. If video-tapping, they’ll need to sign a consent form. And you may want to have your participants sign a non-disclosure agreement if they’ll be viewing sensitive or proprietary information.

Once the session is complete, thank the participant for their time. If there’s compensation, give it to them at this time.

6. Gather the results and wrap it up

Compile the results so you understand how each participant responded to each of the questions and tasks. Review the questions and tasks one by one and look for opportunities for improvements. Make the changes and re-test if time allows. Keep the results for your records.

When should I test?

The answer is as early and often as possible. The sooner you obtain consumer feedback, the less time you’ll spend re-doing time-consuming efforts. Sometimes it’s best to just layout what you want to do on paper and run it by some people before you even begin. This may save you considerable time and money in the long run. The more testing and the sooner you involve testing, the better the end result. And you don’t even have to get dirty.

To read more on the basics of usability testing, I recommend two great reads: A Practical Guide to Usability Testing by Joseph S. Dumas and Janice C. Redish and Observing the User Experience by Mike Kuniavsky.

----------------------------------------------------------------------------------