Observation

Once you know what you don't know, start interacting with customers and users.

Card sorting

What

A categorization exercise in which participants divide concepts into different groups based on their understanding of those concepts.

Why

To gain insights from users about how to organize content in an intuitive way.

Phase

Observation

Time required

15–30 minutes per user

How to do it

There are two types of card sorting: open and closed. Most card sorts are performed with one user at a time, but you can also do the exercise with groups of two to three people.

Open card sort

  1. Give users a collection of content represented on cards.
  2. Ask users to separate the cards into whatever categories make sense to them.
  3. Ask users to label those categories.
  4. Ask users to tell you why they grouped the cards and labeled the categories as they did.

Closed card sort

  1. Give users a collection of content represented on cards.
  2. Ask users to separate the cards into a list of categories you have predefined.
  3. Ask users to tell you why they assigned cards to the categories they did.

Example from 18F

Considerations for use in government

No PRA implications. The PRA explicitly exempts direct observation and non-standardized conversation, 5 CFR 1320.3(h)3. It also explicitly excludes tests of knowledge or aptitude, 5 CFR 1320.3(h)7, which is essentially what a card sort tests (though in our case, a poor result is our fault).

18F

Cognitive walkthrough

What

An evaluation method in which people work through a set of representative tasks and ask questions about the task as they go.

Why

To get quick and early feedback on whether a design solution is easy for a new or infrequent user to learn, and why it is or isn’t easy. This method is useful for catching big issues at any stage in the design process when you don’t have access to real users, but it is not a substitute for user evaluation.

Phase

Observation

Time required

30 minutes to one hour per person

How to do it

  1. Identify specific traits for new or infrequent users of a design solution.
  2. Develop a set of representative tasks that emphasize new use or infrequent use.
  3. Designate a member of the design team to play the role of a user. That person will use the traits you’ve identified to participate in a moderated usability testing session. (The traits can overlap.)
  4. Ask the user to accomplish their goal using a printed or interactive design. As they go, ask what they would attempt to do next or how they would learn.
  5. Don’t lead the user through the task, but encourage them to stay focused on what they’re trying to accomplish.
  6. Pay attention to expected outcomes and how quickly/easily participants are able to pick up a task.
  7. Analyze the walkthrough results to highlight where the user struggled and what needs improvement.

Additional resources

Considerations for use in government

No PRA implications. The PRA explicitly exempts direct observation and non-standardized conversation (e.g., not a survey) that a cognitive walkthrough entails, 5 CFR 1320.3(h)3.

If you are not working with government employees, you will need to observe standard precautions for archiving personally identifiable information.

18F

Contextual inquiry

What

The product team unobtrusively observes participants at work, with their permission, then asks questions.

Why

To learn how and why users do what they do; to discover needs and attitudes that might not emerge in an interview to map how tools, digital and otherwise, interact during complex activities.

Phase

Observation

Time required

1-2 hours per user

How to do it

  1. With permission from a supervisor and from the participant, schedule a time to watch a typical work activity and record data.
  2. While observing, ask the participant to act normally. Pretend you’re a student learning how to do the job. Ask questions to help you understand what the person is doing and why.
  3. At the end of the session, explain what you have learned and check for errors.
  4. Immediately after, write up your notes.

Example from 18F

A pair of 18F team members visited two Department of Labor/Wage Hour Division investigators as they interviewed home health care workers who were subject to unpaid overtime and other infractions. Since it was a sensitive subject, the 18F team did not question the health care workers directly, but instead asked the investigators clarifying questions in private. 18F staff also made sure that photos did not include faces.

Additional resources

Considerations for use in government

No PRA implications, if done properly. Contextual interviews should be non-standardized, conversational, and based on observation. The PRA explicitly exempts direct observation and non-standardized conversation, 5 CFR 1320.3(h)3. See the methods for Recruiting and Privacy for more tips on taking input from the public.

For internal folks, get permission from the right level of management. If participants could be under union agreements, contact the agency’s labor relations team.

18F

Heuristic evaluation

What

A quick way to find common, large usability problems on a website.

Why

To quickly identify common design problems that make websites hard to use without conducting more involved user research.

Phase

Observation

Time required

1–2 hours

How to do it

  1. Recruit a group of three to five people familiar with evaluation methods. These people are not necessarily designers, but are familiar with common usability best practices. They are usually not users.
  2. Ask each person to individually create a list of “heuristics” or general usability best practices. Examples of heuristics from Nielsen’s “10 Usability Heuristics for User Interface Design” include:
    1. The website should keep users informed about what is going on, through appropriate feedback within reasonable time.
    2. The system should speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms.
    3. Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue.
  3. Ask each person to evaluate the website against their list and write down possible problems.
  4. After individual evaluations, gather people to discuss what they found and prioritize potential problems.

Additional resources

Considerations for use in government

No PRA Implications, as heuristic evaluations usually include a small number of evaluators. If conducted with nine or fewer members of the public, the PRA does not apply, 5 CFR 1320.5(c)4. If participants are employees, the PRA does not apply. See the methods for Recruiting and Privacy for more tips on taking input from the public.

18F

Task flow analysis

What

A step-by-step analysis of how a user will interact with a system in order to reach a goal. This analysis is documented in a diagram that traces a user’s possible paths through sequences of tasks and decision points in pursuit of their goal. The tasks and decision points should represent steps taken by the user, as well as steps taken by the system.

Why

To validate a design team’s understanding of users’ goals, common scenarios, and tasks, and to illustrate in a solution-agnostic way the overall flow of tasks through which a user progresses to accomplish a goal. Task flow diagrams also help surface obstacles in the way of users achieving their goal.

Phase

Observation

Time required

2-3 hours per user goal

How to do it

  1. Based on user research, identify target users’ goals that need to be analyzed.
  2. For each goal, identify common scenarios and the tasks and decisions that the user or system will perform in each scenario. Don’t assume you and your stakeholders share the same understanding of the tasks. The idea is to make the flow of tasks explicit in the diagram, so that you can check your understanding by walking through the diagram with users (steps 4 & 5).
  3. Produce a diagram that includes each task and decision point that a user might encounter on their way toward their goal. While there are several diagrammatic languages that can be used to produce task flow diagrams, the basic look is a flow chart of boxes for tasks and decision points and arrows showing directionality and dependencies among tasks. The diagram should cover the common scenarios identified in step 2.
  4. Present the diagram to a subject matter expert who knows the task(s) well enough to check for accuracy.
  5. In collaboration with users and/or subject matter exprts, annotate the task flow diagram to pinpoint areas of interest, risk, or potential frustration.

Additional resources

Considerations for use in government

No PRA implications. No information is collected from members of the public.

18F