Formative usability testing – Frequently Asked Questions?

Formative usability testing is not a regulatory requirement, but it is necessary if you want to successfully develop medical devices.

What is the difference between formative and summative usability testing?

“Formative” tests are any usability tests that you perform during the development process, while “summative” testing is the final usability testing you perform to validate that your chosen user interface is effective. Many design teams perform formative testing of one kind or another without even realizing that is what they are doing. Unfortunately, design teams often forget to document the testing they performed during prototyping and product development. Formative usability testing probably always existed as part of product development, but not everyone recognizes the term and identifies the work they have done as “formative.” The most important reason for documenting formative usability testing is to identify which user interface designs failed and why so that future design teams can learn from your failures.

Why don’t more companies do usability testing?

Everyone likes to believe they can skip steps in the learning process, but some lessons can only be learned the hard way. When a medical device design team is developing a user interface for a new product, they need to learn which designs will fail and why before they can fully understand how to design the best user interface for the device. Therefore, most design and development teams will select a user interface that they are familiar with or they see used by a competitor product. The team will not always test the proposed design solution, because they have no reason to believe that the chosen interface will fail. Unfortunately, this can lead to failure later in the design process. Then the team will need to backtrack and repeat the evaluation of various interface designs.

What is the best approach?

“Fail small and fail fast” is the best advice for anyone performing formative usability testing. Instead of writing a lengthy protocol and recruiting 10 subjects to evaluate your proposed user interface, you might consider building a couple of different prototypes and asking two or three people which prototype they prefer and why? Another simple question is, “Tell me what you think of this design?” Iterative formative testing over time with different users is better than one single testing session with a lot of users. It is also better to start collecting formative usability testing data as early in the development process as possible. Gathering data earlier in the process will ensure that users direct the development of your device instead of the design team developing a new device in a direction that is not preferred by users.

When during the design process should formative testing be planned?

Formative testing should be planned during the development phase of the design process. During this phase, medical device manufacturers evaluate multiple design solutions as risk controls for their devices. Use-related risks should be included in this, and the formative usability testing is intended to identify which user interface will do the best job of eliminating the use errors. It is important to evaluate these potential user interfaces and to verify that there are no use errors that the design team overlooked during this phase of the design. This is also the phase of design when the instructions for use are developed and user training is developed. All of this formative usability testing should be completed prior to your design freeze and the start of the verification and validation testing.

What are the different types of formative testing?

Formative usability testing can be used as a pilot for your summative usability testing protocol prior to scheduling the final testing. However, there are many other types of formative testing. The most common reason for doing this testing is to identify any potential use errors that were not originally identified in your user-related risk analysis (URRA). Another type of testing is to simulate use of the device to make sure that every user task is identified in the instructions for use. Finally, design teams will conduct formative usability testing to develop training materials for training new users on how to properly use your medical device.

Which types of formative tests are the most useful?

Use-related risks are difficult to identify unless you conduct simulated use testing with your device. Therefore, you need to get your device in the hands of your intended users, in the intended use environment, and ask them to simulate the use of the device. It is not critical to evaluate a specific number of users. Two or three users might be enough, but simulated use by intended users in the intended use environment is essential to give you the information you need regarding potential use errors. It is also important to avoid “leading” the users. Instead of asking users to perform a specific task, ask users to show you how they would use the device. Ask them what they like about the device, and ask them what they don’t like about the device. Ask users what they think about the device, and ask them how it compares to other devices they are already using.

Who should you recruit for your formative usability testing?

You should start your human factors process by defining the intended user of your device and by defining if there is more than one user group. You then should recruit subjects that are within this user group(s). You can use employees or friends to help you with initial feedback about the usability of your device’s user interface. However, what seems intuitive to one person may be the opposite for other people with different experiences. Even the sequence of steps in which users perform the same tasks can impact usability. Therefore, be very cautious about relying upon data collected only from subjects that are outside your intended user group. Most companies disregard this advice because they are unsure of how to recruit their intended users. However, if your company has difficulty identifying intended users for testing, you will also have difficulty marketing and selling your device later. This struggle may be an indicator that you need to involve marketing and salespeople that can get your prototypes in the hands of the intended users.

How should you document formative studies?

When you are performing summative usability testing you already know exactly what your use-related risks are and you have a list of critical tasks that you are trying to verify users can perform without use errors. Because these tasks are clearly defined, it is easier to write a protocol and it is easier to design data collection forms for study moderators to use. In contrast, when you are conducting formative usability testing you are trying to identify use errors that you are not already aware of. Therefore, it is much harder to write a detailed protocol and design a data collection form. For this reason, it is critical to capture the data with video recordings. This is a safety measure you are taking to ensure that you will not miss valuable use errors or use tasks that you had not already identified. The use of video to record data allows the moderator to focus on observation and interviewing users with open-ended questions. This will generate the most value for your design team during the development process.

Where is testing performed?

While the design team is developing the list of design inputs for your new device, the team must create a definition for the intended users and the intended use environment. The formative usability testing and summative testing should be conducted in the intended use environment or you will need to simulate that use environment. If you are struggling to figure out how to simulate the intended use environment, you should systematically identify the characteristics of the intended use environment. These characteristics include temperature, humidity, ambient noise, other equipment that is present, the number of people present, and the dimensions of the space. If you have a room available with temperature and humidity control, you can add ambient noise by recording the intended use environment. You can rent equipment, or you can place objects of the same size in the space. You can also identify the workspace restrictions by taping the floor to establish boundaries for the simulation. By adding these characteristics to a simulated environment, you open the possibilities for additional places that can be used for formative usability testing.

What will happen if you skip formative testing?

If you skip formative usability testing, you will increase the possibility of failing your summative usability testing. If this happens, then your summative testing becomes your formative usability testing. After you fail, you will need to revise your testing protocol and repeat the study. Another possibility is that you will fail to identify a potential use error. If the FDA identifies this use error you will need to repeat your testing. If the use error is never identified, then you may end up with complaints or medical device reporting of use errors. In extreme cases, this could result in serious injuries or death.