Blog

Archive for Usability

Human factors process, can we make this easy to understand?

90% of usability testing submitted to the FDA is unacceptable and the root cause is simply a failure to understand the human factors process.

If you submitted no usability testing to the FDA in your 510(k) submission, it would be obvious why the FDA reviewer identified usability as a major deficiency. However, you spent tens of thousands of dollars on usability testing that delayed the 510(k) submission by six months. Despite all of the time and money your company invested in the human factors process, it appears that you need to start over and repeat the entire process again. The CEO is furious, and he wants you to show him where in the 49-page FDA guidance it says that you have to do things differently.

Benefits from the human factors process

  1. Use errors result in serious injuries and death
  2. Easy to use products sell
  3. You will prevent delays in regulatory approval

Why was your rationale for no usability testing rejected?

Unlike CE Marking technical files, the FDA does not require a usability engineering file for all products. Instead, the FDA determines if usability testing is required based upon a comparison of your device’s user interface and a competitor’s user interface (i.e. predicate device user interface). If the user interface is identical, then usability testing may not be required. Instead, your company should be able to write a rationale for not doing usability testing based upon equivalence with the predicate device. If there are differences in your user interface, you will need to provide use-related risk analysis (URRA), identify critical tasks, implement risk controls, and provide verification testing to demonstrate the effectiveness of the risk controls. Even if your device is “easier to use” or “simpler”, you still need to provide the documentation to support this claim in your submission. The FDA also does not allow comparative claims in your marketing for 510(k) cleared devices. Comparative claims require the support of clinical data.

What is the 10-step human factors process?

  1. Define human factors for your device or IVD
  2. Identify use errors
  3. Conduct a URRA
  4. Perform a critical task analysis
  5. Conduct a risk control option analysis
  6. Conduct formative usability testing
  7. Implement risk controls
  8. Conduct summative usability testing
  9. Prepare HFE/UE documentation
  10. Collect post-market surveillance data specific to use errors

There is a YouTube video describing these 10 steps at the bottom of this blog posting.

Why is formative testing needed?

  • Observational study to identify unforeseen use errors
  • Observational study to evaluate risk control options
  • What are the other types of studies?
  • Development of indications for use
  • Development of training materials

Why is the human factors process crazy expensive to outsource?

  • Human factors consultants need time to learn about your device
  • Consultants are more conservative because they cannot afford to fail
  • Justifying your choice of risk controls is difficult because you started too late
  • Your instructions for use (IFU) are inadequate
  • Consultants need to explain the human factors process to you
  • Recruiting subjects is marketing (which may not be their expertise)
  • You are paying for infrastructure (specialized testing facilities)
  • This is a team effort that requires many consulting hours collectively

Why was your Usability Engineering File refused?

  1. Your company provided an application failure modes and effects analysis (aFMEA) to support your justification that residual risks are acceptable. The FDA guidance suggests using risk analysis tools such as an FMEA or fault-tree analysis, but deficiency letters from FDA reviewers recommend a use-related risk analysis (URRA) format that is totally different.

    URRA table example from the FDA 1024x399 Human factors process, can we make this easy to understand?

    Example of a URRA Table provided by the FDA for the Human Factors Process

    The primary problem with using an FMEA or Fault-Tree risk analysis tool is that these tools involve estimation of the severity of harm and the probability of occurrence of harm, while the FDA does not feel it is appropriate to estimate the probability of occurrence of harm. Instead, the FDA instructs companies to assume that use errors will occur and to implement risk controls to mitigate those risks (see URRA example above). Although “mitigation” is unlikely, and use risks will only be reduced, this is the approach the FDA wants companies to use. In addition, the FDA expects your company to provide traceability of risk control implementation to each use-related risk you identified and the FDA expects documentation of verification testing (i.e. usability testing) that shows your risk controls are effective. Finally, the FDA (and ISO 14971, Clause 10) expects you to collect and perform a trend analysis of use errors. Any use errors that are reported should be evaluated for the need to implement additional corrective actions to prevent future use errors. Blaming “user error” is not an acceptable approach. 

  2. You provided risk analysis and human factors testing in your 510(k) submission, but the FDA reviewer said you need to identify critical tasks and provide traceability to each critical task in your summative validation report. – Critical tasks are specifically mentioned in section 3.2 of the FDA guidance on applying human factors and usability engineering–and a total of 49 times throughout the guidance. However, “critical tasks” are not mentioned even once in ISO 14971:2019 or ISO/TR 24971:2020. The term “critical tasks” is not even found in IEC 62366-1:2015. There is mention of “tasks”, and “task” is a formal definition (i.e. Definition 3.14, “Task – one or more USER interactions with a MEDICAL DEVICE to achieve a desired result”). Therefore, companies that are familiar with the ISO Standards and CE Marking process frequently need training on the FDA requirements for the human factors process. After receiving training, then your company will be prepared to modify your usability engineering file documentation to comply with the FDA requirements for human factors.
  3. You completed a summative validation protocol, but the FDA disagrees with your definition of user groups. – Each user has a different level of experience, training, and competency. Therefore, if you define the intended user population too broadly (e.g. healthcare practitioners), the FDA may not accept your summative usability testing. This is the reason that the human factors process begins with defining the human factors for your IVD or device. Radiologists, for example, have the following training pathway:
    • graduate from medical school;
    • complete an internship;
    • pass state licensing exam;
    • complete a residency in radiology;
    • become board certified; and
    • complete an optional fellowship.

Therefore, if you are developing imaging software, you need to make sure your user group includes radiologists that cover the entire range of competencies. In addition, most radiology images are taken by radiology technicians and then reviewed by the radiologist. Therefore, radiology technicians should be considered a completely different user group due to the differences in experience, training, and competency when compared to a radiologist. This simple example doubles the number of users needed because you have two user groups instead of one.

  1. You evaluated 15 users, but the FDA reviewer is asking you to evaluate a larger number of users based upon a special controls guidance document. – The FDA guidance on human factors testing specifies a minimum of 15 users for each user group–not a minimum of 15 users. Therefore, for a device that is for Rx-only and OTC use, you will have at least two user groups that need to be evaluated independently. In addition, some devices have special controls guidance documents that specify usability testing requirements. For example, an OTC blood glucose meter must pass a 350-person lay-user study. Covid-19 self-tests are expected to pass a 30-person lay-user study as another example.
  2. Your usability study was conducted in Australia, but the FDA insists that your usability study must be repeated in the USA. – Most people think of language being the primary difference between two countries, and therefore the author of a study protocol may not perceive any difference between the USA and Australia, Ireland, Canada, or the UK. However, this lack of ability to identify differences between cultural norms shows our own ignorance of cultural differences. International travelers learn quickly about the differences in the interface used for electrical outlets between the USA and other countries. There are also more subtle differences between cultures, such as in which direction do you toggle a light switch to turn on a light, up or down? For devices that are used in a hospital environment, it is critical to understand how your device will interact with other devices and how different hospital protocols might impact human factors.
  3. The FDA reviewer indicated that your usability engineering file does not assess the ability of laypersons to self-select whether your OTC device is appropriate for them. – Devices and IVD devices may have contraindications or indications for use that are specific to an intended patient population or intended user population. In these cases, the user of the device or IVD needs to be able to “self-select” as included or excluded from use. The ability to self-select should be assessed as part of any OTC usability study. The ability to identify suitable and unsuitable patients for treatment is also a common criterion for a usability study involving prescription devices where a physician is the subject of the study.
  4. The FDA reviewer indicated that you did not provide raw data collected by the study moderator. – Data collected during a human factors study is usually subjective in nature, and the FDA may want to conduct their own review and analysis of your data. Therefore, you cannot provide only a testing report that summarizes the results of your study. You must also provide the raw data for the study. It is permitted to provide the data in a tabular format that has been transcribed from paper case report forms or was recorded electronically. You should also consider scanning any paper forms for permanent retention or retaining the paper forms in case there is any question of accuracy in the transcription of the data collected. Finally, it is best practice to record videos of the study participants performing each task and answering interview questions. This will help in filling any gaps in the notes recorded by the moderator, and the recording provides additional objective evidence of the study results.
  5. The FDA reviewer indicated that your study is not valid, because the training provided by moderators was not scripted and training decay was not considered in the design of the study. – Summative usability testing requires that users complete all of the critical tasks identified in your critical task analysis without assistance. It is permitted to provide training to the user prior to conducting the study if the device or IVD is for prescription use and healthcare practitioners are responsible for providing instruction to the user. However, any training provided must be scripted in advance and approved as part of the summative usability testing protocol. This ensures that every subject in the study receives consistent training. Unfortunately, the FDA may still not be satisfied with the design of your study if you do not allow sufficient time to pass between the time that training is provided to the user and when the subject uses the device or IVD for the first time. In general, one hour is the minimum amount of time that should pass between providing user training and when the device or IVD is used for the first time. This is referred to as “training decay” and the duration of time between your scripted training and the user performing critical tasks for the first time should be specified in your summative usability protocol. One solution to address both issues is to provide a video of the instructions to each subject 24-hours in advance of participation in the study.

Additional resources for the human factors process and usability testing

Posted in: 510(k), Design Control, Usability

Leave a Comment (3) →

Integrating usability testing into your design process

This article explains how you should be integrating usability testing into your design control process–especially formative usability testing.

Integrating Usability Engineering and Risk Management into your Design Control Process Integrating usability testing into your design process

Why you should be integrating usability testing into the design

We recently recorded an updated usability webinar and released a usability procedure (SYS-048) with help from Research Collective–a firm specializing in human factors testing. After listening carefully to the webinar, and reading through the new usability procedure, I felt we needed to update our combined design/risk management plan to specify formative testing during phase 3 and summative (validation) testing during phase 4 of the design process. This is necessary to ensure your usability testing is interwoven with your risk management process. Integrating usability testing into all phases of your design process is critical–especially design planning (phase 1), feasibility (phase 2), and development (phase 3).

Integrating usability testing into your design plan helps identify issues earlier

During the usability training webinar, Research Collective provided a diagram showing the various steps in the usability engineering process. The first five steps should be included in Phases 1 and 2 of your design process. Phase 1 of the design process is planning. In that phase, you should identify all of the usability engineering tasks that need to be performed during the design process and estimate when each activity will be performed. The first of these usability activities is the identification of usability factors related to your device. Identifying usability factors is performed during Phase 2 of your design process before hazard identification.

Indentifying Usability Issues 300x209 Integrating usability testing into your design process

Before performing hazard identification, which should include identifying potential use errors, you need to identify five key usability elements associated with your device:

  1. prospective device users during all stages of use must be defined
  2. use environments must be identified
  3. user interfaces must be identified
  4. known use errors with similar devices and previous generations of your device must be researched
  5. critical tasks must be described in detail and analyzed for potential use errors

Defining users must include the following characteristics: physical condition, education, literacy, dexterity, experience, etc. Use environment considerations may consist of low lighting, extreme temperatures or humidity, or excessive uncontrolled motion (e.g., ambulatory devices). User interfaces may include keyboards, knobs, buttons, switches, remote controllers, or even a touch screen display.

Often the best reason for developing a new device is to address an everyday use error that is inherent to the design of your current device model or a competitor’s product. Therefore, a thorough review of adverse event databases and literature searches for potential use errors is an important task to perform before hazard identification. This review of adverse event data and literature searches of clinical literature are key elements of performing post-market surveillance, and now ISO 13485:2016 requires that post-market surveillance shall be an input to your design process.

Finally, the step-by-step process of using your device should be analyzed carefully to identify each critical user task. User tasks are defined as “critical” for “a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care.” Not every task is critical, all critical tasks must be identified, and ultimately you need to verify that each critical task is performed correctly during your summative (validation) usability testing.

Evaluating Risk Control Options – Formative Usability Testing in Phase 3 (Development)

Once your design team has conducted hazard identification and identified your design inputs (i.e., design phase 2), you will begin to evaluate risks and compare various risk control options. Risk control option analysis requires testing multiple prototype versions to assess which design has the optimum benefit/risk ratio. This is an iterative process that involves screening tests. For any use risks you identify, formative usability testing should be performed. Sometimes the risk controls you implement will create new use errors or new risks of other types. In this case, you must compare the risks before implementing a risk control with risks created by the risk control.

Formative Usability Testing Process 220x300 Integrating usability testing into your design process

Ideally, each design iteration will reduce the risks further until all risks have been eliminated. The international risk management standard (ISO 14971) states that risks shall be reduced as low as reasonably practicable (ALARP). However, the European medical devices regulations require risks to be reduced as far as possible, considering the state-of-the-art. For example, all small-bore connectors in the USA are now required to have unique connectors that are incompatible with IV tubing Luer lock connections to prevent potential use errors. That requirement is considered “state-of-the-art.” If your device is marketed in both the USA and Europe, you will need to reduce errors as far as possible–before writing warnings and precautions in your instructions for use.

Reaching the point where use errors cannot be reduced any further may require many design iterations, and each iteration should be subsequently evaluated with formative usability testing. Formative testing can be performed with prototypes, rather than production equivalents, but the formative testing conditions should also address factors such as the use environment and users with different levels of education and/or experience. Ultimately, if the formative testing is done well, summative (validation) testing will be a formality.

Risk Control Effectiveness During Phase 4 – Summative Usability Testing during Verification

Once your team freezes the design, you will need to conduct verification testing. This includes integrating usability testing into the verification testing process. Summative (validation) testing must be performed once your design is “frozen.” If you are developing an electrical medical device, then you will need to provide evidence of usability testing as part of your documentation for submission to an electrical safety testing lab for IEC 60601-1 testing. There is a collateral standard for usability (i.e., IEC 60601-1-6). For software as a medical device (SaMD), you will also be expected to conduct usability testing to demonstrate that the user interface does not create any user errors.

Summative Usability Testing Process 174x300 Integrating usability testing into your design process

When you conduct summative (validation) testing, it is critical to make sure that you are using samples that are production equivalents rather than prototypes. Also, it is crucial to have your instructions for use (IFU) finalized. Any residual risks for use errors should be identified in the precautions section of your IFU, and the use of video is encouraged as a training aid to ensure use errors are identified, and the user understands any potential harm. When the summative testing is performed, there should be no deviations and no use errors. Inadequate identification of usability factors during Phase 2, or inadequate formative testing during Phase 3, is usually the root cause of failed summative testing. If your team prepared sufficiently in Phase 2 and 3, the Phase 4 results would be unsurprisingly successful.  

Additional Training Resources for Usability Engineering

The following additional training resources for usability engineering may be helpful to you:

Posted in: Design Control, Usability

Leave a Comment (1) →