Blog

Formative usability testing – Frequently Asked Questions?

Formative usability testing is not a regulatory requirement, but it is necessary if you want to successfully develop medical devices.

Formative Usability Testing FAQs 1024x169 Formative usability testing   Frequently Asked Questions?

What is the difference between formative and summative usability testing?

“Formative” tests are any usability tests that you perform during the development process, while “summative” testing is the final usability testing you perform to validate that your chosen user interface is effective. Many design teams perform formative testing of one kind or another without even realizing that is what they are doing. Unfortunately, design teams often forget to document the testing they performed during prototyping and product development. Formative usability testing probably always existed as part of product development, but not everyone recognizes the term and identifies the work they have done as “formative.” The most important reason for documenting formative usability testing is to identify which user interface designs failed and why so that future design teams can learn from your failures.

Why don’t more companies do usability testing?

Everyone likes to believe they can skip steps in the learning process, but some lessons can only be learned the hard way. When a medical device design team is developing a user interface for a new product, they need to learn which designs will fail and why before they can fully understand how to design the best user interface for the device. Therefore, most design and development teams will select a user interface that they are familiar with or they see used by a competitor product. The team will not always test the proposed design solution, because they have no reason to believe that the chosen interface will fail. Unfortunately, this can lead to failure later in the design process. Then the team will need to backtrack and repeat the evaluation of various interface designs.

What is the best approach?

“Fail small and fail fast” is the best advice for anyone performing formative usability testing. Instead of writing a lengthy protocol and recruiting 10 subjects to evaluate your proposed user interface, you might consider building a couple of different prototypes and asking two or three people which prototype they prefer and why? Another simple question is, “Tell me what you think of this design?” Iterative formative testing over time with different users is better than one single testing session with a lot of users. It is also better to start collecting formative usability testing data as early in the development process as possible. Gathering data earlier in the process will ensure that users direct the development of your device instead of the design team developing a new device in a direction that is not preferred by users.

When during the design process should formative testing be planned?

Formative testing should be planned during the development phase of the design process. During this phase, medical device manufacturers evaluate multiple design solutions as risk controls for their devices. Use-related risks should be included in this, and the formative usability testing is intended to identify which user interface will do the best job of eliminating the use errors. It is important to evaluate these potential user interfaces and to verify that there are no use errors that the design team overlooked during this phase of the design. This is also the phase of design when the instructions for use are developed and user training is developed. All of this formative usability testing should be completed prior to your design freeze and the start of the verification and validation testing.

What are the different types of formative testing?

Formative usability testing can be used as a pilot for your summative usability testing protocol prior to scheduling the final testing. However, there are many other types of formative testing. The most common reason for doing this testing is to identify any potential use errors that were not originally identified in your user-related risk analysis (URRA). Another type of testing is to simulate use of the device to make sure that every user task is identified in the instructions for use. Finally, design teams will conduct formative usability testing to develop training materials for training new users on how to properly use your medical device.

Which types of formative tests are the most useful?

Use-related risks are difficult to identify unless you conduct simulated use testing with your device. Therefore, you need to get your device in the hands of your intended users, in the intended use environment, and ask them to simulate the use of the device. It is not critical to evaluate a specific number of users. Two or three users might be enough, but simulated use by intended users in the intended use environment is essential to give you the information you need regarding potential use errors. It is also important to avoid “leading” the users. Instead of asking users to perform a specific task, ask users to show you how they would use the device. Ask them what they like about the device, and ask them what they don’t like about the device. Ask users what they think about the device, and ask them how it compares to other devices they are already using.

Who should you recruit for your formative usability testing?

You should start your human factors process by defining the intended user of your device and by defining if there is more than one user group. You then should recruit subjects that are within this user group(s). You can use employees or friends to help you with initial feedback about the usability of your device’s user interface. However, what seems intuitive to one person may be the opposite for other people with different experiences. Even the sequence of steps in which users perform the same tasks can impact usability. Therefore, be very cautious about relying upon data collected only from subjects that are outside your intended user group. Most companies disregard this advice because they are unsure of how to recruit their intended users. However, if your company has difficulty identifying intended users for testing, you will also have difficulty marketing and selling your device later. This struggle may be an indicator that you need to involve marketing and salespeople that can get your prototypes in the hands of the intended users.

How should you document formative studies?

When you are performing summative usability testing you already know exactly what your use-related risks are and you have a list of critical tasks that you are trying to verify users can perform without use errors. Because these tasks are clearly defined, it is easier to write a protocol and it is easier to design data collection forms for study moderators to use. In contrast, when you are conducting formative usability testing you are trying to identify use errors that you are not already aware of. Therefore, it is much harder to write a detailed protocol and design a data collection form. For this reason, it is critical to capture the data with video recordings. This is a safety measure you are taking to ensure that you will not miss valuable use errors or use tasks that you had not already identified. The use of video to record data allows the moderator to focus on observation and interviewing users with open-ended questions. This will generate the most value for your design team during the development process.

Where is testing performed?

While the design team is developing the list of design inputs for your new device, the team must create a definition for the intended users and the intended use environment. The formative usability testing and summative testing should be conducted in the intended use environment or you will need to simulate that use environment. If you are struggling to figure out how to simulate the intended use environment, you should systematically identify the characteristics of the intended use environment. These characteristics include temperature, humidity, ambient noise, other equipment that is present, the number of people present, and the dimensions of the space. If you have a room available with temperature and humidity control, you can add ambient noise by recording the intended use environment. You can rent equipment, or you can place objects of the same size in the space. You can also identify the workspace restrictions by taping the floor to establish boundaries for the simulation. By adding these characteristics to a simulated environment, you open the possibilities for additional places that can be used for formative usability testing.

What will happen if you skip formative testing?

If you skip formative usability testing, you will increase the possibility of failing your summative usability testing. If this happens, then your summative testing becomes your formative usability testing. After you fail, you will need to revise your testing protocol and repeat the study. Another possibility is that you will fail to identify a potential use error. If the FDA identifies this use error you will need to repeat your testing. If the use error is never identified, then you may end up with complaints or medical device reporting of use errors. In extreme cases, this could result in serious injuries or death.

Posted in: Uncategorized

Leave a Comment (0) →

Is monitoring every procedure required?

Process monitoring is required but do you know whether monitoring every procedure is required by the FDA QSR or ISO 13485?

One of the elements that Medical Device Academy has incorporated into each procedure we created in our turnkey quality system is a section titled, “monitoring and measurement.” The purpose of this section is to force each process owner to identify a process metric for monitoring every procedure. In some cases, we suggest a metric that would be appropriate for most companies establishing a new quality system. In other procedures, we use the following default text:

Enter a quality metric that you want to track for this process in accordance with ISO 13485:2016, Clause 8.2.5 and the procedure for Monitoring, Measurement, and Analysis (SYS-017).

Where are the requirements for process monitoring in 21 CFR 820?

Some of the companies that have purchased our turnkey quality system have asked, “Is it required to monitor and measure something in every procedure?” In general, it is not a specific requirement to have a metric specified in each procedure. In fact, if your quality system is not ISO 13485 certified, there are actually only a few places where the US FDA requires monitoring. The FDA does not have a section specific to monitoring and measurement of processes, but there is a section of the regulations specific to statistical techniques (i.e. 21 CFR 820.250). However, it does not state in the QSR that statistical analysis is required for all processes. In fact, there are only six instances where the word “statistical” is used:

  • 21 CFR 820.100(a)(1) – “Analyzing processes, work operations, concessions, quality audit reports, quality records, service records, complaints, returned product, and other sources of quality data to identify existing and potential causes of nonconforming product, or other quality problems. Appropriate statistical methodology shall be employed where necessary to detect recurring quality problems;”
  • 21 CFR 820.200(b) – “Each manufacturer shall analyze service reports with appropriate statistical methodology in accordance with § 820.100.”
  • 21 CFR 820.250 – “(a) Where appropriate, each manufacturer shall establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics. (b) Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.” Note: the other two instances are the title of 21 CFR 820.250.

The word “monitoring” is equally rare (i.e. 4x) in the QSR:

  • 21 CFR 820.70(a) – “Each manufacturer shall develop, conduct, control, and monitor production processes to ensure that a device conforms to its specifications…Where process controls are needed…(2) Monitoring and control of process parameters and component and device characteristics during production.”
  • 21 CFR 820.75(b) – “Each manufacturer shall establish and maintain procedures for monitoring and control of process parameters for validated processes to ensure that the specified requirements continue to be met…(2) For validated processes, the monitoring and control methods and data, the date performed, and, where appropriate, the individual(s) performing the process or the major equipment used shall be documented.”

Where are the requirements for process monitoring in ISO 13485:2016?

ISO 13485:2016 has a section specific to monitoring and measurement of processes (i.e. Clause 8.2.5). In addition, the word “monitoring” occurs 52 times in the standard and there are 60 incidents of some variant or the exact word. , but there is a section of the regulations specific to statistical techniques (i.e. 21 CFR 820.250). However, it does not state in the QSR that statistical analysis is required for all processes. In fact, there are only six instances where the word “statistical” is used. There are four Clause headings that actually include the word monitoring:

  • Clause 7.6, Control of monitoring and measuring equipment
  • Clause 8.2, Monitoring and measurement
  • Clause 8.2.5, Monitoring and measurement of processes
  • Clause 8.2.6, Monitoring and measurement of product

In Clause 1, Scope, and Clause 4.1.5, the Standard states that any outsourced processes remain the responsibility of the company and must be accounted for in the quality system by monitoring, maintaining, and controlling the processes.

Monitoring of risk is included in the definition of “risk management” in the Standard (i.e. Clause 3.18).

Clause 4.1.3 states that the organization shall, “b) ensure the availability of resources and information necessary to support the operation and monitoring of these processes…d) monitor, measure as appropriate, and analyze these processes.”

Clause 4.2.3 states that the contents of the Medical Device File (i.e. MDR or TF), shall include, “d) procedures for measuring and monitoring.”

Monitoring and measurement of processes and product are required inputs to the Management Review in Clauses 5.6.2e) and f).

Clause 6.4.1 requires a procedure for monitoring the work environment if it can have an effect on product quality.

Clause 7.1 requires the company to consider including monitoring in product realization planning.

Clause 7.4.1 requires a plan for monitoring suppliers.

Clause 7.5.1 requires monitoring production and service, including the monitoring of process parameters and product characteristics.

Clause 7.5.6 requires monitoring of validated process parameters.

Clause 7.5.8 requires identification of status with regard to product monitoring and measurement (i.e. inspection status).

Clause 7.6 requires monitoring and measurement of calibrated devices and validation of any computer software used to monitor calibrated devices.

Clause 8.1 states that companies shall plan and implement monitoring and measurement of processes.

Clause 8.2 is titled, “Monitoring and measurement.”

Clause 8.2.1 requires monitoring of customer feedback.

Clause 8.2.5 requires monitoring of processes to ensure planned results are achieved.

Clause 8.2.6 requires monitoring of products to ensure product requirements have been met.

Clause 8.4 requires data analysis of monitoring data from at least six different processes:

  1. Feedback
  2. Conformity to product requirements
  3. Characteristics and trends of processes and products, including opportunities for improvement
  4. Suppliers
  5. Audits
  6. Service reports, as appropriate

In summary, while not every single clause that requires a procedure includes a requirement for monitoring, there are a number of processes where the requirement to monitor the process is explicitly stated.

Why do all of our procedures include the requirement for metrics?

Medical Device Academy expanded the requirement for monitoring to all procedures for five reasons:

  1. Quality objectives must be “established at relevant functions and levels within the organization.” Therefore, establishing monitoring requirements for each procedure ensures that top management has metrics for every process and a lack of data is never an excuse for not establishing a new quality objective when improvement is needed.
  2. If every procedure has a requirement for monitoring, then employees don’t have to remember which processes require monitoring and which processes do not explicitly require monitoring.
  3. The process approach to auditing includes metrics of the process as one of the seven items that are included in every process turtle diagram, and therefore, including metrics for each procedure facilitates the process approach to auditing.
  4. If a company does not have a process metric already established, it is often difficult to perform an investigation of the root cause of quality issues. If a metric is already being monitored for the process, this facilitates the investigation of the root cause and you can use the baseline monitoring data to help you establish effectiveness criteria for the corrective action.quantitative effectiveness check 300x209 Is monitoring every procedure required?
  5. Finally, most companies struggle to identify preventive actions as required by Clause 8.5.3, and we have found that data analysis of monitoring data is the best source of identifying new preventive actions.

What are the disadvantages when you monitor and measure something in every procedure?

The primary reason for resistance to identifying a metric for monitoring in every procedure is that it will increase the workload for the employees responsible for that process. However, monitoring of data does not always increase workload. In fact, when process data is recorded in real-time on a run chart it is often possible to identify a trend much earlier than when data is simply recorded and subjected to monitoring.

  • Example #1: The automatic tracking of toner in a printer tells HP when to ship you a new toner cartridge before you need it. This ensures that there is no loss in productivity because you never run out of ink or the ability to print documents.
  • Example #2: Companies will use project management software (e.g. Asana) to monitor labor utilization. This will help identify when a specific resource is nearing capacity. When this occurs, the project manager can add time buffers to prerequisite steps and adjust the starting date of the resource-limited tasks to an earlier starting date. This ensures that more time is available to finish the task or to take advantage of resource availability at an earlier date.
  • Example #3: Monitoring the revision date for procedures helps the document control process owner identify procedures that should be evaluated for the need to be revised and updated. Often this is articulated as a quality objective of reviewing and updating all procedures within 2 years. This also ensures that procedures remain current and compliant with regulatory requirements.

What are the advantages of monitoring every procedure?

The phrase “what gets measured gets managed” is a popular business philosophy that implies measuring employee activity increases the likelihood that employees will complete a task or perform it well. In contrast, if a process is not monitored, employees may assume that it is not important and the tasks may be skipped or completely forgotten. Setting quantitative goals is also sometimes integrated with economic incentives or bonuses that are granted to individuals and teams.

FDA transition from QSR to ISO 13485

The US FDA is planning its transition from 21 CFR 820 to ISO 13485 as the quality system criteria. This will force companies to make adjustments to their quality systems and increase the amount of process monitoring performed. My general advice is to work with employees that are performing tasks to identify streamlined methods for monitoring those tasks without being overly burdensome. Then you and the employees you manage can analyze the data together and identify opportunities for improvement. When you do this, experiment with manual methods using whiteboards and paper charts that are visible in public areas first. Only implement automated solutions after you have optimized the data being collected and the frequency of data collection, and remember that not every process will benefit from automated statistical process control. Sometimes the simple approach is best.

Posted in: Uncategorized

Leave a Comment (0) →

ISO 13485 – need training?

Learn how to become ISO 13485 certified while avoiding the stress that tortures other quality system managers.

Your cart is empty

What is ISO 13485?

ISO 13485 is an international standard for quality management systems that is specific to the medical device industry. ISO 13485:2016 is the most recent version of the standard, and it has become the blueprint for medical device company quality systems globally. If your company wants to design, manufacture, or distribute medical devices you should consider becoming ISO 13485 certified.

Table of contents for this page

  1. What is ISO 13485?
  2. Do you have to purchase a copy of ISO 13485?
  3. Medical Device Academy’s experience with ISO 13485 training
  4. Fast-forward to 2022
  5. Previous versions of the ISO 13485 webinars
  6. The most recent version of ISO 13485 webinars
  7. Webinar duration & format
  8. Purchase the ISO 13485 training bundle
  9. Exam and training certificate available
  10. Step 1 – Planning for ISO 13485 certification
    1. Task 1 – Purchase applicable standards
    2. Task 2 – Identify which processes are applicable
    3. Task 3 – Assign a process owner to each process
    4. Task 4 – Prioritize and schedule the implementation of each process
    5. Task 5 – Create forms, flowcharts, and procedures for each process
    6. Task 6 – Perform a gap analysis of each procedure
    7. Task 7 – Train applicable personnel for each process
    8. Task 8 – Approve the procedure
    9. Task 9 – Start using the procedure and generating records
  11. Step 2 – Conducting your first internal audit
  12. Step 3 – Initiating corrective actions
  13. Step 4 – Conducting your first management review
  14. Step 5 – Stage 1, Initial ISO 13485 Certification Audit
  15. Step 6 – Stage 2, Initial ISO 13485 Certification Audit
  16. Q&A

Do you have to purchase a copy of ISO 13485?

Yes, you need to maintain a copy of the ISO 13485 standard as a “document of external origin.” This is needed for reference when you are making updates to procedures in your quality system. If you are looking for the best place to purchase a copy of the ISO 13485:2016 standard, we recommend the Estonian Centre for Standardisation and Accreditation. If you purchase a copy, we recommend selecting the option for a multi-user license so the standard can be used by more than one person in your company and printed. The only difference between the EN ISO version and the International ISO version is that the EN ISO version includes harmonization Annex ZA for compliance with the EU MDR and Annex ZB for compliance with the EU IVDR. This version is also referred to as A11:2021. Here’s a copy of the text from the beginning of the Standard:

“This Estonian standard EVS-EN ISO 13485:2016/A11:2021 consists of the English text of the European standard EN ISO 13485:2016/A11:2021. This standard has been endorsed with a notification published in the official bulletin of the Estonian Centre for Standardisation and Accreditation. Date of Availability of the European standard is 08.09.2021. The standard is available from the Estonian Centre for Standardisation and Accreditation.”

Screenshot 2016 03 08 at 4.51.04 PM ISO 13485   need training?

Medical Device Academy’s experience with ISO 13485 training

Rob Packard created his first quality system in the Spring of 2004. In October of 2009, after successfully managing quality systems for three different medical device manufacturers, Rob joined BSI as a Lead Auditor and instructor. In April of 2010, he purchased the 13485cert.com URL and he began to help companies implement quality systems as a consultant (while continuing to audit and train 140 days per year for BSI). In 2011 his medical device blog postings began as a way to help medical device companies. In 2012, Rob began building a library of quality system procedures for a turn-key quality system and selling the procedures from the Medical Device Academy website. Dozens and dozens of consulting clients have successfully achieved ISO 13485 certification with Medical Device Academy’s turnkey quality system procedures, and hundreds of quality systems were audited and/or improved. This ISO 13485 training webinar is also included as part of our turnkey quality system.

Fast-forward to 2022

Medical Device Academy is currently helping multiple device manufacturers implement their first quality system for compliance–including SaMD, electromedical devices, implants, and IVD devices. We have turnkey quality systems for the US FDA, European MDR, and MDSAP, and we are finishing our most recent turnkey system for compliance with the European IVDR. We have four qualified lead auditors as employees and three subcontractors that are ISO 13485 lead auditors. Our current turnkey quality system clients are located in countries all over the world, including Finland, Japan, Australia, France, and Canada.

Previous versions of the ISO 13485 webinars

This 2-part webinar has been previously recorded three different times. Our previous webinar on the 2003 version of ISO 13485 was split into two parts: Stage 1 and Stage 2. That first webinar was recorded in 2015. The webinars were updated in 2016 and again in 2018. We followed the same format, 2-part Stage 1 and Stage 2, for all of the subsequent ISO 13485 training webinars. The Stage 1 webinar focuses on the following processes:

  1. Management Review
  2. CAPA
  3. Internal Auditing
  4. Quality System Documentation

The Stage 2 webinar on the rest of the standard, including but not limited to:

  1. Change Control
  2. Customer Related Processes
  3. Design Controls
  4. Supplier Controls
  5. Servicing
  6. Process Validation
  7. Acceptance Activities
  8. Incoming Inspection
  9. UDI Requirements

The most recent version of ISO 13485 webinars

The webinars explaining the requirements for ISO 13485 were last updated for 2020. Anyone that purchases these webinars will receive free access to updated versions of the ISO 13485 training webinars. If you are making a new purchase of these two training webinars, the webinars are only being sold as a bundle for $258. You get:

  • access to the Stage 1 webinar recorded July 24, 2020
  • access to the Stage 2 webinar recorded July 28, 2020
  • native slide decks for both webinars

This pair of ISO 13485 training webinars explain precisely what you need to do to implement a quality system compliant with ISO 13485. After you create your own plan (a free template is provided with a subscription), then you can show the recording of these two webinars to your management team so they can implement your plan in the next several months. All deliveries of content will be sent via Aweber emails to confirmed subscribers.

Webinar duration & format

Webinars were hosted live via Zoom in 2020. The Stage 1 webinar was 64 minutes, and the duration of the Stage 2 webinar was 82 minutes. When you purchase this webinar bundle, you will receive a link to download both recorded webinars from our Dropbox folder. In addition, you will receive links to download the native slide deck for each webinar from Dropbox. 

Purchase the ISO 13485 training bundle

ISO 13485 2016 150x150 ISO 13485   need training?
ISO 13485:2016 Training Webinars – Stage 1 & Stage 2

The webinars explaining the requirements for ISO 13485 were last updated for 2020. Anyone that purchases these webinars will receive free access to updated versions of the ISO 13485 training webinars. If you are making a new purchase of these two training webinars, the webinars are only being sold as a bundle for $258. You get:
1 – access to the Stage 1 webinar recorded July 24, 2020
2 – access to the Stage 2 webinar recorded July 28, 2020
3 – native slide decks for both webinars

Price: $258.00

Exam and Training Certificate available

Training Effectiveness Exam4 300x223 ISO 13485   need training?
Exam – ISO 13485:2016 update

This is a 20 question quiz with multiple choice and fill in the blank questions. The completed quiz is to be submitted by email to Rob Packard as an MS Word document. Rob will provide a corrected exam with explanations for incorrect answers and a training effectiveness certificate for grades of 70% or higher.

Price: $49.00

ISO 13485 Certified ISO 13485   need training?

There is a big difference between being ISO 13485 certified and being compliant with ISO 13485:2016, the medical devices quality management systems standard. Anyone can claim compliance with the standard. Certification, however, requires that an accredited certification body has followed the requirements of ISO 17021:2015, and they have verified that your quality system is compliant with the standard. To maintain that certification, you must maintain your quality system’s effectiveness and endure both annual surveillance audits and a re-certification audit once every three years.

Step 1 – Planning for ISO 13485 certification

There are six steps in the ISO 13485 certification process, but that does not mean there are only six tasks. The first step in every quality system is planning. Most people refer to the Deming Cycle or Plan-Do-Check-Act (PDCA) Cycle when they describe how to implement a quality system. However, when you are implementing a full quality system, you need to break the “doing” part of the PDCA cycle into many small tasks rather than one big task. You also can’t implement a quality system alone. Quality systems are not the responsibility of the quality manager alone. Implementing a quality system is the responsibility of everyone in top management.

Below you will find seven tasks listed. I did NOT identify these nine tasks as “Steps” in the ISO 13485 certification process, because these tasks are typically repeated for each process in your quality system. Most quality systems are implemented over time, and the scope of the quality system usually grows. Therefore, you are almost certain to have to perform all of the following nine tasks multiple times–even after you receive the initial ISO 13485 certification. As the saying goes, “How do you eat an elephant? One bite at a time.” Therefore, avoid the inevitable heartburn caused by trying to do too much at one time. Implement your quality system one “bite” at a time.

Task 1 – Purchase applicable standards

The first task in implementing an ISO 13485 quality system is to purchase a copy of the ISO 13485:2016 standard, such as the MDSAP Companion Document. You will also need other applicable medical device standards. Some of these standards are general standards that apply to most, if not all, medical devices, such as ISO 14971:2019 for risk management. There are also guidance documents that explain how to use these general standards, such as ISO/TR 24971:2020, and guidance on how to apply ISO 14971:2019. Finally, there are testing standards that identify testing methods and acceptance criteria for things such as biocompatibility and electrical safety. You will need to monitor these standards for new and revised versions. When these standards are updated, you will need to identify the revised standard and develop a plan for addressing the changes.

When you purchase a standard, be sure to buy an electronic version of the standard so you can search the standard for keywords efficiently. You should also consider purchasing a multi-user license for the standard because every manager in your company will need to look up information in the standard. Alternatively, you could buy a paper copy of the standard and locate the standard where everyone in your company can access it. Often I am asked what the difference is between the EN version of the standard and the ISO version of the standard. “EN” is an abbreviation meaning European Standards or “European Norms,” which is based upon the literal translation from the French (i.e., “normes”) and German (i.e. “norm”) languages. “ISO” versions are international standards. In general, the body of the standard is typically identical but harmonized EN standards for medical devices include annexes ZA, ZB, and ZC that identify any deviations from the requirements in three medical device directives (i.e., MDD, AIMD, and IVDD).

Task 2 – Identify which processes are applicable

Clause 1 of ISO 13485 is specific to the scope of a quality system. ISO 9001, the general quality system standard, allows you to “exclude” any clause from your quality system certification. However, ISO 13485 will only allow you to exclude design controls (i.e., clause 7.3). Other clauses within ISO 13485 may be identified as “non-applicable” based upon the nature of your medical device or service. You must also document the reason for non-applicability in your quality manual. Typically, the following clauses are common clauses identified for non-applicability:

  1. Clause 4.1.6 – quality system software
  2. Clause 6.4 – work environment
  3. Clause 7.5.2 – cleanliness of the product
  4. Clause 7.5.3 – installation
  5. Clause 7.5.4 – servicing
  6. Clause 7.5.5 – sterile devices
  7. Clause 7.5.6 – process validation
  8. Clause 7.5.7 – sterilization validation
  9. Clause 7.5.9.2 – implantable devices
  10. Clause 7.5.10 – customer property
  11. Clause 8.3.4 – rework

Task 3 – Assign a process owner to each process 

The third task is to assign a process owner to each of the processes in your quality system. Typically, you create a master list of each of the required processes. Usually, the assignments are made to managers in the company who may delegate some or all of a specific process. You should expect most managers to be responsible for more than one process because there are 28 required procedures in ISO 13485:2016, but most companies have fewer than ten people when they first implement a quality system.

Task 4 – Prioritize and schedule the implementation of each process

The fourth task is to identify which processes need to be created first and to schedule the implementation of procedures from first to last. You can and should build flexibility into the schedule, but some procedures are needed at the beginning. For example, you need document control, record control, and training processes to manage all of your other procedures. You also need to implement the following processes to document your Design History File (DHF): 1) design controls, 2) risk management, 3) software development (if applicable), and 4) usability. Therefore, these represent the seven procedures that most companies will implement as early as possible. Procedures such as complaint handling, medical device reporting, and advisory notice procedures are usually reserved for last. These procedures are last because they are not needed until you have a medical device in use.

Task 5 – Create forms, flowcharts, and procedures for each process

Forms create the structure for records in your quality system, and a well-designed form can reduce the need for lengthy explanations in a procedure or work instruction. Therefore, you should consider developing forms first. The form should include all required information that is specified in the applicable standard or regulations, and the cells for that information should be presented in the order that the requirements are listed in the standard. You might even consider numbering the cells of the form to provide an easy cross-reference to the corresponding section of the procedure. Once you create a form, you might consider creating a flowchart next. Flowcharts provide a visual representation of the process. You might consider including numbers in the flow chart that cross-reference to the form as well.

Once you have created a form and a flowchart, you are now ready to write your quality system procedure. Many sections are typically included in a procedure template. It is recommended that you use a template to ensure that none of the basic elements of a procedure are omitted. You might also consider adding two sections that are uncommon to a procedure: 1) risk analysis of the procedure with the identification of risk controls to prevent risks associated with the procedure, and 2) a section for monitoring and measurement of the process to objectively measure the effectiveness of the process. These metrics are the best sources of preventive actions, and some of the metrics might be potential quality objectives to be identified by top management. 

Task 6 – Perform a gap analysis of each procedure

Most companies rely upon internal audits to catch missing elements in their procedures. However, audits are intended to be a sampling rather than a 100% comprehensive assessment. Therefore, when a draft procedure is being reviewed and approved for the first time, or a major re-write of a procedure is conducted, a thorough gap analysis should be done before the approval of the draft procedure. Matthew Walker created an article explaining how to conduct a gap analysis of procedures. In addition, Matthew has been gradually adding cross-references to ISO 13485:2016 requirements in each procedure. He is color-coding the cross-referenced clauses in blue font as well. This makes it much easier for auditors to verify that a procedure is compliant with the regulations with minimal effort. The success of these two methods has taught us the importance of conducting a gap analysis of all new procedures.

Task 7 – Train applicable personnel for each process 

You are required to document the training requirements for each person or each job in your company. Documentation of training requirements may be in a job description or within a procedure. In addition to defining who should be trained, you also need to identify what type of training should be provided. We recommend recording your training to ensure that new future employees receive the same training. This will ensure consistency. You are also required to maintain records of the training. You must verify that the training was effective, and you need to check whether the person is competent in performing the tasks. This training may require days or weeks to complete. Therefore, you may want to start training people several weeks before your procedure is approved. Alternatively, you can swap the order of tasks and conduct training after the procedure approval. If that approach is taken, then the procedure should indicate the date the procedure becomes effective–typical 30 days after approval to allow time for training.

Task 8 – Approve the procedure 

Approval of a procedure may be accomplished by signing and dating the procedure itself, while another approach is to create a document that lists all the procedures and forms being approved at one time. The second method is the method we use in our turn-key quality system. Companies can review and approve as many procedures at one time as they wish. Since this process needs to be defined to ensure that all of the procedures you implement are approved, the document control process is typically the first procedure that companies will approve in a new quality system. The second procedure generally is for the control of records. Then the next procedures implemented will typically be focused on the documentation of design controls: design controls, risk management, usability testing, and software development. The last procedures to be approved are typically complaint handling, medical device reporting, and recalls. These procedures are left for last because you don’t need them until you are selling your medical device.

Task 9 – Start using the procedure and generating records

The last task required for the implementation of a new quality system is to start using the procedures to generate records. All of the procedures will need records before the process can be verified to be effective. Records can be paper-based, or the records can be electronic. Whichever format you use for the record retention needs to be communicated to everyone in the company through your Control of Records procedure and/or within each procedure. If you include the information in each procedure, the records of each procedure should be listed in the procedure, and the location where those records are stored should be identified. Generally, there is no specific minimum number of records to have for a certification audit, but you should have at least a few records for each process that you implement.

Step 2 – Conducting your first internal audit

The purpose of the internal audit is to verify the effectiveness of the quality system and to identify nonconformities before the certification body auditor finds them. To successfully achieve this secondary objective, it is essential to have a more rigorous internal audit than you expect for the certification audit. Therefore, the internal audit should be of equal duration or longer in duration than the certification audit. The internal audit should not consist of a desktop review of procedures. Reviewing procedures should be part of gap analysis (i.e., task 6 above) that is conducted on draft procedures before they are approved. Internal audits should utilize the process approach to auditing, and the auditor should apply a risk-based approach (i.e., focus on those processes that are most likely to contribute to the nonconforming products, result in a complaint, or cause severe injuries and death).

After your internal audit, you will receive an internal audit report from the auditor. You should also expect findings from the internal auditor, and you should expect opportunities for improvement (OFI) to be identified. Experienced auditors can typically identify the root cause of a nonconformity more quickly than most process owners. Therefore, it is recommended for each process owner and subject matter expert to review nonconformities with the auditor and discuss how the nonconformity should be investigated. The root cause must be correctly identified during the CAPA process, and the effectiveness check must be objective to ensure that problems do not recur.

Step 3 – Initiating corrective actions

Corrective actions should be initiated for each internal audit finding immediately, to make sure the findings are corrected and prevented from repeat occurrence before the Stage 1 audit. It will take a minimum of 30 days to implement the most corrective actions. Depending upon the scheduling of the internal audit, there may not be sufficient time to complete the corrective actions. However, you should at least initiate a CAPA for each finding, perform an investigation of the root cause, and begin to implement corrective actions.

Also, to take corrective actions related to internal audit findings, you should look for internal audits from other sources. The diagram below shows several different sources of potential corrective and preventive actions.

Risk based CAPA Process Diagram ISO 13485   need training?

Monitoring and measuring each process is the best source of preventive actions, while internal audits are typically the best source of corrective actions.  Any quality problems identified during validation are also excellent sources of corrective actions because the validation can be repeated as a method of demonstrating that the corrective actions are effective. However, your ISO 13485 certification auditor will focus on non-conforming products, complaints, and services as the most critical sources of corrective actions. These three sources are prioritized because these three sources have the greatest potential for resulting in a serious injury, death, or recall if corrective actions are not implemented to prevent problems from recurring.

Step 4 – Conducting your first management review 

In addition to completing a full quality system audit before your stage 1 audit, you are also expected to complete at least one management review. To make sure that you have inputs for each of the 12 requirements in the ISO 13485:2016 standard, it is recommended to conduct your management review only after you have completed your full quality system audit and initiated some corrective actions. If possible, you should also conduct supplier audits for any contract manufacturers or contract sterilizers. It is recommended to use a template for that management review that is organized in the order of the required inputs to ensure that none of the necessary inputs are skipped. Quality objectives will need to be established long before the management review so that the top management team has sufficient time to gather data regarding each of the quality objectives. Also, you should consider delegating the responsibility for creating the various slides for each input to different members of top management. This will ensure that everyone invited to the meeting is engaged in the process, and it will spread the workload for meeting preparation across multiple people.

At the end of the meeting, top management will need to create a list of action items to be completed before the next management review meeting. Meeting minutes will need to be documented for the meeting, including the list of action items and each of the four required outputs of the management review process. We recommend using the notes section of a presentation slide deck to document the meeting minutes related to each slide. Then the slide deck can be converted into notes pages and saved as a PDF. The PDF notes pages will be your final meeting minutes for the management review. An example of one of these notes pages is provided in the figure below.

Print PDF of Meeting Minutes Notes Page Example ISO 13485   need training?

One of the more common non-value-added findings by auditors is when an auditor issues a nonconformity because you do not have your next internal audit and your next management review scheduled–even though each may have occurred only a month prior to the Stage 1 audit. Therefore, we recommend that you document your next 12-month cycle for internal audits and schedule your next management review as action items in every management review meeting. The schedule can be adjusted if needed, but this allows top management to emphasize various areas in internal audits that may need improvement. You might even set a quality objective to conduct a minimum of three management reviews per year at the end of your first management review.

Step 5 – Stage 1, Initial ISO 13485 Certification Audit

In 2006, the ISO 17021 Standard was introduced for assessing certification bodies. This is the standard that defines how certification bodies shall go about conducting your initial certification audit, annual surveillance of your quality system, and the re-certification of your quality system. In the past, certification bodies would typically conduct a “desktop” audit of your company before the on-site visit to make sure that you have all the required procedures. However, ISO 17021 requires that certification bodies conduct a Stage 1 audit that assesses the readiness of your company before conducting a Stage 2 audit. Therefore, even if the Stage 1 audit is conducted remotely, the certification body is expected to interview process owners and sample records to verify that the quality system has been implemented. Certification body auditors will also typically verify that your company has conducted a full quality system audit and at least one management review. Finally, the auditor will usually select a process such as corrective action and preventive action (CAPA) to make sure that you are identifying problems with the quality system and taking appropriate measures to address those problems.

Your goal for the Stage 1 audit should not be perfection. Instead, your focus is to make sure that there are no “major” nonconformities. The term “major” used to have a specific definition:

  1. Absence of a documented procedure or process
  2. Release of nonconforming product
  3. Repeat nonconformities (not possible during a Stage 1)

Under the MDSAP, the grading system for nonconformities now uses a numbering system for grading nonconformities: “Nonconformity Grading System for Regulatory Purposes and Information Exchange Study Group 3 Final Document GHTF/SG3/N19:2012.” Any nonconformity is graded on a scale of one to four, and then two potential escalation rules are applied. If any nonconformities are graded as a four or a 5, then the auditor must assess whether a five-day notice to Regulatory Authorities is required. A five-day notice is required in either of the following situations: 1) one or more findings grading of “5”; or 2) three or more findings graded as “4.” If your Stage 1 audit results in a five-day notice, then you are not ready for your Stage 2 audit. For example, a complete absence of two required procedures in clauses 6.4 through 8.5 of ISO 13485:2016 would result in two findings with a grading of “4.” This would not result in a five-day notice, but the absence of a third required procedure would result in a five-day notice.

The duration of your Stage 1 audit will be one or two days, but a 1.5-day audit is quite common for MDSAP Stage 1 audits. The reason for the 1.5-day Stage 1 audit is that it is challenging to assess readiness for Stage 2 in one day, and if the total duration of Stage 1 and Stage 2 is 5.5 days, then the Stage 2 audit could be completed in four days. The four-day audit is more convenient than a three-day audit for a two-person audit team.

After your Stage 1 audit, you will receive an audit report, and you should expect findings. You should initiate corrective actions for each finding immediately, to make sure the findings are corrected and prevented from repeat occurrence before the Stage 2 audit. The duration between the audits is typically about 4-6 weeks. That does not leave much time for you to initiate a CAPA, perform an investigation of the root cause, and implement corrective action. At a minimum, you must submit a corrective action plan for each finding to your MDSAP auditing organization (AO) within 15 calendar days of receiving the finding. For any findings graded as a “4” or higher, you will need to provide evidence of implementing the corrective action plan to the AO within 30 calendar days of receiving the finding. You are also unlikely to have enough time to conduct an effectiveness check prior to the Stage 2 audit.

Step 6 – Stage 2, Initial ISO 13485 Certification Audit

The Stage 2 initial ISO 13485 certification audit will verify that all regulatory requirements have been met for any market you plan to distribute in. The auditor will complete an MDSAP checklist that includes all of the regulatory requirements for each of the countries that recognize MDSAP: 1) the USA, 2) Canada, 3) Brazil, 4) Austria, and 5) Japan. The auditor will also sample records from every process in your quality system to verify that the procedures and processes are fully implemented. This audit will typically be at least four days in duration unless multiple auditors are working in an audit team.

The audit objectives for the Stage 2 ISO 13485 certification audit specifically include evaluating the effectiveness of your quality system in the following areas:

  1. Applicable regulatory requirements
  2. Product and process-related technologies
  3. Technical documentation

All procedures will be reviewed for compliance with ISO 13485:2016 and the applicable regulations. The auditor will also sample records from each process. If the auditor identifies any nonconformities during the audit, it is important to record the findings and begin planning corrective actions immediately. If you have any questions regarding the expectations for the investigation of the root cause, corrections, corrective actions, and effectiveness checks, you should ask the auditor during the audit or the closing meeting. At a minimum, you must submit a corrective action plan for each finding to your MDSAP auditing organization (AO) within 15 calendar days of receiving the finding. For any findings graded as a “4” or higher, you will need to provide evidence of implementing the corrective action plan to the AO within 30 calendar days of receiving the finding. The auditor will not be able to recommend you for ISO 13485 certification until your corrective action plans are accepted.

If you receive a finding with a grading of “5,” or three or more findings graded as “4,” then the MDSAP auditor is required to issue a five-day notification to the regulators. The auditor will also need to return to your facility for a follow-up audit to close as many findings as they can. It is not necessary to eliminate all of the findings in order to be recommended for ISO 13485 certification, but the grading of the findings must be reduced to at least a “3” before recommending the company for certification. The number of findings also determines whether the auditor recommends your company for certification.

In addition to reviewing the findings and conclusions of the audit during the closing meeting, the auditor will also review the plan for the annual surveillance and re-certification with you. Each certification cycle is three years in duration. There will be two surveillance audits of approximately one-third the duration of the combined duration of stage 1 and stage 2 initial certification audits, and the first surveillance audit must be completed within 12 months of the initial certification audit. In the third year, there will be a re-certification audit for two-thirds of the duration of the combined duration of stage 1 and stage 2 initial certification audits. The initial ISO 13485 certificate will be issued with a three-year expiration, and the certificate is typically received about one month after the acceptance of your corrective action plan.

Q&A

There are no stupid questions, and we can save your weeks of wasted time if you just ask for help. We are always looking for new ideas for blogs, webinars, and videos on our YouTube channel. If you have any general questions about obtaining ISO 13485:2016 certification, please email Rob Packard at rob@13485cert.com. If you have a suggestion for new ISO 13485 training materials, you can also use our “Suggestion Box.” You can also schedule an initial free consultation with Rob using his calendly link.

Posted in: ISO 13485:2016, ISO Certification, Quality Management System

Leave a Comment (37) →

What’s new in the 2022 draft cybersecurity guidance?

On April 8, 2022, the FDA released a new draft cybersecurity guidance document to replace the 2018 draft that the industry does not support.

Why was the draft cybersecurity guidance created?

Due to the ubiquitous nature of software and networked devices in the medical industry, the impact of cybersecurity attacks is becoming more frequent and more severe. The WannaCry Ransomeware Attack is just one example of this global cybersecurity issue. The FDA is responding to the need for stronger cybersecurity controls by issuing a new draft cybersecurity guidance for 2022.

The first four paragraphs of the introduction explain why we need this, and WannaCry is mentioned in the second paragraph of the background section. This new guidance is only a draft, but this is the FDA’s third attempt at regulating the cybersecurity of medical devices. The first guidance was finalized in 2014. That’s the 9-page guidance we currently have in effect. The guidance mentions risk 11 times and there is no mention of testing requirements or a bill of materials (BOM). The 2018 draft guidance (24-pages) met with resistance from the industry for a lot of reasons. One of the reasons mentioned by Suzanne Schwartz in an interview is the inclusion of a cybersecurity bill of materials (CBOM). The industry felt it would be too burdensome to disclose all of the hardware elements that are related to cybersecurity. Therefore, the FDA rewrote the 2018 draft and released a new draft on April 8, 2022 (49-pages).

Untitled presentation e1650071404761 What’s new in the 2022 draft cybersecurity guidance?

You might have expected the FDA to soften its requirements in the face of resistance from industry, but the new draft does not appear to be less robust. It is true that the CBOM was replaced by a software bill of materials (SBOM). However, the SBOM must be electronically readable and it must include:

  • the asset(s) where the software resides;
  • the software component name;
  • the software component version;
  • the software component manufacturer;
  • the software level of support provided through monitoring and maintenance from the software component manufacturer;
  • the software component’s end-of-support date; and
  • any known vulnerabilities.

You can be sure that the medical device industry will view providing an SBOM as a hefty burden. After all, a machine-readable SBOM is more complex than UDI labeling requirements. An SBOM will not fit on the “Splash Screen” for anyone’s software application. Companies may provide documentation through the company website with a link in their software to that information. The format of the information could be in the “Manufacturer Disclosure Statement for Medical Device Security (MDS2).” However, MDS2 is a 349-line item Excel spreadsheet to be used as a checklist (i.e. quite a bit longer than the GUDID data elements spreadsheet), and it took the FDA eight years to complete the transition for the UDI Final Rule (i.e. 2013 – 2021).

The 2018 draft cybersecurity guidance document from the FDA required a cybersecurity bill of materials (CBOM). CBOM was defined as “a list that includes but is not limited to commercial, open source, and off-the-shelf software and hardware components that are or could become susceptible to vulnerabilities.” Therefore, the FDA’s change from a CBOM to an SBOM eliminated the requirement to disclose the hardware components. Despite the change in disclosure requirements, manufacturers will still be expected to monitor potential hardware vulnerabilities to cybersecurity attacks. It should also be noted that the language in the PATCH Act (a new bill submitted to the House of Representatives and to the Senate for ensuring the cybersecurity of medical devices) specifically requires manufacturers “to furnish a software bill of materials as required under section 524B (relating to ensuring the cybersecurity).”

 Structure of the draft cybersecurity guidance

The 2022 draft cybersecurity guidance organizes the requirements into four major principles:

  1. cybersecurity as part of device safety and the quality system regulations
  2. designing for security
  3. transparency
  4. submission documentation

The draft cybersecurity guidance recommends the implementation of a Secure Product Development Framework (SPDF). However, there is not much detail provided in the guidance for a SPDF. In the past, the term for this type of process was referred to as a Secure Software Development Lifecycle (i.e. Secure SDLC). However, in February 2022, the NIST Computer Security Resource Center (CSRC) released version 1.1 of the Secure SDLC guidance which is now titled “Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities.” This guidance provides guidance on the implementation of best practices for reducing the risk of software vulnerabilities because existing standards for managing the software development lifecycle do not explicitly address software security (e.g. IEC 62304-1:2015). The SSDF recommends implementing a core set of high-level secure software development practices that can be integrated into your SDLC process. Your software development team will also require cybersecurity training.

Design for security is the second principle of the draft cybersecurity guidance

Under this new draft cybersecurity guidance, the FDA will be evaluating the cybersecurity of devices based on the ability of the device to provide and implement the following security objectives:

  • Authenticity, which includes integrity;
  • Authorization;
  • Availability;
  • Confidentiality; and
  • Secure and timely updatability and patchability.

Transparency of cybersecurity information is for users

The draft cybersecurity guidance seeks to give device users more information pertaining to the device’s cybersecurity controls, potential risks, and other relevant information. This information will be in the form of an SBOM that is electronically readable. This information shall include disclosure of 1) known vulnerabilities or risks, 2) information to securely configure and update devices, and 3) communication interfaces and third-party software.

In addition to providing an SBOM, the FDA draft cybersecurity guidance includes requirements for cybersecurity labeling in section VI(A). There are 15 specific labeling requirements identified by the FDA for sharing with device users to improve the transparency of cybersecurity information. The first of these requirements is recommendations from the manufacturer for cybersecurity controls appropriate for the intended use environment (e.g., antimalware software, use of a firewall, password requirements). This first labeling requirement is identical to the 2018 draft guidance. Several of the other requirements are copied from the 2018 draft guidance, but others are new and/or reworded cybersecurity labeling requirements.

FDA Submission Documentation Requirements

The 2022 FDA draft cybersecurity guidance includes requirements for FDA submission documentation. Submission documentation must include a security risk management plan and report. The draft cybersecurity guidance explains on page 13 (numbered 9) that “performing security risk management is a distinct process from performing safety risk management as described in ISO 14971:2019.” Therefore, instead of using your safety risk management process, your software development team will need to have a different risk management process for software security. Details on the content for security risk management plans and reports can be found in AAMI TIR57:2016 – Principles for medical device security—Risk management. Appendix 2 also provides guidance for the inclusion of a) call flow diagrams, and b) information details for an architecture view.

Cybersecurity testing requirements for your FDA submission

The biggest impact of this new draft guidance may be the requirement for testing. The 2014 guidance has no testing requirement, the 2018 draft guidance mentioned testing 5 times in a few bullet points, but this new draft guidance mentions testing 43 times. The testing requirements for cybersecurity risk management verification include:

  1. Security requirements
  2. Threat mitigation
  3. Vulnerability testing
  4. Penetration testing

This guidance also includes a paragraph with multiple bullets of requirements for each of the four types of testing. This would essentially double the size and scope of the current software section for a 510k submission, and manufacturers will need to create new procedures and templates for their cybersecurity risk management process. For example, penetration testing requirements include the following elements:

  • Independence and technical expertise of testers,
  • Scope of testing,
  • Duration of testing,
  • Testing methods employed, and
  • Test results, findings, and observations.

Differences between the cybersecurity guidance documents

The following table provides a high-level overview comparing the four cybersecurity guidance documents released by the FDA, including the 2016 guidance on post-market management of cybersecurity:

Screenshot 2022 04 16 12.48.51 AM 1024x291 What’s new in the 2022 draft cybersecurity guidance?

Vulnerability management plans

The FDA draft cybersecurity guidance document also has a requirement for manufacturers to develop a plan for identifying and communicating vulnerabilities to device users after the release of the device. The FDA requires this plan to be included in your device submission. The vulnerability management plan should include the following information (in addition to the requirements of the 2016 guidance for postmarket cybersecurity management):

  • Personnel responsible;
  • Sources, methods, and frequency for monitoring for and identifying vulnerabilities (e.g. researchers, NIST NVD, third-party software manufacturers, etc.);
  • Periodic security testing to test identified vulnerability impact;
  • Timeline to develop and release patches;
  • Update processes;
  • Patching capability (i.e. rate at which update can be delivered to devices);
  • Description of their coordinated vulnerability disclosure process; and
  • Description of how manufacturer intends to communicate forthcoming remediations, patches, and updates to customers.

What’s the next step for the draft cybersecurity guidance?

In March the “Protecting and Transforming Cyber Health Care Act of 2022 (PATCH Act)” was introduced to the House of Representatives and the Senate. The goal of the PATCH Act is to enhance medical device security by requiring manufacturers to create a cybersecurity risk management plan for monitoring and addressing potential postmarket cybersecurity vulnerabilities. The FDA seeks comments on the draft cybersecurity guidance through July 7, 2022. Given the support of the new bill in the House of Representatives and Congress, it is likely that the FDA will get the support it needs for this new guidance. 

Posted in: 510(k), Cybersecurity

Leave a Comment (0) →

Human factors process, can we make this easy to understand?

90% of usability testing submitted to the FDA is unacceptable and the root cause is simply a failure to understand the human factors process.

If you submitted no usability testing to the FDA in your 510(k) submission, it would be obvious why the FDA reviewer identified usability as a major deficiency. However, you spent tens of thousands of dollars on usability testing that delayed the 510(k) submission by six months. Despite all of the time and money your company invested in the human factors process, it appears that you need to start over and repeat the entire process again. The CEO is furious, and he wants you to show him where in the 49-page FDA guidance it says that you have to do things differently.

Benefits from the human factors process

  1. Use errors result in serious injuries and death
  2. Easy to use products sell
  3. You will prevent delays in regulatory approval

Why was your rationale for no usability testing rejected?

Unlike CE Marking technical files, the FDA does not require a usability engineering file for all products. Instead, the FDA determines if usability testing is required based upon a comparison of your device’s user interface and a competitor’s user interface (i.e. predicate device user interface). If the user interface is identical, then usability testing may not be required. Instead, your company should be able to write a rationale for not doing usability testing based upon equivalence with the predicate device. If there are differences in your user interface, you will need to provide use-related risk analysis (URRA), identify critical tasks, implement risk controls, and provide verification testing to demonstrate the effectiveness of the risk controls. Even if your device is “easier to use” or “simpler”, you still need to provide the documentation to support this claim in your submission. The FDA also does not allow comparative claims in your marketing for 510(k) cleared devices. Comparative claims require the support of clinical data.

What is the 10-step human factors process?

  1. Define human factors for your device or IVD
  2. Identify use errors
  3. Conduct a URRA
  4. Perform a critical task analysis
  5. Conduct a risk control option analysis
  6. Conduct formative usability testing
  7. Implement risk controls
  8. Conduct summative usability testing
  9. Prepare HFE/UE documentation
  10. Collect post-market surveillance data specific to use errors

There is a YouTube video describing these 10 steps at the bottom of this blog posting.

Why is formative testing needed?

  • Observational study to identify unforeseen use errors
  • Observational study to evaluate risk control options
  • What are the other types of studies?
  • Development of indications for use
  • Development of training materials

Why is the human factors process crazy expensive to outsource?

  • Human factors consultants need time to learn about your device
  • Consultants are more conservative because they cannot afford to fail
  • Justifying your choice of risk controls is difficult because you started too late
  • Your instructions for use (IFU) are inadequate
  • Consultants need to explain the human factors process to you
  • Recruiting subjects is marketing (which may not be their expertise)
  • You are paying for infrastructure (specialized testing facilities)
  • This is a team effort that requires many consulting hours collectively

Why was your Usability Engineering File refused?

  1. Your company provided an application failure modes and effects analysis (aFMEA) to support your justification that residual risks are acceptable. The FDA guidance suggests using risk analysis tools such as an FMEA or fault-tree analysis, but deficiency letters from FDA reviewers recommend a use-related risk analysis (URRA) format that is totally different.

    URRA table example from the FDA 1024x399 Human factors process, can we make this easy to understand?

    Example of a URRA Table provided by the FDA for the Human Factors Process

    The primary problem with using an FMEA or Fault-Tree risk analysis tool is that these tools involve estimation of the severity of harm and the probability of occurrence of harm, while the FDA does not feel it is appropriate to estimate the probability of occurrence of harm. Instead, the FDA instructs companies to assume that use errors will occur and to implement risk controls to mitigate those risks (see URRA example above). Although “mitigation” is unlikely, and use risks will only be reduced, this is the approach the FDA wants companies to use. In addition, the FDA expects your company to provide traceability of risk control implementation to each use-related risk you identified and the FDA expects documentation of verification testing (i.e. usability testing) that shows your risk controls are effective. Finally, the FDA (and ISO 14971, Clause 10) expects you to collect and perform a trend analysis of use errors. Any use errors that are reported should be evaluated for the need to implement additional corrective actions to prevent future use errors. Blaming “user error” is not an acceptable approach. 

  2. You provided risk analysis and human factors testing in your 510(k) submission, but the FDA reviewer said you need to identify critical tasks and provide traceability to each critical task in your summative validation report. – Critical tasks are specifically mentioned in section 3.2 of the FDA guidance on applying human factors and usability engineering–and a total of 49 times throughout the guidance. However, “critical tasks” are not mentioned even once in ISO 14971:2019 or ISO/TR 24971:2020. The term “critical tasks” is not even found in IEC 62366-1:2015. There is mention of “tasks”, and “task” is a formal definition (i.e. Definition 3.14, “Task – one or more USER interactions with a MEDICAL DEVICE to achieve a desired result”). Therefore, companies that are familiar with the ISO Standards and CE Marking process frequently need training on the FDA requirements for the human factors process. After receiving training, then your company will be prepared to modify your usability engineering file documentation to comply with the FDA requirements for human factors.
  3. You completed a summative validation protocol, but the FDA disagrees with your definition of user groups. – Each user has a different level of experience, training, and competency. Therefore, if you define the intended user population too broadly (e.g. healthcare practitioners), the FDA may not accept your summative usability testing. This is the reason that the human factors process begins with defining the human factors for your IVD or device. Radiologists, for example, have the following training pathway:
    • graduate from medical school;
    • complete an internship;
    • pass state licensing exam;
    • complete a residency in radiology;
    • become board certified; and
    • complete an optional fellowship.

Therefore, if you are developing imaging software, you need to make sure your user group includes radiologists that cover the entire range of competencies. In addition, most radiology images are taken by radiology technicians and then reviewed by the radiologist. Therefore, radiology technicians should be considered a completely different user group due to the differences in experience, training, and competency when compared to a radiologist. This simple example doubles the number of users needed because you have two user groups instead of one.

  1. You evaluated 15 users, but the FDA reviewer is asking you to evaluate a larger number of users based upon a special controls guidance document. – The FDA guidance on human factors testing specifies a minimum of 15 users for each user group–not a minimum of 15 users. Therefore, for a device that is for Rx-only and OTC use, you will have at least two user groups that need to be evaluated independently. In addition, some devices have special controls guidance documents that specify usability testing requirements. For example, an OTC blood glucose meter must pass a 350-person lay-user study. Covid-19 self-tests are expected to pass a 30-person lay-user study as another example.
  2. Your usability study was conducted in Australia, but the FDA insists that your usability study must be repeated in the USA. – Most people think of language being the primary difference between two countries, and therefore the author of a study protocol may not perceive any difference between the USA and Australia, Ireland, Canada, or the UK. However, this lack of ability to identify differences between cultural norms shows our own ignorance of cultural differences. International travelers learn quickly about the differences in the interface used for electrical outlets between the USA and other countries. There are also more subtle differences between cultures, such as in which direction do you toggle a light switch to turn on a light, up or down? For devices that are used in a hospital environment, it is critical to understand how your device will interact with other devices and how different hospital protocols might impact human factors.
  3. The FDA reviewer indicated that your usability engineering file does not assess the ability of laypersons to self-select whether your OTC device is appropriate for them. – Devices and IVD devices may have contraindications or indications for use that are specific to an intended patient population or intended user population. In these cases, the user of the device or IVD needs to be able to “self-select” as included or excluded from use. The ability to self-select should be assessed as part of any OTC usability study. The ability to identify suitable and unsuitable patients for treatment is also a common criterion for a usability study involving prescription devices where a physician is the subject of the study.
  4. The FDA reviewer indicated that you did not provide raw data collected by the study moderator. – Data collected during a human factors study is usually subjective in nature, and the FDA may want to conduct their own review and analysis of your data. Therefore, you cannot provide only a testing report that summarizes the results of your study. You must also provide the raw data for the study. It is permitted to provide the data in a tabular format that has been transcribed from paper case report forms or was recorded electronically. You should also consider scanning any paper forms for permanent retention or retaining the paper forms in case there is any question of accuracy in the transcription of the data collected. Finally, it is best practice to record videos of the study participants performing each task and answering interview questions. This will help in filling any gaps in the notes recorded by the moderator, and the recording provides additional objective evidence of the study results.
  5. The FDA reviewer indicated that your study is not valid, because the training provided by moderators was not scripted and training decay was not considered in the design of the study. – Summative usability testing requires that users complete all of the critical tasks identified in your critical task analysis without assistance. It is permitted to provide training to the user prior to conducting the study if the device or IVD is for prescription use and healthcare practitioners are responsible for providing instruction to the user. However, any training provided must be scripted in advance and approved as part of the summative usability testing protocol. This ensures that every subject in the study receives consistent training. Unfortunately, the FDA may still not be satisfied with the design of your study if you do not allow sufficient time to pass between the time that training is provided to the user and when the subject uses the device or IVD for the first time. In general, one hour is the minimum amount of time that should pass between providing user training and when the device or IVD is used for the first time. This is referred to as “training decay” and the duration of time between your scripted training and the user performing critical tasks for the first time should be specified in your summative usability protocol. One solution to address both issues is to provide a video of the instructions to each subject 24-hours in advance of participation in the study.

Additional resources for the human factors process and usability testing

Posted in: 510(k), Design Control, Usability

Leave a Comment (3) →

What are the IVDR risk management requirements?

This article reviews unique IVDR risk management requirements for CE Marking of in vitro diagnostic (IVD) devices in Europe.

Last week I posted a blog about “How to create an IVDR checklist.” The article was very popular because we included a form for downloading a free IVDR checklist. That form included the opportunity for people to ask a question about the IVDR. One of the subscribers, a gentleman from New Zealand, entered a very simple comment: “risk management requirements.” My first thought was that the risk management file is the required technical documentation for the IVDR. Then I quickly remembered that in 2012, EN ISO 14971:2012 was released with three new annexes for the three directives: ZA (for the MDD), ZB (for the AIMD), and ZC (for the IVDD). In Annex ZC there were seven deviations, and even though ISO 14971 was updated in 2019, the international standard continues to deviate from the European regulations in significant ways. Therefore, this blog provides an overview of the IVDR risk management requirements.

If you are already compliant with ISO 14971:2019, do you meet the IVDR risk management requirements?

The biggest difference between the ISO 14971:2019 standard and the IVDR risk management requirements is that the standard only requires a benefit-risk analysis to be performed if risks are unacceptable. In contrast, the IVDR requires that a benefit/risk analysis be performed for all risks and the overall residual risk. Therefore, you must include a benefit/risk analysis in your technical file submission regardless of risk acceptability. The harmonized version of the standard (i.e. EN ISO 14971:2019/A:11:2021) was released in December of 2021. If you already purchased ISO 14971:2019, you only need to purchase the amendment which consists of Annex ZA (comparison between the standard and Annex I of MDR) and ZB (comparison between the standard and Annex I of the IVDR).

In the amendment, it states that manufacturers must have a risk management policy that is compliant with Annex I of the EU regulation. There are notes at the beginning of each harmonization annex that indicates that the risk management process needs to be compliant with the IVDR, which means risks have to be ‘reduced as far as possible’, ‘reduced to a level as low as reasonably practicable’, ‘reduced to the lowest possible level’, ‘reduced as far as possible and appropriate’, ‘removed or reduced as far as possible’, ‘eliminated or reduced as far as possible’, ‘prevented’ or ‘minimized’, according to the wording of the corresponding section in Annex I of the IVDR. The comparison table has a column with remarks/notes. In most cases, the deficiency identified states, “Device-specific execution of the process is not covered.” There are also two remarks/notes that state “Device-specific and usability-specific execution of the process is not covered.” 

Where are the IVDR risk management requirements?

Blog tip 1024x183 What are the IVDR risk management requirements?

IVDR risk management requirements are found in Annex II, Section 5 of the IVDR. However, there are 228 references to the word risk throughout the IVDR. The following risk-related requirements in IVDR are particularly important:

  • Article 10, Sections 2 & 8(e) – risk management procedure requirement
  • Annex I, Section 3 – reiteration of risk management procedure requirement, but specific steps in the risk management process are identified (e.g. a risk management plan, hazard identification, estimation of risks, evaluation of risks, etc.)
  • Annex I, Section 4 – Priority of risk control measures
  • Annex I, Section 5 – Elimination or reduction of use-related risks
  • Annex III, Section 1(b) – Reassessment of the benefit-risk analysis and risk management using post-market surveillance data

How should you document your risk management file for the IVDR?

In your risk management file of course. There is no format requirement for risk management files, but there are requirements for the content and there is a GHTF guidance document for risk management, and ISO/TR 24971:2020 is a new guidance document on the application of ISO 14971 to medical devices. Neither of these guidance documents is specific to IVDR risk management requirements. Annex H of ISO/TR 24971:2020, however, provides guidance specific to IVD devices.

What do the IVDR risk management requirements include for risk analysis?

In our March 23, 2022 blog posting, I described four types of risk analysis:

  1. Design risk analysis
  2. Process risk analysis
  3. Software hazard analysis
  4. User-related risk analysis (URRA)

Of these four types of risk analysis, only the software hazard analysis is sometimes not applicable. For an FDA 510(k) submission, you would need to provide software hazard analysis and URRA in the actual submission. The other two types of risk analysis would only be included in your design history file (DHF), and the FDA would review the design and process risk analysis during a routine inspection when the DHF is sampled as part of the design control process.

In contrast, the IVDR requires that a complete risk management file be submitted as part of the technical file (see Annex II, Section 5):

“Benefit-risk analysis and risk management

The documentation shall contain information on:

    • the benefit-risk analysis referred to in Sections 1 and 8 of Annex I, and
    • the solution adopted and the results of the risk management referred to in Section 3 of Annex I.”

The above documentation typically consists of design risk analysis and does not typically include process risk analysis, software hazard analysis, or use-related risk analysis. These other three risk analysis documents are IVDR risk management requirements, but they are referenced by the technical file in other sections. The most obvious IVDR risk management requirements are referenced in Annex I, Sections 1-9. These are referred to as the General Safety and Performance Requirements (GSPRs), and this requirement is typically met by including a GSPR checklist in the technical file to meet the requirement of Annex II, Section 4.

The process risk analysis is typically included with manufacturing information to meet the requirement of Annex II, Section 3.2. This documentation may include, any and all of the following elements:

  1. a process failure mode and effects analysis (pFMEA)
  2. a risk control plan including all processes from receiving inspection to final inspection and product release
  3. a process validation plan that is risk-based and linked to the risk control plan

The best practice for estimation of process risks is to link the probability of occurrence and probability of detection to the quantitative data gathered during process validation. In addition, you may establish a risk management policy that prescribes specific types of process risk controls (e.g. automated inspection) for the highest risk processes where manufacturing process errors are not acceptable residual risks. For example, an inspection of printed circuit board assemblies (PCBAs) typically requires automated optical inspection (AOI) methods, because visual inspection is not sufficient by itself and not all PCBAs allow sufficient ICT coverage, and functional testing is limited.

The software hazard analysis, if applicable, is typically performed in accordance with IEC/TR 80002-1:2009, Guidance on the application of ISO 14971 to medical device software. In the software hazard analysis, it is unnecessary to estimate the probability of occurrence of harm. Instead, it is only necessary to identify hazards and estimate harm. Examples of these hazards include loss of communication, mix-up of data, loss of data, etc. Software failures are systemic in nature and the probability of occurrence cannot be determined using traditional statistical methods. Therefore, we recommend that you assume that the failure will occur and estimate software risks based on the severity of the hazard resulting from the failure. For these reasons, it is recommended that software hazard analysis documentation is maintained as a separate document from your design risk analysis. The software hazard analysis documentation should be referenced in your risk management report, but the software hazard analysis should be included as part of your software verification and validation. The IVDR requires that you include a summary of software verification and validation in Annex II, Section 6.4 rather than the complete hazard analysis document.

A use-related risk analysis should be part of your useability engineering file for IVD devices as required by EN 62366-1:2015. Use-related risks are mentioned in Annex I, Section 5:

“In eliminating or reducing risks related to use error, the manufacturer shall:

    • reduce as far as possible the risks related to ergonomic features of the device and the environment in which the device is intended to be used (design for patient safety), and
    • Give consideration to the technical knowledge, experience, education, training and use environment, where applicable, and the medical and physical conditions of intended users (design for lay, and professional, disabled or other users).”

The above requirement includes not only the ability to read and interpret test results of IVD devices but also the ability of laypersons to properly self-select if an IVD is intended to be sold as an over-the-counter product. Usability also is mentioned in Article 78, Section 3(f):

“for the identification of options to improve the usability, performance and safety of the device;”

Therefore, there should be specific elements of your post-market surveillance plan that are designed to gather feedback on the usability of your IVD device. 

When should risk management activities be performed for IVD devices?

The IVDR does not specifically define when in the design and development process the various risk management activities shall be performed. However, the required risk management activities are specified in the IVDR within Annex I. ISO 14971:2019, however, is more descriptive of the risk management activities and the risk management process. Therefore, your risk management plan should align with the process defined in ISO 14971:2019, Clause 4.1.

Unfortunately, most companies do not include risk management as an integral part of the design and development process. Instead, risk management documentation is created retroactively as part of the documentation preparation for technical file submission. For this reason, most medical device executives fail to see the benefit associated with the risk management process. Even biomedical engineers struggle to appreciate the necessity of following the process outlined in the risk management standard in order to prevent device malfunctions and use errors.

The following is a list of the required risk management activities in the order that they should be occurring. Each activity also references the applicable clause of ISO 14971:2019. We have also grouped the activities into the five phases of design and development:

Design Controls with risk 1024x542 What are the IVDR risk management requirements?

Design Planning

    • Risk management planning (Clause 4.4)

Design Inputs

    • Identification of hazards and hazardous situations (Clause 5.4)

Design & Development

    • Risk estimation (Clause 5.5)
    • Risk evaluation (Clause 6)
    • Risk control option analysis (Clause 7.1)
    • Implementation of risk control measures (Clause 7.2)
    • Residual risk evaluation (Clause 7.3)

Design Verification and Validation

    • Benefit/risk analysis (Clause 7.4)
    • Risk control effectiveness verification (Clause 7.6)

Design Release

    • Evaluation of overall residual risk (Clause 8)
    • Risk management review (Clause 9)

If your company is preparing a 510(k), the company may be able to submit the 510(k) immediately after completion of risk control effectiveness verification. You may also be able to postpone the benefit/risk analysis until you submit your IVD technical file for CE Marking approval. The benefit/risk analysis is not required by ISO 14971 unless the risks are unacceptable, and the FDA does not require a benefit/risk analysis except for novel devices seeking market authorization through a De Novo Classification Request or a Pre-Market Approval (PMA). The FDA also does not require the submission of the complete risk management file.

IVDR risk management requirements are quite different than the US FDA requirements for risk management. An IVD technical file must include a risk management summary report that summarizes all activities that were performed according to the risk management plan. A benefit/risk analysis is required for each risk and the overall risk. The Notified Body auditor is also expected to sample the complete risk management file during quality system audits. Finally, the IVDR includes a requirement for a post-market surveillance plan that includes the collection of production and post-production data as feedback on the risk management process and a post-market clinical performance follow-up (PMPF) plan. 

What production and post-production information should you be collecting for IVD devices?

Medical device manufacturers struggle to see the benefits of requiring a post-market surveillance system, and smaller companies, in particular, complain that the cost of the new European post-market surveillance requirements is excessive and prohibits innovation. However, the primary role of post-market surveillance is to ensure rapid initiation of containment and corrective actions for devices that malfunction and/or present unacceptable risks to the intended users and intended patient population. The purpose of generating the post-market surveillance data is defined in the IVDR within Article 78, Section 3.

The minimum requirements for post-market surveillance are defined in Annex III, Section 1(a):

  • Information concerning serious incidents, including information from PSURs, and field safety corrective actions;
  • records referring to non-serious incidents and data on any undesirable side-effects;
  • information from trend reporting;
  • relevant specialist or technical literature, databases and/or registers;
  • information, including feedback and complaints, provided by users, distributors, and importers; and
  • publicly-available information about similar medical devices.

The IVDR is not prescriptive regarding what production data shall be collected for post-market surveillance, but the reason for this is that there are many different types of manufacturing processes with different process risks. In addition, the IVDR includes software as a medical device where there is no manufacturing process at all. Therefore, the best approach for determining what production data to collect is the review your process risk analysis (e.g. pFMEA). The process risk analysis for each manufacturing process should allow you to identify the manufacturing process steps that have the greatest residual risks (e.g. risk priority number or RPN) and potentially the highest severity of the effect. The risks should be identified as a priority for post-market surveillance. You should also include process parameter monitoring data for any validated processes (e.g. sterilization time, temperature, and pressure). Finally, you should also monitor rejects at incoming inspection, in-process inspection, and final inspection operations.  

Other IVD Risk Management Resources

The following resources may be helpful for creating and maintaining your IVD risk management file:

  1. EN ISO 14971:2019 + A11:2021
  2. ISO/TR 24971:2020
  3. GHTF/SG3/N15R8
  4. Regulation (EU) 2017/746 (i.e. IVDR)
  5. IEC/TR 80002-1:2009
  6. EN 62366-1:2015 + A1:2020

Note: Whenever possible, hyperlinks to the Estonian Centre for Standardization and Accreditation (EVS) are provided for procedures, because we find that this source is frequently the least expensive, and digital versions are available on-demand as a multi-user license.

Posted in: ISO 14971:2019 (Risk Management), IVDR - Regulation (EU) 2017/746

Leave a Comment (0) →

How to create an IVDR checklist

This article provides an IVDR checklist for updating your ISO 13485 quality system to comply with EU Regulation 2017/746.

IVD Checklist 1024x474 How to create an IVDR checklist

Why I created an IVDR checklist?

Hundreds (if not thousands) of IVD manufacturers are currently updating their ISO 13485:2016 certified quality system from compliance with the In Vitro Diagnostic Directive (i.e. Directive 98/79/EC) or IVDD to the new EU In Vitro Diagnostic Regulation (i.e. Regulation 2017/746). Revision of technical files and the associated procedures for creating your technical files is a big part of these updates. However, there is much more that needs to be updated than just the technical documentation. Therefore, IVD manufacturers are asking Medical Device Academy to conduct remote internal audits of their quality system to identify any gaps. Usually, we conduct internal audits using the process approach to auditing, but this is one of the scenarios where the element approach and an audit checklist are invaluable.

If you would like to download our IVDR checklist for FREE, please fill in the form below.

How do you use an audit checklist?

An audit checklist is used by quality system auditors to collect objective evidence during an audit. This objective evidence verifies compliance with regulatory requirements or internal procedural requirements. If the auditor is unable to find supporting evidence of compliance, the auditor may continue to search for data or identify the requirement as a nonconformity. Typically the checklist is in four columns using a tabular form. The left-hand column lists each requirement. The next column is where the auditor documents records sampled, procedures reviewed, and personnel interviewed. In the third column, the auditor indicates what they were looking for in the records, procedures, or during the interview. Some of the information in the second and third columns can often be entered prior to starting the audit by reviewing audit preparation documents (e.g. procedures and previous audit reports). In the fourth column the auditor will enter the objective evidence for conformity collected during the audit.

How to create an IVDR quality plan

Most of the companies that are preparing for an IVDR audit by their notified body already have ISO 13485:2016 certification and they are using the self-declaration pathway for CE Marking under the IVDD. Under the IVDR, a notified body must now review and approve the technical file. The notified body must also confirm that their quality system has been updated to include the IVDR requirements. The Technical File requirements are found in Annex II and III; while most of the quality system requirements are found in the Articles.  The quality system requirements include:

  1. a risk management process in accordance with Annex I – deviations from ISO 14971:2019 will be necessary)
  2. conduct a performance evaluation–including a post-market performance follow-up (PMPF). This requirement is defined in Article 52 and Annex XIII
  3. create and maintain a technical file in accordance with Annex II & III
  4. create and maintain a Declaration of Conformity in accordance with Article 17
  5. CE Mark the product in accordance with Article 18
  6. implement a UDI system in accordance with Article 24, 26, and 28
  7. record retention requirements for the technical file, Declaration of Conformity, and certificates shall be increased from 5 years to 10 years
  8. set-up, implement, and maintain a post-market surveillance system in accordance with Article 78
  9. document a procedure for communication with Competent Authorities, Notified Bodies, Economic Operators, Customers, and/or other Stakeholders
  10. update procedures for reporting of serious incidents and field safety corrective actions in the context of vigilance to require reporting within 15 calendar days
  11. update the product labeling to comply with Annex I, section 20
  12. revise the translation procedure to ensure translations of the instructions for use are available in all required languages of the member states, and make sure these translations are available on the company website
  13. create a procedure for utilization of the Eudamed database for registration, CE Marking applications, UDI data entry, and vigilance reporting

Which IVDR requirements are already met by your quality system?

Some companies also manufacture medical devices that must comply with Regulation (EU) 2017/745. For those companies, many of the above requirements are already incorporated into their quality system. In this case, you should still include all of the IVDR checklist requirements in your plan, but you should indicate that the requirement has already been met and audited previously.

Content related to our IVDR checklist

On Friday, April 1, 2022 @ 11 am EDT (8 am Pacific), Rob Packard will be Joe Hage’s guest speaker on the weekly MDG Premium Live video (please click on the link to register). The topic of the live presentation will be “How to create an IVDR quality plan.” #MedicalDevices #MDGpremium

Posted in: CE Marking, In Vitro Diagnostic (IVD) Devices, ISO 13485:2016, ISO Auditing, ISO Certification, IVDR - Regulation (EU) 2017/746

Leave a Comment (3) →

What are the four types of risk analysis?

You are familiar with design and process risk analysis, but do you know all four types of risk analysis?

Last week’s YouTube live streaming video answered the question, “What are the four different types of risk analysis?” Everyone in the medical device industry is familiar with ISO 14971:2019 as the standard for medical device risk management, but most of us are only familiar with two or three ways to analyze risks. Most people immediately think that this is going to be a tutorial about four different tools for risk management (e.g. FMEA, Fault Tree Analysis, HAZOP, HACCP, etc.). Instead, this article is describing the four different quality system processes that need risk analysis.

What are the four different types?

The one most people are familiar with is risk analysis associated with the design of a medical device. Do you know what the other three are? The second type is process risk management where you document your risk estimation in a process risk analysis. The third type is part of the medical device software development process, specifically a software hazard analysis. Finally, the fourth type is a Use-Related Risk Analysis (URRA) which is part of your usability engineering and human factors testing. Each type of risk analysis requires different information and there are reasons why you should not combine these into one risk management document or template.

Design Risk Analysis

Design risk analysis is the first type of risk analysis we are reviewing in this article. The most common types of design risk analysis are the design failure modes and effects analysis (dFMEA) and the fault-tree analysis (FTA). The dFMEA is referred to as a bottom-up method because you being by identifying all of the possible failure modes for each component of the medical device and you work your way backward to the resulting effects of each failure mode. In contrast, the FTA is a top-down approach, because you begin with the resulting failure and work your way down to each of the potential causes of the failure. The dFMEA is typically preferred by engineers on a development team because they designed each of the components. However, during a complaint investigation, the FTA is preferred, because you will be informed of the alleged failure of the device by the complainant, but you need to investigate the complaint to determine the cause of the failure. Regardless of which risk analysis tool is used for estimating design risks, the risk management process requires that production and post-production risks be monitored. Therefore, the dFMEA or the FTA will need to be reviewed and updated as post-market data is gathered. If a change to the risk analysis is required, it may also be necessary to update the instructions for use to include new warnings or precautions to prevent use errors.

Process Risk Analysis

Process risk analysis is the second type of risk analysis. The purpose of process risk analysis is to minimize the risk of devices being manufactured incorrectly. The most common method of analyzing risks is to use a process failure modes and effects analysis (i.e. pFMEA). This method is referred to as a bottom-up method because you begin by identifying all of the possible failure modes for each manufacturing process step. Next, the effects of the process failure are identified. After you identify the effects of failure for each process step, the severity of harm is estimated. Then the probability of occurrence of harm is estimated, and the ability to detect the failure is estimated. Each of the three estimates (i.e. Severity, Occurrence, and Detectability) are multiplied to calculate a risk priority number (RPN). The resulting RPN is used to prioritize the development of risk controls for each process step.

As risk controls are implemented, the occurrence and detectability scores estimated again. This is usually where people end the pFMEA process, but to complete one cycle of the pFMEA the risk management team should document the verification of the effectiveness of the risk controls implemented. For example, if the step of the process is sterilization then documentation of effectiveness consists of a sterilization validation report. This is the last step of one cycle in the pFMEA, but the risk management process includes monitoring production and post-production risks. Therefore, as new process failures occur the pFMEA is reviewed to determine if any adjustments are needed in the estimates for severity, occurrence, or detectability. If any of the risks increase, then additional risk controls may be necessary. This process is continuously updated with production and post-production information to ensure that process risks remain acceptable.

Software Hazard Analysis

Sofware hazard analysis is becoming more important to medical devices as physical devices are integrated with hospital information systems and with the development of software as a medical device (SaMD). Software risk analysis is typically referred to as hazard analysis because it is unnecessary to estimate the probability of occurrence of harm. Instead, it is only necessary to identify hazards and estimate harm. Examples of these hazards include loss of communication, mix-up of data, loss of data, etc. For guidance on software hazard identification, IEC/TR 80002-1:2009 is a resource. FDA software validation guidance indicates that software failures are systemic in nature and the probability of occurrence cannot be determined using traditional statistical methods. Therefore, the FDA recommends that you assume that the failure will occur and estimate software risks based on the severity of the hazard resulting from the failure. 

Use-Related Risk Analysis

The fourth and final type of risk analysis is use-related risk analysis (URRA). Most development teams assume that they are able to use traditional hazard identification techniques to identify the potential use-related risks. However, use-related risks are inextricably linked to the experiences of the user. The development team has unique knowledge of the device they are developing, and therefore it is likely that use-related risks associated with a lack of knowledge about the device will result in use errors that the development team would not realize. For this reason, formative testing is necessary to identify unforeseen use-related risks. Once formative testing identifies these risks, additional formative usability testing can be used to create and refine the instructions for the use of a medical device. Finally, formative testing can be used to develop user training programs that prevent potential use errors. Once the development team has completed the necessary formative testing, then summative usability testing is used to validate the effectiveness of the risk controls that were implemented.
 
In the past, I believed that the FDA’s focus on usability was the review of summative usability testing. However, I have learned that the FDA feels it is equally important to begin the human factors testing process by first performing a use-related risk analysis and then identifying the critical tasks. Without identifying these critical tasks, it is not possible for the FDA to determine if the moderator of the summative testing has observed all of the critical tasks being performed correctly. An example of a Use-Related Risk Analysis (URRA) was provided by the FDA in a 510(k) AI deficiency letter that we received. The example is provided below.
URRA table example from the FDA 300x117 What are the four types of risk analysis? Example of a URRA Table provided by the FDA
 

Can you use only the IFU to prevent use-related risks?

Instructions for use (IFU) are required to include warnings and precautions. This information provided by the manufacturer explains how to use a medical device correctly and identifies the residual risks. This is a form of risk control, but it is the least effective form of risk control and should be the risk control of last resort. Not everyone reads the IFU, and you cannot guarantee that everyone will understand the instructions. You certainly can’t be sure that users will remember all your warnings or precautions when they are tired, stressed, or acting in an emergency situation. Design controls and protective measures should be implemented as the first and second priority for risk controls, and the IFU should be your lowest priority.
 
This is the reason why we have color-coding, design features that eliminate the possibility of a use error, we provide training to users, and we are required to monitor use-related risks for medical devices. Formative usability testing is intended to identify use errors we did not anticipate, to help us develop instructions for use (IFU), and help us develop training for users. Summative testing is intended to validate that the design, training, and IFU are effective at preventing use errors. All three of these aspects work together–not the IFU alone. In fact, there is an entire alarms standard that identifies protective measures that shall be used for electromedical devices to prevent use errors (i.e. – IEC 60601-1-8).

 

Facilitating Risk Management Activities – An Interview with Rick Stockton

I listened to our YouTube video about the four different types of risk analysis, you may have heard my reference to Rick Stockton’s interview that we posted on our YouTube channel and embedded above. In our interview with Rick Stockton, we discussed how to facilitate risk management activities during the design and development of medical devices. If you are interested in learning more about Rick and facilitating risk management activities, please watch the video of our interview with Rick.
 

Posted in: ISO 14971:2019 (Risk Management)

Leave a Comment (1) →

Medical Device Shortage Reporting

The FDA and Health Canada both have executive-level orders requiring medical device shortage reporting or supply-chain disruptions.

In a previous article, we discussed supply-chain disruptions and mentioned that there might be medical device shortage reporting requirements if that disruption causes a market shortage of the manufactured device. Both the United States and Canada have reporting requirements for supply disruptions or the market’s ability to meet the demand of specific types of devices.

Both the U.S. FDA and Health Canada have executive-level orders that require reporting of shortages or disruptions to the supply of medical devices deemed necessary for the COVID-19 Health Emergency. There is some overlap, but each country is monitoring and experiencing shortages and disruptions of different devices.

Where did medical device shortage reporting responsibilities come from?

Check 21 CFR 820, ISO 13485:2016, and even peek at SOR 98-282 and see if you can find your obligations for reporting. Go ahead. I’ll wait… Not much in there, right? Adverse events, complaints, etc., but not market shortages.
Medical device shortage reporting is specific to health emergencies. The U.S. FDA and Health Canada happen to be two authorities having jurisdiction with reporting requirements for shortages concerning the COVID-19 Health Emergency. However, there may be others, so having your organization’s regulatory affairs manager verify the reporting requirements for the markets in which you are engaged might not be bad.

U.S. FDA 506J reporting-

fda logo Medical Device Shortage Reporting
U.S. FDA logo

In the United States, an Amendment to the U.S. Food, Drug, and Cosmetics Act requires regulatory reporting by medical device manufacturers to the U.S. FDA. It is sometimes called 506J reporting for the Section of the U.S. FD&C Act where it is located.

You will find the statutory requirements outlined within 21 USC 356J.

21 USC 356j screenshot from uscode.house .gov cropped title Medical Device Shortage Reporting
21 USC 356J Discontinuance or interruption in the production of medical devices

For the full text read, 21 USC 356j: Discontinuance or interruption in the production of medical devices. (Interestingly enough, the website where this information is available is not an HTTPS site, so visit at your own discretion).

http://uscode.house.gov/browse.xhtml

What devices are subject to 506J reporting?

There are two types of devices that the FDA is monitoring. “Critical” devices and an FDA-published list of devices for which COVID-19 is causing a higher than expected demand.

The FDA has released a guidance document that contains criteria for what is considered to be a “Critical Device”. This includes devices such as those used during surgery, emergency medical care, and those intended to treat, diagnose, prevent, or mitigate COVID-19.

fda guidance criteria for 506j critical devices Medical Device Shortage Reporting
Screenshot of the Critical Device Criteria for 506J reporting

There is also a published list of concerned devices that the FDA is specifically monitoring. The FDA website lists these devices by product code, but include the following device types;

  • Clinical Chemistry Products
  • Dialysis-Related Products
  • General ICU/Hospital Products
  • Hematology Products
  • Infusion Pumps and Related Accessories
  • Microbiology Products
  • Needles and Syringes
  • Personal Protective Equipment (PPE)
  • Sterilization Products
  • Testing Supplies and Equipment
  • Ventilation-Related Products
  • Vital Sign Monitoring
fda 506j shortage list screenshot Medical Device Shortage Reporting
Screenshot of the FDA Shortage List

Understandably this process may not be intuitive, and for this, the FDA has released a guidance document that addresses;

  • Who must make the notification
  • When you should make a notification
  • What information needs to be included within your 506J notification
  • How to make a notification, and
  • Penalties for failure to make a notification

The referenced product codes may not be an all-inclusive list or entirely up to date. The best suggestion for full compliance is to go straight to the source of the regulation, in part because noncompliance can result in enforcement action from the FDA. If you think that your device might require notification to the FDA but isn’t in the reference table, you should contact the FDA for notification clarification. Below is the quote from the FDA website, and it includes the contact email for asking these specific questions to ‘the agency.’

“If a device type is not included in this table, but you believe it requires a notification under section 506J of the FD&C Act, or if you have questions regarding the device types in this table, you should contact FDA at CDRHManufacturerShortage@fda.hhs.gov and include “Question” in the subject line of the email.”

Link to the FDA Guidance Document for 506J Reporting- HERE

How to make a 506J report to the U.S. FDA?

The FDA accepts 506J reports in multiple ways. For example, you may use the 506J Reporting web form or submit a notification by email directly to (Include Email Here). In addition, Medical Device Academy has developed a Work Instruction and Form to determine if your company is experiencing a reportable discontinuance or meaningful disruption in manufacturing a medical device as well as compiling the report for submission.

There are a few methods of notification, a web form for individual notifications and spreadsheet options for multiple notifications at once, or emailing a report directly to the FDA reporting email included below;

CDRHManufacturerShortage@fda.hhs.gov

fda 506j webform screenshot Medical Device Shortage Reporting
Screenshot of the FDA 506J reporting Webforms from https://fdaprod.force.com/shortages

It is for this process that Medical Device Academy developed WI-010 506J Shortage Reporting to the U.S. FDA. This work instruction and associated form, FRM-053 506J Reporting Form are designed to walk you through the process of determining reportability and compiling the information necessary to either complete the webform or email the report directly to the shortage reporting email.

Medical Device Shortage Reporting to Health Canada

health canada logo sante canada 1024x224 1 Medical Device Shortage Reporting
Health Canada logo

Rather than discontinuance and disruption of manufacture, Health Canada is monitoring for shortages of specific devices. Therefore, Health Canada wants Medical Device Shortage Reports regardless of the reason for the shortage. It also shows that this is not identical reporting of the same conditions to two different authorities. Health Canada will also accept reports from Importers because the frame of reference is Canada’s supply of medical devices concerning Canada’s needs.

As an Authority Having Jurisdiction, Health Canada also has reporting requirements for medical device shortage reporting of specific types of medical devices. Health Canada is also an independent authority that uses a different device classification system than the U.S. FDA.

The table below shows the device types by their classification level that HC requires supply chain disruption notifications for. This information is current as of September 5th, 2021, and the link below will take you to the HC website page for the most up-to-date list.

https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/shortages/covid19-mandatory-reporting.html

Class I Medical Devices
Masks (surgical, procedure or medical masks) – Level 1, 2, 3 (ATSM)
N95 respirators for medical use
KN95 respirators for medical use
Face shields
Gowns (isolation or surgical gowns) – Level 2, 3 and 4
Gowns (chemotherapy gowns)
Class II Medical Devices
Ventilators (including bi-level positive airway pressure or BiPAP machines, and continuous positive airway pressure or CPAP machines)
Infrared thermometers
Digital thermometers
Oxygen Concentrators
Pulse Oximeters (single measurement)
Aspirators/suction pumps (portable and stationary)
Laryngoscopes
Endotracheal tubes
Manual resuscitation bags (individually or part of a kit)
Medical Gloves – Examination and Surgical (Nitrile, Vinyl)
Oxygen Delivery Devices
Class III Medical Devices
Ventilators (including bi-level positive airway pressure or BiPAP machines)
Pulse Oximeters (continuous monitoring)
Vital Signs Monitors
Dialyzers
Infusion Pumps
Anesthesia Delivery Devices
Class IV Medical Devices
Extracorporeal Membrane Oxygenation (ECMO) Devices
List of ‘Specified Devices’ that Health Canada is monitoring for shortage reporting

One of the things that Health Canada does an excellent job of is defining its expectations. In the Second Interim Order Respecting Drugs, Medical Devices and Foods for a Special Dietary Purpose in Relation to COVID-19, it is explained the Manufacturers or Importers should report to the Minister actual or expected shortages of the device, OR components, accessories, or parts. These notifications must be made within 5-days of becoming aware of the shortage or the anticipated shortage date. Update reports must be made within 2-days of becoming aware of new information regarding the shortage, and a closing report must be made within 2-days of the end of the shortage.

(This link is to the HC website for the 2nd Interim Order referenced above)

https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/drug-medical-device-food-shortages/interim-order-2021.html

How to make a shortage report to Health Canada?

These reports are submitted online through the Health Canada Website. They have an entire section dedicated to medical device shortages, and the reporting links can be found there (Link here). If you have any questions or are on the fence about notification, you can email Health Canada at MD.shortages-penurie.de.IM@canada.ca.

Inkedhc reporting shortages overview screenshot edited LI 1024x384 Medical Device Shortage Reporting
Health Canada Webforms for reporting a shortage and the end of a shortage

The webform for reporting a shortage is the same webform that is used for providing update reports to Health Canada as well. This is both for manufacturers of specified medical devices as well as importers.

Posted in: FDA, Health Canada

Leave a Comment (0) →

Feedback options for your pre-sub meeting request

This article analyzes feedback options offered for a pre-submission meeting request and gives you insight into which option is best for you.

Pre submission meeting request feedback options Feedback options for your pre sub meeting request

In 2021 the FDA published an updated guidance document about pre-submission meeting requests (i.e., pre-sub meetings). The purpose of a pre-submission meeting is to ask and obtain answers to your questions directly from the FDA. The guidance document has great advice on what to ask the FDA and what you should not ask. The best time to be asking the FDA questions is before you begin your verification and validation testing. The FDA can give you valuable feedback on your testing plan to demonstrate safety and efficacy, but if you already started your testing it’s too late. Unfortunately, the guidance document has no advice on which method of feedback to select or why.

What is the last section of your pre-sub?

The last section of your pre-submission meeting request should indicate what method of feedback you prefer and what your preferred dates are for a potential meeting with the FDA. There are three options offered for methods of feedback:

  1. a face-to-face meeting
  2. a conference call
  3. an email response

Feedback option 1 – A Face-to-Face Meeting

Some executives believe that face-to-face meetings are critical in establishing relationships with people. However, you need to understand the culture of the people your are trying to build a relationship with. The FDA is an overworked bureaucracy, and government agencies have security concerns. When the FDA meets with visitors they must go to a different building and arrange for their guests to pass through security. This is more work and takes more time. To justify the extra work and time, you need a compelling reason why a face-to-face meeting with the FDA is necessary.

Traveling to the FDA will cost your team money and time that conference calls and emails will not. More importantly, you are limited to one hour for a pre-submission meeting. One hour is barely enough time to ask questions and listen to the answers. You only have minutes to introduce your company, your team and the describe the product. There is no time for relationship building. The best way to impress the FDA is to: 1) prepare thoroughly, 2) conduct an efficient meeting, and 3) ask smart questions.

There is one time when you should visit the FDA face-to-face–if you have a powerful demonstration and video just isn’t good enough.

Feedback option 2 – Conference Call

Conference calls save you time and money, but conference calls also save the FDA time and effort. You won’t personally meet people from the agency, but you can communicate information prior to the meeting and you can provide videos of simulated use for your device. Conference calls do have the advantage of allowing you to mute the call for a moment and make a comment among your team members without the agency listening as well. Whenever you are discussing a performance testing plan or a clinical study protocol with the FDA, you will probably want a conference call to enable clarification questions.

Feedback option 3 – Email

Email responses from the FDA are highly underrated in value. When you specify an email response, you generally receive a response to your questions sooner. You also should receive more information, because each person from the agency is able to provide an hour of their time to write detailed feedback. In a conference call, you are speaking for part of the hour and only one person from the FDA can speak at a time. Therefore, you almost always have less feedback during conference calls and face-to-face meetings. The primary downside to email as a feedback method is that it is not interactive.

Update Related to Covid-19 Pandemic

The FDA is not allowing face-to-face meetings during the Covid-19 pandemic. Three of the pre-subs Medical Device Academy submitted during the pandemic were rejected by the FDA due to insufficient FDA resources. We are also noticing increased delays in the pre-sub timeline. Two pre-subs had a 5-month scheduling lead-time instead of 60-75 days. Due to these delays, we have advised many clients to skip the presub if testing requirements are well defined in guidance documents and predicate 510k summaries. Althought the email option should theoretically result in a faster response from the FDA, during the pandemic we have actually seen that the teleconference options has been faster. My theory is that the teleconferences are require coordinating the schedules of multiple people, and therefore there is more focus by lead reviewers in making sure the feedback is provided in time for the scheduled teleconference. 

Which feedback option will you pick?

Regardless of which feedback method you choose, you can always follow-up with supplemental questions and obtain additional feedback from the FDA after you receive the initial response to your pre-submission meeting request. If you are planning a clinical study, you might seek interactive feedback in a conference call during the pre-submission meeting. Then you can follow-up with a clinical study protocol as a supplement to obtain additional feedback from the FDA.

Additional Resources

If you are interested in learning more about a pre-submission meeting request to the FDA, consider watching and listening to a webinar on the topic.

Posted in: 510(k)

Leave a Comment (1) →
Page 1 of 29 12345...»