This article explains the requirements for a benefit-risk analysis as defined in ISO 14971:2019, Clause 7.4 and in the EU regulations.
What is a benefit-risk analysis?
A benefit-risk analysis is one of the risk management activities explained in ISO 14971:2019. Specifically, this requirement is found in clause 7.4 of the medical device risk management standard. Originally, the requirement was described as “risk-benefit analysis” in the second edition of the medical device risk management standard. The US FDA revised their policies for novel devices (e.g., De Novo and PMA submissions) to emphasize that novel devices must demonstrate clinical benefits or they will not be approved. Therefore, the US FDA revised the wording to place the word “benefit” before the word “risk.” This approach and the revised wording was adopted by the committee that was drafting the third edition of the ISO 14971 standard. The wording was also adopted by 2012 European version of the standard, the EU MDR, and EU IVDR. In general, this risk management activity involves a semi-quantitative comparison of clinical benefits with risks of harm. The ISO 14971 standard indicates that if risks are unacceptable, a device can still be recommended for commercial release by a design team if the clinical benefits outweigh the risk of harm.
Is there discretion as to whether a benefit-risk analysis needs to take place?
The ISO 14971 Standard implies that a benefit-risk analysis is only required if the risks of harm exceed a threshold of acceptability. In the ISO/TIR 24971:2020 guidance, the committee clarified that acceptability of risk must be documented in a risk management policy (see Annex C2 for guidance and recommended content for a risk management policy). However, the EU MDR and IVDR regulations require that you perform a benefit-risk analysis for each individual risk and overall residual risk of a medical device. This is stated in Annex I, the General Safety and Performance Requirements. Therefore, if your company distributes devices only in the USA that are Class 1 or Class 2, and the submission type is not a De Novo or Humanitarian Device Exemption (HDE), then you are only required to perform a benefit-risk analysis if the risks of harm are unacceptable. If the device requires a De Novo application, and HDE, or a Class 3 PMA, then you are required to submit a benefit-risk analysis to the FDA in your premarket submission. For companies that distribute devices in Europe, the companies do not have discretion with regard to performing a benefit-risk analysis and they must include it in the risk management file. Since some of Medical Device Academy’s clients are seeking approval for a De Novo, HDE, or PMA, or the companies are distributing in the EU, our risk management procedure does not allow discretion regarding whether a benefit-risk analysis needs to be performed. The template we created for this is TMP-034 in SYS-010.
As Low As Reasonably Practicable (ALARP)
Your company may have a risk management procedure which includes a matrix for severity and probability. The matrix is probably color-coded to identify red cells as unacceptable risks that require a benefit-risk analysis, yellow cells that are ALARP, and green cells that are acceptable. This practice was criticized in 2012 by the European Commission. “Acceptability” of risk is no longer permitted using the principles of “ALARP.”
The EU regulations require that the analysis of benefit-risk ratios be performed for each risk and all residual risks—not just the risks you identify as unacceptable. However, the EU regulations also do not permit that financial considerations be used as part of the determination of risk acceptability. Financial considerations are implied in the ALARP principle. To clarify this, notes were added to ISO 14971:2019, the guidance on risk acceptability was moved to ISO/TIR 24971:2020, and the concept of ALARP was removed from the risk management standard and the guidance. Therefore, we recommend that your risk management policy reference the need for a benefit-risk analysis, regulatory requirements, the requirements of recognized medical device standards, and stakeholder requirements–not ALARP.
Integrating benefit-risk analysis into your design process
The best way to integrate benefit-risk considerations into your design process is by performing a clinical evaluation. In addition to using clinical literature, clinical study data, and post-market surveillance as inputs for your clinical evaluation, your company should also be using residual risks as inputs to the evaluation. The clinical evaluation should be used to assess the significance of these residual risks, and verify that there are not any risks identified in the clinical evaluation that were not considered in the risk analysis.
In order to document that your company has performed a benefit-risk analysis for each residual risk, you will need to reference the risk management report in the clinical evaluation and vice-versa. Both documents will need to provide traceability to each risk identified in the risk analysis, and conclusions of risk acceptability will need to be based upon the conclusions of the clinical evaluation.
Once your device is commercialized, you will need to update the clinical evaluation with adverse events and other post-market surveillance information. As part of updating clinical evaluations, you will need to determine the acceptability of the risk when weighed against the clinical benefits. These conclusions will then need to be updated in the risk management report—including any new or revised risks.
Anyone can sign and date training records, but how do you demonstrate the effectiveness of training and competence?
What are the requirements for training?
The requirements for training are found in ISO 13485:2016, Clause 6.2 (i.e., Human Resources). Auditors and inspectors usually only ask for training records, but the requirement for records is the last item in the clause (i.e., “6.2(e) – maintain appropriate records of education, training, skills, and experience (see 4.2.5).”). The first four items in Clause 6.2 all include one word, but it’s not “records.” The most critical word in the requirements is “competence.”
What is the difference between training requirements, training records, training effectiveness, and training competence?
Training requirements define the “appropriate education, training, skills, and experience.” Specifically, what degree(s) is required, if any? What training curriculum needs to be completed? What skills are needed to perform the work? And how many years of experience are needed?
Training records are documented evidence that all training requirements have been met. Records may include a job application, resume, individual training records, group training records, or a training certificate. Records may also include the results of any quizzes to demonstrate training effectiveness and any evaluation of training competence.
Training effectiveness is how well a person understands the information communicated during training. Using terms from human factors engineering (i.e., the PCA Process), the person must have the correct perception and cognition. Cognition can be evaluated by giving people quizzes or asking them questions. Written and verbal exams may also include pictures and/or video. Performing these types of tasks on a quiz or exam are “knowledge tasks.”
Training competence is possessing the skill or ability to perform tasks. To demonstrate competence, someone that is already competent (i.e., a subject matter expert) must observe the trainee performing tasks. A person that once possessed the skill or ability can also judge competency, and a person that does not possess the skill or ability can be trained to evaluate skill and ability (e.g., an Olympic Judge or referee). Skills and abilities can be physical or cognitive, but there are also social skills. Examples of each are provided in the table below:
What keeps me awake at night? (a story from April 2, 2011)
I am in Canada, it’s almost midnight, and I can’t sleep. I’m here to teach a Canadian client about ISO 14971—the ISO Standard for Risk Management of medical devices. Most companies requesting this training are doing so for one of two reasons: 1) some design engineers have no risk management training, or 2) the engineers have previous training on risk management, but the training is out-of-date. This Canadian company falls into the second category, and engineers with previous training ask the most challenging questions. This group of engineers forced me to re-read the Standard several times and reflect on the nuances of almost every single phrase. While teaching this risk management course, I learned more about risk management than I ever knew.
The four levels of the Learning Pyramid
The image at the beginning of this blog is a learning model that explains my experiences training Canadian design engineers. I call this model the “Learning Pyramid.” At the base of the pyramid, there are “Newbies.”
This is the first of four levels. At the base, students read policies and procedures, hoping to understand.
The student is now asked to watch someone else demonstrate proper procedures in the second level of the pyramid. One of my former colleagues has a saying that explains the purpose of this process well, “A picture tells a thousand words, but a demonstration is like a thousand pictures.” Our children call this “sharing time,” but everyone over 50 remembers this as “show and tell.”
The student is now asked to perform their tasks in the third level of the pyramid. This is described as “doing,” but in my auditing courses, I refer to this process as “shadowing.” Trainees will first read the procedures for Internal Auditing (level 1). Next, trainees will shadow the trainer during an audit to demonstrate the proper technique (level 2). During subsequent audits, the trainees will audit, and the trainer will shadow the trainee (level 3). During this “doing” phase, the trainer must watch, listen, and wait for what I call the “teachable moment.” This is a moment when the trainee makes a mistake, and you can use this mistake as an opportunity to demonstrate a complex subject.
Finally, in the fourth level of the Learning Pyramid, we now allow the trainee to become a trainer. This is where I am at. I am an instructor, but I am still learning. I am learning what I don’t know.
Teaching forces you back to the bottom of the Learning Pyramid
After teaching, the next step in the learning process is to return to the first level. I re-read the Standards and procedures until I understood the nuances I was unaware of. Then, I search for examples in the real world that demonstrate the complex concepts I am learning. After searching for examples, I tested my knowledge by applying the newly acquired knowledge to an FDA 510(k) or De Novo submission for a medical device client. Finally, I prepared to teach again. This reiterative process reminds me of the game Chutes and Ladders, but one key difference is that we never really reach the level of “Guru.” We continue to improve but never reach our goal of perfection.
Where is training competence in the Learning Pyramid?
Most people feel that a person is competent in a position when they can consistently perform a task. However, the ISO 13485 standard uses the phrase “necessary competence” to suggest that competency for any given task might not be the same for all people. For example, in some cases, it may be sufficient to perform a task with written instructions in front of you, while an instructor might be expected to perform the task without written instructions. The speed at which a person performs a task might also differ. For example, a secretary might be required to touch type at a speed of 60+ wpm with a QWERTY keyboard, while a stenographer must be able to use a STENO keyboard and write at 225 wpm. The accuracy requirements may differ for those two positions as well. Therefore, your company may decide that the training competence requirement for a design engineer is that they can pass an exam on risk management. However, the training competency requirement for the design project lead or Engineering Manager might include teaching inexperienced design engineers to apply the basic risk management principles. Demonstrating the ability to teach inexperienced design engineers might be demonstrated by an auditor interviewing members of your design team.
What do you look at and look for when you are auditing risk management files to ISO 14971 and the new Regulation (EU) 2017/745?
Your cart is empty
Next week, November 15th @ Noon EST, you will have the opportunity to watch a live webinar teaching you what to look at and what to look for when you are auditing risk management files to Regulation (EU) 2017/745 and ISO 14971. Risk Management Files are one of the essential requirements of technical documentation required for CE Marking of medical devices. Most quality system auditors are trained on how to audit to ISO 13485:2016 (or an earlier version of that standard), but very few quality system auditors have the training necessary to audit risk management files.
Why you are not qualified to audit risk management files
Being a qualified lead auditor is not enough to audit the risk management process. When you are auditing a risk management file, you need risk management training and lead auditor training. To audit the risk management process, you will also need training on applicable guidance documents (i.e., ISO/TR 24971:2020) and applicable regulations (i.e., Regulation 2017/745 and/or Regulation 2017/746). There may also be device-specific guidance documents that specify known risks and risk controls that are considered state-of-the-art.
Creating an audit agenda
Once you have scheduled an audit of risk management files, and assigned a lead auditor, then the lead auditor needs to create an audit agenda. The audit can be a desktop audit that is performed remotely, or it can be an on-site audit. Regardless of the approach, the audit should include interviewing participants in the risk management process documented in the risk management file. As a rule of thumb, I expect a minimum of 30 minutes to be spent interviewing the process owner and one or more other participants. Then I spend an additional 60 minutes of auditing time reviewing documents and records.
Your audit agenda should specify the following items at a minimum:
the method of auditing to be used,
date(s) of the audit,
the duration of the audit,
the location of the audit, and
the auditing criteria.
The auditor(s) and the auditee participants should be identified in the audit agenda. Finally, you should specify which documents and records are required for audit preparation. These documents will be used to help identify audit checklist questions and to determine a sampling plan for the audit. At a minimum, you will need a copy of the risk management procedure and a list of the risk management files that are available to audit. You may also want to request the audit plan for each of those risk management files.
What did you look at and look for during your risk management audit?
When you audit the risk management process, you could take any of the following approaches or a combination of more than one. You could audit the process according to the risk management procedure. You could audit the process according to the risk management plan(s) for each risk management file. You could audit using the process approach to auditing. Finally, you could audit in accordance with specific requirements in the ISO 14971:2019 standard and applicable regulations (i.e., Regulation 2017/745). Regardless of which approach you take, your audit notes and the audit report should identify which documents and records you sampled and what you looked for in each document. Providing only a list of the documents is not enough detail.
Creating an auditing checklist for risk management files
Auditors with limited experience are taught to create an audit checklist by creating a table that includes each of the requirements of the audit criteria. For a risk management file, this would include a list of each of the requirements in ISO 14971 for a risk management file (i.e., Clause 9???). However, this approach is more like the approach that you should be using for a gap analysis. The better approach for creating an audit checklist for risk management files is to start by creating a turtle diagram. In the “process inputs” section (i.e., step 2 of 7), you would add questions derived from your review of the risk management plan(s). In the “process outputs” section (i.e., step 3 of 7), you would add questions specific to the risk management report and other records required in a risk management file. In the “with whom” section (i.e., step 5 of 7), you would add questions related to training and competency. You might also identify additional people involved in the risk management process, other than the process owner, to interview as a follow-up trail. In the “how done” section (i.e., step 6 of 7), you would add questions specific to the procedure and forms used for the risk management process. Finally, in the “metrics” section (i.e., 7 of 7), you would verify that the company is conducting risk management reviews and updating risk management documentation in accordance with the risk management procedure and individual risk management plan(s).
Audits are just samples
Just because you can generate a lot of questions for an audit checklist does not mean that you are required to address every question. Audits are intended to be a “spot check” to verify the effectiveness of a process. You should allocate your auditing resources based on the importance of a process and the results of previous audits. I recommend approximately three days for a full quality system audit, and approximately 90-minutes should be devoted to a process unless it is the design control process (i.e., Clause 7.3 of ISO 13485) which typically requires three to four hours due to the importance and complexity of the design controls process. Therefore, you should schedule approximately 30 minutes to interview people for the risk management process and approximately 60 minutes should be reserved for reviewing documents and records. With this limited amount of time, you will not be able to review every record or interview everyone that was involved in the risk management process. This is why auditors always remind auditees that an audit is just a sampling.
Which records are required in a risk management file?
The contents of a risk management file is specified in ISO 14971:2019, Clause 4.5. There are only four bullets in that section, but the preceding sentence says, “In addition to the requirements of other clauses of this document.” Therefore, your risk management file should address all of the requirements in ISO 14971:2019. What I recommend is a virtual risk management folder for each risk management file. As the auditor, you should also request a copy of the risk management policy and procedure. An example of what this would look like is provided below. The numbers in front of each subfolder correspond to the sub-clause or clause for that requirement in ISO 14971:2019.
Which records are most valuable when auditing risk management files?
As an auditor, I typically focus on three types of targets when auditing any process. First, I will sample any corrective actions implemented in response to previous audit findings. Second, will sample documents and records associated with any changes made to the process. Changes would also include any changes that were made to individual risk management files or the creation of a new risk management file. Finally, my third target for audit sampling is any item that I feel is at risk for safety or performance failures. The severity of the safety or performance failure is also considered when prioritizing audit sampling. In the context of a risk management file, I always verify that production and post-production activities are being conducted as planned. I try to verify that risk analysis documentation was reviewed for the need to update the documentation in response to complaints and adverse events.
More auditor training on risk management files
We are recording a live webinar intended to teach internal auditors and consultants how to perform a thorough audit of risk management files against the requirements of the new European Regulation (EU) 2017/745 and ISO 14971.
Auditing Risk Management Files
In this new webinar, you will learn how to conduct a process audit of risk management files. You will learn what to look at and what to look for in order to verify compliance with Regulation (EU) 2017/745 and ISO 14971:2019. The webinar will be approximately one hour in duration. Attendees will be invited to participate in the live webinar and receive a copy of the native slide deck. Anyone purchasing after the live event will receive a link to download the recording of the live event and the native slide deck.
Price: $64.50
In addition to this webinar on auditing risk management files, we also have other risk management training webinars available. The webinar on auditing risk management files will be hosted live on November 15, 2022 @ Noon EST (incorrect in the live video announcement).
ISO 14971:2019 includes a requirement for top management to define and document a riskmanagement policy, but do you have one?
Your risk management procedure is not your risk management policy
ISO 14971:2019 includes a requirement for a risk management policy and a risk management procedure. The word procedure is defined (Clause 3.13), a “specified way to carry out an activity or a process,” but there is no definition for policy. Both of these words begin with the letter “p,” but they are not the same. There is no guidance for a risk management policy in either of the European device regulations for CE Marking and there is no guidance in the US FDA’s regulations. In fact, there is not even a specific cause of the international risk management standard that is specific to the requirement for a risk management policy. The word “policy” only appears in ISO 14971 seven times, but the last occurrence provides the best explanation:
Appendix A2.4.2 states that “because [ISO 14971] does not define acceptable risk levels, top management is required to establish a policy on how acceptable risks will be determined.”
If someone responsible for risk management activities does not understand this distinction, this shows that risk management training may not be adequate.
Can you have a different policy for each product family?
The purpose of the policy is to establish how the acceptability of risks will be determined. However, not all devices have the same benefit-risk ratio. Therefore, if you have product families with high and low risks, then you should address this in your policy with specific criteria for each device family or create a separate risk management policy for each product family. For example, if your company is focused on designing and developing products for diabetics, you will not have the same benefit-risk profile for a Class 2 glucose reader and lancet for Type 2 diabetics that you have for an automated Class 3 insulin pumps for Type 1 diabetics. In general, separate criteria within one policy are preferred over separate policies to reduce the number of documents that must be managed.
Is there a required format for a risk management policy?
The ISO 14971:2019 standard does not include a specific format or content requirement for your risk management policy. Instead, information about the format and content of a risk management policy is provided in Annex C of ISO/TR 24971:2020. This is a guidance document, and therefore you can choose an alternate approach if you provide a justification for its equivalence. If you choose the approach recommended in Annex C, the following elements should be included:
purpose;
scope;
factors and considerations for determining acceptable risk;
approaches to risk control;
requirements for approval and review.
You can download Medical Device Academy’s template for a risk management policy (POL-005) by completing the form below.
What are the factors for determining acceptable risk?
There are four possible factors to consider when determining your risk management policy:
Applicable regulatory requirements;
Relevant international standards;
State-of-the-Art;
And stakeholder concerns.
An example of regulatory requirements being applied to the determination of acceptable risks is the special controls defined in 21 CFR 880.5730 for insulin pumps. The special controls requirements outlined by the FDA specify design inputs as well as verification and validation requirements. The requirements are also organized into systems that comprise an insulin pump. For the digital interface requirements, the regulation specifies:
secure pairing to external devices;
secure data communication between the pump and connected devices;
sharing of state information between devices;
ensuring the pump continues to operate safely when receiving data that is outside of the boundary limits that are specified as inputs;
a detailed process and procedure for sharing pump interface specifications with connected devices.
The hazard implied by the fourth requirement above is that the pump will stop without warning or deliver the incorrect amount of insulin if the data from a continuous glucose sensor is outside of the input specifications. This design input is then addressed by a software design specification established by your company. To verify that your software risk controls are adequate, you will need to execute a verification protocol that automatically inputs a series of values that are outside of the boundary limits specified. Every time a change is made to the software, these boundary limits will need to be re-verified as part of your automated regression analysis to make sure software changes did not have an unintended effect on the device.
For software and use-related hazards, you will not be able to estimate the probability of occurrence of harm. Therefore, you shall assess the acceptability of risks based upon the severity of harm alone. Risk acceptability criteria shall be recorded in your risk management plan and the criteria shall align with your risk management policy. Ideally, these criteria are based upon international standards. For the example of an interoperable insulin pump, the following international standards are applicable:
ISO 14971, application of risk management to medical devices
IEC 62366-1, application of usability engineering to medical devices
IEC 62304, medical device software – software lifecycle processes
For the state-of-the-art, there are three examples provided in the ISO/TR 24971 guidance for how to this relates to your risk management policy:
“Leakage currents of the medical device are state of the art, demonstrated by compliance to the limits and tests regarding leakage current of IEC 60601-1.
Dose accuracy of the delivery device are state of the art, as demonstrated by compliance to the limits and tests regarding dose accuracy of ISO 11608-1.
Protection against mechanical failure caused by impact is on the same level as or better than a similar medical device, as demonstrated by comparative test such as drop test.”
Stake holder concerns is the fourth factor to consider when creating your risk management policy. Stakeholder concerns may be identified in clinical literature. However, the current trend is an emphasis on patient-reported outcome (PRO) data and post-market surveillance. Post-market surveillance is a requirement in ISO 13485, Clause 8.2.1. However, the new European MDR and IVDR have new requirements for post-market surveillance data in the technical documentation. Health Canada updated the medical device regulations to include post-market surveillance summary reports, and even the FDA is trying to develop methods for using real-world data and real-world evidence to make regulatory decisions.
Approaches to risk acceptability
The European device regulations require that a benefit/risk analysis be conducted for all risks and the overall residual risk of your device. The EU regulations also do not permit risk acceptability to consider economic impact. The EU regulations also require that risks are reduced as far as possible. Therefore, if your company is seeking CE Marking, there is only one acceptable approach suggested in ISO/TR 24971, Annex C.2: “reducing risk as far as possible without adversely affecting the benefit-risk ratio.” This is also the approach specified in our risk management procedure (SYS-010).
Requirements for review and approval of the risk policy
Requirements for approval and review of the risk management policy should be specified in the policy itself. This should specify who needs to approve that the policy is acceptable and how often the policy needs to be reviewed. Section 4.2.2 of ISO 14971 also requires that top management review the risk management process for its effectiveness. In general, we recommend that this review of the risk management process be incorporated into the management review process. Therefore, we also believe that this would be the ideal time to review the risk management policy. Generally, this is more frequently than is typically required, but if your risk management process is being reviewed for effectiveness then you have all of the necessary inputs available to review the policy as well.
This article reviews unique IVDR risk management requirements for CE Marking of in vitro diagnostic (IVD) devices in Europe.
Last week I posted a blog about “How to create an IVDR checklist.” The article was very popular because we included a form for downloading a free IVDR checklist. That form included the opportunity for people to ask a question about the IVDR. One of the subscribers, a gentleman from New Zealand, entered a very simple comment: “risk management requirements.” My first thought was that the risk management file is the required technical documentation for the IVDR. Then I quickly remembered that in 2012, EN ISO 14971:2012 was released with three new annexes for the three directives: ZA (for the MDD), ZB (for the AIMD), and ZC (for the IVDD). In Annex ZC there were seven deviations, and even though ISO 14971 was updated in 2019, the international standard continues to deviate from the European regulations in significant ways. Therefore, this blog provides an overview of the IVDR risk management requirements.
If you are already compliant with ISO 14971:2019, do you meet the IVDR risk management requirements?
The biggest difference between the ISO 14971:2019 standard and the IVDR risk management requirements is that the standard only requires a benefit-risk analysis to be performed if risks are unacceptable. In contrast, the IVDR requires that a benefit/risk analysis be performed for all risks and the overall residual risk. Therefore, you must include a benefit/risk analysis in your technical file submission regardless of risk acceptability. The harmonized version of the standard (i.e. EN ISO 14971:2019/A:11:2021) was released in December of 2021. If you already purchased ISO 14971:2019, you only need to purchase the amendment which consists of Annex ZA (comparison between the standard and Annex I of MDR) and ZB (comparison between the standard and Annex I of the IVDR).
In the amendment, it states that manufacturers must have a risk management policy that is compliant with Annex I of the EU regulation. There are notes at the beginning of each harmonization annex that indicates that the risk management process needs to be compliant with the IVDR, which means risks have to be ‘reduced as far as possible’, ‘reduced to a level as low as reasonably practicable’, ‘reduced to the lowest possible level’, ‘reduced as far as possible and appropriate’, ‘removed or reduced as far as possible’, ‘eliminated or reduced as far as possible’, ‘prevented’ or ‘minimized’, according to the wording of the corresponding section in Annex I of the IVDR. The comparison table has a column with remarks/notes. In most cases, the deficiency identified states, “Device-specific execution of the process is not covered.” There are also two remarks/notes that state “Device-specific and usability-specific execution of the process is not covered.”
Where are the IVDR risk management requirements?
IVDR risk management requirements are found in Annex II, Section 5 of the IVDR. However, there are 228 references to the word risk throughout the IVDR. The following risk-related requirements in IVDR are particularly important:
Annex I, Section 3 – reiteration of risk management procedure requirement, but specific steps in the risk management process are identified (e.g. a risk management plan, hazard identification, estimation of risks, evaluation of risks, etc.)
Annex I, Section 4 – Priority of risk control measures
Annex I, Section 5 – Elimination or reduction of use-related risks
Annex III, Section 1(b) – Reassessment of the benefit-risk analysis and risk management using post-market surveillance data
How should you document your risk management file for the IVDR?
In your risk management file of course. There is no format requirement for risk management files, but there are requirements for the content and there is a GHTF guidance document for risk management, and ISO/TR 24971:2020 is a new guidance document on the application of ISO 14971 to medical devices. Neither of these guidance documents is specific to IVDR risk management requirements. Annex H of ISO/TR 24971:2020, however, provides guidance specific to IVD devices.
What do the IVDR risk management requirements include for risk analysis?
Of these four types of risk analysis, only the software hazard analysis is sometimes not applicable. For an FDA 510(k) submission, you would need to provide software hazard analysis and URRA in the actual submission. The other two types of risk analysis would only be included in your design history file (DHF), and the FDA would review the design and process risk analysis during a routine inspection when the DHF is sampled as part of the design control process.
In contrast, the IVDR requires that a complete risk management file be submitted as part of the technical file (see Annex II, Section 5):
“Benefit-risk analysis and risk management
The documentation shall contain information on:
the benefit-risk analysis referred to in Sections 1 and 8 of Annex I, and
the solution adopted and the results of the risk management referred to in Section 3 of Annex I.”
The above documentation typically consists of design risk analysis and does not typically include process risk analysis, software hazard analysis, or use-related risk analysis. These other three risk analysis documents are IVDR risk management requirements, but they are referenced by the technical file in other sections. The most obvious IVDR risk management requirements are referenced in Annex I, Sections 1-9. These are referred to as the General Safety and Performance Requirements (GSPRs), and this requirement is typically met by including a GSPR checklist in the technical file to meet the requirement of Annex II, Section 4.
The process risk analysis is typically included with manufacturing information to meet the requirement of Annex II, Section 3.2. This documentation may include, any and all of the following elements:
a process failure mode and effects analysis (pFMEA)
a risk control plan including all processes from receiving inspection to final inspection and product release
a process validation plan that is risk-based and linked to the risk control plan
The best practice for estimation of process risks is to link the probability of occurrence and probability of detection to the quantitative data gathered during process validation. In addition, you may establish a risk management policy that prescribes specific types of process risk controls (e.g. automated inspection) for the highest risk processes where manufacturing process errors are not acceptable residual risks. For example, an inspection of printed circuit board assemblies (PCBAs) typically requires automated optical inspection (AOI) methods, because visual inspection is not sufficient by itself and not all PCBAs allow sufficient ICT coverage, and functional testing is limited.
The software hazard analysis, if applicable, is typically performed in accordance with IEC/TR 80002-1:2009, Guidance on the application of ISO 14971 to medical device software. In the software hazard analysis, it is unnecessary to estimate the probability of occurrence of harm. Instead, it is only necessary to identify hazards and estimate harm. Examples of these hazards include loss of communication, mix-up of data, loss of data, etc. Software failures are systemic in nature and the probability of occurrence cannot be determined using traditional statistical methods. Therefore, we recommend that you assume that the failure will occur and estimate software risks based on the severity of the hazard resulting from the failure. For these reasons, it is recommended that software hazard analysis documentation is maintained as a separate document from your design risk analysis. The software hazard analysis documentation should be referenced in your risk management report, but the software hazard analysis should be included as part of your software verification and validation. The IVDR requires that you include a summary of software verification and validation in Annex II, Section 6.4 rather than the complete hazard analysis document.
A use-related risk analysis should be part of your useability engineering file for IVD devices as required by EN 62366-1:2015. Use-related risks are mentioned in Annex I, Section 5:
“In eliminating or reducing risks related to use error, the manufacturer shall:
reduce as far as possible the risks related to ergonomic features of the device and the environment in which the device is intended to be used (design for patient safety), and
Give consideration to the technical knowledge, experience, education, training and use environment, where applicable, and the medical and physical conditions of intended users (design for lay, and professional, disabled or other users).”
The above requirement includes not only the ability to read and interpret test results of IVD devices but also the ability of laypersons to properly self-select if an IVD is intended to be sold as an over-the-counter product. Usability also is mentioned in Article 78, Section 3(f):
“for the identification of options to improve the usability, performance and safety of the device;”
Therefore, there should be specific elements of your post-market surveillance plan that are designed to gather feedback on the usability of your IVD device.
When should risk management activities be performed for IVD devices?
The IVDR does not specifically define when in the design and development process the various risk management activities shall be performed. However, the required risk management activities are specified in the IVDR within Annex I. ISO 14971:2019, however, is more descriptive of the risk management activities and the risk management process. Therefore, your risk management plan should align with the process defined in ISO 14971:2019, Clause 4.1.
Unfortunately, most companies do not include risk management as an integral part of the design and development process. Instead, risk management documentation is created retroactively as part of the documentation preparation for technical file submission. For this reason, most medical device executives fail to see the benefit associated with the risk management process. Even biomedical engineers struggle to appreciate the necessity of following the process outlined in the risk management standard in order to prevent device malfunctions and use errors.
The following is a list of the required risk management activities in the order that they should be occurring. Each activity also references the applicable clause of ISO 14971:2019. We have also grouped the activities into the five phases of design and development:
Design Planning
Risk management planning (Clause 4.4)
Design Inputs
Identification of hazards and hazardous situations (Clause 5.4)
Design & Development
Risk estimation (Clause 5.5)
Risk evaluation (Clause 6)
Risk control option analysis (Clause 7.1)
Implementation of risk control measures (Clause 7.2)
Residual risk evaluation (Clause 7.3)
Design Verification and Validation
Benefit/risk analysis (Clause 7.4)
Risk control effectiveness verification (Clause 7.6)
Design Release
Evaluation of overall residual risk (Clause 8)
Risk management review (Clause 9)
If your company is preparing a 510(k), the company may be able to submit the 510(k) immediately after completion of risk control effectiveness verification. You may also be able to postpone the benefit/risk analysis until you submit your IVD technical file for CE Marking approval. The benefit/risk analysis is not required by ISO 14971 unless the risks are unacceptable, and the FDA does not require a benefit/risk analysis except for novel devices seeking market authorization through a De Novo Classification Request or a Pre-Market Approval (PMA). The FDA also does not require the submission of the complete risk management file.
IVDR risk management requirements are quite different than the US FDA requirements for risk management. An IVD technical file must include a risk management summary report that summarizes all activities that were performed according to the risk management plan. A benefit/risk analysis is required for each risk and the overall risk. The Notified Body auditor is also expected to sample the complete risk management file during quality system audits. Finally, the IVDR includes a requirement for a post-market surveillance plan that includes the collection of production and post-production data as feedback on the risk management process and a post-market clinical performance follow-up (PMPF) plan.
What production and post-production information should you be collecting for IVD devices?
Medical device manufacturers struggle to see the benefits of requiring a post-market surveillance system, and smaller companies, in particular, complain that the cost of the new European post-market surveillance requirements is excessive and prohibits innovation. However, the primary role of post-market surveillance is to ensure rapid initiation of containment and corrective actions for devices that malfunction and/or present unacceptable risks to the intended users and intended patient population. The purpose of generating the post-market surveillance data is defined in the IVDR within Article 78, Section 3.
Information concerning serious incidents, including information from PSURs, and field safety corrective actions;
records referring to non-serious incidents and data on any undesirable side-effects;
information from trend reporting;
relevant specialist or technical literature, databases and/or registers;
information, including feedback and complaints, provided by users, distributors, and importers; and
publicly-available information about similar medical devices.
The IVDR is not prescriptive regarding what production data shall be collected for post-market surveillance, but the reason for this is that there are many different types of manufacturing processes with different process risks. In addition, the IVDR includes software as a medical device where there is no manufacturing process at all. Therefore, the best approach for determining what production data to collect is the review your process risk analysis (e.g. pFMEA). The process risk analysis for each manufacturing process should allow you to identify the manufacturing process steps that have the greatest residual risks (e.g. risk priority number or RPN) and potentially the highest severity of the effect. The risks should be identified as a priority for post-market surveillance. You should also include process parameter monitoring data for any validated processes (e.g. sterilization time, temperature, and pressure). Finally, you should also monitor rejects at incoming inspection, in-process inspection, and final inspection operations.
Other IVD Risk Management Resources
The following resources may be helpful for creating and maintaining your IVD risk management file:
Note: Whenever possible, hyperlinks to the Estonian Centre for Standardization and Accreditation (EVS) are provided for procedures, because we find that this source is frequently the least expensive, and digital versions are available on-demand as a multi-user license.
You are familiar with design and process risk analysis, but do you know all four types of risk analysis?
Last week’s YouTube live streaming video answered the question, “What are the four different types of risk analysis?” Everyone in the medical device industry is familiar with ISO 14971:2019 as the standard for medical device risk management, but most of us are only familiar with two or three ways to analyze risks. Most people immediately think that this is going to be a tutorial about four different tools for risk management (e.g. FMEA, Fault Tree Analysis, HAZOP, HACCP, etc.). Instead, this article is describing the four different quality system processes that need risk analysis.
What are the four different types?
The one most people are familiar with is risk analysis associated with the design of a medical device. Do you know what the other three are? The second type is process risk management where you document your risk estimation in a process risk analysis. The third type is part of the medical device software development process, specifically a software hazard analysis. Finally, the fourth type is a Use-Related Risk Analysis (URRA) which is part of your usability engineering and human factors testing. Each type of risk analysis requires different information and there are reasons why you should not combine these into one risk management document or template.
Design Risk Analysis
Design risk analysis is the first type of risk analysis we are reviewing in this article. The most common types of design risk analysis are the design failure modes and effects analysis (dFMEA) and the fault-tree analysis (FTA). The dFMEA is referred to as a bottom-up method because you being by identifying all of the possible failure modes for each component of the medical device and you work your way backward to the resulting effects of each failure mode. In contrast, the FTA is a top-down approach, because you begin with the resulting failure and work your way down to each of the potential causes of the failure. The dFMEA is typically preferred by engineers on a development team because they designed each of the components. However, during a complaint investigation, the FTA is preferred, because you will be informed of the alleged failure of the device by the complainant, but you need to investigate the complaint to determine the cause of the failure. Regardless of which risk analysis tool is used for estimating design risks, the risk management process requires that production and post-production risks be monitored. Therefore, the dFMEA or the FTA will need to be reviewed and updated as post-market data is gathered. If a change to the risk analysis is required, it may also be necessary to update the instructions for use to include new warnings or precautions to prevent use errors.
Process Risk Analysis
Process risk analysis is the second type of risk analysis. The purpose of process risk analysis is to minimize the risk of devices being manufactured incorrectly. The most common method of analyzing risks is to use a process failure modes and effects analysis (i.e. pFMEA). This method is referred to as a bottom-up method because you begin by identifying all of the possible failure modes for each manufacturing process step. Next, the effects of the process failure are identified. After you identify the effects of failure for each process step, the severity of harm is estimated. Then the probability of occurrence of harm is estimated, and the ability to detect the failure is estimated. Each of the three estimates (i.e. Severity, Occurrence, and Detectability) are multiplied to calculate a risk priority number (RPN). The resulting RPN is used to prioritize the development of risk controls for each process step.
As risk controls are implemented, the occurrence and detectability scores estimated again. This is usually where people end the pFMEA process, but to complete one cycle of the pFMEA the risk management team should document the verification of the effectiveness of the risk controls implemented. For example, if the step of the process is sterilization then documentation of effectiveness consists of a sterilization validation report. This is the last step of one cycle in the pFMEA, but the risk management process includes monitoring production and post-production risks. Therefore, as new process failures occur the pFMEA is reviewed to determine if any adjustments are needed in the estimates for severity, occurrence, or detectability. If any of the risks increase, then additional risk controls may be necessary. This process is continuously updated with production and post-production information to ensure that process risks remain acceptable.
Software Hazard Analysis
Sofware hazard analysis is becoming more important to medical devices as physical devices are integrated with hospital information systems and with the development of software as a medical device (SaMD). Software risk analysis is typically referred to as hazard analysis because it is unnecessary to estimate the probability of occurrence of harm. Instead, it is only necessary to identify hazards and estimate harm. Examples of these hazards include loss of communication, mix-up of data, loss of data, etc. For guidance on software hazard identification, IEC/TR 80002-1:2009 is a resource. FDA software validation guidance indicates that software failures are systemic in nature and the probability of occurrence cannot be determined using traditional statistical methods. Therefore, the FDA recommends that you assume that the failure will occur and estimate software risks based on the severity of the hazard resulting from the failure.
Use-Related Risk Analysis
The fourth and final type of risk analysis is use-related risk analysis (URRA). Most development teams assume that they are able to use traditional hazard identification techniques to identify the potential use-related risks. However, use-related risks are inextricably linked to the experiences of the user. The development team has unique knowledge of the device they are developing, and therefore it is likely that use-related risks associated with a lack of knowledge about the device will result in use errors that the development team would not realize. For this reason, formative testing is necessary to identify unforeseen use-related risks. Once formative testing identifies these risks, additional formative usability testing can be used to create and refine the instructions for the use of a medical device. Finally, formative testing can be used to develop user training programs that prevent potential use errors. Once the development team has completed the necessary formative testing, then summative usability testing is used to validate the effectiveness of the risk controls that were implemented.
In the past, I believed that the FDA’s focus on usability was the review of summative usability testing. However, I have learned that the FDA feels it is equally important to begin the human factors testing process by first performing a use-related risk analysis and then identifying the critical tasks. Without identifying these critical tasks, it is not possible for the FDA to determine if the moderator of the summative testing has observed all of the critical tasks being performed correctly. An example of a Use-Related Risk Analysis (URRA) was provided by the FDA in a 510(k) AI deficiency letter that we received. The example is provided below.
Example of a URRA Table provided by the FDA
Can you use only the IFU to prevent use-related risks?
Instructions for use (IFU) are required to include warnings and precautions. This information provided by the manufacturer explains how to use a medical device correctly and identifies the residual risks. This is a form of risk control, but it is the least effective form of risk control and should be the risk control of last resort. Not everyone reads the IFU, and you cannot guarantee that everyone will understand the instructions. You certainly can’t be sure that users will remember all your warnings or precautions when they are tired, stressed, or acting in an emergency situation. Design controls and protective measures should be implemented as the first and second priority for risk controls, and the IFU should be your lowest priority.
This is the reason why we have color-coding, design features that eliminate the possibility of a use error, we provide training to users, and we are required to monitor use-related risks for medical devices. Formative usability testing is intended to identify use errors we did not anticipate, to help us develop instructions for use (IFU), and help us develop training for users. Summative testing is intended to validate that the design, training, and IFU are effective at preventing use errors. All three of these aspects work together–not the IFU alone. In fact, there is an entire alarms standard that identifies protective measures that shall be used for electromedical devices to prevent use errors (i.e. – IEC 60601-1-8).
Facilitating Risk Management Activities – An Interview with Rick Stockton
I listened to our YouTube video about the four different types of risk analysis, you may have heard my reference to Rick Stockton’s interview that we posted on our YouTube channel and embedded above. In our interview with Rick Stockton, we discussed how to facilitate risk management activities during the design and development of medical devices. If you are interested in learning more about Rick and facilitating risk management activities, please watch the video of our interview with Rick.
This blog discusses why contract manufacturers need to have a strong risk management process, and your company needs to help your contract manufacturers. This article was updated on April 28, 2022, and the original publication was January 05, 2011. Please ignore the date of publication at the top of the post for articles that are more than a year old.
Can contract manufacturers exclude risk management from the scope of their quality system?
Most contract manufacturers in the medical device industry exclude design from their Quality Management Systems. Unfortunately, most of the contract manufacturers also associate risk management with only the design process. Risk Management cannot be “not applicable” in anISO 13485Quality Management System. The requirement of section 7.1 is: “The organization shall establish documented requirements for risk management throughout product realization. Records arising from risk management shall be maintained.” The Standard also referencesISO 14971as a source of guidance on Risk Management.
Have you experienced an audit dialogue at a contract manufacturer similar to this?
The auditor asks, “How do you manage risk throughout the production process?” Then the auditee responds, “That is the responsibility of our customers. We will prepare a risk analysis if customers pay for it, but usually, customers do the risk analysis.”
For a contract manufacturer, compliance with ISO 14971 is not my primary concern as an auditor. My primary concern is to verify that contract manufacturers analyze risks associated with the processes that they perform and do their best to minimize those risks. What I don’t understand is why more companies don’t want to have strong risk management processes. Risk management is how we prevent bad things from happening. Bad stuff like scrap, complaints, and recalls. Should we expect our suppliers to have a strong risk management process?
Duh.
Why your company needs to be involved in the risk management process?
Contract manufacturers should be doing everything they can to get better at risk management. During pre-production planning, they should be asking, “What happens if…” The contract manufacturer knows best HOW things will fail in production, while the customer knows best WHAT happens when things fail in production. To be safe and effective, both companies need to collaborate on risk analysis.
In any risk analysis, you need to estimate the severity of potential harm and the probability of occurrence of that harm. For production defects, the contract manufacturer can estimate the probability of occurrence of defects (i.e., P1 in Annex E of ISO 14971:2007), but the likelihood of occurrence of harm is less. The probability of occurrence of harm is the product of multiplying P1 and P2. The probability that occurrence will result in harm is P2, and P2 is a number that is less than 100% or 1. Your company can gather pre-market clinical data and post-market clinical data to estimate P2, but before launching your product, you can only guess at the value of P2. Your contract manufacturer, however, is not able to estimate P2 at all. It’s ok to estimate risk without P2 during the design phase because this will overestimate risks and result in more conservative decisions.
In addition to P2, your contract manufacturer is also not capable of estimating the severity of potential harm. As the designer of the medical device, you will know best how your device is used and what the likely clinical outcomes are when a device malfunctions. There may even be multiple possible clinical outcomes. The contract manufacturer knows what can go wrong during manufacturing, but you will need to define the clinical outcomes due to malfunctions.
Why do contract manufacturers avoid doing risk analysis?
The reason contract manufacturers avoid doing risk analysis is because it’s time-consuming and tedious.
Too bad, so sad.
Balancing my checkbook is time-consuming and tedious too, but I balance my checkbook to prevent an overdraft charge. Not doing a risk analysis can be much more painful. Scrapping out a part can cost tens or hundreds of dollars. Complaints can cost thousands of dollars. Recalls can cost millions of dollars.
If I owned a contract manufacturing company, I would ensure that everyone in the company is involved in risk management. We don’t want scrap, we can’t afford mistakes that lead to complaints, and a recall could put us out of business.
How Medical Device Academy Can Help?
Medical device academy can help both the contract manufacturer and the specification developer utilizing a contract manufacturer as a supplier! We offer training on 14971:2019 as well as procedures on risk management and supplier quality management.
Two-part Risk Management Training Webinar for ISO 14971:2019 – Part 1 of this webinar will be presented live on Tuesday, March 29 @ 9-10:30 am EDT. Part 2 of this webinar series will be presented live on Tuesday, April 5 at 9-10:30 am EDT. Purchase of this webinar series will grant the customer access to both live webinars. They will also receive the native slide decks and recording for the two webinars.
The 3rd edition of the risk management standard for medical devices, ISO 14971:2019, was released on December 16, 2019.
ISO 14971:2019 – Risk Management Standard, 3rd edition
In October of 2018, I wrote a blog on the draft version of ISO 14971:2019 for risk management of medical devices. That article explained the differences between the different versions of the ISO 14971 standard (i.e., 2000, 2007, 2009, and 2012). I also explained what changed between ISO 14971:2007 and ISO/DIS 14971:2018. The final 2019 version of ISO 14971 3rd edition is now available.
The changes proposed in the draft included subtle changes to the names of the processes and a minor adjustment to the numbering of the clauses. Many of the annexes were also moved to ISO/TR 24971 guidance–which was released in 2020. The draft did not, however, result in a change in the overall process of risk management.
All of the changes that were discussed in my 2018 review were maintained in the final 2019 version that was released, but the ISO/TR 24971 guidance was not released at the same time as the committee had hoped for.
There are not any surprises in the 3rd edition (i.e., ISO 14971:2019). Therefore, I plan to wait until the ISO/TR 24971 guidance is released and then prepare a new blog specific to the guidance. If you are interested in training on the ISO 14971:2019 standard, the training I recorded on October 19, 2019, provides an excellent overview of these changes and highlights some of the challenges that you will encounter when trying to harmonize your risk management procedure between the ISO 14971:2019 standard and Regulation (EU) 2017/745.
This is a lot of information to absorb. Therefore, I recommend purchasing the October 2019 webinar and your copy of the ISO 14971 standard from AAMI. Anyone that has already purchased either the webinar or the procedure will receive an email offering them a discount on this new bundle that credits them for their previous purchase. If you have purchased both, you will receive credits for both purchases. Just think you can watch the video and read the new version of the standard while you are working out at the gym in January. Learn and burn!
This article explains what a pFMEA is (i.e. Process Failure Modes and Effects Analysis) and how to use them as part of your risk management process.
I recently had someone ask for help understanding the Process Failure Mode and Effect Analysis (pFMEA) a little better. I can’t blame them, because I was lost the first time I tried to fill out a form for one. It can be confusing and overwhelming if you have never created one before.
First things first, what is a pFMEA
FMEA= Failure Modes and Effects Analysis
A lower-case letter will come before the FMEA, and that denotes the ‘what’, of what the failure is that is being analyzed. A pFMEA will often be examining process failures where a dFMEA might evaluate design failures.
Some systems capitalize all the letters. Some capitalize none. That is not what is important as long as it is consistent throughout your system. Everyone should be able to easily understand that whatever variation of pfmea is used; it means “process failure modes and effects analysis.”
What does a pFMEA do?
A pFMEA will break down your manufacturing process into its individual steps and methodically examine them for potential risks or failures. For companies that utilize ourTurn-Key Quality Management System, FRM-025 process Failure Modes, and Effects Analysis can be used as a template.
For this example, we will look at receiving inspection of injection-molded casing parts for a medical device. This receiving inspection includes a manual inspection of 10 randomly selected parts out of each delivery of 100 using an optical overlay.
Process Step
This area, as the section title suggests, is the process step. When looking at the process as a whole, the pFMEA will break it down into each and every step included in that process. This area is simply that individual step that is going to be examined.
The Process Step or item function depending on what your form uses for this scenario, is going to be part of the random sampling for manual inspection of the received parts using an optical overlay. Our example is going to be the backlighting element of the optical overlay display. The backlighting element will illuminate the inspected part against the template to verify that the part is within specific dimension criteria.
Potential Failure Modes
A failure mode is a way in which that process step might fail. Since it is failure modeS, it needs to be considered that there may be more than one way for the process step to fail. Do not be fooled that because this box on the form has been filled in that the pFMEA will be complete. A thorough examination of all of the possible failures should be investigated.
Our example in this process requires the backlighting element to illuminate a visual template over the parts. The light not illuminating properly is a potential failure mode of this process.
Potential Effects of Failure
the potential effects of the failure is a look into what the ramifications would be if that failure for that process step actually happened.
In our scenario, one of the potential effects of the lighting not functioning properly is that parts outside of the designated sizing acceptance criteria may be accepted rather than rejected as non-conforming parts.
S (Severity)
The next area is the first area that requires an estimated grading of the failure. That is ‘Severity’ which is abbreviated as S. There is a scale provided in the rating section of FRM-025 that outlines the numbering system that Medical Device Academy uses.
In this case, our example is using molded plastic pieces of the outside casing of a medical device. Pieces that are too large or too small will not fit when making the final assembly of the device. These plastic pieces do not happen to be patient contacting, and do not affect the function of the device.
The evaluation of this failure is determined to have no potential effect on patient safety or increase any potential for risk of harm, therefore the severity is assigned as a ‘business risk’ meaning that it bears no risk for the user or the patient. This makes the Severity Score 0.
Causes of Failure
This column is exactly that. What might cause this identified failure to happen? In our example might be the light bulbs in the overlay machine may slowly burn out over time with use. This burnout causes potential failure.
If the bulb is expected to only have a lifetime of 100 hours, then the more hours the bulb is used, the dimmer the light may become. A slowly dimming light decreases the sharpness of the overlay template and our parts that are supposed to have a + or – size criteria of 10% now have a fuzzy template that in reality changes the overlay to show closer to + or – 13%. Now parts that are too small or too large may be accepted.
O (Occurrence/Probability)
This grading criterion is also found in the Rating section of FRM-025. This is how often the failure is expected to occur. How often will the lighting element of our optical overlay fail to function in the appropriate manner for this cause?
Hopefully not very often. In fact, regularly scheduled maintenance and calibration of the overlay machine could prevent this from ever happening in the best-case scenario. Our evaluations determine that the probability of this happening is low. However, since we cannot be certain it will never happen the potential for this risk exists and makes the Occurrence score a 4.
Current Process Controls
What is currently being done to control this risk? Our example uses regularly scheduled maintenance and calibration to prevent bulb burnout affecting the overlay.
D (Detectability)
Our current process is based on routine maintenance and visual inspection. This means that the bulb burnout is something that is visually inspected for and visual inspections for detectability on the rating scale are graded as 8. This chart is found in the Rating Section of FRM-025.
RPN (Risk Priority Number)
This is a number that is found by multiplying the Severity, by the Probability, by the Detectability. In our example, the numbers RPN is 0X4X8=32 for an RPN of 32 which is considered LOW.
pFMEA math
Below is a short video explaining the math behind calculating the Risk Priority Number
What if anything can be done to improve this process? In our example, a recommended action may be to transfer from visual only inspections to verification of light output by the meter. This makes the Detectability of the failure measurable by meter or gage which is a detectability score of 4.
This changes the RPN now to 0X4X4=16
The pFMEA shouldn’t be a solo thing
If it can be avoided this type of analysis should be done by a multidisciplinary team. Sometimes in smaller companies, people end up having to wear more than one hat. There are many entrepreneurs that have to function as the CEO/CFO/Design Engineer/RA/QA manager.
Ideally, a team approach should be used if feasible. Have the management level staff who have ownership of the processes participating in this analysis. They should know the process more intimately than anyone else in the company and should have more insight into the possible failure modes of the processes as they have likely seen them first hand. They are also the type of employee who would know the types of recommended actions to control the risk of those failures as well.
The pFMEA should also be a living document
As new failure modes are discovered they should be added to your pFMEA. A new failure mode might be discovered through a CAPA because the process had an actual failure that was not originally analyzed. Take an instance like that as an opportunity for improvement and to update your pFMEA as part of a living breathing risk management system. Also, use this as a time to re-brainstorm potentially similar failure modes that may not have been considered previously so that they can be controlled before they happen.
If you took the time to watch the video above it is also mentioned that in some instances the very first FMEA must be based on estimates because there is no data. Managers and engineers may be forced to estimate the probability of occurrence. If that is the case the FMEA should be updated in the future to adjust the (O) score to reflect what is occurring in actuality based on real data and not the theoretical data that was used for the initial estimate.
This article describes updates being made to the ISO 14971 Standard in the new draft version released for comment in July 2018.
There are two versions of ISO 14971 that are currently available. The first is the international version: ISO 14971:2007. The second is the European normative version: EN ISO 14971:2012. There is also a new draft being created by the TC210 committee for release in 2019.
Explanation of the different versions of the ISO 14971 standard
In 2000, the first edition of ISO 14971 was released as the international standard for risk management of medical devices. In 2007, the second edition of ISO 14971 was released. When new international standards are released, a European normative version is also released. The “European Norm” or EN version is intended to identify any gaps between the international standard and the requirements of the applicable European directives (i.e., the MDD, AIMD, and the IVDD). These gaps historically were included in the ZA annex at the end of the EN version. However, in 2009 this annex was split into three annexes (i.e., ZA, ZB, and ZC) to address each of the three directives separately. In reality, the 2009 annex only differed concerning the directive referenced. In 2012, a new EN version was released. This new standard included seven deviations, which were controversial. These deviations were intended to identify contradictions between the directives and the international standard, but the interpretations were not agreed with by companies or most of the Notified Bodies. Ultimately, the seven deviations were required to be addressed in the risk management files for any medical device that was CE Marked.
What changed between ISO 14971:2007 and ISO/DIS 14971:2018?
The TC210 working group assigned to update the ISO 14971 standard (JWG1) was tasked with improving guidance for the implementation of ISO 14971. Still, the committee was also tasked with making these improvements without changing the risk management process. Also, the committee was asked to move the informative annexes at the end of ISO 14971 from the standard to the guidance document ISO/TR 24971. Therefore, in July, the committee released a draft for comment and voting. Draft versions are identified with the prefix “ISO/DIS.” The ISO/DIS 14971 standard released in July has only three annexes: A) Rationale for the requirements, B) Risk management process for medical devices, and C) Fundamental risk concepts (formerly Annex E). The other seven annexes were moved to the draft of ISO/TR 24971. The reason stated for moving these Annexes to the guidance document was to make future revisions to the guidance easier to implement because it is a guidance rather than a standard. However, there were also some objectionable recommendations in the informative annexes that were the subject of deviation #3—ALARP from Annex D.8 vs. “As far as possible,” in the first indent of section 2 of Annex I in the MDD.
Although the committee was tasked to make improvements in the implementation of ISO 14971 without changing the process, the new draft has subtle changes in the process. Most of these changes can be identified quickly by reviewing the updated risk management flow chart provided in Figure 1. The updated flow chart now has two places where risks are evaluated. The first place is identical to the original Figure 1, but now the associated section is clarified to be specific to evaluating individual risks. The second place in the flow chart is new and specific to the evaluation of overall residual risks. The draft standard also states that different acceptability criteria and methods of evaluation may be used for each evaluation phase in the process. There have also been subtle changes to the names of process phases:
Section 7.4 is now “Benefit/Risk” analysis instead of “Risk/Benefit” analysis—although the draft flow chart does not reflect this.
Section 9 is now “Risk Management Review” instead of “Risk Management Report”
Section 10 is now “Production and post-production activities” instead of “Production and post-production information”
There is also more detail in the diagram under the phases for 1) risk analysis, 2) risk control, and 3) production and post-production activities.
Three new definitions are introduced in the draft standard: 3.2, benefit; 3.15, reasonably foreseeable misuse; and 3.28, state of the art. The section for identification of hazards, Clause 5.4, was reworded and expanded to consider the reasonably foreseeable sequences or combinations of events that can result in a hazardous situation. The draft standard now states that your risk management plan must also include a method to evaluate the overall residual risk and the criteria for the acceptability of the overall residual risk. In the section for risk estimation, Clause 5.5, the draft standard states that if the probability of the occurrence of harm cannot be estimated, the possible consequences shall be listed for use in the risk evaluation and risk control. The risk control option analysis priorities in section 7.1 are updated to match the new MDR, Regulation (EU) 2017/745, nearly exactly. In section 9, risk management reports were changed to risk management review, and the Clause now requires determining when to conduct subsequent reviews and when to update reports. This emphasizes the requirement to continuously update risk management documentation with input from production and post-production information. This mirrors the emphasis on continually updating post-market clinical follow-up in Regulation (EU) 2017/745, Annex XIV, Part B, Section 5, and continuously updating clinical evaluations in Regulation (EU) 2017/745, Annex XIV, Part A, Section 1.
Will ISO 14971:2019 address the 7 Deviations in EN ISO 14971:2012?
The new MDR, Regulation (EU) 2017/745, revised and clarified the wording of the essential requirements in the MDD. The MDR attempts to explain the requirements for risk management files of CE Marked products, but the MDR remains different from the requirements of ISO 14971. Unfortunately, because the ISO/DIS 14971 was not intended to change the risk management process of ISO 14971:2007, there will continue to be “deviations” between the MDR and standard.
Some people have tried to use ISO/TR 24971, the risk management guidance, as the official interpretation of how the risk management standard. However, the guidance is also a product of the TC210 committee, and it does not meet all requirements of the MDD or the MDR.
The new draft does, however, include changes that address some of the deviations in EN ISO 14971:2012. Below, each of the seven variations is listed, and hyperlinks are provided to other articles on each deviation.
Negligible Risks – The word “negligible” was only in one location in the body of the standard as a note referring to Annex D.8. In the draft, Annex D was removed and relocated to ISO/TR 24971, and the note was eliminated from Clause 3.4—now Clause 4.4 in the draft. The draft should fully resolve this deviation.
Risk Acceptability – Clause 7 was renumbered to Clause 8 in the draft. Still, the title of this Clause was also changed from “Evaluation of overall residual risk acceptability” to “Evaluation of overall residual risk.” However, if you read the Clause it still refers to determining the acceptability of risks. In note 2 of Annex ZA of the draft, it states that determining acceptable risk must comply with Essential Requirements 1, 2, 5, 6, 7, 8, 9, 11, and 12 of the Directive. The draft should fully resolve this deviation.
ALARP vs. “As far as possible” – The European Commission believes that the concept of “ALARP” implies economic considerations, and some companies have used economics as a reason for not implementing certain risk controls. ALARP was eliminated from the notes in the risk management plan clause and by moving Annex D.8 to ISO/TR 24971 and adding note 1 in Annex ZA. The draft should fully resolve this deviation.
Benefit/Risk Analysis – The contradiction in requirements between the International Standard and the MDD, as it relates to determining when a benefit/risk analysis must be conducted, has not been updated. The draft does not resolve this deviation. Companies that CE Mark products will need to perform a benefit/risk analysis for all residual risks and all individual risks—despite the wording of the standard.
Risk Control – The contradiction in requirements between the International Standard and the MDD, as it relates to determining when risk controls must be implemented. The International Standard gives companies the option to avoid the implementation of risk controls if the risk is acceptable. At the same time, the MDD requires that risk controls be implemented for all risks unless the risk controls create additional risks that increase risks, or the risk controls do not reduce risks further. The draft does not resolve this deviation. Companies that CE Mark products will need to implement risk controls for all individual risks—despite the wording of the standard.
Risk Control Options – The intent of Clause 6.2 in ISO 14971:2007 was likely to be the same as the MDD. However, the European Commission identified the missing word “construction” as being significant. Therefore, to prevent any misunderstandings, the TC210 committee copied the wording of Regulation (EU) 2017/745. The draft should fully resolve this deviation.
IFU Validation – Again, to prevent any misunderstandings, the TC210 committee copied the wording of Regulation (EU) 2017/745. However, the examples of information for safety (i.e., warnings, precautions, and contraindications) were not included. Hopefully, the final version of the 3rd edition will consist of these examples. Clause 8, evaluation of overall residual risk, was also reworded to state, “the manufacturer shall decide which residual risks to disclose and what information is necessary to include in the accompanying documentation to disclose those residual risks.” The draft should fully resolve this deviation.
Recommendations for your Risk Management Process?
The most important consideration when establishing a risk management process for medical devices is whether you plan to CE Mark products. If you intend to CE Mark products, then you should write a procedure that is compliant with the current requirements of the MDD and future requirements of Regulation (EU) 2017/745. Therefore, the seven deviations should be addressed. Also, you need to maintain compliance with the current version of the standard.
I recommend creating a process based upon the newly updated process diagram in the latest draft. The process should begin with a risk management plan. For your plan, you may want to create a template and maintain it as a controlled document. It could also be part of your design and development plan template, but the plan should include each of the following risk management activities:
Hazard identification
Risk estimation
Risk evaluation
Risk control option analysis
Risk control verification of effectiveness
Benefit/Risk analysis
Evaluation of overall residual risk
Risk management review
Production and post-production activities
Your procedure should also be integrated with other processes, such as 1) design control, 2) post-marketing surveillance, and 3) clinical evaluation. Your procedure must indicate the priority for the implementation of risk control options. The best strategy for ensuring risk control priorities are compliant is to copy the wording of the new EU Regulations verbatim. Your process should include performing benefit/risk analysis. You should also define your process for risk management review. Your review process should specify when subsequent reviews will be done, and when your risk management report will be updated. Finally, you should identify a post-market surveillance plan for each device or device family, and use that post-market surveillance data as feedback in the risk management process.
The one element that appears to be weakly addressed in the body of the standard is the requirement for traceability of each hazard to the other aspects of the risk management process. Although traceability is mentioned in Clause 3.5 of the 2nd edition, and Clause 4.5 of the draft 3rd edition of ISO 14971, that is the only place is mentioned in the body of the standard. Traceability is mentioned several more times in Annex A, but the focus seems to be on the risk management file. Companies need more guidance on how to achieve this traceability. The appropriate place for this guidance is probably in ISO/TR 24971. Still, in order to maintain this documentation, a software database will likely be critical to maintaining traceability as changes are made during design iterations and after commercialization. This type of software tool is also needed to expedite the review of risk management documentation during a complaint investigation.
Which Risk Analysis Tool should you use?
In Annex G of ISO 14971:2007 and the EN 2012 version, there are five different risk analysis tools described. The word “described” is emphasized because informative annexes are not “recommended.” The committee that created the 2nd edition of ISO 14971 wanted to provide several suggestions for possible risk analysis tools to consider. However, each tool has strengths and weaknesses. Additionally, the widespread use of the failure-mode-and-effects analysis (FMEA) tool in the automotive and aerospace industries has spread to the medical device industry, and companies seem to believe that regulators prefer the FMEA tool. This is not true. Companies should be trained in all of these tools, and training should consist of more than just reading Annex G, and the tools should be used where they are most beneficial. My recommendations are below:
Preliminary Hazard Analysis (PHA) – This process is critical during the development of design inputs. It is also the most underutilized analysis tool. I have not seen a single example of this tool written in a procedure by any medical device company. I believe this process should be continuously updated as part of training new design team members and should be both product and project-specific.
Fault-tree Analysis (FTA) – This process is a top-down approach to risk analysis. It is heavily utilized by transportation engineers when intersections are designed, and accidents are investigated. This tool depicts risk analysis pictorial as a tree of fault modes representing each possible root cause for failure. At each level of the tree, fault mode combinations are described with logical operators (i.e., AND, OR). The information displays the frequency of each fault mode quantitatively. Therefore, when you are investigating a complaint, the tree can be used to help identify possible fault modes that may have been the root cause of device failure. You may also be interested in the standard specific to Fault tree analysis (FTA): IEC 61025:2006.
Failure Mode and Effects Analysis (FMEA) – This process is a bottom-up approach to risk analysis. The automotive and aerospace industries heavily utilize it. This tool systematically lists all failure modes in groups organized by component. Risks are estimated based upon the severity of effect, probability of occurrence, and detectability. Over time, the FMEA process split into three tools: 1) process FMEA (pFMEA), 2) design FMEA (dFMEA), and 3) use FMEA (uFMEA). The first is ideal for analyzing and reducing risks associated with the manufacturing of devices. In particular, the detectability factor can be linked closely with process validation. The second evolved from the realization that the detection of a risk after the device is in the user’s hands does not reduce risk. A risk reduction only occurs if detectability is proactive. Therefore, this was stated in Annex G.4, and companies began to eliminate detectability and continued to use FMEA as their primary tool. Due to the widespread familiarity with the FMEA tool, usability FMEAs became popular for documenting risks associated with the use of a device. Unfortunately, the only real advantages of a dFMEA and uFMEA are familiarity with the tool. You may also be interested in the standard specific to FMEA: IEC 60812:2018.
Hazard and Operability Study (HAZOP) – In addition to the risks of using devices, there are also risks associated with the production of devices. Processes related to coating, cleaning, and sterilization are all processes that typically involve hazardous chemicals. The chemical and pharmaceutical industries use HAZAP as a tool to analyze these process risks and prevent injuries. You may also be interested in the standard specific to HAZOP: IEC 61882:2016.
Hazard Analysis and Critical Control Point (HACCP) – This process is primarily used by the food industry to prevent the spread of contaminated food supplies. Even though medical device manufacturers do not typically use it, it should be considered as a tool for managing the supply chain for devices. This model is useful when manufacturing is outsourced, or secondary processing is conducted at second and third-party suppliers. Since many FDA inspectors started in the food industry as inspectors, this is also a method that is supported by the FDA as a risk control process for outsourced processes.
How to document your risks?
For simple devices, risk management documentation is a burdensome task. For complex devices, a spreadsheet could include hundreds of lines or more than even one thousand individual lines. Also, the requirement for traceability requires additional columns in a table. Therefore, it becomes nearly impossible for you to include all the required information on a page that is 11 inches wide. If you expand your page to 17 inches wide, the size of your font will need to be very small. If you make a change, your spreadsheet can be challenging to update quickly. You could purchase a 43” widescreen TV for your monitor, or you can use dual monitors for your display, but changes remain challenging to implement without a mistake.
You need to stop relying upon spreadsheets. Use a database, and don’t use Microsoft Access. Purchase a database that is designed to document design controls and risk management traceability. If your company has software expertise, develop your software tool to do this. You should also design standardized templates for exporting your reports. By doing this, it will only take minutes to create an updated report when you make design changes. If you describe the risk management activities as notes in your software, the description of these activities can also be automatically converted into summary pages for each report summarizing that risk management activity. You can even prompt the user to answer questions in the software to populate a templated document. For example, you can prompt users to input subsequent updates of your risk management reviews, and that can be automatically converted into a summary paragraph. This reporting capability is especially helpful when responding to FDA review questions asking for cybersecurity risks.
Additional Training Resources for ISO 14971
The risk management training webinar has been completely rewritten for the second time (i.e. the first time was on October 19, 2018). The newest version will be a two-part webinar series. Part one of two will focus on Clauses 1 through 7.1 of the ISO 14971:2019 standard. Part two of two will focus on Clauses 7.2 through 10. We selected Clause 7.2 to begin the second part of this webinar series, because it marks the beginning of the verification of the risk controls your company has implemented (i.e. – Post “Design Freeze”). Part 1 will be hosted live on March 29, 2022 @ 9-10:30 am EDT, and Part 2 will be hosted live on April 5, 2022 @ 9-10:30 am EDT. Both sessions will be recorded if you are unable to participate in the live sessions.
SYS-010, Medical Device Academy’s Risk Management Procedure, is compliant with EN ISO 14971:2019. The procedure includes templates for documentation of design risk management and process risk management. The procedure is also compliant with ISO/TR 24971:2020 and Regulation (EU) 2017/745. Both the two-part risk management training webinar, and the risk management procedure, are included in Medical Device Academy’s turnkey quality system.
