This article review explains how to conduct a quantitative CAPA effectiveness check, and you will also learn three methods NOT recommended.
There are three methods NOT recommended for a CAPA effectiveness check:
- verifying the procedure was revised,
- verifying employees were retrained, and
- making sure mistakes don’t occur 3x in a row.
The best method is to establish quantitative criteria for effectiveness based upon data collected during the investigation of the root cause. The graph above is an example of objective evidence that preventive action was effective. The chart shows that the process capability (Cpk) was improved from 0.837 to 2.50 by changing a process set-point to adjust the mean of the dimension closer to the center of the specification range. This is typical of adjustments made during process validation and revalidation activities.
Incorrect Method 1: Verify the Procedure was Revised
When a nonconformity is identified during an ISO 13485 audit, the laziest way to “fix” the problem is to revise your procedure. Despite the fact that most FDA 483s identify inadequate procedures as the reason for observation, your procedures are seldom the problem. Your employees may not even be following the procedures. Repeatedly revising procedures may be part of the problem. If you must revise your procedures, please involve the people that use the procedures.
Incorrect Method 2: Verify Employees were Retrained
During your last surveillance audit, you may have revised the procedure, but your auditor noticed that there were no retraining records for employees that were performing the revised procedure. One interviewee was unable to identify where the new inspection step could be found in the revised procedure. It’s too bad the interviewee didn’t notice the bold and underlined text indicating recent revisions. Your auditor wonders how effective your retraining process is.
Incorrect Method 3: No Mistakes 3x in a Row
Last month a manufacturing engineer was assigned to perform an effectiveness check related to corrective actions implemented in the incoming inspection process. The procedure was revised to clarify the proper procedure for a statistical sampling of rolls of plastic film as a corrective action. The engineer sampled the three most recent lots of the same plastic film that was incorrectly sampled in the past. All three lots were correctly sampled in accordance with the revised procedure. The engineer reported that the corrective actions implemented were effective. However, you have two new nonconformities on your desk from manufacturing related to incorrect sampling procedures during an incoming inspection of other raw materials. Now you wonder if the incoming inspection procedure was the real root cause.
Corrective actions that are actually effective
Instead of adding something to your procedures each time someone makes a mistake, you might want to think about how you can simplify and streamline your procedures with fewer words. You can say things more clearly with pictures and flow charts instead of hundreds of words. Training effectiveness can be verified with exams that ensure employees “read and understand” your revised procedures. Finally, when you identify a nonconformity with one product, you need to ensure that you consider how similar mistakes might occur with similar products. Maybe you need a process for incoming inspection that doesn’t rely upon someone reading procedures.
You need to be objective to perform an effectiveness check
The biggest weakness of the auditing process is that it relies heavily upon the subjective opinion of an auditor. This is why auditors are supposed to audit against objective audit criteria in an international standard. The need for objectivity is also why there are guidance documents to clarify a consistent interpretation of those standards. Therefore, when you perform an effectiveness check, you also need objectivity. The best way to ensure objectivity is to establish documented criteria for effectiveness prior to finalizing your corrective action plan. Ideally, that will be in the form of a prospective process validation protocol with quantitative acceptance criteria.
How to ensure objectivity
The single best way to ensure objectivity when you are performing a CAPA effectiveness check is to define the post-implementation goal in terms of a quantitative quality objective. Ideally, you can graph the quality metric using historical data and current data. If you need statistical analysis to see a difference between pre- and post-implementation of the CAPA, then your CAPA was not effective. If your graph looks like a miracle happened and the metric changed almost overnight, and timing corresponds to the date your corrective action(s) was implemented, then your CAPA was effective.
How to set a quality objective for you CAPA effectiveness check
Some people have trouble with using a quantitative approach in performing effectiveness checks because some things are harder to measure than others. However, you can measure anything. For example, you can even measure employees forgetting to initial and date changes to quality records. This can be done by identifying critical control points where quality records are reviewed, and documentation errors are measured. You can measure by the employee, by form, by month, etc. The key to monitoring and measuring a process is to answer the following questions:
- Who will measure it?
- What will be measured?
- Where will it be measured?
- When will it be measured?
- How will it be measured?
- How will measurements be analyzed?
- Who will data analysis be communicated to?
When to Perform a CAPA Effectiveness Check
Many companies set arbitrary deadlines for performing an effectiveness check (e.g., – between 30 and 60 days of implementation of corrective actions). Some companies use a risk-based approach to their CAPA process, and the urgency of effectiveness checks may be a function of risk. I recommend a completely different approach. Instead of using an arbitrary or risk-based approach, I recommend monitoring your new quality metric to estimate how long it will take to reach your new quality objective.
If you are interested in learning more about CAPA, click here to register for Medical Device Academy’s Risk-Based CAPA webinar.
It clear that the Effectiveness Check must be done by the owner of the system where the deviation was found once the Corrective and Preventive Actions were closed.
Then, the deviation is already closed – with last action performed – when the effectiveness check is performed.
Right?
If the Effectiveness check is a review of metrics to find if the issue was effectively solved. Is not this an approach really close to a KPI? Then What is the difference?
I hope to hear about you soon.
Ciao!
Thank you Humberto. The effectiveness check should probably not be done by the owner of the system. It should be reviewed by an independent person–probably an internal auditor or owner of the CAPA process. Using metrics is the same as using Key Process Indicators (KPIs), but not every company uses KPIs–especially for non-production processes.
Pingback: 15 Tips for Creating an Effective CAPA Form -
Pingback: 7 Steps to Writing a CAPA Procedure Compliant with ISO 13485:2016 -
Rob,
I think you nailed the key elements of a CAPA effectiveness check plan. About the only thing I would add to your blog is where you mention audit criteria is the plan needs to be defensible when audited. Thanks.
Dear Henry,
Thank you for your feedback on the CAPA plans. You are correct that you need to be able to defend your CAPA plan to an auditor. However, that’s not actually a requirement of regulators or the Standard. The way you “defend” you plan is to demonstrate effectiveness. I recommend using quantitative metrics to demonstrate effectiveness. Effectiveness check are a requirement of the standard and the regulations. That was the primary focus of this article.
Rob
What are your thoughts on performing an effectiveness check for a CAPA initiated by a single one-time occurring high-impact non-conformity that escaped QC inspection? Due to the low frequency, it’s hard to show improvement, Even one customer complaint would fail our CAPA effectiveness check and if no customers complain, it could be a fluke rather than effectiveness. An effectiveness check based on internal QC data is also challenging: how do we measure improved detection and delineate from manufacturing error rate?
Looking forward to your reply!
This type of effectiveness check is often required by 8D responses in automotive companies. If the process is manual, then you need to provide retraining of inspectors and test their competency. You do this by providing a bunch of samples to inspect, but there are a few “red herrings” in the bunch for the inspector to catch. The key is to not tell them how many defects there are and to keep the frequency small (e.g., 1-5%). If the process is automated, then you would create test articles to use as positive and negative controls. The automated system would be validated to catch the error, and then you would run the test articles before and after each shift at start-up. This will verify that the automated inspection can catch the mistakes. This is often a useful tool for automated optical inspection systems in the manufacture of PCBs.
How do verify effectiveness for something that may have been overlooked during testing. Corrective Action is to re-do the testing with the missing test component, but how do you verify it was effective if it doesn’t change anything for the device itself.
Example – an auditor identified to us we did not test “worst case” scenario (drop test with device open vs closed). We went ahead and tested the worst case scenario. The results of our testing were no different than our original testing results. How do you verify effectiveness associated with a corrective action such as that?
Hi Stephen,
In this case, it doesn’t sound like you ever got to the root cause. You need to dig deeper using a 5-Why analysis. Why did you forget to test “worst case scenario.” Once you get to the cause, then you might have a better ability to identify whether corrective actions are effective. So far, you only corrected the problem. Retesting is not a corrective action.
What if the CAPA was initiated by an auditor on a procedure? And the procedure solely required revision to add a phrase. How is an effectiveness check performed at that point? And what is an appropriate effectiveness check?
Rabia,
The answer to your question depends entirely on the conclusion of the investigation. If the investigation demonstrates the failure mode is an isolated instance (Special Case Failure) then only correction is required. On the other hand, if the investigation demonstrates the failure to be part of a pattern of similar failures (Systemic Failure) the cause must be identified and that would require a corrective action and subsequent Verification of Effectiveness (VoE).
Is the root cause systemic or special case? That is the question that must be answered before a decision regarding the need for VoE can be made.
Pingback: CAPA Form - 15 tips to avoid CAPA failure Medical Device Academy
Pingback: 7 Steps to Writing a CAPA Procedure Compliant with ISO 13485:2016 Medical Device Academy
Pingback: How to Prepare for an ISO 13485 Certification Stage 2 Audit - Part 1 of 2 Medical Device Academy
Pingback: What is a CAPA Board? and Do you need one? - Medical Device Academy Medical Device Academy
Pingback: Is monitoring every procedure required? Medical Device Academy
Pingback: CAPA - Corrective Action and Preventative Action Medical Device Academy