CAPA effectiveness check

This article review explains how to conduct a quantitative CAPA effectiveness check, and you will also learn three methods NOT recommended.

There are three methods NOT recommended for a CAPA effectiveness check:

1. verifying the procedure was revised,
2. verifying employees were retrained, and
3. making sure mistakes don’t occur 3x in a row.

The best method is to establish quantitative criteria for effectiveness based upon data collected during the investigation of the root cause. The graph above is an example of objective evidence that preventive action was effective. The chart shows that the process capability (Cpk) was improved from 0.837 to 2.50 by changing a process set-point to adjust the mean of the dimension closer to the center of the specification range. This is typical of adjustments made during process validation and revalidation activities.

Incorrect Method 1: Verify the Procedure was Revised

When a nonconformity is identified during an ISO 13485 audit, the laziest way to “fix” the problem is to revise your procedure. Despite the fact that most FDA 483s identify inadequate procedures as the reason for observation, your procedures are seldom the problem. Your employees may not even be following the procedures. Repeatedly revising procedures may be part of the problem. If you must revise your procedures, please involve the people that use the procedures.

Incorrect Method 2: Verify Employees were Retrained

During your last surveillance audit, you may have revised the procedure, but your auditor noticed that there were no retraining records for employees that were performing the revised procedure. One interviewee was unable to identify where the new inspection step could be found in the revised procedure. It’s too bad the interviewee didn’t notice the bold and underlined text indicating recent revisions. Your auditor wonders how effective your retraining process is.

Incorrect Method 3: No Mistakes 3x in a Row

Last month a manufacturing engineer was assigned to perform an effectiveness check related to corrective actions implemented in the incoming inspection process. The procedure was revised to clarify the proper procedure for a statistical sampling of rolls of plastic film as a corrective action. The engineer sampled the three most recent lots of the same plastic film that was incorrectly sampled in the past. All three lots were correctly sampled in accordance with the revised procedure. The engineer reported that the corrective actions implemented were effective. However, you have two new nonconformities on your desk from manufacturing related to incorrect sampling procedures during an incoming inspection of other raw materials. Now you wonder if the incoming inspection procedure was the real root cause.

Corrective actions that are actually effective

Instead of adding something to your procedures each time someone makes a mistake, you might want to think about how you can simplify and streamline your procedures with fewer words. You can say things more clearly with pictures and flow charts instead of hundreds of words. Training effectiveness can be verified with exams that ensure employees “read and understand” your revised procedures. Finally, when you identify a nonconformity with one product, you need to ensure that you consider how similar mistakes might occur with similar products. Maybe you need a process for incoming inspection that doesn’t rely upon someone reading procedures.

You need to be objective to perform an effectiveness check

The biggest weakness of the auditing process is that it relies heavily upon the subjective opinion of an auditor. This is why auditors are supposed to audit against objective audit criteria in an international standard. The need for objectivity is also why there are guidance documents to clarify a consistent interpretation of those standards. Therefore, when you perform an effectiveness check, you also need objectivity. The best way to ensure objectivity is to establish documented criteria for effectiveness prior to finalizing your corrective action plan. Ideally, that will be in the form of a prospective process validation protocol with quantitative acceptance criteria.

How to ensure objectivity

The single best way to ensure objectivity when you are performing a CAPA effectiveness check is to define the post-implementation goal in terms of a quantitative quality objective. Ideally, you can graph the quality metric using historical data and current data. If you need statistical analysis to see a difference between pre- and post-implementation of the CAPA, then your CAPA was not effective. If your graph looks like a miracle happened and the metric changed almost overnight, and timing corresponds to the date your corrective action(s) was implemented, then your CAPA was effective.

How to set a quality objective for you CAPA effectiveness check

Some people have trouble with using a quantitative approach in performing effectiveness checks because some things are harder to measure than others. However, you can measure anything. For example, you can even measure employees forgetting to initial and date changes to quality records. This can be done by identifying critical control points where quality records are reviewed, and documentation errors are measured. You can measure by the employee, by form, by month, etc. The key to monitoring and measuring a process is to answer the following questions:

1. Who will measure it?
2. What will be measured?
3. Where will it be measured?
4. When will it be measured?
5. How will it be measured?
6. How will measurements be analyzed?
7. Who will data analysis be communicated to?

When to Perform a CAPA Effectiveness Check

Many companies set arbitrary deadlines for performing an effectiveness check (e.g., – between 30 and 60 days of implementation of corrective actions). Some companies use a risk-based approach to their CAPA process, and the urgency of effectiveness checks may be a function of risk. I recommend a completely different approach. Instead of using an arbitrary or risk-based approach, I recommend monitoring your new quality metric to estimate how long it will take to reach your new quality objective.

If you are interested in learning more about CAPA, click here to register for Medical Device Academy’s Risk-Based CAPA webinar.