How to perform a quantitative CAPA effectiveness check

This article reviews explains how to conduct a quantitative CAPA effectiveness check, and you will also learn three methods NOT recommended.

quantitative effectiveness check 1024x713 How to perform a quantitative CAPA effectiveness check

There are three methods NOT recommended for a CAPA effectiveness check:

  1. verifying the procedure was revised,
  2. verifying employees were retrained, and
  3. making sure mistakes don’t occur 3x in a row.

The best method is to establish quantitative criteria for effectiveness based upon data collected during the investigation of root cause. The graph above is an example of objective evidence that a preventive action was effective. The graph shows that the process capability (Cpk) was improved from 0.837 to 2.50 by changing a process set-point to adjust the mean of the dimension closer to the center of the specification range. This is typical of adjustments made during process validation and re-validation activities.

Incorrect Method 1: Verify the Procedure was Revised

When a nonconformity is identified during an audit, the laziest way to “fix” the problem is to revise your procedure. Despite the fact that most FDA 483s identify inadequate procedures as the reason for an observation, your procedures are seldom the problem. Your employees may not even be following the procedures. In fact, repeatedly revising procedures may actually be part of the problem. If you must revise your procedures, please involve the people that actually use the procedures.

Incorrect Method 2: Verify Employees were Retrained

During your last surveillance audit you may have revised the procedure, but your auditor noticed that there were no retraining records for employees that were performing the revised procedure. In fact, one interviewee was unable to identify where the new inspection step could be found in the revised procedure. It’s too bad the interviewee didn’t notice the bold and underlined text indicating recent revisions. Your auditor wonders how effective your retraining process is.

Incorrect Method 3: No Mistakes 3x in a Row

Last month a manufacturing engineer was assigned to perform an effectiveness check related to corrective actions implemented in the incoming inspection process. The procedure was revised to clarify the proper procedure for statistical sampling of rolls of plastic film as a corrective action. The engineer sampled the three most recent lots of the same plastic film that was incorrectly sampled in the past. All three lots were correctly sampled in accordance with the revised procedure. The engineer reported that the corrective actions implemented were effective. However, you have two new nonconformities on your desk from manufacturing related to incorrect sampling procedures during incoming inspection of other raw materials. Now you wonder if the incoming inspection procedure was the real root cause.

Corrective actions that are actually effective

Instead of adding something to your procedures each time someone makes a mistake, you might want to think about how you can simplify and streamline your procedures with fewer words. You can say things more clearly with pictures and flow charts instead of hundreds of words. Training effectiveness can be verified with exams that ensure employees “read and understand” your revised procedures. Finally, when you identify a nonconformity with one product, you need to ensure that you consider how similar mistakes might occur with similar products. Maybe you need a process for incoming inspection that doesn’t rely upon someone reading procedures.

You need to be objective to perform an effectiveness check

The biggest weakness of the auditing process is that it relies heavily upon the subjective opinion of an auditor. This is why auditors are supposed to audit against objective audit criteria in an international standard. The need for objectivity is also why there are guidance documents to clarify a consistent interpretation of those standards. Therefore, when you perform an effectiveness check, you also need objectivity. The best way to ensure objectivity is to establish documented criteria for effectiveness prior to finalizing your corrective action plan. Ideally, that will be in the form of a prospective process validation protocol with quantitative acceptance criteria.

How to ensure objectivity

The single best way to ensure objectivity when you are performing a CAPA effectiveness check is to define the post-implementation goal in terms of a quantitative quality objective. Ideally, you can graph the quality metric using historical data and current data. If you need statistical analysis to see a difference between pre- and post-implementation of the CAPA, then your CAPA was not effective. If your graph looks like a miracle happened and the metric changed almost overnight, and timing corresponds to the date your corrective action(s) was implemented, then your CAPA was effective.

How to set a quality objective for you CAPA effectiveness check

Some people have trouble with using a quantitative approach in performing effectiveness checks, because some things are harder to measure than others. However, you can measure anything. For example, you can even measure employees forgetting to initial and date changes to quality records. This can be done by identifying critical control points where quality records are reviewed and documentation errors are measured. You can measure by employee, by form, by month, etc. The key to monitoring and measuring a process is to answer the following questions:

  1. Who will measure it?
  2. What will be measured?
  3. Where will it be measured?
  4. When will it be measured?
  5. How will it be measured?
  6. How will measurements be analyzed?
  7. Who will data analysis be communicated to?

When to Perform a CAPA Effectiveness Check

Many companies set arbitrary deadlines for performing an effectiveness check (e.g., – between 30 and 60 days of implementation of corrective actions). Some companies use a risk-based approach to their CAPA process, and the urgency of effectiveness checks may be a function of risk. I recommend a completely different approach. Instead of using an arbitrary or risk-based approach, I recommend monitoring your new quality metric to estimate how long it will take to reach your new quality objective.
%name How to perform a quantitative CAPA effectiveness check

If you are interested in learning more about CAPA, click here to register for Medical Device Academy’s Risk-Based CAPA webinar.

Posted in: CAPA

Leave a Comment (8) ↓


  1. Humberto September 4, 2015

    It clear that the Effectiveness Check must be done by the owner of the system where the deviation was found once the Corrective and Preventive Actions were closed.
    Then, the deviation is already closed – with last action performed – when the effectiveness check is performed.
    If the Effectiveness check is a review of metrics to find if the issue was effectively solved. Is not this an approach really close to a KPI? Then What is the difference?
    I hope to hear about you soon.

    • Rob Packard November 6, 2015

      Thank you Humberto. The effectiveness check should probably not be done by the owner of the system. It should be reviewed by an independent person–probably an internal auditor or owner of the CAPA process. Using metrics is the same as using Key Process Indicators (KPIs), but not every company uses KPIs–especially for non-production processes.

  2. Henry Zelman November 8, 2017

    I think you nailed the key elements of a CAPA effectiveness check plan. About the only thing I would add to your blog is where you mention audit criteria is the plan needs to be defensible when audited. Thanks.

  3. Rob Packard December 3, 2017

    Dear Henry,
    Thank you for your feedback on the CAPA plans. You are correct that you need to be able to defend your CAPA plan to an auditor. However, that’s not actually a requirement of regulators or the Standard. The way you “defend” you plan is to demonstrate effectiveness. I recommend using quantitative metrics to demonstrate effectiveness. Effectiveness check are a requirement of the standard and the regulations. That was the primary focus of this article.

  4. Michelle January 4, 2018

    What are your thoughts on performing an effectiveness check for a CAPA initiated by a single one-time occurring high-impact non-conformity that escaped QC inspection? Due to the low frequency, it’s hard to show improvement, Even one customer complaint would fail our CAPA effectiveness check and if no customers complain, it could be a fluke rather than effectiveness. An effectiveness check based on internal QC data is also challenging: how do we measure improved detection and delineate from manufacturing error rate?
    Looking forward to your reply!

    • Rob Packard February 18, 2018

      This type of effectiveness check is often required by 8D responses in automotive companies. If the process is manual, then you need to provide retraining of inspectors and test their competency. You do this by providing a bunch of samples to inspect, but there are a few “red herrings” in the bunch for the inspector to catch. The key is to not tell them how many defects there are and to keep the frequency small (e.g., 1-5%). If the process is automated, then you would create test articles to use as positive and negative controls. The automated system would be validated to catch the error, and then you would run the test articles before and after each shift at start-up. This will verify that the automated inspection can catch the mistakes. This is often a useful tool for automated optical inspection systems in the manufacture of PCBs.


Leave a Comment

Time limit is exhausted. Please reload the CAPTCHA.


Get every new post on this blog delivered to your Inbox.

Join other followers:

Simple Share Buttons
Simple Share Buttons