ISO Auditing Medical Device Academy

Archive for ISO Auditing

Auditing Medical Device Software Vendors

Posted by Rob Packard on June 20, 2012

This blog presents some thoughts related to auditing medical device software companies.

Software medical devices are used to assist medical professionals. For example, radiologists use software with identifying areas of interest for medical imaging. Do you know how to audit a software company?

As a third-party auditor, I have had the pleasure of auditing software companies for CE Marking. When you audit a software company for the first time, this forces you to re-learn the entire ISO 13485 Standard. For example, if a company only produces software (i.e. software as a medical device or SaMD) there is very little to sample for incoming inspection and purchasing records. This is because the product is not physical—it’s software. Clauses of ISO 13485 related to sterility, implants, and servicing are also not applicable to software products. If the software is web-based, the shipping and distribution clauses (i.e., – 7.5.5) might present a challenge to an auditor as well.

The aspects of the ISO 13485 Standard that I found to be the most important to auditing software products were design controls and customer communication. Many auditors are trained in auditing the design and development of software, but very few auditors have experience auditing technical support call centers. When auditing a call center, most of the calls represent potential complaints related to software “bugs,” system incompatibilities with the operating system or hardware, and use errors resulting from the design of the user interface.

In most technical support call centers, the support person tries to find a work-around for problems that are identified. The problem with a “work-around” is that it is the opposite approach to the CAPA process. To meet ISO 13485 requirements, software companies must show evidence of monitoring and measuring these “bugs.” There must also be evidence of management identifying negative trends and implementing corrective actions when appropriate.

As an auditor, you should focus on how the company prioritizes “bugs” for corrective actions. Most software companies focus on the severity of software operations and the probability of occurrence. This is the wrong approach. Failure to operate is not the most severe result of medical device software failure. Medical device software can result in injury or death to patients. Therefore, it is critical to use a risk-based approach to the prioritization of CAPAs. This risk-based approach should focus on the severity of effects upon patients—not users. This focus on safety and performance is emphasized throughout the EU Medical Device Regulations and it is a risk management requirement in ISO 14971.

Tags: auditing, software vendor

Posted in: ISO Auditing

Process Approach to Auditing – 7 Steps to Training Auditors

Posted by Rob Packard on June 8, 2012

The process approach to auditing is demonstrated using Turtle Diagrams as a tool instead of using traditional auditor checklists.

tutle diagram1 Process Approach to Auditing 7 Steps to Training Auditors

I have been reviewing trends for how people find my website, and a large number of you appear to be interested in my auditing schedules and other audit-related topics. Therefore, this week’s blog is dedicated to training auditors on the process approach.

First, the process approach is just a different way of organizing audits. Instead of auditing by clause, or by procedure, instead, you audit each process. Typical processes include:

Design & Development
Purchasing
Incoming inspection
Assembly
Final Inspection
Packaging
Sterilization
Customer Service
Shipping
Management Review
CAPA
Internal Auditing

Why the Process Approach is Recommended

First, the process approach identifies linkages between processes as inputs and outputs. Therefore, if there is a problem with communication between departments, the process approach will expose it. If only a procedural audit is performed, the lack of communication to the next process is often overlooked.

Second, the process approach is a more efficient way to cover all the clauses of the ISO Standard than auditing each clause (i.e.,– the element approach). My rationale for the claim of greater efficiency is simple: there are 19 required procedures in the ISO 13485 Standard, but there are only 12 processes identified above. The “missing” procedures are incorporated into each process audit.

For example, each process audit requires a review of records as input and outputs. Also, training records should be sampled for each employee interviewed during an audit. Finally, nonconforming materials can be identified and sampled at incoming inspection, in assembly processes, during final inspection, during packaging, and even during shipment. The tool that BSI uses to teach the process approach is the “Turtle Diagram.” The diagram above illustrates where the name came from.

Interviewing with the Process Approach

The first skill to teach a new auditor is the interview. Each process approach audit should begin with an interview of the process owner. The process owner and the name of the process are typically documented in the center of the turtle diagram. Next, most auditors will ask, “Do you have a procedure for ‘x process’?” This is a weak auditing technique because it is a “closed-ended” or yes/no. This type of question does little to help the auditor gather objective evidence. Therefore, I prefer to start with the question, “Could you please describe the process?” This should give you a general overview of the process if you are unfamiliar with it.

After getting a general overview, I like to ask the question: “How do you know how to start the process.” For example, inspectors know that there is material for incoming inspection because raw materials are in the quarantine area. I have seen visual systems, electronic and paper-based systems for notifying QC inspectors of product to inspect. If there is a record indicating that material needs to be inspected—that is the ideal scenario. A follow-up question is, “What are the outputs of the inspection process?” Once again, the auditor should be looking for paperwork. Sampling these records and other supporting records is how the process approach addresses Clause 4.2.4—control of records.

The next step of this approach is to “determine what resources are used by incoming inspection.” This includes gauges used for measurement, cleanliness of the work environment, etc. This portion of the process approach is where an auditor can review calibration, gowning procedures, and software validation. After “With What Resources,” the auditor then needs to identify all the incoming inspectors on all shifts. From this list, the auditor should select people to interview and follow-up with a request for training records.

The sixth step is to request procedures and forms. Many auditors believe that they need to read the procedure. However, if a company has long procedures, this could potentially waste valuable time. Instead, I like to ask the inspector to show me where I can find various regulatory requirements in the procedures. This approach has the added benefit of forcing the inspector to demonstrate they are trained in the procedures—a more effective assessment of competency than reviewing a training record.

Challenging Process Owners

The seventh and final step of the turtle diagram seems to challenge process owners the most. This is where the auditor should be looking for department Quality Objectives and assessing if the department objectives are linked with company quality objectives. Manufacturing often measures first pass yield and reject rates, but every process can be measured. If the process owner doesn’t measure performance, how does the process owner know that all the required work is getting done? The seventh step also is where the auditor can sample and review the monitoring and measurement of processes, and the trend analysis can be verified to be input into the CAPA process.

In my brief description of the process approach, I used the incoming inspection process. I typically choose this process for training new auditors because it is a process that is quite similar in almost every company, and it is easy to understand. More importantly, however, the incoming inspection process does an effective job of covering more clauses of the Standard than most audits. Therefore, new auditors get an appreciation for how almost all the clauses can be addressed in one process audit. If you are interested in learning more about Turtle Diagrams and the process approach to auditing, please register for our webinar on the process approach to auditing.

Tags: auditing, ISO, training auditors

Auditor Job Responsibilities: The Toughest Thing to Do

Posted by Rob Packard on February 29, 2012

The author reveals his thoughts related to auditor job responsibilities and what the toughest thing to do is.

Today was my last day as an external resource for BSI, and tomorrow will be my last day as an independent consultant. On March 1^st, I begin a new job as Sr. Regulatory Affairs Manager for Delcath Systems, Inc. in Queensbury, NY. I am grateful to everyone that I had the pleasure of meeting during the past two-and-half years as a 3^rd party auditor, instructor, and consultant. I have learned so much from you all. Your parting wishes were very kind and supportive. I sent out emails to as many of you as I could to notify you of this change. Instead of brief acknowledgment and “Good Luck!” I received genuine words of thanks and compliments that made me feel very lucky that we have had such an unusual relationship for an auditor and auditee—very much like cats and dogs that learn to live together in the same house.

One of you described the typical relationship with an auditor quite well, “Having an auditor come to your place is always a somehow stressful time. You are always afraid of failing somewhere.” This same person sent me an email last night saying, “I feel like you are one of my friends.” Another auditee walked by another team member and me a few weeks ago while we were waiting for a ride. Instead of avoiding eye contact and walking right on by, he stopped and thanked us for really helping to bring attention to areas that need improvement. This same gentleman had endured a tough interview by me, where I pointed out mistakes in drawings, procedures, and his own QC inspection of incoming raw materials. This person has the right attitude.

Auditor Job Responsibilities

As an auditor, we must come to a conclusion as to whether the evidence we collect demonstrates conformity or nonconformity. When we identify nonconformities, we must explain our findings. The toughest part of the job is how to “break the news.” If you do it well, the auditee will agree with you and thank you for helping to improve the quality system. If you do it poorly, the auditee will resent you and may even toss you out on your ear.

In my first-ever ISO certification audit, I was the auditee, and the auditor that interrogated our team was horrible. Not only did the auditor “break the news” poorly, but the conclusions were also wrong in several instances. To make matters worse, the CEO and the regulatory consultant I had hired were so upset with our auditor that I had to play referee just to keep them from killing the auditor. We received a recommendation for ISO 13485 certification at the end of that audit, but I learned a valuable lesson: “Always look at an audit as an opportunity to improve.” The worst that can happen is that the auditor will require you to implement corrective action. The best that can happen is that you will need to perform internal audits to identify opportunities for corrective actions on your own. Who cares who finds the opportunities to improve?

Auditors and auditees maybe cats and dogs, but we should learn to help each other get better without getting upset or feeling anxious.

My third-party auditing days may be done, but I will continue to share my thoughts through this blog, and I hope you will share your feedback too.

Tags: auditor, auditor job responsibilities

Posted in: ISO Auditing

Improving Your ISO Internal Auditing Schedule

Posted by Rob Packard on May 27, 2011

The author provides tips on how to improve the efficiency and effectiveness of your internal auditing schedule.

Each week I audit a different company, or I teach a group of students how to audit. In the courses I teach, I use a slide that gives an example of an internal auditing schedule (see the example above). On the surface, this example seems like a good audit schedule. There are 12 auditors performing two audits each year. If each auditor spends a day auditing, and another day writing the report, the combined resources equal 48 days (~$20,000) allocated to auditing, and each person spends less than two percent of their work year auditing.

Unfortunately, I have learned that the quality of auditing is directly related to how much time you spend auditing. Therefore, I recommend using fewer auditors. There is no perfect number, but “less is more.” My example also has another fundamental weakness. The internal auditing schedule does not take full advantage of the process approach to auditing. Instead of performing an independent audit of document control and training, these two clauses/procedures should be incorporated into every audit. The same is true of maintenance and calibration. Wherever maintenance and calibration are relevant, these clauses should be investigated as part of auditing that area.

For example, when the incoming inspection process is audited, it only makes sense to look for evidence of calibration for any devices used to perform measurements in that area. For a second example…when the production area is being audited, it only makes sense to audit maintenance of production equipment too.

If the concept of process auditing is fully implemented, the following ISO 13485 clauses can easily be audited in the regular course of reviewing other processes: 4.2.1), Quality System Documentation, 4.2.3), Document Control, 4.2.4), Record Control, 5.3), Quality Policy, 5.4.1), Quality Objectives, 6.2.2), Training, 6.3), Maintenance, 6.4), Work Environment, 7.1), Planning of Product Realization & Risk Management, 7.6), Calibration, 8.2.3), Monitoring & Measurement of Processes, 8.5.2), Corrective Action, and 8.5.3) Preventive Action. This strategy reduces the number of audits needed by more than half.

Internal Auditing: Upstream/Downstream Examples

Another way to embrace the process approach to auditing is to assign auditors to processes that are upstream or downstream in the product realization process from their own area. For example, Manufacturing can audit Customer Service to understand better how customer requirements are confirmed during the order confirmation process. This is an example of auditing upstream because Manufacturing receives the orders from Customer Service—often indirectly through an MRP system. Using this approach allows someone from Manufacturing to identify opportunities for miscommunication between the two departments. If Regulatory Affairs audits the engineering process, this is an example of auditing downstream. Regulatory Affairs is often defining the requirements for the Technical Files and Design History Files that Engineering creates. If someone from Regulatory Affairs audits these processes, the auditor will realize what aspects of technical documentation are poorly understood by Engineering, and quickly identify retraining opportunities.

One final aspect of the example internal auditing schedule that I think can be improved is the practice of auditing the same process twice per year. This practice doesn’t seem to work very well for a few reasons. First, it requires that an auditor prepare for an audit twice per year and write two reports, instead of one. This doubles the number of time auditors spends in preparation and follow-up activities associated with an audit. Second, increasing the number of audits naturally shortens the duration of each audit. It is more difficult for auditors to cover all the applicable clauses in a shorter audit because it takes time to locate records and pursue follow-up trails. Longer audits, covering more clauses, make it easier for the auditor to switch to a different clause while they are waiting for information. Third, if an area is audited every six months, it is often difficult to implement corrective actions and produce evidence of effectiveness before the area is due for auditing again.

I can’t provide a generic internal auditing schedule that will work for every company or even show how all the clauses will be addressed in one table. I can, however, provide an example of an improved schedule that illustrates the above concepts. This example (see below) uses four auditors instead of 12, and the number of days planned for each audit is two days instead of one. The preparation and reporting time is still one day per audit. Therefore the combined resources equal 24 days (~$10,000) allocated to auditing, and each person spends two and one-half percent of their work year auditing. My intention is not to create the perfect plan, but to give audit program managers some new ideas for more efficient utilization of resources. I hope this helps, and please share your own ideas as comments to this posting.

Tags: Audit, audit schedule, Internal Auditing

Posted in: ISO Auditing

7 Steps to Effective Auditor Training

Posted by Rob Packard on January 7, 2011

A five-day lead auditor course is never enough. Effective auditor training must include practical feedback from an experienced auditor.

Recently, a client asked me to create a training course on how to train operators. I could have taught the operators myself, but there were so many people that needed training, that we felt it would be more cost-effective to train the trainers. Usually, I have multiple presentations archived that I can draw upon, but this time I had nothing. I had never trained engineers on how to be trainers before—at least not formally. I thought about what kinds of problems other quality managers have had in training internal auditors, and how I have helped the auditors improve. The one theme I recognized was that effective auditor training needs to include practical feedback from an experienced auditor.

How do you audit the auditing process?

Most quality managers are experienced and have little trouble planning an audit schedule. The next step is to conduct the audit. The problem is that there is very little objective oversight of the auditing process. The ISO 13485 standard for medical devices requires that “Auditors shall not audit their own work.” Therefore, most companies will opt for one of two solutions for auditing the internal audit process: 1) hire a consultant, or 2) ask the Director of Regulatory Affairs to audit the internal auditing process.

Both of the above strategies for auditing the internal audit process meet the requirements of ISO 13485, but neither approach helps to improve an internal auditor’s performance. I have interviewed hundreds of audit program managers over the years, and the most common feedback audit program managers give is “change the wording of this finding” or “you forgot to close this previous finding.” This type of feedback is related to the report writing phase of the audit process. I rarely hear program managers explain how they help auditors improve at the other parts of the process.

When auditors are first being trained, we typically provide examples of best practices for audit preparation, checklists, interviewing techniques, AND reports. After the auditors have been “shadowed” by the program manager for an arbitrary three times, the auditors are now miraculously “trained.” Let’s see if I can draw an analogy that will make my point…

That kind of sounds like watching your 16-year-old drive the family car three times and then giving them a license. I guess that’s why my new Ford Festiva was severely dented on all four sides within six months. You may think my father was a Saint, but I think he might have totaled his tenth car by age 18. At least I contained the damage to one vehicle.

7 Steps to Effective Auditor Training

The key to training auditors to audit is consistent follow-up over a long period of time (1-2 years depending upon the frequency of audits). I recommend following the same training process that accredited auditors must complete. I have adapted that process and developed seven (7) specific recommendations:

Have a new auditor observe a few audits before they are allowed to participate (make sure they take notes and explain what you are doing and why, as you conduct audits they are observing)
Have new auditors join as team members for 10-20 audits, before they are allowed to act as a lead auditor
Have new lead auditors conduct team audits with another qualified lead auditor for 10-20 audits before you allow them to conduct an audit alone
Shadow new auditors for 100% of their first audit and gradually observe less with each subsequent audit; try to plan the shadowing into your audit agenda
Review the notes of new auditors periodically throughout the audit to provide suggestions for improvement and identify missing information
Have new lead auditors submit a draft audit agenda to you before sending it to the supplier or department manager
Have new lead auditors rehearse their first few opening and closing meetings with you in private before conducting the opening and closing meeting (make sure they have an opening/closing meeting checklist to help them)

About the Author

Robert Packard is a regulatory consultant with 25+ years of experience in the medical device, pharmaceutical, and biotechnology industries. He is a graduate of UConn in Chemical Engineering. Robert was a senior manager at several medical device companies—including the President/CEO of a laparoscopic imaging company. His Quality Management System expertise covers all aspects of developing, training, implementing, and maintaining ISO 13485 and ISO 14971 certification. From 2009-2012, he was a lead auditor and instructor for one of the largest Notified Bodies. Robert’s specialty is regulatory submissions for high-risk medical devices, such as implants and drug/device combination products for CE marking applications, Canadian medical device applications, and 510(k) submissions. The most favorite part of his job is training others. He can be reached via phone 802.258.1881 or email. You can also follow him on Google+, LinkedIn or Twitter.

Tags: Internal Auditor, Train the trainer, Training

Posted in: ISO Auditing