ISO Auditing

Internal Auditing, Supplier Auditing, Lead Auditor, Internal Auditor, ISO 19011, and ISO 17021.

Hiring an Auditor

1 Comment / ISO Auditing / By /

In this article, you will learn how to hire an auditor to conduct medical device internal audits and supplier audits.

help wanted Hiring an Auditor
Stop begging people to help you audit. Learn how to recruit auditors more effectively.

Hiring and Auditor

Hiring an auditor, whether as a consultant or a permanent team member, is a critical decision that can drastically improve your quality management system and foster a culture of quality, or it can add no value and lead to disruption and frustration.  The purpose of this blog is to identify the qualities and training that make the best auditor to help you elevate your internal audit programs.  

Audit Program Structures 

Companies typically take one of the following approaches to address their internal audit requirements: 

  1. Train internal personnel with other primary functions as auditors and have them audit other departments.
  2. Hire an independent 3rd party to conduct the internal audits.
  3. Build an internal audit team that is independent of all other processes.  

Hiring an Auditor from Within 

Option 1 is common across the industry and is a personnel-efficient means of achieving the audit objectives. While this type of approach can sometimes be effective and may satisfy the basic requirement to conduct internal audits, there can be some drawbacks to this structure. Sometimes, when people were not hired specifically to be auditors and auditing is something they were asked to do in addition to their regular job, there is little to no motivation to develop auditing skills, and the audits lack a depth and thoroughness that ultimately reduces the value of the audit program. Proper internal recruiting and training of these auditors is crucial to ensuring audits are a useful value-added exercise and not a box-checking chore. 

To successfully recruit internal auditors serving in other roles, it’s important to motivate people to want to be an auditor. Let potential recruits know that employees with audit experience are more valuable to companies than those without. It exposes employees to upstream or downstream processes to better understand the overall operations and provides them the opportunity to make process improvements in both directions to their workflow. If you want to be effective and get promoted, you need to demonstrate value to your boss and top management. If you don’t understand what other departments need, how can you help them? No manager will promote a selfish, power-hungry hog. They promote team players that make others better. Auditing gives you the insight necessary to understand how you can do that.  

Once motivated and recruited, it’s important to ensure these employees have the skills and resources to be successful as auditors. To help develop their skills, training on audit processes and the responsibilities and role of an auditor in accordance with ISO 19011 will provide guidance on conducting audits and the basics of how to audit. Auditors should also be trained against the specific standard or regulation they are auditing against, which may include ISO 13485, 21 CFR 820, ISO 14971, EU MDR, and others. Resources that will support their activities may include process audit diagrams, checklists, examples of record requests, strategies for intelligent sample selection, and, of course, a clear definition of the regulatory and procedural requirements of the process that they are auditing.  

If you are looking for support in training your own employees to be internal auditors, we would be happy to outline or provide a training program specific to your company’s processes and products to ensure your auditors are competent and effective in their new role.  

Hiring a 3rd Party Auditor  

Option 2 can be useful to any company, but selecting the right auditor is essential to the success of this approach. The basic qualifications and qualities that I recommend companies look for when hiring an outside auditor are: 

  1. Experience – this includes industry experience and regulatory knowledge. An auditor with experience auditing or working for a company with similar devices, manufacturing processes, etc., will provide more value than an unfamiliar auditor. Regulatory knowledge and experience within your targeted markets are also important to evaluate to ensure that they are familiar with the standards and regulations against which they will be auditing.  
  1. Communication Skills – This is a make-or-break quality of auditors that can shift the substance of an audit from a value-added exercise to a disrupting and frustrating experience. You want to ensure that auditors are affable yet confident, able to communicate the usefulness of the audit for the purpose of process improvement and facilitate a productive dialogue, offering education and suggestions when issues or nonconformances arise.   
  1. Reputation and References – ask the auditor for references from previous clients. Contact the references to get feedback on their performance, reliability, and professionalism. This is a great way to evaluate an auditor’s communication skills and whether previous auditees gained value from the interaction.  
  1. Auditor Training – acceptable qualifications for an auditor can be defined by the company but may include lead auditor certification, demonstrated training on relevant standards with experience shadowing experienced auditors, and documented training on other relevant standards/regulations.  
  1. Audit Methodology – Inquire about how auditors plan, execute, and report on audits. What audit methodology does the auditor prefer for the scope of your audit, and why?  

There are many companies and consultants that offer 3rd party auditor services, but not all are created equal. Like the CAPA process, the internal audit program is a window into the culture surrounding quality that your company has, and by demonstrating that you are proactively policing yourself and seeing continuous improvement through an effective internal audit program will show regulators that your company has a commitment to quality.  

Hiring a Full-time Audit Team 

Option 3 is generally reserved for the resource-rich industry with operations that demand expansive continuous audit processes to justify the support of a full-time auditor or audit team. Hiring your own team benefits from the same considerations that come with hiring a 3rd party auditor; the ability for the auditors to become intimately familiar with the company, devices, and processes is valuable. For companies that do not have the need for full-time auditors, the same value of familiarity can come from building a trusted relationship with a third-party auditor or audit team, who can support your audit program year after year.  

Hiring an Auditor from Medical Device Academy  

Our goal at Medical Device Academy is to help you improve your quality system and provide valuable consulting advice to achieve improvements. We specialize in helping start-up companies achieve initial ISO 13485 certification, MDSAP certification, and CE Certification. Based on the scope of the audit and medical device, we will assign the most qualified team member. Some of our specific areas of expertise include auditing companies with manufacturing and machining, aseptic processing, agile software development, sterile products, medical device reprocessors, 3D printed manufacturing, and more. If you are interested in outsourcing any supplier or internal audit activities, you can check out our Audit Services page to get in touch or to learn more about our audit team.

Auditor shadowing as an effective auditor training technique

2 Comments / ISO Auditing / By /

This article reviews auditor shadowing as an effective auditor training technique, but we also identify five common auditor shadowing mistakes.

How do you evaluate auditor competency?

Somewhere in your procedure for quality audits, I’ll bet there is a section on auditor competency. Most companies require that the auditor has completed either a course for an internal auditor or a lead auditor course. If the course had an exam, you might even have evidence of training effectiveness. Demonstrating training competence is much more challenging. One way is to review internal audit reports, but writing reports is part of what an auditor does. How can you evaluate an auditor’s ability to interview people, take notes, follow audit trails, and manage their time? The most common solution is to require the auditor “shadow” a more experienced auditor several times, and then the trainer will “shadow” the trainee.

auditor with clip board 203x300 Auditor shadowing as an effective auditor training technique
If you are shadowing, you are taking notes, so you can discuss your observations with the person you are shadowing later. 

Auditor shadowing in 1st party audits

ISO 19011:2018 defines first-party audits as internal audits. When first-party auditors are being shadowed by a trainer or vice versa, there are many opportunities for training. The key to the successful training of auditors is to recognize teachable moments.

When the trainer is auditing, the trainer should look for opportunities to ask the trainee, “What should I do now?” or “What information do I need to record?” In these situations, the trainer asks the trainee what they should do BEFORE doing it. If the trainee is unsure, the trainer should immediately explain what, why, and how with real examples.

When the trainer is shadowing, the trainer should watch and wait for a missed opportunity to gather important information. In these situations, the trainer must resist guiding the trainee until after the trainee appears to be done. When it happens, sometimes the best tool is simply asking, “Are you sure you got all the information you came for?”

Here are five (5) mistakes that I observed trainers made when they were shadowing:

1. Splitting up, instead of staying together, is one of the more common mistakes I have observed. This happens when people are more interested in completing an audit rather than taking advantage of training opportunities. The trainee may be capable of auditing independently, but this is unfair to the trainee because they need feedback on their auditing technique. This is also unfair to the auditee because it is challenging to support multiple auditors simultaneously. When it is unplanned, trainers may not be available for both auditors. If an audit is running behind schedule, this is the perfect time to teach a trainee how to recover sometime in their schedule. Time management is, after all, one of the most challenging skills for auditors to master.

2. Staying in the conference room instead of going to where the work is done is a common criticism of auditors. If the information you need to audit can be found in a conference room, you could have completed the audit remotely. This type of audit only teaches new auditors how to take notes. These are necessary skills that auditors should master in a classroom before shadowing.

3. Choosing an administrative process is a mistake because administrative processes limit the number of aspects of the process approach that an auditor-in-training can practice. Administrative processes rarely have equipment that requires validation or calibration, and the process inputs and outputs consist only of paperwork, forms, or computer records. With raw materials and finished goods to process, the auditor’s job is more challenging because there is more to be aware of.

4. Not providing honest feedback is a huge mistake. Auditors need to be thick-skinned, or they don’t belong in a role where they will criticize others. Before you begin telling others how to improve, you must self-reflect and identify your strengths and weaknesses. Understanding your perspective, strengths, weaknesses, and prejudices is critical to being a practical assessor. As a trainer, it is your job to help new auditors to self-reflect and accurately rate their performance against objective standards.

5. “Silent Shadowing” has no value at all. By this, I mean shadowing another auditor without asking questions. You should mentally pretend you are doing the audit if you are a trainee. Whenever the trainer does something different from how you would do things, you should make a note to ask, “Why did you do that?” If you are the trainer, you should also mentally pretend you are doing the audit. It is not enough to be present. Your job is to identify opportunities for the trainee to improve. The better the trainee, the more challenging it becomes to identify areas for improvement.  This is why training other auditors have helped me improve my auditing skills.

Auditor shadowing in 2nd party audits

supply chain weakest link Auditor shadowing as an effective auditor training technique

Auditors responsible for supplier auditing are critical to supplier selection, supplier evaluation, re-evaluation, and the investigation of the root cause for any non-conformities related to a supplier. Auditor shadowing is a great tool to teach supplier auditors and other people responsible for supply-chain management what to look at and what to look for when they audit a supplier. If you are developing a new supplier quality engineer responsible for performing supplier audits, observing the auditor during some actual supplier audits is recommended. Supplier audits are defined as second-party audits in the ISO 19011 Standard. The purpose of these audits is not to verify conformity to all the aspects of ISO 13485. Instead, the primary purpose of these audits is to verify that the supplier has adequate controls to manufacture conforming products for your company consistently. Therefore, processes such as Management Review (Clause 5.6) and Internal Auditing (Clause 8.2.2) are not typically sampled during a second-party audit.

The two most valuable processes for a second-party auditor to sample are 1) incoming inspection and 2) production controls. Using the process approach to auditing, the second-party auditor will have an opportunity to verify that the supplier has adequate controls for documents and records for both of these processes. Training records for personnel performing these activities can be sampled. The adequacy of raw material storage can be evaluated by following the flow of accepted raw materials, leaving the incoming inspection area. Calibration records can be sampled by gathering equipment numbers from calibrated equipment used by both processes. Even process validation procedures can be assessed by comparing the actual process parameters being used in manufacturing with the documented process parameters in the most recent validation or re-validation reports.

I recommend having the trainee shadow the trainer during the process audit of the incoming inspection process and for the trainer to shadow the trainee during the process audit of production processes. The trainee should ask questions between the two process audits to help them fully understand the process approach to auditing. Supplier auditors should also be coached on techniques for overcoming resistance to observing processes involving trade secrets or where competitor products may also be present. During the audit of production processes, the trainer may periodically prompt the trainee to gather the information that will be needed for following audit trails to calibration records, document control, or for comparison with the validated process parameters. The “teachable moment” is immediately after the trainee misses an opportunity, but while the trainee is still close enough to go back and capture the missing details.

Are you allowed to shadow a 3rd party auditor or FDA inspector?

qsit inspection Auditor shadowing as an effective auditor training technique

Consider using 3rd party audits and inspections as an opportunity to shadow experienced auditors to learn what they are looking at and what they look for. In addition to shadowing an expert within your own company or an auditor/consultant you hire for an internal audit, you can also shadow a 3rd party auditor or an FDA inspector. This concept was the subject of a discussion thread I ran across on Elsmar Cove from 2005. The comments in the discussion thread supported the idea of shadowing a 3rd party auditor. The process owner (i.e., the manager responsible for that process) should be the guide for whichever process is being audited, and the process owner is responsible for addressing any non-conformities found in the area., The process owner should be present during interviews, but the process owner should refrain from commenting. The 3rd party auditor and the process owner need to know if the person being interviewed was effectively trained and if they can explain the process under the pressure of an audit or FDA inspection. If you are interested in implementing this idea, I recommend using one of two strategies (or both):

  1. Consider having the internal auditor that audited each process shadow the certification body auditor for the processes they audited during their internal audit. This approach will teach your internal auditor what they might have missed, and they will learn what the 3rd party auditors look for to simulate a 3rd party audit more effectively when conducting internal audits.
  2. Consider having the internal auditor that is assigned to conduct the next process audit of each process shadow the certification body auditor for that process. This approach will ensure that any nonconformities observed during the 3rd party audit are checked for the effectiveness of corrective actions during the next internal auditor. Your internal auditor will know precisely how the original nonconformity was identified and the context of the finding.

Seven ways to improve quality auditor training

2 Comments / ISO Auditing / By / , ,

A five-day lead auditor course is never enough. Effective quality auditor training must include practical feedback from an expert.

What is required for quality auditor training?

The key to training auditors to audit is consistent follow-up over a long period of time (1-2 years, depending upon the frequency of audits). I recommend following the same training process that accredited auditors must complete. I have adapted that process and developed seven (7) specific recommendations.

Training the trainer

One of my clients asked me to create a training course on how to train operators. I could have taught the operators myself, but so many people needed training that we felt it would be more cost-effective to train the trainers. Usually, I have multiple presentations archived that I can draw upon, but this time I had nothing. I had never trained engineers on how to be trainers before—at least not formally. I thought about the problems other quality managers have had in training internal auditors and how I have helped the auditors improve. The one theme I recognized was that effective quality auditor training needs to include practical feedback from an experienced auditor. An expert auditor that is training new auditors needs to identify systematic ways to provide feedback, and setting a benchmark for the number of times feedback will be provided is really helpful.

Improve by observing yourself and other quality auditors

Observing someone else is a great way to learn when you are learning any new skill. Interns often do this, which is also a technique used to train new auditors. This technique is called shadowing. You can learn by watching, but eventually, you need to try to do tasks that are beyond your comfort level, and it is best to practice auditing with an expert watching you.

Practice team member audit preparation

Many of the internal auditing procedures we see require new auditors to conduct three audits as team members before they can audit independently. In contrast, notified body auditors join as team members for 10-20 audits before they can act as lead auditors. During the training period, auditors in training observe multiple lead auditors and multiple quality systems. Each audit allows auditors in training to write nonconformities and receive feedback from a lead auditor. At the beginning of quality auditor training, the focus must be on audit preparation. What are the areas of importance, what are the results of previous audits, are there any previous audit findings to close, etc? This preparation can even be done as practice for a hypothetical audit.

During quality auditor training, practice the opening and closing meetings

Opening and closing meetings are one of the first things to teach a new lead auditor. Have new lead auditors rehearse their first few opening and closing meetings with you in private before conducting the opening and closing meetings. Ensure the lead auditor has an opening/closing meeting checklist to help them. Recording practice sessions is enormously helpful because the trainee can watch and observe their mistakes. As trainees get more experience, the opening and closing meetings should have time limits. Finally, you might ask members of top management to challenge the lead auditor with questions. The lead auditor needs to be comfortable with their decisions and the grading of the audit findings.

How to practice audit team leadership

Have new lead auditors conduct team audits with another qualified lead auditor for 10-20 audits before you allow them to conduct an audit alone. Leading the opening and closing meetings is usually the first area new lead auditors master. The most complicated area to learn is managing a team of auditors. Team members will fall behind schedule during audits, or someone will forget to audit a process. As a lead auditor, you must complete the audits for your assigned processes and communicate with the entire team to ensure everyone else is on schedule. As an observer, you must let lead auditors make mistakes and help them realize them. Initially, a trainer will encourage new lead auditors to give themselves more than enough time. As their training progresses, the timing needs to be shorter and more challenging. Ultimately, you have to push the team beyond its capability to teach new lead auditors to recognize problem signs and teach them how to fix the problems.

Shadow auditors virtually with recordings

Live shadowing is challenging for experts and trainees because you are distracted by listening to the auditee and observing the auditor. However, if an audit is recorded, the person shadowing can watch the recording. The audit is already completed, and there is little need to concentrate on the auditee. A recording allows the observer to focus on the auditor. If a new auditor is conducting their first audit, an expert should shadow the trainee for 100% of the audit. Gradually the observation can decrease with each subsequent audit.

Practice note-taking with recorded audits

Taking detailed notes is something that experts take for granted, but I learned a lot by watching FDA inspectors take notes during an inspection. Have a new auditor observe a few audits before they are allowed to participate. Make sure they take notes and explain what you are doing and why they are observing as you conduct audits. Review the notes of new auditors periodically throughout the audit to provide suggestions for improvement and identify missing information. You can also record a supplier audit or internal audit and let a new trainee take notes on the pre-recorded webinar. This eliminates the need to coordinate schedules to involve the trainee.

Quality auditor training should include practicing audit agenda creation

Have new lead auditors submit a draft audit agenda to you before sending it to the supplier or department manager. Usually, the first audit agenda will need revision and possibly multiple revisions. Make sure you train the person to include enough detail in the agenda, and using a checklist or template is recommended. The agenda creation will be part of the audit preparation, and it can be done without time pressure.

How do you audit the auditing process?

Most quality managers are experienced and have little trouble planning an audit schedule. The next step is to conduct the audit. The problem is that there is very little objective oversight of the auditing process. The ISO 13485 standard for medical devices requires that “Auditors shall not audit their own work.” Therefore, most companies will opt for one of two solutions for auditing the internal audit process: 1) hire a consultant or 2) ask the Director of Regulatory Affairs to audit the internal auditing process.

Both of the above strategies for auditing the internal audit process meet the requirements of ISO 13485, but neither approach helps to improve an internal auditor’s performance. I have interviewed hundreds of audit program managers over the years, and the most common feedback audit program managers give is “Change the wording of this finding” or “You forgot to close this previous finding.” This type of feedback is related to the report-writing phase of the audit process. I rarely hear program managers explain how they help auditors improve at the other parts of the process.

When auditors are first being trained, we typically provide examples of best practices for audit preparation, checklists, interviewing techniques, AND reports. After auditors are “shadowed” by the audit program manager for an arbitrary three times, the auditors are now miraculously “trained.” Let’s see if I can draw an analogy to make my point.

That kind of sounds like watching your 16-year-old drive the family car three times and then giving them a license.

About the Author

Rob Packard 150x150 Seven ways to improve quality auditor trainingRobert Packard is a regulatory consultant with 25+ years of experience in the medical device, pharmaceutical, and biotechnology industries. He is a graduate of UConn in Chemical Engineering. Robert was a senior manager at several medical device companies—including the President/CEO of a laparoscopic imaging company. His Quality Management System expertise covers all aspects of developing, training, implementing, and maintaining ISO 13485 and ISO 14971 certifications. 2009-2012, he was a lead auditor and instructor for one of the largest Notified Bodies. Robert’s specialty is regulatory submissions for high-risk medical devices, such as implants and drug/device combination products for CE marking applications, Canadian medical device applications, and 510(k) submissions. The most favorite part of his job is training others. He can be reached via phone at 802.258.1881 or by email. You can also follow him on LinkedIn, Twitter, and YouTube.

Audit schedule and an audit agenda, what’s the difference?

8 Comments / ISO Auditing / By / , ,

Internal audit and supplier audit programs both require an audit schedule and an audit agenda, but what’s the difference between them?

What is an audit schedule?

An “audit schedule” is not a formal definition in ISO 19011:2018. However, section 5.1 of that standard states that your audit program should include nine different requirements. Item “d” is “d) schedule (number/duration/frequency) of the audits.” Typically, the audit program manager will maintain an annual audit schedule with a date indicating the date the schedule was last revised. The most common example in lead auditor training is a matrix like the one shown below. The left-hand column will list all of the individual processes that are identified in the company’s process interaction diagram, and the top row of the matrix will indicate the month when each process audit will be conducted. Typically, the expectation is to complete the audit sometime during that month, but some quality auditing procedures specify that the audit may be completed the month before or the month after to accommodate the process owner. The regulations only require that you document and maintain an audit schedule, and the standard is only considered guidance.

%name Audit schedule and an audit agenda, what’s the difference?
Example of an Internal Audit Schedule

I use a slide in lead auditor courses that gives the example of an internal auditing schedule provided above. On the surface, this example seems like a good audit schedule. Twelve auditors perform two audits each year. If each audit requires approximately two days, each auditor spends less than two percent of their work year auditing. Unfortunately, a two-percent allocation of time is insufficient to become or remain proficient at auditing. An improvement to the auditing schedule would be to assign fewer auditors so each auditor gets more experience. There is no perfect number, but assigning a few specialists will improve the chances of becoming and remaining proficient at auditing.

What is an audit agenda?

An “audit agenda” is not a formal definition in ISO 19011:2018 either. In fact, the word “agenda” is not even used in ISO 19011. Instead, section 5.5.5 of ISO 19011 states that “The assignment [of the individual audit] should be made in sufficient time before the scheduled date of the audit, in order to ensure the effective planning of the audit.” The audit plan must also be part of the records [i.e., Clause 5.5.7(b)]. Therefore, “agenda” and “plan” may be used interchangeably. Details of audit planning are provided in 6.3.2 of ISO 19011.

6 Steps to Creating an Audit Schedule

There are six steps to creating an audit schedule:

  1. What were the results of previous audits?
  2. Which processes are the most important to audit?
  3. Who should conduct your internal audit?
  4. How long should your internal audit be?
  5. Should you conduct one full quality system audit or several audits?
  6. Is a remote audit good enough?

We will address each of the six steps below.

How do the results of previous audits impact your audit schedule?

The results of an audit include nonconformities, observations for improvement (OFI), and a conclusion regarding whether the quality system is effective or not. Usually, most processes are effective, and there are no nonconformities or OFIs. Therefore, any processes that had a nonconformity or OFI should be prioritized in the audit schedule and audit planning for the future. For internal and supplier audits, a best practice is for the auditor and the process owner to discuss the corrective actions planned and determine the appropriate timeline or implementation of actions planned. Then the auditor can indicate a timeframe for re-auditing the nonconforming process after corrective actions are implemented. This strategy allows the auditor to be part of the effectiveness check. This approach is appropriate for individual process audits but not for a full-quality system audit.

Which processes are the most important to audit?

The primary element impacting the importance of processes is the risk to product quality associated with the process. Usually, support processes are of lower importance because they do not directly impact product quality. In contrast, core processes directly involved in a device’s design, manufacture, and distribution are critical. Most auditors and audit program managers emphasize design controls and production process controls as important areas to audit. However, the distribution area is often neglected. Other core processes are purchasing, sales, customer service, and servicing. Not every process is equally important when comparing two companies. For example, device manufacturers that only make software as a medical device (SaMD) often have very limited purchasing and incoming inspection activities to audit.

Who should the audit program manager assign to each internal audit?

%name Audit schedule and an audit agenda, what’s the difference?The example of a revised audit schedule provided above identifies the departments where each of the auditors works with color coding. This is done to ensure that auditors are not assigned to audit processes where they might have a conflict of interest (i.e., they would be auditing their own work). This is the most important aspect of assigning auditors. The second most important aspect is to make sure the auditor has the technical knowledge to audit the process. It is challenging to conduct an audit of manufacturing if you have not spent any time in manufacturing before. If auditors are new and their training is in progress, then the audit program manager may assign the auditor to a process specifically to give them more experience with that type of process. Inexperienced auditors often are assigned less important processes that have not changed recently. However, a better approach to training auditors is to give them a challenge with support. Having the new auditor prepare a detailed sampling plan and list of questions before the audit can prepare them for auditing a more challenging, important process that is likely to have one or more nonconformities. Auditing processes that have nonconformities is also the best way to teach a new auditor how to write the audit findings.

What should be the duration of each internal audit in your schedule?

The duration of an audit should be based on the results of previous audits, but other important factors include: 1) the number of personnel involved in the process, 2) the complexity of the process, and 3) the risk to product quality associated with the process. The MDSAP program uses a procedure for audit time determination (i.e., MDSAP AU P0008.007: Audit Time Determination Procedure), and the MDSAP audit approach document (i.e., MDSAP AU P0002.008 Audit Approach) classifies processes as having either a “direct” or “indirect” impact upon product quality based upon the applicable clause of the process (i.e., Clauses 0-6.3 are indirect, and Clauses 6.4-8.5.3 are direct). For example, the production processes and design and development processes both involve a large number of people in most organizations, the processes are complex, and both processes directly impact product quality. Therefore, I typically allocate 3-4 hours to each of those processes during an audit. In comparison, incoming inspection often involves one or two people, and the process often involves only one procedure. Incoming inspection is a “direct” process, but less time (e.g., 1 hour) should be allocated to auditing incoming inspection than the other two processes–unless there was a nonconformity in the incoming inspection process during a recent audit or unless the process was recently changed.

Should you conduct one full quality system audit or several audits?

Both approaches have strengths and weaknesses, but there is not a single best way. If I am using employees to conduct an audit, then I typically restrict the scope of the audit to a single process. Alternatively, when I use a consultant to conduct an audit, I typically conduct a full-quality system audit to minimize travel costs. Another strategy I have recommended is to identify the processes that are most important to audit first (e.g., processes with recent changes and/or nonconformities), and these processes are scheduled for individual process audits during the first half of the audit schedule. Then I schedule a full-quality system audit in the second half of the audit schedule. The strategy ensures that all important processes will be audited twice in one year, but every process will be audited at least once.

Remote audits vs On-site audits

Prior to the Covid-19 pandemic, remote audits were rare in the medical device industry. Many NBs insisted that remote audits were not permitted or effective. The pandemic forced the entire industry to create policies for remote auditing and to use remote auditing whenever possible. Now that the pandemic has ended, many companies continue to conduct remote audits to save money. Even NBs are conducting more remote audits for Stage 1 readiness audits during the ISO 13485 certification process. ISO 19011 has a section in the Appendices outlining the differences between remote and on-site audits. However, there is a minimal advantage to conducting an on-site audit of a process where the auditor is expected to spend all of their time in a conference room during the audit. If the audit is going to be done in a conference room, then why not conduct it remotely? The one exception is when most records are paper-based and unavailable electronically. Alternatively, an on-site audit is generally more effective if the process involves observing inspection activities or assembly operations. Remote audits of inspection activities and assembly operations should be reserved for re-auditing or when a process has been audited on-site in the past, but an on-site audit would still be more effective for those processes.

How many times should a process be audited annually?

Many notified bodies will expect companies to audit all processes at least once during the year. However, it doesn’t expressly state this as a requirement in the regulations, and some companies justify skipping processes that are functioning well and have not changed in the past year. Our team is seeing this more frequently as the number of lead auditors worldwide has become scarce due to the requirements of MDSAP, the MDR/IVDR implementation, and unannounced audits. However, I almost never see the opposite justification (i.e., auditing a process more than once a year). If a process has been changed significantly, or there were nonconformities, then re-auditing the process may be used to verify the effectiveness of corrective actions or to verify that personnel are compliant with the revised process.

How to take advantage of the process approach to auditing

Another improvement that can be made to the revised example of an audit schedule is to use the process approach to auditing. Instead of performing an independent document control and training audit, these two clauses/procedures can be incorporated into every audit. The same is true of maintenance and calibration support processes. Wherever maintenance and calibration are relevant, these clauses should be investigated as part of auditing that area. For example, when the incoming inspection process is audited, it makes sense to look for evidence of calibration for any devices used to perform measurements in that area. When production process controls are being audited, maintenance records of production equipment should also be sampled.

If the concept of process auditing is fully implemented, the following ISO 13485 clauses can easily be audited in the regular course of reviewing other processes:

  • 4.2.1), Quality System Documentation;
  • 4.2.3), Document Control;
  • 4.2.4), Record Control;
  • 5.3), Quality Policy;
  • 5.4.1), Quality Objectives;
  • 6.2.2), Training;
  • 6.3), Maintenance;
  • 6.4), Work Environment;
  • 7.1), Planning of Product Realization & Risk Management
  • 7.6), Calibration;
  • 8.2.3), Monitoring & Measurement of Processes
  • 8.5.2), Corrective Action; and
  • 8.5.3) Preventive Action.

This strategy reduces the number of process audits needed by more than half.

Internal Auditing: Upstream/Downstream Examples

Another way to embrace the process approach to auditing is to assign auditors to upstream or downstream processes in the product realization process from their own area. For example, Manufacturing can audit Customer Service to understand better how customer requirements are confirmed during the order confirmation process. This is an example of auditing upstream because Manufacturing receives the orders from Customer Service—often indirectly through an MRP system. Using this approach allows someone from Manufacturing to identify opportunities for miscommunication between the two departments. If Regulatory Affairs audits the engineering process, this is an example of auditing downstream. Regulatory Affairs is often defining the requirements for the Technical Files and Design History Files that Engineering creates. If someone from Regulatory Affairs audits these processes, the auditor will realize what aspects of technical documentation are poorly understood by Engineering and quickly identify retraining opportunities.

ISO 19011 – Do you need this quality system auditing standard?

5 Comments / Auditing, ISO Auditing, Remote Auditing / By / , ,

Read this article to learn why ISO 19011 standard is a vital guidance for anyone that audits quality systems or manages an audit program.

What is ISO 19011?

ISO 19011 is a seven-part international standard for auditing management systems. The standard defines the eight principles of auditing (e.g., the process approach to auditing), provides guidance on managing audit programs and conducting audits, and includes recommendations for evaluating people for competency. There is also an appendix with details on conducting on-site and remote audits.

If you have ever taken a lead auditor course for ISO 13485, or one of the other quality management system standards, one of the critical handouts for the class should have been ISO 19011. The title is “Guidelines for Auditing Quality Management Systems.” In 2018, ISO 19011 was updated, and the changes were not superficial. If you need to purchase a copy of ISO 19011:2018, the Estonian Center for Standardization and Accreditation is the least expensive source we know.

ISO 19011 covers the topic of quality management system auditing. This Standard provides guidance on managing audit programs, conducting internal and external audits, and determining auditor competency.  One of the most common points of confusion in the lead auditor course is the difference between first, second, and third-party audits. In the first edition of this Standard, the difference between first, second, and third-party audits was just a note at the bottom of page one and the top of page two. The note was also not clear. In the second edition of 19011, in Table 1 (reproduced below), the difference between these three types of auditing is crystal clear. Table 1 was modified further in the 3rd edition to include a bottom row that remains unchanged in the 3rd edition, released in 2018.

Types of Audits Table 1 1024x205 ISO 19011 Do you need this quality system auditing standard?

Figure 1, found in Clause 5.1 of the 2nd edition, was combined with Figure 2, found in Clause 6.1 of the 2nd edition. The combined figure is now Figure 1 in the 3rd edition. The combined scope of Figure 1 is now a “Process flow for the management of an audit program” and a “Process flow for conducting an audit.” The figure categorizes the various stages of audit program management and conducting an audit into the Plan-Do-Check-Act (PDCA) cycle. We highly recommend this style for presenting any process in your internal procedures as an example of best practices in writing an SOP. The flow chart even references each of the clauses in the Standard.

The 2018 version still includes an opening meeting checklist (i.e., Clause 6.4.3) and a closing meeting checklist (i.e., Clause 6.4.10). Figure 3 in the 2nd edition, “Overview of the process of collecting and verifying information,” was a poor example of a flow chart. The committee did not update the figure when the standard was updated for the 3rd edition. Therefore, we updated the figure below to provide additional traceability to the Clauses of the Standard. If you incorporate this figure into your quality auditing procedure, you should substitute references to your procedure’s sections instead of the clauses of the standard.

Figure 2 ISO 19011 2018 1024x702 ISO 19011 Do you need this quality system auditing standard?

Competency Requirements in ISO 19011

Many audit procedures neglect to define the qualifications and methods for determining the competency of the audit program manager. Clause 5.3.2 tells you how. Put it in your own procedure. Most of the procedures we read include qualifications for a “Lead Auditor,” but we seldom see anything regarding competency. Unfortunately, this Standard only explicitly addresses the “Lead Auditor” competency in a two-sentence paragraph—Clause 7.2.5. When we teach people how to be Lead Auditors, we spend more than an hour on this topic alone.

The Standard would be more effective by providing an example of how third-party auditors become qualified as a Lead Auditor. Third-party accreditation requires the auditor to be an “acting lead” for audit preparation, opening meetings, conducting the audit, closing meetings, and final preparation/distribution of the audit report. This must be performed for 15 certification audits (i.e., – Stage 2 certification or re-certification), and another qualified lead auditor must evaluate you and provide feedback.

Appendices in ISO 19011

The appendices were the last significant additions to this Standard in 2011 (i.e., 2nd edition). Annex A provided examples of discipline-specific knowledge and skills of auditors. This section was eliminated from the 3rd edition of ISO 19011:

“Due to the large number of individual management system standards, it would not be practical to include competence requirements for all disciplines.” – Copied from the Foreward

I think providing adding a short Annex to each management system standard that defines recommended discipline-specific knowledge would be helpful. Still, that kind of change would need to be initiated with the next version of ISO 9001.

Appendix B in the 2nd edition is now Appendix A in the 3rd edition of ISO 19011. A table (Table A.1 – Audit Methods) compares conducting on-site and remote audits. We were pleased to see that conducting interviews is a significant part of remote auditing in this table. Section A.17 in the appendix provides suggestions for conducting interviews. Still, if you exhibit all 13 professional behavior traits found in Clause 7.2.2, you don’t need advice on speaking with people. For the rest of us mortals, we could use a five-day course on interviewing alone. To improve your skills in this area, ask an experienced auditor with solid interviewing skills to watch and comment on a recording of a virtual audit you perform. Watching yourself audit is cringe-worthy, but we guarantee you will improve.

What are the primary changes to the 2018 version of the standard?

There are seven main differences between the second edition, published in 2011, and the third edition of ISO 19011, released in 2018:

  1. addition of a seventh principle of auditing in sub-clause 4(g) (i.e., risk-based approach);
  2. more guidance on audit program management in Clause 5, including audit program risk;
  3. expansion of Clause 6 on conducting an audit–especially Clause 6.3 on audit planning;
  4. expansion of auditor competence requirements in Clause 7;
  5. updating of terminology to emphasize processes rather than objects;
  6. removal of an annex containing competence requirements for specific quality management systems;
  7. expansion of Annex A to include guidance on new auditing concepts such as remote audits.

Risk-based auditing is the most significant change in the 2018 version of ISO 19011

One of the main differences between ISO 19011:2018 and the previous 2011 version is the addition of a “risk-based approach” to the principles of auditing. Specifically, clause 4(g) of the guidelines for auditing management systems is, “The risk-based approach should substantively influence the planning, conducting and reporting of audits to ensure that audits are focused on matters that are significant for the audit client, and for achieving the audit program objectives.” A lot of people are unsure of what is meant by a risk-based approach. Still, the key to understanding this is to focus on the definition of risk. From a product perspective, the risk is the “combination of the probability of occurrence of harm and the severity of that harm.” From a process perspective, the risk is the “effect of uncertainty on an expected result” (ISO 9001:2015, clause 3.09). Therefore, auditors should emphasize medical devices with the highest severity of harm and devices with a high probability of hazards or hazardous situations. When an auditor focuses on a process rather than a specific medical device, auditors should emphasize any processes that are not under control and any recent process changes.

animal nature reptile animal world ISO 19011 Do you need this quality system auditing standard?

What is risk-based auditing?

Risk-based auditing considers the risks of failing to achieve audit objectives and the opportunities created by choosing various audit methods and strategies. For example, a desktop audit of procedures might be appropriate if you are conducting your first internal audit for a new quality system. Alternatively, a desktop audit would be a waste of time if you are auditing a mature quality system where very few changes to procedures have been made in the past year. Using the element approach to auditing is unlikely to add much value. Audits are meant to be a sampling. Therefore, you should focus on areas of importance where previous nonconformities were identified, any new products or processes, and anything that changed significantly.

Auditor selection should also be risk-based

Suppose you are conducting a supplier audit as part of your initial supplier qualification for a critical component supplier or contract manufacturer. In that case, you should consider doing a team audit with a multi-disciplinary team. This is a risk-based approach to the supplier qualification process, which ensures that subject matter experts evaluate each process instead of auditors with a general quality assurance background. This approach also forces more of your personnel to introduce themselves to the new supplier, and the audit will develop more reliable communication channels between your two companies. Alternatively, if you are conducting a routine internal audit of a production process, you might select a new lead auditor to conduct the audit. You don’t expect any significant findings in a routine internal audit of an established production process. In your role as an audit program manager, you need to match the new lead auditor to a process that will force them to look at all aspects of the process approach to auditing. Specifically, process validation, calibration, maintenance, and process monitoring may not apply to other administrative process areas, such as purchasing.

Risk-based auditing should influence your auditing schedule

The frequency of auditing suppliers and internal process areas should reflect the associated risks. Therefore, when you create or update your auditing schedule, you should consider the risk level of the products being audited and the process being audited. Production processes with a moderate or high level of non-conforming products may need to be audited more than once yearly. Still, a supplier with an excellent track record of extremely high quality and on-time delivery may be audited in alternating years. If you previously scheduled a remote audit, you may want to alternate to conducting an on-site audit the next time.

The duration of your audits should not always be the same either. Suppose one production process makes one product in low volume, and another production process makes multiple products in high volume. In that case, you should not schedule a two-hour internal audit for both processes every year. The low-volume production process may only need a one-hour audit once per year. In contrast, the high-volume process may require a four-hour internal audit or multiple annual audits.

Risk-based auditing applied to remote supplier auditing

The risk-based auditing approach was added to ISO 19011:2018 as the seventh principle of auditing. This represents the most significant change to that standard, but how does it apply to remote auditing? Despite the opportunities created by remote auditing, there are also risks associated with auditing suppliers remotely. People worry about auditees hiding hazardous situations or unacceptable environmental conditions such as filth or disrepair. However, unacceptable cleanliness and maintenance practices don’t happen overnight. Therefore, you should expect a clean and well-maintained facility to remain that way. One approach is to alternate between remote and on-site audits to verify the overall condition of a supplier’s facility. Therefore, the risk of auditees hiding objective evidence is more an issue of trust than a highly probable occurrence.

The more probable risks associated with remote auditing are related to the potential lack of availability of records. This is especially important for paper-based quality systems. Most people try to address this risk by scanning paper documents and records, but scanning documents have limited value. Scanning paper documents is more efficiently performed in a large batch by an automated or semi-automated process. Also, auditors and inspectors typically focus on the most recent records, and auditors and inspectors rarely sample 100% of the records. Therefore, the best risk controls include the following:

  • Ask a guide to send a digital picture of the record.
  • Use a tripod-mounted HD webcam focused on a music stand or similar surface.
  • Ask the auditee to read the document while you take notes.

In our experience, you will probably rely on all three risk controls, but it is unlikely to delay the audit. However, in response to the limited physical access to medical device facilities and personnel, certification bodies are sending out questionnaires to assess the risk of being unable to achieve audit objectives or cover the required scope of surveillance and recertification audits. As the audit program manager, you can reduce these risks by working with supply chain managers to develop new supplier questionnaires that specifically ask questions about the capability of supporting audits remotely. In particular, it would be essential to obtain facility maps to identify areas with inadequate cellular coverage and identify records that are only available in hardcopy format.

Auditing Risk Management Files

3 Comments / Auditing, CE Marking, ISO 14971:2019 (Risk Management), ISO Auditing, Remote Auditing / By / ,

What do you look at and look for when you are auditing risk management files to ISO 14971 and the new Regulation (EU) 2017/745?

Your cart is empty

Next week, November 15th @ Noon EST, you will have the opportunity to watch a live webinar teaching you what to look at and what to look for when you are auditing risk management files to Regulation (EU) 2017/745 and ISO 14971. Risk Management Files are one of the essential requirements of technical documentation required for CE Marking of medical devices. Most quality system auditors are trained on how to audit to ISO 13485:2016 (or an earlier version of that standard), but very few quality system auditors have the training necessary to audit risk management files.

Why you are not qualified to audit risk management files

Being a qualified lead auditor is not enough to audit the risk management process. When you are auditing a risk management file, you need risk management training and lead auditor training. To audit the risk management process, you will also need training on applicable guidance documents (i.e., ISO/TR 24971:2020) and applicable regulations (i.e., Regulation 2017/745 and/or Regulation 2017/746). There may also be device-specific guidance documents that specify known risks and risk controls that are considered state-of-the-art.

Creating an audit agenda

Once you have scheduled an audit of risk management files, and assigned a lead auditor, then the lead auditor needs to create an audit agenda. The audit can be a desktop audit that is performed remotely, or it can be an on-site audit. Regardless of the approach, the audit should include interviewing participants in the risk management process documented in the risk management file. As a rule of thumb, I expect a minimum of 30 minutes to be spent interviewing the process owner and one or more other participants. Then I spend an additional 60 minutes of auditing time reviewing documents and records.

Your audit agenda should specify the following items at a minimum:

  1. the method of auditing to be used,
  2. date(s) of the audit,
  3. the duration of the audit,
  4. the location of the audit, and
  5. the auditing criteria.

The auditor(s) and the auditee participants should be identified in the audit agenda. Finally, you should specify which documents and records are required for audit preparation. These documents will be used to help identify audit checklist questions and to determine a sampling plan for the audit. At a minimum, you will need a copy of the risk management procedure and a list of the risk management files that are available to audit. You may also want to request the audit plan for each of those risk management files.

What did you look at and look for during your risk management audit?

When you audit the risk management process, you could take any of the following approaches or a combination of more than one. You could audit the process according to the risk management procedure. You could audit the process according to the risk management plan(s) for each risk management file. You could audit using the process approach to auditing. Finally, you could audit in accordance with specific requirements in the ISO 14971:2019 standard and applicable regulations (i.e., Regulation 2017/745). Regardless of which approach you take, your audit notes and the audit report should identify which documents and records you sampled and what you looked for in each document. Providing only a list of the documents is not enough detail.

Creating an auditing checklist for risk management files

Auditors with limited experience are taught to create an audit checklist by creating a table that includes each of the requirements of the audit criteria. For a risk management file, this would include a list of each of the requirements in ISO 14971 for a risk management file (i.e., Clause 9???). However, this approach is more like the approach that you should be using for a gap analysis. The better approach for creating an audit checklist for risk management files is to start by creating a turtle diagram. In the “process inputs” section (i.e., step 2 of 7), you would add questions derived from your review of the risk management plan(s). In the “process outputs” section (i.e., step 3 of 7), you would add questions specific to the risk management report and other records required in a risk management file. In the “with whom” section (i.e., step 5 of 7), you would add questions related to training and competency. You might also identify additional people involved in the risk management process, other than the process owner, to interview as a follow-up trail. In the “how done” section (i.e., step 6 of 7), you would add questions specific to the procedure and forms used for the risk management process. Finally, in the “metrics” section (i.e., 7 of 7), you would verify that the company is conducting risk management reviews and updating risk management documentation in accordance with the risk management procedure and individual risk management plan(s).

Audits are just samples

Just because you can generate a lot of questions for an audit checklist does not mean that you are required to address every question. Audits are intended to be a “spot check” to verify the effectiveness of a process. You should allocate your auditing resources based on the importance of a process and the results of previous audits. I recommend approximately three days for a full quality system audit, and approximately 90-minutes should be devoted to a process unless it is the design control process (i.e., Clause 7.3 of ISO 13485) which typically requires three to four hours due to the importance and complexity of the design controls process. Therefore, you should schedule approximately 30 minutes to interview people for the risk management process and approximately 60 minutes should be reserved for reviewing documents and records. With this limited amount of time, you will not be able to review every record or interview everyone that was involved in the risk management process. This is why auditors always remind auditees that an audit is just a sampling.

Which records are required in a risk management file?

The contents of a risk management file is specified in ISO 14971:2019, Clause 4.5. There are only four bullets in that section, but the preceding sentence says, “In addition to the requirements of other clauses of this document.” Therefore, your risk management file should address all of the requirements in ISO 14971:2019. What I recommend is a virtual risk management folder for each risk management file. As the auditor, you should also request a copy of the risk management policy and procedure. An example of what this would look like is provided below. The numbers in front of each subfolder correspond to the sub-clause or clause for that requirement in ISO 14971:2019.

Risk Management File Example Auditing Risk Management FilesWhich records are most valuable when auditing risk management files?

As an auditor, I typically focus on three types of targets when auditing any process. First, I will sample any corrective actions implemented in response to previous audit findings. Second, will sample documents and records associated with any changes made to the process. Changes would also include any changes that were made to individual risk management files or the creation of a new risk management file. Finally, my third target for audit sampling is any item that I feel is at risk for safety or performance failures. The severity of the safety or performance failure is also considered when prioritizing audit sampling. In the context of a risk management file, I always verify that production and post-production activities are being conducted as planned. I try to verify that risk analysis documentation was reviewed for the need to update the documentation in response to complaints and adverse events.

More auditor training on risk management files

We are recording a live webinar intended to teach internal auditors and consultants how to perform a thorough audit of risk management files against the requirements of the new European Regulation (EU) 2017/745 and ISO 14971.

PXL 20221101 183748328 Auditing Risk Management Files
Auditing Risk Management Files
In this new webinar, you will learn how to conduct a process audit of risk management files. You will learn what to look at and what to look for in order to verify compliance with Regulation (EU) 2017/745 and ISO 14971:2019. The webinar will be approximately one hour in duration. Attendees will be invited to participate in the live webinar and receive a copy of the native slide deck. Anyone purchasing after the live event will receive a link to download the recording of the live event and the native slide deck.
Price: $64.50

In addition to this webinar on auditing risk management files, we also have other risk management training webinars available. The webinar on auditing risk management files will be hosted live on November 15, 2022 @ Noon EST (incorrect in the live video announcement).

How to create an IVDR checklist

4 Comments / CE Marking, In Vitro Diagnostic (IVD) Devices, ISO 13485:2016, ISO Auditing, ISO Certification, IVDR - Regulation (EU) 2017/746 / By

This article provides an IVDR checklist for updating your ISO 13485 quality system to comply with EU Regulation 2017/746.

IVD Checklist 1024x474 How to create an IVDR checklist

Why I created an IVDR checklist?

Hundreds (if not thousands) of IVD manufacturers are currently updating their ISO 13485:2016 certified quality system from compliance with the In Vitro Diagnostic Directive (i.e. Directive 98/79/EC) or IVDD to the new EU In Vitro Diagnostic Regulation (i.e. Regulation 2017/746). Revision of technical files and the associated procedures for creating your technical files is a big part of these updates. However, there is much more that needs to be updated than just the technical documentation. Therefore, IVD manufacturers are asking Medical Device Academy to conduct remote internal audits of their quality system to identify any gaps. Usually, we conduct internal audits using the process approach to auditing, but this is one of the scenarios where the element approach and an audit checklist are invaluable.

If you would like to download our IVDR checklist for FREE, please fill in the form below.

How do you use an audit checklist?

An audit checklist is used by quality system auditors to collect objective evidence during an audit. This objective evidence verifies compliance with regulatory requirements or internal procedural requirements. If the auditor is unable to find supporting evidence of compliance, the auditor may continue to search for data or identify the requirement as a nonconformity. Typically the checklist is in four columns using a tabular form. The left-hand column lists each requirement. The next column is where the auditor documents records sampled, procedures reviewed, and personnel interviewed. In the third column, the auditor indicates what they were looking for in the records, procedures, or during the interview. Some of the information in the second and third columns can often be entered prior to starting the audit by reviewing audit preparation documents (e.g. procedures and previous audit reports). In the fourth column the auditor will enter the objective evidence for conformity collected during the audit.

How to create an IVDR quality plan

Most of the companies that are preparing for an IVDR audit by their notified body already have ISO 13485:2016 certification and they are using the self-declaration pathway for CE Marking under the IVDD. Under the IVDR, a notified body must now review and approve the technical file. The notified body must also confirm that their quality system has been updated to include the IVDR requirements. The Technical File requirements are found in Annex II and III; while most of the quality system requirements are found in the Articles.  The quality system requirements include:

  1. a risk management process in accordance with Annex I – deviations from ISO 14971:2019 will be necessary)
  2. conduct a performance evaluation–including a post-market performance follow-up (PMPF). This requirement is defined in Article 52 and Annex XIII
  3. create and maintain a technical file in accordance with Annex II & III
  4. create and maintain a Declaration of Conformity in accordance with Article 17
  5. CE Mark the product in accordance with Article 18
  6. implement a UDI system in accordance with Article 24, 26, and 28
  7. record retention requirements for the technical file, Declaration of Conformity, and certificates shall be increased from 5 years to 10 years
  8. set-up, implement, and maintain a post-market surveillance system in accordance with Article 78
  9. document a procedure for communication with Competent Authorities, Notified Bodies, Economic Operators, Customers, and/or other Stakeholders
  10. update procedures for reporting of serious incidents and field safety corrective actions in the context of vigilance to require reporting within 15 calendar days
  11. update the product labeling to comply with Annex I, section 20
  12. revise the translation procedure to ensure translations of the instructions for use are available in all required languages of the member states, and make sure these translations are available on the company website
  13. create a procedure for utilization of the Eudamed database for registration, CE Marking applications, UDI data entry, and vigilance reporting

Which IVDR requirements are already met by your quality system?

Some companies also manufacture medical devices that must comply with Regulation (EU) 2017/745. For those companies, many of the above requirements are already incorporated into their quality system. In this case, you should still include all of the IVDR checklist requirements in your plan, but you should indicate that the requirement has already been met and audited previously.

Content related to our IVDR checklist

On Friday, April 1, 2022 @ 11 am EDT (8 am Pacific), Rob Packard will be Joe Hage’s guest speaker on the weekly MDG Premium Live video (please click on the link to register). The topic of the live presentation will be “How to create an IVDR quality plan.” #MedicalDevices #MDGpremium

Individual process audits or one full quality system audit, which is better?

3 Comments / ISO Auditing / By

You can conduct multiple individual process audits or you can conduct one full quality system audit, but which solution is better?

What are individual process audits?

There are 25 processes that require procedures for compliance with the US FDA quality system regulations and ISO 13485:2016 has 28 required procedures. Individual process audits focus on one of these procedures, the process it controls, the equipment and software used by that process, the work environment where the process is performed, the people responsible for the process, the records resulting from that process, and any metrics or quality objectives associated with that process. An individual process audit can be completed in remotely or on-site, and these audits will be much shorter in duration than a full quality system audit. Another way to think of an individual process audit is to realize that a full quality system audit is comprised of many individual process audits scheduled back-to-back. Auditing one process might be as short in duration as 30 minutes (e.g. control of records) but individual process audits can take as long as four hours (e.g. design controls and technical file audits).

What is a full quality system audit?

A full quality system audit is typically a single audit conducted annually to address all the requirements for conducting an internal audit of your quality system. In this type of audit, all of the procedures and processes should be covered. Therefore, full quality system audits are necessarily longer. If the person assigned to conduct the full quality system audit is an employee, that person cannot audit their own work. This can be addressed in two ways: 1) the audit can be a team audit, and the other team members can audit areas the lead auditor was responsible for; and 2) the process(es) that the lead auditor is responsible for can be audited as individual process audits by another auditor at another time.

If the person assigned to conduct the full quality system audit is a consultant from outside the company, there is still potential for conflicts regarding independence. If the consultant audited the company in the previous year, then the auditor cannot audit last year’s internal audit. In our consulting firm we address this issue in two ways: 1) we rotate who is assigned to audits so that the same auditor does not conduct a full quality system audit two years in a row, or 2) we assign another auditor in our company to conduct the audit of internal auditing as a team member.

How do you evaluate auditing effectiveness?

Some companies perceive that auditing is a necessary evil and they want to put as little effort and resources into the audit as possible. In this situation, auditing might be evaluated based upon whether it was completed on-time, by how much the audit cost the company, and the fewer nonconformities identified the better the perceived outcome. This perspective typically results in a single full quality system audit that is three days in duration or shorter if an auditor can manage to complete the audit in less time. Of course the shorter the audit is, the fewer records that an auditor has time to review. Therefore, shorter audits typically have fewer findings and management is pleased at the outcome because the audit required fewer resources and had little or no nonconformities.

The better approach is to look at auditing as a method for identifying areas that need improvement. Identifying areas where your quality system needs improvement is the intent of requiring internal audits. Therefore, the amount of time your company allocates to auditing should reflect the benefits for improvement that are identified. Top management of your company needs to identify which process areas they feel needs improvement. Only then can the audit program manager design an audit schedule that will focus on identifying opportunities for improvement and nonconformities in the process areas where management feels improvement is most needed. Ideally, this approach to auditing will focus on looking for inefficiency and metrics with negative trends. These findings result in preventive actions instead of corrective actions, because the process is not yet nonconforming. In general, the more opportunities for CAPAs that are identified the more valuable the audit was.

What advantages do one full quality system audit present?

Sometimes a single full quality system audit is easier to schedule, because it is only once per year. The rest of the year your company will not need to spend much time discussing audits or even thinking about them. If your company perceives audits as a necessary evil, then the less disruption caused by scheduling an audit the better.

Another advantage of conducting full quality system audits is that you can more easily afford to use external consultant auditors, because the travel costs for auditing are limited to one trip per year. If you had more than twenty individual process audits each year, and external consultant auditors conducted all of the audits, then you would have to pay for travel costs twenty times each year. Unless the consultant lives locally, these travel costs can be substantial.

What advantages exist for individual process audits?

Individual process audits are much easier for the auditor to complete within the time established in the audit agenda, because the auditor does not have another audit process immediately proceeding or immediately after the process they are auditing. There are also fewer people that need to attend an opening or closing meeting for an individual process audit, because only one process is being audited. Managers from other departments are seldom needed for participation in the opening or closing meeting.  The combined benefits result in the auditor being more likely to start the opening meeting on-time and to start the closing meeting on-time.

The shorter duration of individual process audits is also an advantage. There are very few times in a year when none of your department managers will be traveling, sick, or on vacation. These rare weeks only happen a few times each year, and sometimes auditors must proceed with an audit even if someone is absent because they have no alternative. If you are preparing for an audit remotely, you face-to-face audit time is only 90 minutes, and your report writing time is also conducted remotely, then finding 90-minutes of available time in an department manager’s schedule is usually quite easy.

Can both approaches to internal audit scheduling coexist?

You can combine both approaches to audit scheduling in several possible ways. First you can schedule one full quality system audit each year in order to make sure that the minimum audit requirements are met, and then top management can review the results of the full quality system audit to decide which processes would benefit from individual process audits.

A second strategy would include conducting individual process audits for each process that resulted in a nonconformity during 3rd party certification audits or during the one full quality system audit. In this scenario, you might have a 3rd party audit in November, a full quality system audit in May, and top management might select 10 other individual processes to audit during the other 10 months of the year.

A third strategy would be to alternate between individual process audits and single full quality system audits each year. During “odd” years the audit program manager would only schedule one full quality system audit, and during “even” years the audit program manager would schedule multiple individual process audits.

A fourth strategy would be for top management to select a few processes that they would like the audit program manager to focus on with individual process audits, and all of the remaining processes would be incorporated into a single audit that covers the remaining 70% of the quality system.

Each of these four strategies for combining the two approaches to audit scheduling is viable and may result in multiple opportunities for improvement being identified. There is no regulation that favors one approach over another, but all four strategies require more time an effort on the part of the audit program manager and top management to discuss and plan the annual audit schedule.

Next steps if you would like to try individual process audits

If your company has always scheduled a single full quality system audit each year, you can test the concept of conducting an individual process audit by selecting just one process to audit. The best choice for this approach is to pick a process that has one or more CAPAs that are in progress or to select a process that top management feels is performing efficiently. The more frustration that top management experiences with a process, the greater the need is to identify opportunities for improvement. If the company has not already identified CAPAs to initiate for that process, you might just need an outsider to state the obvious: “I think we need a CAPA in this department.” The outsider might be a consultant, but it could also be a person from another department. If you would like a quote for an individual process audit, please visit our audit quote webpage.

About the Author

Rob Packard 150x150 Individual process audits or one full quality system audit, which is better?

Rob Packard is a regulatory consultant with 25+ years of experience in the medical device, pharmaceutical, and biotechnology industries. He is a graduate of UConn in Chemical Engineering. Robert was a senior manager at several medical device companies—including the President/CEO of a laparoscopic imaging company. His Quality Management System expertise covers all aspects of developing, training, implementing, and maintaining ISO 13485 and ISO 14971 certification. From 2009-2012, he was a lead auditor and instructor for one of the largest Notified Bodies. Robert’s specialty is regulatory submissions for high-risk medical devices, such as implants and drug/device combination products for CE marking applications, Canadian medical device applications, and 510(k) submissions. The most favorite part of his job is training others. He can be reached via phone 802.258.1881 or email. You can also follow him on Google+, LinkedIn or Twitter.

Why remote audit duration should never exceed 90 minutes

1 Comment / Auditing, ISO Auditing, Remote Auditing / By

This article explains why remote audit duration should not exceed 90 minutes and the unique opportunities created by a series of short remote audits.

download 3 Why remote audit duration should never exceed 90 minutes

Parkinson’s Law and the subject of audit duration

On November 19, 1995, Cyril Northcote Parkinson published an essay in the Economist. The title of the article was “Parkinson’s Law.” In the first sentence of the essay, Parkinson says, “It is a commonplace observation that work expands to fill the time available for its completion.” This essay refers to the observation that work is elastic concerning the demands on time when completing paperwork. When I first trained as an auditor, trainers emphasized that the most significant challenge faced by auditors is to complete an audit within the time available. An auditor’s task is to achieve the audit objectives within the time specified by the audit program manager. Time is precious, and you cannot easily extend the audit duration after scheduling the audit.

How much time is needed for a full quality system audit?

This question is a silly question to ask a consultant that works on an hourly basis. A consultant working on an hourly basis will make more money if they work more hours. Therefore, there is little incentive to underestimate the time required to complete the objectives of an audit. However, after completing hundreds of audits, I can honestly state that eight hours is not enough time to perform a full quality system audit of a medical device company’s quality system. However, I completed a full quality system audit of a small company in less than two days. I also had difficulty completing an audit of a larger company in four days. An FDA inspector typically requires four days to complete a routine inspection, even at foreign manufacturers where English is a second language, and they only need to return on the fifth day to prepare their FDA 483 observations to give to the company. Therefore, three days is typically the absolute minimum time required to complete a full quality system audit.

Does Parkinson’s Law apply to audit duration?

Parkinson’s Law certainly applies to the audit duration. If the lead auditor assigns a team member to review the CAPA process, the task is unlikely to be completed in 30 minutes, and most auditors would struggle to appear busy for more than three hours. You need enough notes to provide objective evidence of conformity for your audit report, but if you finish too quickly, then others may perceive that you were not thorough. Therefore, most auditors will begin any process audit by asking for a copy of the procedure and a log of the records available. The auditor will quickly review the procedure’s revision history to determine when the last revision was made and if there have been any significant revisions since the last audit. Next, the auditor will review the log to estimate how many records should be sampled. The auditor will then estimate how much time is needed to review the sampled records. Finally, a quick mental calculation is made to determine how much time remains for procedure review before the auditor must move on to interview the next subject matter expert.

Why are auditors always behind schedule?

An auditor begins with small, close-ended questions that are designed to put the auditee at ease. The auditor may even comment on unrelated subjects to build rapport first. Records may not be readily available, but auditors almost always have to wait for record retrieval. The request is recorded, copies are made, and the subject matter expert may need a little time to review before handing the auditor the requested record. Auditors will ask clarifying questions, and auditees will need a few moments to check their facts. Any one of these delays is insignificant by itself, but collectively there may be two-and-half minutes of delay cumulatively for each record requested if you sample five records, which represents a combined delay 12.5 minutes. If you average only seven minutes to review each record, then a sampling of five records will require 47.5 minutes. This will leave you only 12.5 minutes for introductions, review of the procedure, and conclusions. If you want to interview any of the people that investigated root-cause, then you will need more than an hour to complete your audit, and you will not finish in the one hour scheduled.

Why is it so hard to complete a full quality system audit in three days?

Most of your process audits require a few more minutes than you expected, but you will also need time to walk to the next subject matter expert, or you will be waiting for the next subject matter expert to enter the conference room. If the quality system consists of only the minimum twenty-eight required procedures, your full quality system audit will require more than 28 hours to complete. If there are additional regulatory requirements for CE Marking or ISO 13485 certification, you will need even more time to audit every process. You should also expect certain processes to require more time to properly sample records, such as technical documentation and design controls. Even the most experienced auditors struggle to review a technical file and/or design history file in less than two hours.

What happens to an auditor after auditing all day?

As a Notified Body auditor, I used to leave my home in Vermont on Sunday afternoon and drive two hours to the nearest major airport. Then I would be gone all week conducting audits. On Friday, I would drive home and arrive in the middle of the night. Each day audits would begin early in the morning, and I would complete the day after 8.5 to 9 hours of work. Jet lag, sleep deprivation, too little exercise, and constantly eating at restaurants took its toll. I would consult my Google calendar to learn what city I was in each morning, and to remember what company I was on my schedule for the day. I would purposely try to do as much walking around during the day just to keep my blood flowing and to help stay awake. I would read documents while pacing back-and-forth in conference rooms, and I would always make sure that we had to audit the most remote area of a facility after lunch to make sure that I didn’t fall asleep. I will tell stories and jokes to entertain my hosts, but it was necessary to break up the monotony of auditing quality systems seven days a week. I would make sure I drank at least six liters of water each day for health, but this also gave me an excuse to go to take frequent bathroom breaks. Somehow I managed to survive that lifestyle for more than three years. Each day my feet, legs, back, and neck were in severe pain. I had constant headaches, and I know the quality of my work gradually declined throughout each day. The most valuable lesson I learned was, you need to move frequently, or you will die.

unnamed Why remote audit duration should never exceed 90 minutes

What happens when you sit in front of a computer for eight hours?

I can sit in front of a computer longer than almost anyone I know. When I focus on work, four hours can elapse without me getting up from a chair even once. I might pick up my empty coffee mug four or five times to take a sip before I am conscious of the need to get another cup. On days where my schedule consists primarily of Zoom meetings, I may sit through as many as six consecutive meetings before I take the time to get up and go to the bathroom and get a drink of water. Clients may perceive that I have tremendous endurance, but there are negative consequences to this work pattern. My wrist becomes sore, and I need to switch my mouse pad and the style of the mouse I am using every day. I change computers, switch microphones, and take a short walk. My neck, back, and legs will hurt worse than any of the audits during my years as a Notified Body auditor. Sitting at a computer all day has resulted in mild symptoms of restless legs syndrome. Sitting at a computer continuously for the audit duration is physically exhausting and tedious. If you must complete a remote audit on a continuous eight-hour day, you can, but it is not healthy or productive. The negative health consequences and negative impact on productivity are equally applicable to auditees.

What can you do to reduce audit fatigue during a remote audit?

The most straightforward strategy for reducing fatigue is to take breaks. Instead of auditing for eight hours continuously, try auditing in two or three 90-minutes segments each day. If you are auditing someone in a different time zone, you may only be able to accommodate an audit duration of one 90-minute session per day without working through the night. Taking breaks will allow you to leave your computer, eat food, and even go to the bathroom. You can recharge your headset during a break too. You should consider taking a walk outside. It is incredible how much better you feel when you get some exercise, stretch, and experience a little natural light instead of the unnatural glow of your computer’s monitor. The person you are auditing will appreciate the breaks, but they will also enjoy the improvement in your overall demeanor. A simple smile after a 30-minute break has a tremendous positive impact.

How can we utilize breaks more effectively during remote audits?

Auditors need documents and records to review as objective evidence. The most obvious way to make use of breaks is for the auditor to give the auditee a list of documents and records to gather during the break. This will give the auditee an excuse to go and get the documents and records if they are stored in another location. The auditee might also scan records during a break. A break also gives subject matter experts time to re-familiarize themselves with the documents and records before resuming the audit. Auditees and auditors will need to recharge batteries, but the auditor might take time to convert their notes into a summary for the final audit report. The auditor might also review the audit criteria one more time before writing a nonconformity. The auditee might take advantage of the break to initiate a new CAPA and write a draft of the corrective action plan. Then when the audit resumes, the auditee can review the draft plan with the auditor to ensure that the plan is appropriate and nothing was accidentally omitted from the CAPA plan.

unnamed 1 Why remote audit duration should never exceed 90 minutes

Why are 90 minutes a magical audit duration?

Auditing one process in a single 45-60 minute session is ok, but if you audit two processes in a single 90-minute session, you can reduce the time spend starting and stopping the audit session by half. Adding a third process to a single session will have a smaller impact, and the meeting will need to be so long that most participants will begin to lose concentration, and fatigue becomes a significant factor. Ninety minutes is not quite long enough to audit two processes effectively. Still, an auditor can request procedures in advance of the session or spend time after the session reviewing procedures. Therefore, by paying an additional 30 minutes reviewing two procedures “off-line,” the auditor can dedicate 100% of the “on-line” time to reviewing records and interviewing subject matter experts. The result is a fast-paced, 90-minute session where each subject matter expert typically is only needed for 45 minutes. Alternatively, if you are auditing more complex records like a design history file, you can spend all 90 minutes discussing that area.

How to make a supplier questionnaire for remote auditing

Leave a Comment / ISO Auditing, Supplier Quality Management / By

You already have a supplier questionnaire, but do you know how to make a supplier questionnaire to assess a supplier’s ability to support a remote audit?

FRM 004 Supplier Questionnaire How to make a supplier questionnaire for remote auditing

The four most significant mistakes people make when designing a supplier questionnaire

In Medical Device Academy’s supplier qualification webinar, you learn how to improve your supplier qualification process by replacing the traditional methods of supplier qualification with more effective approaches to supplier evaluation. The following are four examples of how to improve your supplier questionnaire.

Supplier questionnaires should be specific to the product or service provided

The first mistake people make is to use a generic questionnaire. It would be best if you asked your supplier questions that are important to the work that the supplier will be performing. Therefore, each category of product or service should have its own set of questions. For example, important questions related to ethylene oxide contract sterilization services are the maximum size limitations for pallets in the sterilization chamber and whether the facility can conduct sterility testing on-site. However, an injection molding supplier might delay the return of your supplier questionnaire if these questions were on the survey that you send to them because they don’t understand the questions.

Supplier surveys should be more than checkboxes

The second mistake people make is to ask questions that can be answered with a “yes” or “no” response or a checkbox. These are closed-ended questions. It would be best if you always were asking open-ended questions because the response will give you more information about the supplier. In addition, most people resist responding with a “no” response even if the real answer is “no.” For example, “What is your FDA registration number?” is more useful than “Is your company FDA registered?” Another example is, “How many production lines use SPC charts?” instead of “Do you use SPC charts?” In fact, in the open-ended version of this question, you will learn if the use of SPC charts is widespread, and you learn how many production lines the supplier has.

Remember to ask suppliers to update survey surveys every year

The third mistake people make is to request that a supplier questionnaire be completed only during the initial supplier qualification process. Every year companies grow, shrink, or change. If you ask suppliers to update their questionnaire, you can use that information to determine the health of your supplier’s business. You might also discover that one supplier just added a new production capability that will allow you to consolidate more of your outsourced work with that supplier and eliminate another problem supplier. Every company has a turnover in personnel as well. It is a great idea to ask suppliers to provide contact information for multiple people in the organization, such as quality contact, billing contact, and a production planner. Eventually, you will probably need to speak with each of these people, and if one of the contacts is no longer at your supplier, you will still have two other contacts. Updating this information also gives you a hint of whether turnover is widespread or limited to a specific individual.

Supplier questionnaires should be in spreadsheet format

The fourth mistake people make is to send a Word Document for suppliers to complete (PDF format is even worse). Word and PDF formats are time-consuming to complete, and they are harder for you to analyze than a spreadsheet. Most people provide a Word document or a PDF because they are focusing on the requirement for control of records. However, if you have an electronic quality system, the supplier survey information will be part of your electronic system as soon as you enter the data into your software. Alternatively, if you have a paper-based quality system, then you can print the spreadsheet out, sign it, and date it. The huge advantage of using Excel spreadsheets is that you can copy the new data into a column next to the previous year’s responses. Then you can quickly see what changes your supplier made in the past year.

What should you add to your supplier questionnaire?

Most private companies will not share what their revenues are for the business, but as a customer, you should be more concerned with how many human resources your supplier has. Therefore, you should consider asking, “How many employees, or full-time equivalents (FTEs), work for your company?” You might also want to know if your supplier is relying on a temporary workforce. For example, “What percentage of the FTEs are temporary workers?” Many questionnaires will ask for the square footage of the facility, but this doesn’t provide you with any details about the facility layout. Alternatively, you could ask for a copy of the pest-control map for the facility. This would give you a detailed layout of the facility, and it also confirms that your supplier has a pest control plan for the facility. Another related question to ask is, “Please describe any expansion/construction projects that have been implemented in the past year or projects that are in progress (e.g., the addition of a mezzanine).” If the company added 30,000 square feet to their production area, but there was no change to the pest control plan, you might have some clarification questions for your supplier. In general, a good strategy for developing your questionnaire is to think of at least one open-ended question related to each clause of the ISO 13485:2016 standard without referencing the standard. The following are some examples that might help you:

  1. When was the last software re-validation for quality system software?
  2. How many active external standards is your company currently maintaining?
  3. Please provide a list of procedures and identify the person who would be interviewed during an audit for each procedure (i.e., process owner or subject matter expert).
  4. In the absence of the management representative, who is designated as the liaison for an FDA inspector?
  5. What are the upper control limits for particulate counts, air viable counts, and surface viable counts in your controlled environment(s)?
  6. On what dates was the environmental monitoring of controlled environments conducted in the last year?
  7. Please identify how many quality inspectors are responsible for the incoming inspection?
  8. Please list the calibration ID and equipment name for any inspection equipment that requires specialized training (e.g., CMM)?
  9. How many suppliers are on your approved supplier list (ASL)? And how many suppliers did you audit in the past year?
  10. How many nonconforming material reports (NCMRs) were opened in the past year? And how many NCMRs currently remain open?
  11. How many partial or complete lots were returned to your company by customers in the past year?
  12. Please list any corrections and removals (i.e., recalls) that your company has been involved in during the past year and the current status?

How many questions should your supplier questionnaire include?

There are 28 required procedures in ISO 13485:2016, and there are even more subclauses within the standard. It is an excellent idea to create a list of questions you might ask for each subclause, but a supplier questionnaire should not include all of those questions. Just as audits are just a sampling, your supplier survey questions should be sampling as well. You should review last year’s questions and eliminate questions that you think are not especially useful for that supplier. Some questions should be asked each year to assess if the quality system has changed significantly, and you should consider adding a few new questions each year. The best questions will require the person to perform some research to answer the questions. But it is unreasonable to expect a supplier to spend more than two hours completing a supplier questionnaire if you plan to purchase less than $20,000 in product or services.

Supplier questionnaires specific to remote auditing

In many ways, a well-designed supplier questionnaire is similar to a remote audit, because you are asking the supplier to answer multiple open-ended questions about their quality system to verify that the quality system is fully implemented and remains effective. However, due to the Covid-19 pandemic, many employees are now required to work from home, and it is not possible to physically visit certain facilities. Therefore, you should be adding three elements to your supplier questionnaire to assess your supplier’s ability to support a remote audit and to determine their ability to maintain the effectiveness of the quality system during a viral outbreak. The three elements are 1) policies for personal protective equipment for employees and visitors, 2) business continuity plans to maintain internal operations and to ensure redundancy of crucial suppliers, and 3) availability of digital documents and records or paper documents and records via video conference software. These three areas were also the subject of a previous blog on changes triggered by Covid-19. It would help if you also asked about the availability of hardware and software communication tools for conducting a remote audit. You might ask your supplier, “Which areas of your facility can we observe during a remote audit using live video conferencing (e.g., Zoom mobile application)?” and “What experience does your company have in the use of Zoom as a video conferencing tool?”

Gimbal How to make a supplier questionnaire for remote auditing

Access to documents and records during remote audits

During a remote audit, you will need to access documents and records virtually. If your supplier can participate via a video conferencing tool with a high definition web camera or smartphone, then you should be able to see any documents and records that you could normally see during an on-site audit. However, your supplier will need to hold the document or records steady, possibly by using a music stand and a camera tripod so that you can take notes regarding the contents of the document or record. You will also need a way to record your notes. You might try using a Pixelbook or similar computer to write your audit notes. At the same time, you watch the video conference using a second computer–possibly on a conference room projector screen or large flat screen monitor. You could also use a tablet, such as remarkable. Of course, you can always use a pad of paper and a pen and then transcribe your notes later. All of these methods will be faster and more convenient than digitally scanning each document and uploading the documents to a shared folder or sending the scanned document by email.

It would help if you also were asking your supplier which records are already available digitally. You can expect all of the quality system procedures to be available in digital formats, but many records may already be available electronically as well. For example, purchase orders, quality system certificates, drawings, and blank forms should be available in digital format. In a supplier audit, you typically will focus on a subset of the quality system records that are related to production process controls, purchasing, incoming inspection, shipping, and control of the nonconforming product. Asking your supplier which of these records are available in digital format will help you determine which records you need to request from the supplier in advance and which records can be requested on-demand.

How to obtain our supplier questionnaire template (FRM-004)

If you are interested in purchasing our supplier questionnaire template, FRM-004, it is included with the purchase of our supplier qualification webinar. If you think of any new questions to add to this template, please email me at rob@13485cert.com. Just put “FRM-004 Suggestion” in the subject line.

Scroll to Top