Audit Archives

Jul 5 2023

Audit schedule and an audit agenda, what’s the difference?

8 Comments / ISO Auditing / By Robert Packard / Audit, audit schedule, Internal Auditing

Internal audit and supplier audit programs both require an audit schedule and an audit agenda, but what’s the difference between them?

What is an audit schedule?

An “audit schedule” is not a formal definition in ISO 19011:2018. However, section 5.1 of that standard states that your audit program should include nine different requirements. Item “d” is “d) schedule (number/duration/frequency) of the audits.” Typically, the audit program manager will maintain an annual audit schedule with a date indicating the date the schedule was last revised. The most common example in lead auditor training is a matrix like the one shown below. The left-hand column will list all of the individual processes that are identified in the company’s process interaction diagram, and the top row of the matrix will indicate the month when each process audit will be conducted. Typically, the expectation is to complete the audit sometime during that month, but some quality auditing procedures specify that the audit may be completed the month before or the month after to accommodate the process owner. The regulations only require that you document and maintain an audit schedule, and the standard is only considered guidance.

%name Audit schedule and an audit agenda, what’s the difference? — Example of an Internal Audit Schedule

I use a slide in lead auditor courses that gives the example of an internal auditing schedule provided above. On the surface, this example seems like a good audit schedule. Twelve auditors perform two audits each year. If each audit requires approximately two days, each auditor spends less than two percent of their work year auditing. Unfortunately, a two-percent allocation of time is insufficient to become or remain proficient at auditing. An improvement to the auditing schedule would be to assign fewer auditors so each auditor gets more experience. There is no perfect number, but assigning a few specialists will improve the chances of becoming and remaining proficient at auditing.

What is an audit agenda?

An “audit agenda” is not a formal definition in ISO 19011:2018 either. In fact, the word “agenda” is not even used in ISO 19011. Instead, section 5.5.5 of ISO 19011 states that “The assignment [of the individual audit] should be made in sufficient time before the scheduled date of the audit, in order to ensure the effective planning of the audit.” The audit plan must also be part of the records [i.e., Clause 5.5.7(b)]. Therefore, “agenda” and “plan” may be used interchangeably. Details of audit planning are provided in 6.3.2 of ISO 19011.

6 Steps to Creating an Audit Schedule

There are six steps to creating an audit schedule:

What were the results of previous audits?
Which processes are the most important to audit?
Who should conduct your internal audit?
How long should your internal audit be?
Should you conduct one full quality system audit or several audits?
Is a remote audit good enough?

We will address each of the six steps below.

How do the results of previous audits impact your audit schedule?

The results of an audit include nonconformities, observations for improvement (OFI), and a conclusion regarding whether the quality system is effective or not. Usually, most processes are effective, and there are no nonconformities or OFIs. Therefore, any processes that had a nonconformity or OFI should be prioritized in the audit schedule and audit planning for the future. For internal and supplier audits, a best practice is for the auditor and the process owner to discuss the corrective actions planned and determine the appropriate timeline or implementation of actions planned. Then the auditor can indicate a timeframe for re-auditing the nonconforming process after corrective actions are implemented. This strategy allows the auditor to be part of the effectiveness check. This approach is appropriate for individual process audits but not for a full-quality system audit.

Which processes are the most important to audit?

The primary element impacting the importance of processes is the risk to product quality associated with the process. Usually, support processes are of lower importance because they do not directly impact product quality. In contrast, core processes directly involved in a device’s design, manufacture, and distribution are critical. Most auditors and audit program managers emphasize design controls and production process controls as important areas to audit. However, the distribution area is often neglected. Other core processes are purchasing, sales, customer service, and servicing. Not every process is equally important when comparing two companies. For example, device manufacturers that only make software as a medical device (SaMD) often have very limited purchasing and incoming inspection activities to audit.

Who should the audit program manager assign to each internal audit?

%name Audit schedule and an audit agenda, what’s the difference? The example of a revised audit schedule provided above identifies the departments where each of the auditors works with color coding. This is done to ensure that auditors are not assigned to audit processes where they might have a conflict of interest (i.e., they would be auditing their own work). This is the most important aspect of assigning auditors. The second most important aspect is to make sure the auditor has the technical knowledge to audit the process. It is challenging to conduct an audit of manufacturing if you have not spent any time in manufacturing before. If auditors are new and their training is in progress, then the audit program manager may assign the auditor to a process specifically to give them more experience with that type of process. Inexperienced auditors often are assigned less important processes that have not changed recently. However, a better approach to training auditors is to give them a challenge with support. Having the new auditor prepare a detailed sampling plan and list of questions before the audit can prepare them for auditing a more challenging, important process that is likely to have one or more nonconformities. Auditing processes that have nonconformities is also the best way to teach a new auditor how to write the audit findings.

What should be the duration of each internal audit in your schedule?

The duration of an audit should be based on the results of previous audits, but other important factors include: 1) the number of personnel involved in the process, 2) the complexity of the process, and 3) the risk to product quality associated with the process. The MDSAP program uses a procedure for audit time determination (i.e., MDSAP AU P0008.007: Audit Time Determination Procedure), and the MDSAP audit approach document (i.e., MDSAP AU P0002.008 Audit Approach) classifies processes as having either a “direct” or “indirect” impact upon product quality based upon the applicable clause of the process (i.e., Clauses 0-6.3 are indirect, and Clauses 6.4-8.5.3 are direct). For example, the production processes and design and development processes both involve a large number of people in most organizations, the processes are complex, and both processes directly impact product quality. Therefore, I typically allocate 3-4 hours to each of those processes during an audit. In comparison, incoming inspection often involves one or two people, and the process often involves only one procedure. Incoming inspection is a “direct” process, but less time (e.g., 1 hour) should be allocated to auditing incoming inspection than the other two processes–unless there was a nonconformity in the incoming inspection process during a recent audit or unless the process was recently changed.

Should you conduct one full quality system audit or several audits?

Both approaches have strengths and weaknesses, but there is not a single best way. If I am using employees to conduct an audit, then I typically restrict the scope of the audit to a single process. Alternatively, when I use a consultant to conduct an audit, I typically conduct a full-quality system audit to minimize travel costs. Another strategy I have recommended is to identify the processes that are most important to audit first (e.g., processes with recent changes and/or nonconformities), and these processes are scheduled for individual process audits during the first half of the audit schedule. Then I schedule a full-quality system audit in the second half of the audit schedule. The strategy ensures that all important processes will be audited twice in one year, but every process will be audited at least once.

Remote audits vs On-site audits

Prior to the Covid-19 pandemic, remote audits were rare in the medical device industry. Many NBs insisted that remote audits were not permitted or effective. The pandemic forced the entire industry to create policies for remote auditing and to use remote auditing whenever possible. Now that the pandemic has ended, many companies continue to conduct remote audits to save money. Even NBs are conducting more remote audits for Stage 1 readiness audits during the ISO 13485 certification process. ISO 19011 has a section in the Appendices outlining the differences between remote and on-site audits. However, there is a minimal advantage to conducting an on-site audit of a process where the auditor is expected to spend all of their time in a conference room during the audit. If the audit is going to be done in a conference room, then why not conduct it remotely? The one exception is when most records are paper-based and unavailable electronically. Alternatively, an on-site audit is generally more effective if the process involves observing inspection activities or assembly operations. Remote audits of inspection activities and assembly operations should be reserved for re-auditing or when a process has been audited on-site in the past, but an on-site audit would still be more effective for those processes.

How many times should a process be audited annually?

Many notified bodies will expect companies to audit all processes at least once during the year. However, it doesn’t expressly state this as a requirement in the regulations, and some companies justify skipping processes that are functioning well and have not changed in the past year. Our team is seeing this more frequently as the number of lead auditors worldwide has become scarce due to the requirements of MDSAP, the MDR/IVDR implementation, and unannounced audits. However, I almost never see the opposite justification (i.e., auditing a process more than once a year). If a process has been changed significantly, or there were nonconformities, then re-auditing the process may be used to verify the effectiveness of corrective actions or to verify that personnel are compliant with the revised process.

How to take advantage of the process approach to auditing

Another improvement that can be made to the revised example of an audit schedule is to use the process approach to auditing. Instead of performing an independent document control and training audit, these two clauses/procedures can be incorporated into every audit. The same is true of maintenance and calibration support processes. Wherever maintenance and calibration are relevant, these clauses should be investigated as part of auditing that area. For example, when the incoming inspection process is audited, it makes sense to look for evidence of calibration for any devices used to perform measurements in that area. When production process controls are being audited, maintenance records of production equipment should also be sampled.

If the concept of process auditing is fully implemented, the following ISO 13485 clauses can easily be audited in the regular course of reviewing other processes:

4.2.1), Quality System Documentation;
4.2.3), Document Control;
4.2.4), Record Control;
5.3), Quality Policy;
5.4.1), Quality Objectives;
6.2.2), Training;
6.3), Maintenance;
6.4), Work Environment;
7.1), Planning of Product Realization & Risk Management
7.6), Calibration;
8.2.3), Monitoring & Measurement of Processes
8.5.2), Corrective Action; and
8.5.3) Preventive Action.

This strategy reduces the number of process audits needed by more than half.

Internal Auditing: Upstream/Downstream Examples

Another way to embrace the process approach to auditing is to assign auditors to upstream or downstream processes in the product realization process from their own area. For example, Manufacturing can audit Customer Service to understand better how customer requirements are confirmed during the order confirmation process. This is an example of auditing upstream because Manufacturing receives the orders from Customer Service—often indirectly through an MRP system. Using this approach allows someone from Manufacturing to identify opportunities for miscommunication between the two departments. If Regulatory Affairs audits the engineering process, this is an example of auditing downstream. Regulatory Affairs is often defining the requirements for the Technical Files and Design History Files that Engineering creates. If someone from Regulatory Affairs audits these processes, the auditor will realize what aspects of technical documentation are poorly understood by Engineering and quickly identify retraining opportunities.

Audit schedule and an audit agenda, what’s the difference? Read More »

Jan 31 2023

ISO 19011 – Do you need this quality system auditing standard?

5 Comments / Auditing, ISO Auditing, Remote Auditing / By Robert Packard / Audit, ISO 19011, PDCA

Read this article to learn why ISO 19011 standard is a vital guidance for anyone that audits quality systems or manages an audit program.

What is ISO 19011?

ISO 19011 is a seven-part international standard for auditing management systems. The standard defines the eight principles of auditing (e.g., the process approach to auditing), provides guidance on managing audit programs and conducting audits, and includes recommendations for evaluating people for competency. There is also an appendix with details on conducting on-site and remote audits.

If you have ever taken a lead auditor course for ISO 13485, or one of the other quality management system standards, one of the critical handouts for the class should have been ISO 19011. The title is “Guidelines for Auditing Quality Management Systems.” In 2018, ISO 19011 was updated, and the changes were not superficial. If you need to purchase a copy of ISO 19011:2018, the Estonian Center for Standardization and Accreditation is the least expensive source we know.

ISO 19011 covers the topic of quality management system auditing. This Standard provides guidance on managing audit programs, conducting internal and external audits, and determining auditor competency. One of the most common points of confusion in the lead auditor course is the difference between first, second, and third-party audits. In the first edition of this Standard, the difference between first, second, and third-party audits was just a note at the bottom of page one and the top of page two. The note was also not clear. In the second edition of 19011, in Table 1 (reproduced below), the difference between these three types of auditing is crystal clear. Table 1 was modified further in the 3rd edition to include a bottom row that remains unchanged in the 3rd edition, released in 2018.

Figure 1, found in Clause 5.1 of the 2nd edition, was combined with Figure 2, found in Clause 6.1 of the 2nd edition. The combined figure is now Figure 1 in the 3rd edition. The combined scope of Figure 1 is now a “Process flow for the management of an audit program” and a “Process flow for conducting an audit.” The figure categorizes the various stages of audit program management and conducting an audit into the Plan-Do-Check-Act (PDCA) cycle. We highly recommend this style for presenting any process in your internal procedures as an example of best practices in writing an SOP. The flow chart even references each of the clauses in the Standard.

The 2018 version still includes an opening meeting checklist (i.e., Clause 6.4.3) and a closing meeting checklist (i.e., Clause 6.4.10). Figure 3 in the 2nd edition, “Overview of the process of collecting and verifying information,” was a poor example of a flow chart. The committee did not update the figure when the standard was updated for the 3rd edition. Therefore, we updated the figure below to provide additional traceability to the Clauses of the Standard. If you incorporate this figure into your quality auditing procedure, you should substitute references to your procedure’s sections instead of the clauses of the standard.

Competency Requirements in ISO 19011

Many audit procedures neglect to define the qualifications and methods for determining the competency of the audit program manager. Clause 5.3.2 tells you how. Put it in your own procedure. Most of the procedures we read include qualifications for a “Lead Auditor,” but we seldom see anything regarding competency. Unfortunately, this Standard only explicitly addresses the “Lead Auditor” competency in a two-sentence paragraph—Clause 7.2.5. When we teach people how to be Lead Auditors, we spend more than an hour on this topic alone.

The Standard would be more effective by providing an example of how third-party auditors become qualified as a Lead Auditor. Third-party accreditation requires the auditor to be an “acting lead” for audit preparation, opening meetings, conducting the audit, closing meetings, and final preparation/distribution of the audit report. This must be performed for 15 certification audits (i.e., – Stage 2 certification or re-certification), and another qualified lead auditor must evaluate you and provide feedback.

Appendices in ISO 19011

The appendices were the last significant additions to this Standard in 2011 (i.e., 2nd edition). Annex A provided examples of discipline-specific knowledge and skills of auditors. This section was eliminated from the 3rd edition of ISO 19011:

“Due to the large number of individual management system standards, it would not be practical to include competence requirements for all disciplines.” – Copied from the Foreward

I think providing adding a short Annex to each management system standard that defines recommended discipline-specific knowledge would be helpful. Still, that kind of change would need to be initiated with the next version of ISO 9001.

Appendix B in the 2nd edition is now Appendix A in the 3rd edition of ISO 19011. A table (Table A.1 – Audit Methods) compares conducting on-site and remote audits. We were pleased to see that conducting interviews is a significant part of remote auditing in this table. Section A.17 in the appendix provides suggestions for conducting interviews. Still, if you exhibit all 13 professional behavior traits found in Clause 7.2.2, you don’t need advice on speaking with people. For the rest of us mortals, we could use a five-day course on interviewing alone. To improve your skills in this area, ask an experienced auditor with solid interviewing skills to watch and comment on a recording of a virtual audit you perform. Watching yourself audit is cringe-worthy, but we guarantee you will improve.

What are the primary changes to the 2018 version of the standard?

There are seven main differences between the second edition, published in 2011, and the third edition of ISO 19011, released in 2018:

addition of a seventh principle of auditing in sub-clause 4(g) (i.e., risk-based approach);
more guidance on audit program management in Clause 5, including audit program risk;
expansion of Clause 6 on conducting an audit–especially Clause 6.3 on audit planning;
expansion of auditor competence requirements in Clause 7;
updating of terminology to emphasize processes rather than objects;
removal of an annex containing competence requirements for specific quality management systems;
expansion of Annex A to include guidance on new auditing concepts such as remote audits.

Risk-based auditing is the most significant change in the 2018 version of ISO 19011

One of the main differences between ISO 19011:2018 and the previous 2011 version is the addition of a “risk-based approach” to the principles of auditing. Specifically, clause 4(g) of the guidelines for auditing management systems is, “The risk-based approach should substantively influence the planning, conducting and reporting of audits to ensure that audits are focused on matters that are significant for the audit client, and for achieving the audit program objectives.” A lot of people are unsure of what is meant by a risk-based approach. Still, the key to understanding this is to focus on the definition of risk. From a product perspective, the risk is the “combination of the probability of occurrence of harm and the severity of that harm.” From a process perspective, the risk is the “effect of uncertainty on an expected result” (ISO 9001:2015, clause 3.09). Therefore, auditors should emphasize medical devices with the highest severity of harm and devices with a high probability of hazards or hazardous situations. When an auditor focuses on a process rather than a specific medical device, auditors should emphasize any processes that are not under control and any recent process changes.

What is risk-based auditing?

Risk-based auditing considers the risks of failing to achieve audit objectives and the opportunities created by choosing various audit methods and strategies. For example, a desktop audit of procedures might be appropriate if you are conducting your first internal audit for a new quality system. Alternatively, a desktop audit would be a waste of time if you are auditing a mature quality system where very few changes to procedures have been made in the past year. Using the element approach to auditing is unlikely to add much value. Audits are meant to be a sampling. Therefore, you should focus on areas of importance where previous nonconformities were identified, any new products or processes, and anything that changed significantly.

Auditor selection should also be risk-based

Suppose you are conducting a supplier audit as part of your initial supplier qualification for a critical component supplier or contract manufacturer. In that case, you should consider doing a team audit with a multi-disciplinary team. This is a risk-based approach to the supplier qualification process, which ensures that subject matter experts evaluate each process instead of auditors with a general quality assurance background. This approach also forces more of your personnel to introduce themselves to the new supplier, and the audit will develop more reliable communication channels between your two companies. Alternatively, if you are conducting a routine internal audit of a production process, you might select a new lead auditor to conduct the audit. You don’t expect any significant findings in a routine internal audit of an established production process. In your role as an audit program manager, you need to match the new lead auditor to a process that will force them to look at all aspects of the process approach to auditing. Specifically, process validation, calibration, maintenance, and process monitoring may not apply to other administrative process areas, such as purchasing.

Risk-based auditing should influence your auditing schedule

The frequency of auditing suppliers and internal process areas should reflect the associated risks. Therefore, when you create or update your auditing schedule, you should consider the risk level of the products being audited and the process being audited. Production processes with a moderate or high level of non-conforming products may need to be audited more than once yearly. Still, a supplier with an excellent track record of extremely high quality and on-time delivery may be audited in alternating years. If you previously scheduled a remote audit, you may want to alternate to conducting an on-site audit the next time.

The duration of your audits should not always be the same either. Suppose one production process makes one product in low volume, and another production process makes multiple products in high volume. In that case, you should not schedule a two-hour internal audit for both processes every year. The low-volume production process may only need a one-hour audit once per year. In contrast, the high-volume process may require a four-hour internal audit or multiple annual audits.

Risk-based auditing applied to remote supplier auditing

The risk-based auditing approach was added to ISO 19011:2018 as the seventh principle of auditing. This represents the most significant change to that standard, but how does it apply to remote auditing? Despite the opportunities created by remote auditing, there are also risks associated with auditing suppliers remotely. People worry about auditees hiding hazardous situations or unacceptable environmental conditions such as filth or disrepair. However, unacceptable cleanliness and maintenance practices don’t happen overnight. Therefore, you should expect a clean and well-maintained facility to remain that way. One approach is to alternate between remote and on-site audits to verify the overall condition of a supplier’s facility. Therefore, the risk of auditees hiding objective evidence is more an issue of trust than a highly probable occurrence.

The more probable risks associated with remote auditing are related to the potential lack of availability of records. This is especially important for paper-based quality systems. Most people try to address this risk by scanning paper documents and records, but scanning documents have limited value. Scanning paper documents is more efficiently performed in a large batch by an automated or semi-automated process. Also, auditors and inspectors typically focus on the most recent records, and auditors and inspectors rarely sample 100% of the records. Therefore, the best risk controls include the following:

Ask a guide to send a digital picture of the record.
Use a tripod-mounted HD webcam focused on a music stand or similar surface.
Ask the auditee to read the document while you take notes.

In our experience, you will probably rely on all three risk controls, but it is unlikely to delay the audit. However, in response to the limited physical access to medical device facilities and personnel, certification bodies are sending out questionnaires to assess the risk of being unable to achieve audit objectives or cover the required scope of surveillance and recertification audits. As the audit program manager, you can reduce these risks by working with supply chain managers to develop new supplier questionnaires that specifically ask questions about the capability of supporting audits remotely. In particular, it would be essential to obtain facility maps to identify areas with inadequate cellular coverage and identify records that are only available in hardcopy format.

ISO 19011 – Do you need this quality system auditing standard? Read More »

Nov 1 2022

Auditing Risk Management Files

3 Comments / Auditing, CE Marking, ISO 14971:2019 (Risk Management), ISO Auditing, Remote Auditing / By Robert Packard / Audit, ISO 14971

What do you look at and look for when you are auditing risk management files to ISO 14971 and the new Regulation (EU) 2017/745?

Your cart is empty

Next week, November 15th @ Noon EST, you will have the opportunity to watch a live webinar teaching you what to look at and what to look for when you are auditing risk management files to Regulation (EU) 2017/745 and ISO 14971. Risk Management Files are one of the essential requirements of technical documentation required for CE Marking of medical devices. Most quality system auditors are trained on how to audit to ISO 13485:2016 (or an earlier version of that standard), but very few quality system auditors have the training necessary to audit risk management files.

Why you are not qualified to audit risk management files

Being a qualified lead auditor is not enough to audit the risk management process. When you are auditing a risk management file, you need risk management training and lead auditor training. To audit the risk management process, you will also need training on applicable guidance documents (i.e., ISO/TR 24971:2020) and applicable regulations (i.e., Regulation 2017/745 and/or Regulation 2017/746). There may also be device-specific guidance documents that specify known risks and risk controls that are considered state-of-the-art.

Creating an audit agenda

Once you have scheduled an audit of risk management files, and assigned a lead auditor, then the lead auditor needs to create an audit agenda. The audit can be a desktop audit that is performed remotely, or it can be an on-site audit. Regardless of the approach, the audit should include interviewing participants in the risk management process documented in the risk management file. As a rule of thumb, I expect a minimum of 30 minutes to be spent interviewing the process owner and one or more other participants. Then I spend an additional 60 minutes of auditing time reviewing documents and records.

Your audit agenda should specify the following items at a minimum:

the method of auditing to be used,
date(s) of the audit,
the duration of the audit,
the location of the audit, and
the auditing criteria.

The auditor(s) and the auditee participants should be identified in the audit agenda. Finally, you should specify which documents and records are required for audit preparation. These documents will be used to help identify audit checklist questions and to determine a sampling plan for the audit. At a minimum, you will need a copy of the risk management procedure and a list of the risk management files that are available to audit. You may also want to request the audit plan for each of those risk management files.

What did you look at and look for during your risk management audit?

When you audit the risk management process, you could take any of the following approaches or a combination of more than one. You could audit the process according to the risk management procedure. You could audit the process according to the risk management plan(s) for each risk management file. You could audit using the process approach to auditing. Finally, you could audit in accordance with specific requirements in the ISO 14971:2019 standard and applicable regulations (i.e., Regulation 2017/745). Regardless of which approach you take, your audit notes and the audit report should identify which documents and records you sampled and what you looked for in each document. Providing only a list of the documents is not enough detail.

Creating an auditing checklist for risk management files

Auditors with limited experience are taught to create an audit checklist by creating a table that includes each of the requirements of the audit criteria. For a risk management file, this would include a list of each of the requirements in ISO 14971 for a risk management file (i.e., Clause 9???). However, this approach is more like the approach that you should be using for a gap analysis. The better approach for creating an audit checklist for risk management files is to start by creating a turtle diagram. In the “process inputs” section (i.e., step 2 of 7), you would add questions derived from your review of the risk management plan(s). In the “process outputs” section (i.e., step 3 of 7), you would add questions specific to the risk management report and other records required in a risk management file. In the “with whom” section (i.e., step 5 of 7), you would add questions related to training and competency. You might also identify additional people involved in the risk management process, other than the process owner, to interview as a follow-up trail. In the “how done” section (i.e., step 6 of 7), you would add questions specific to the procedure and forms used for the risk management process. Finally, in the “metrics” section (i.e., 7 of 7), you would verify that the company is conducting risk management reviews and updating risk management documentation in accordance with the risk management procedure and individual risk management plan(s).

Audits are just samples

Just because you can generate a lot of questions for an audit checklist does not mean that you are required to address every question. Audits are intended to be a “spot check” to verify the effectiveness of a process. You should allocate your auditing resources based on the importance of a process and the results of previous audits. I recommend approximately three days for a full quality system audit, and approximately 90-minutes should be devoted to a process unless it is the design control process (i.e., Clause 7.3 of ISO 13485) which typically requires three to four hours due to the importance and complexity of the design controls process. Therefore, you should schedule approximately 30 minutes to interview people for the risk management process and approximately 60 minutes should be reserved for reviewing documents and records. With this limited amount of time, you will not be able to review every record or interview everyone that was involved in the risk management process. This is why auditors always remind auditees that an audit is just a sampling.

Which records are required in a risk management file?

The contents of a risk management file is specified in ISO 14971:2019, Clause 4.5. There are only four bullets in that section, but the preceding sentence says, “In addition to the requirements of other clauses of this document.” Therefore, your risk management file should address all of the requirements in ISO 14971:2019. What I recommend is a virtual risk management folder for each risk management file. As the auditor, you should also request a copy of the risk management policy and procedure. An example of what this would look like is provided below. The numbers in front of each subfolder correspond to the sub-clause or clause for that requirement in ISO 14971:2019.

Which records are most valuable when auditing risk management files?

As an auditor, I typically focus on three types of targets when auditing any process. First, I will sample any corrective actions implemented in response to previous audit findings. Second, will sample documents and records associated with any changes made to the process. Changes would also include any changes that were made to individual risk management files or the creation of a new risk management file. Finally, my third target for audit sampling is any item that I feel is at risk for safety or performance failures. The severity of the safety or performance failure is also considered when prioritizing audit sampling. In the context of a risk management file, I always verify that production and post-production activities are being conducted as planned. I try to verify that risk analysis documentation was reviewed for the need to update the documentation in response to complaints and adverse events.

More auditor training on risk management files

We are recording a live webinar intended to teach internal auditors and consultants how to perform a thorough audit of risk management files against the requirements of the new European Regulation (EU) 2017/745 and ISO 14971.

Auditing Risk Management Files

In this new webinar, you will learn how to conduct a process audit of risk management files. You will learn what to look at and what to look for in order to verify compliance with Regulation (EU) 2017/745 and ISO 14971:2019. The webinar will be approximately one hour in duration. Attendees will be invited to participate in the live webinar and receive a copy of the native slide deck. Anyone purchasing after the live event will receive a link to download the recording of the live event and the native slide deck.

Price: $64.50

In addition to this webinar on auditing risk management files, we also have other risk management training webinars available. The webinar on auditing risk management files will be hosted live on November 15, 2022 @ Noon EST (incorrect in the live video announcement).

Auditing Risk Management Files Read More »

Mar 18 2013

An Auditor’s Best Practices in Issuing a Major Nonconformity

4 Comments / ISO Auditing / By Robert Packard / Audit, Internal Auditor, ISO

From the opening meeting through the audit and closing meeting, the author describes an auditor’s best practices in issuing a major nonconformity.

As an auditor, one of the most important (and difficult) things to learn is how to issue a nonconformity—especially a major. This is usually done at the closing meeting of an audit, but the closing meeting is not where the process of issuing the nonconformity begins. Issuing a nonconformity starts in the opening meeting.

ISO 19011:2011 is the official guidance document for auditors of Quality Management Systems, and ISO 13485 is the quality system standard for medical device manufacturers. Section 6.4.2 of this Standard explains best practices for an opening meeting. The last five items in this section are critical to preparing the client for potential nonconformities:

Method of reporting audit findings, including grading, if any
Conditions under which the audit may be terminated
Time and place of the closing meeting
How to deal with possible findings during the audit
System for feedback from the auditee on findings or conclusions of the audit
Process for complaints and appeals

Methods of Reporting and Grading Nonconformities

The auditor should be crystal clear in their description of minor and major nonconformities or any other grading that will be used. The auditor should also make it clear that they are looking for conformity rather than nonconformity. This is an audit—not an inspection. Typically, a minor nonconformity is described as “a single lapse in the fulfillment of a requirement,” while a major nonconformity is described as one of the following: 1) “a total absence in the fulfillment of a requirement,” 2) “repetition of a previous nonconformity,” 3) “failure to address a previously identified minor nonconformity,” or 4) “shipment of non-conforming product.” When the auditor is in doubt, then the finding is minor, and never a major. For a major nonconformity to be issued, there can be no doubt.

Conditions for Termination

The option to terminate an audit is typically reserved for a certification audit where a major nonconformity is identified, and there is no point in continuing. Termination is highly discouraged, because it is better to know about all minor and major nonconformities right away, instead of waiting until the certification audit is rescheduled. The certification body will charge you for their time anyway.

Another reason for termination is when an auditor is unreasonable or inappropriate. This is rare, but it happens. If the audit is terminated, you should communicate this to upper management at the certification body and the company—regardless of which side of the table you sit. For FDA inspections, this is not an option. For audits performed by Notified Bodies, there is the possibility of suspension of a certificate in response to audit termination. Therefore, I always recommend appealing after the fact, instead of termination. Appealing also works for FDA inspections.

How to Deal with Findings

All guides and auditees should be made aware of possible findings at the time an issue is discovered. This is important so that an auditee has the opportunity to clarify the evidence being presented. Often, nonconformities are the result of miscommunication between the auditor and the auditee. This frequently happens when the auditor has a poor understanding of the process being audited. It is a tremendous waste of time for both sides when this occurs. If there is an actual nonconformity, it is also important to gather as much objective evidence as possible for the auditor to write a thorough finding and for the auditee to prepare an appropriate corrective action plan in response to the discovery.

Feedback from the Auditee

As an auditor, I always encourage auditees to provide honest feedback to me directly and to management, so that I could continue to improve. If you are giving feedback about an internal auditor or a supplier auditor, you should always give feedback directly before going to the person’s superior. You are both likely to work together in the future, and you should give the person every opportunity to hear the feedback first-hand.

When providing feedback from a third-party certification audit, you should know that there will be no negative repercussions against your company if you complain directly to the certification body. At most, the certification body will assign a new auditor for future audits and investigate the need for taking action against the auditor. In all likelihood, any action taken will be “retraining.” I never fired somebody for a single incident—unless they broke the law or did something unsafe. The key to providing feedback, however, is to be objective. Give specific examples in your complaint, and avoid personal feelings and opinions.

Complaints and Appeals

As the auditee, you should ask for the contact information of the certification body during the opening meeting. Ask with a smile—just in case you disagree, and so you can provide feedback (which might be positive). As the auditor, you should always make contact information for the certification body available. If you are conducting a supplier audit or an internal audit, you probably know the auditor’s boss, and there is perhaps no formal complaint or appeals process. In the case of a supplier audit, the customer is always right—even when they are wrong.

During the Audit

During the audit, you should always make the guide(s) and process owner(s) aware of any potential nonconformities as you find them. This is their opportunity to clarify the objective evidence for you and to explain why there is not a nonconformity. Often, at this point in the audit, I will refer to the Standard. I will identify the specific requirement(s) and show the process owner. I will say, “This is what I am trying to verify. Do you have anything that would help address this requirement?” If the process owner is unsure of how to meet the requirement, often, I will provide an example of how this requirement is addressed in other areas or at other companies.

If the audit is a multi-day audit, I will review the potential nonconformities at the end of the day and allow the auditee to provide additional objective evidence in the morning. If it is the last day of the audit, or it is a single-day audit, I will give auditees until the closing meeting to provide the objective evidence. Often, I will use this opportunity to explain what would be considered a minor nonconformity and what would be a major nonconformity. Usually, I can say, “This is not a major nonconformity because…”

Closing Meeting

The closing meeting should be conducted as scheduled, and the time/location should be communicated to upper management in the audit agenda and during the opening meeting. Top management won’t be happy about nonconformities, but failure to communicate when the closing meeting will be conducted will irritate them further.

At the closing meeting, the auditee should never be surprised. If an issue remains unfulfilled at the closing meeting, the auditee should be expecting a minor nonconformity—unless the issue warrants a major nonconformity. Since a minor nonconformity is described as “a single lapse in the fulfillment of a requirement,” it is difficult for an auditee to argue that an issue does not warrant a minor nonconformity. Typically, the argument is that you are not consistent with other auditors. The most common response to that issue is, “Audits are just a sample, and previous auditors may not have seen the same objective evidence.” The more likely scenario, however, is that the previous auditor interprets requirements, instead of reviewing requirements with the client, and ensuring both parties agree before a finding is issued.

If a finding is major, the auditee should have very few questions. Also, I often find the reason for a major nonconformity is a lack of management commitment to address the root cause of a problem. Issuing a major nonconformity is sometimes necessary to get management’s attention.

Regardless of the grading, all audit findings will require a corrective action plan—even an FDA warning letter requires a CAPA plan. Therefore, a major nonconformity is not a disaster. You just need to create a more urgent plan for action.

An Auditor’s Best Practices in Issuing a Major Nonconformity Read More »

Feb 2 2013

A 6 Step Approach if You Disagree With a Notified Body Auditor

2 Comments / ISO Certification / By Robert Packard / Audit, Notified Body

The author’s first certification audit experience is discussed, and we review six different approaches to take if you disagree with a notified body auditor.

My first certification audit ever didn’t go so well. The reason it didn’t go well is that the auditor wrote nonconformities that my boss and our regulatory consultant didn’t agree with. At the time, I was too inexperienced to know how to handle it. My boss and the consultant, however, totally lost it. I’ve never seen veins that big in someone’s forehead–even in cartoons.

I asked them both to leave the room because I was afraid to “push back” on the auditor. Many Management Representatives feel the same way that I did during that initial certification audit. The best way to summarize our concerns is with the following picture:

Recently another LinkedIn group member emailed me to say that they have seen several auditors for registrars identifying nonconformities that represented their own personal opinions rather than specific requirements of the Standard. For example, there is a requirement to assign management responsibilities and document it, but there is no requirement to have an organization chart.

Another common mistake is when auditors insist that a company must create a turtle diagram for every single process. I support the use of turtle diagrams 100%, but the only requirement in the Standard is to use the process approach–not turtle diagrams specifically.

My favorite is my own personal mistake. I wrote a nonconformity for not having a process for implant registration cards for a company that was planning to ship a high-risk implant product to Canada. There is a requirement for implant registry cards, but I forgot that Canada defines “implants” in this case as only a very short list of implant devices–not implants in general.

Auditors are human. These are audit findings–not a jail sentence. Everyone needs to remember that the worst that can happen is that you receive a nonconformity. If the auditor finds a nonconformity, then you need to develop a CAPA plan. If the auditor finds nothing, you still need to do your own internal audits to identify nonconformities and continuously improve processes.

What Should You Do When an Auditor is Wrong?

I recommend that you “push back,” but you need to know-how. Many consultants suggest saying, “Can you show me in the Standard where it says I have to do that?” That’s just like poking a bear. If you do it once, it’s annoying. If you do it multiple times, an auditor might just eat you.

One Management Representative did that to me after I had taken the time to review the requirements with him. I responded by holding the ISO 13485 Standard in front of him and reciting clause 7.3.2. He responded by saying, “Well, that’s up for interpretation.” I offered to recite the ISO 14969 guidance document for him, but his boss told him to shut up.

This certainly wasn’t the only time a client pushed back during a registration audit, but other clients have had the sense to argue about things they understood.

One of the clients I audited said that he would change the topic to the auditor’s favorite sports team. That’s one approach. I’m sure that more than one client has taken the approach of asking me to explain where they can learn about best practices. I’m sure that they were somewhat successful. Another approach is to slide the lunch menu in front of them; I have only met one auditor that would not be distracted by a lunch menu.

6 Step Approach When You Disagree With an Auditor

1. Shut-up and look it up (before you open your mouth, grab the applicable external Standard and locate the information you are looking for).

2. If you are still convinced that the auditor is wrong, then tell that you are having trouble finding the requirement. Show them where you are looking, and then ask them to help you find the requirement.

3. If the auditor can’t show you where you are wrong, or it appears that the auditor is interpreting the Standard as they see fit, then focus on asking the auditor for guidance on what they will be looking for in your CAPA plan.

4. If the CAPA plan the auditor is looking for is something you think is a good idea, then shut up and implement the improvements. If the CAPA plan is not acceptable to you, then you should ask what the process is for the resolution of disputes.

5. No matter what, don’t start an argument with the registrar. They enjoy it. They like a challenge and resent people with less experience criticizing them.

6. If you still disagree with your auditor, then you should ask if the auditor can explain the process for appealing findings and follow that process.

A 6 Step Approach if You Disagree With a Notified Body Auditor Read More »

Dec 24 2012

The Audit Program Manager: 4 Areas of Auditor Competency

2 Comments / ISO Auditing / By Robert Packard / Audit, Internal audit, Training

Passing a webinar on auditing does not make you competent.

This blog reviews an audit program manager’s four areas of auditor competency: experience, skills, training, and education.

Does your company ask incoming inspectors to update CAD drawings when a design changes occur? Of course not. Your company has engineers trained to use SolidWorks, and it takes a new engineer a while to become proficient with the software. Auditing is a skill that you learn—just like SolidWorks.

I’ve never met a manager who wondered where the value was in having an engineer update a drawing, but many managers view internal and supplier audits as a necessary evil. Instead of asking the expert how few audit days you can get away with, ask the expert: “What is the purpose of auditing?”

Internal auditing aims to confirm that the management system is effective and identify opportunities for improvement. Supplier auditing seeks to verify that a supplier can meet your needs and identify opportunities for improvement; therefore, if an auditor finds no nonconformities or opportunities for improvement, what a waste of time!

To receive value from auditing, you need competent auditors. Clause 6.2.1 of the ISO 13485 Standard states, “Personnel performing work affecting product quality shall be competent based on appropriate education, training, skills, and experience.” As the audit program manager, recruit people who demonstrate auditing competency.

Education

First, educational background is essential for auditors. You cannot expect someone who has never taken a microbiology course to be an effective auditor of sterilization validation. Likewise, someone who has never taken a course in electricity and magnetism will not be effective as an auditor for active implantable devices. Therefore, determine what types of processes the auditor will be auditing. Then, ensure that the person you hire has the necessary education to understand the processes they will be auditing.

Training

Second, auditors need to be trained before they can audit. The auditor needs training in three different aspects: 1) the process they will be auditing, 2) the standard that is the basis for assessing conformity, and 3) auditing techniques. If you are auditing Printed Circuit Board (PCB) manufacturers with Surface-Mount Technology (SMT), then you need to learn about the types of components used to make PCBs and how these components are soldered to a raw board. I know first-hand that anyone can learn how SMT works, but it took me a few months of studying.

If your company only sells medical devices in the United States, you will need to learn 21 CFR 820 (i.e., – the QSR). However, suppose your company also sells devices in Europe or Canada. In that case, you will need to learn ISO 13485, the Medical Device Directive (MDD) (93/42/EEC as modified by 2007/47/EC), and the Canadian Medical Device Regulations (CMDR). I learned about ISO 13485 in a four-and-a-half day lead auditor course in Florida, MDD in a three-day CE Marking Course in Virginia, and the CMDR in a two-day course taught by Health Canada in Ontario. A 50-minute webinar on each regulation is not sufficient for auditing.

Finally, you need training in the techniques of auditing. A two-day course is typically needed. I took a 50-minute webinar and passed a quiz before conducting my first internal audit, but I had not developed my skills.

Skills

Third, an auditor needs communication, organizational, and analytical skills to be helpful as an auditor. Communication skills must include the ability to read and write exceptionally well, and the auditor needs to be able to verbally communicate with auditees during meetings and interviews. The most difficult challenge for auditors is covering all items on their agenda in the time available. The auditor rarely has more time than they need to audit any topic, and audit team leaders must be able to manage their own time and simultaneously manage the time of several other auditors.

Experience

Last but not least, an essential aspect of auditor competency is experience. This is why third-party auditors are required to act as team members under the guidance of a more experienced auditor before they are allowed to perform audits on their own. This is required, regardless of how many internal or supplier audits the person may have conducted. More experienced auditors are also required to observe new auditors and recommend modifications to their techniques. Once a new auditor has completed a sufficient number of audits as a team member, the auditor can practice leading audits while being observed. After six to nine months, a new auditor is finally ready to be a lead auditor on their own. An internal auditor does not need the same degree of experience as a third-party auditor, but being shadowed two or three times is not sufficient experience for an auditor (first or second-party). For more information about this topic, please read my blog posting on auditor shadowing.

The Audit Program Manager: 4 Areas of Auditor Competency Read More »

Dec 24 2012

Internal Audit Training for New Hires

1 Comment / ISO Auditing / By Robert Packard / Audit, Internal audit

The author discusses a few proven internal audit training strategies (i.e., shadowing, auditing process owners) for new hires.

Once you have identified someone that you want to “hire” as an internal auditor, your next step should be to develop an “Onboarding“ plan for them with their boss. If you are hiring someone that will be a dedicated auditor, please ignore my quotation marks above. In most companies, however, the internal auditors are volunteers that report to another hiring manager. Therefore, as the audit program manager, you need to get a firm commitment from the auditor’s boss with regard to the time required to train the new auditor and to perform audits on an ongoing basis.

Winning Over the Boss

In my previous posting, I said that “The biggest reason why you want to be an auditor is that it will make you more valuable to the company.” The auditor’s boss may or may not agree with this statement, but the boss knows that the salary is coming out of their budget either way. Therefore, talk with the auditor’s boss and determine what the auditor’s strengths and weaknesses are. Find out which skills the boss would like to see the auditor develop. By doing this, the two of you can develop a plan for making the auditor more valuable to their boss AND the company.

Making Re-Introductions

Ideally, auditors are extraverted and have worked at the company long enough to know the processes and process owners that they will be assigned to audit—especially if they will be auditing upstream and downstream from their process area. In the past, the auditor may have been a customer or a supplier, but now the relationship with a process owner will change. Auditors are required to interview process owners, and this involves asking tough questions that might not be appropriate in the auditor’s regular job duties. Therefore, as the audit program manager, you should re-introduce the auditor to the process owner in their new capacity as an auditor. During this re-introduction, it is important to make three points:

The auditor is going to be trained first (on auditing and ISO 13485)
You will be shadowing the auditor during the audit, and
The auditor’s job is to help the process owner identify opportunities for improvement

By making the first point, you are reminding the process owner of the scheduled audit—well in advance. You are also informing the process owner that this auditor will have new skills, and the process owner should have some tolerance for mistakes that new employees make. You might also mention that you would like to get the process owner’s feedback after the audit, so the auditor knows which areas they need to improve upon to become better auditors. The second point should put the process owner at ease—assuming the process owner has a good relationship with you as the audit program manager. It is important to be descriptive when “shadowing” is mentioned. Both the process owner and the auditor may not understand the process or the purpose of shadowing. The following blog posting might help with this: “How do you shadow an auditor? Did you learn anything?”

The third point is the most critical step in onboarding a new auditor. For an auditor to be successful, they must ADD VALUE! As an auditor, you cannot pretend to add value. The process owner should know their process, and they probably know which areas are weakest. The audit program manager should encourage the process owner to list some specific areas in which they are having problems. Ideally, the process owner would be informed of this need before the re-introduction. Then the process owner can be better prepared for the meeting, and hopefully, they will have a few target areas already identified. Targets with associated metrics are the best choice for a new auditor because these targets reinforce the process approach to auditing.

Next Steps for Internal Audit Training

Once your new auditor has been re-introduced to the process owners, they will be auditing, and you need to begin the training process. As with any new employee, it is important to document training requirements and to assess the auditor’s qualifications against the requirements of an auditor. Every new auditor will need some training, but the training should be tailored specifically to the needs of the auditor. The training plan for a new auditor should include the following:

A reading list of company procedures specific to auditing and external standards that are relevant
Scheduled dates for the auditor to shadow another experienced auditor
Scheduled dates for an experienced auditor to shadow the auditor during the first two process audits (upstream and downstream)
Goals and objectives for the internal audit program; and
Any training goals that the auditor’s boss has identified for the auditor

Internal Audit Training for New Hires Read More »