Blog

Archive for Auditing

Why remote audit duration should never exceed 90 minutes

This article explains why remote audit duration should not exceed 90 minutes and the unique opportunities created by a series of short remote audits.

download 3 Why remote audit duration should never exceed 90 minutes

Parkinson’s Law and the subject of audit duration

On November 19, 1995, Cyril Northcote Parkinson published an essay in the Economist. The title of the article was “Parkinson’s Law.” In the first sentence of the essay, Parkinson says, “It is a commonplace observation that work expands to fill the time available for its completion.” This essay refers to the observation that work is elastic concerning the demands on time when completing paperwork. When I first trained as an auditor, trainers emphasized that the most significant challenge faced by auditors is to complete an audit within the time available. An auditor’s task is to achieve the audit objectives within the time specified by the audit program manager. Time is precious, and you cannot easily extend the audit duration after scheduling the audit.

How much time is needed for a full quality system audit?

This question is a silly question to ask a consultant that works on an hourly basis. A consultant working on an hourly basis will make more money if they work more hours. Therefore, there is little incentive to underestimate the time required to complete the objectives of an audit. However, after completing hundreds of audits, I can honestly state that eight hours is not enough time to perform a full quality system audit of a medical device company’s quality system. However, I completed a full quality system audit of a small company in less than two days. I also had difficulty completing an audit of a larger company in four days. An FDA inspector typically requires four days to complete a routine inspection, even at foreign manufacturers where English is a second language, and they only need to return on the fifth day to prepare their FDA 483 observations to give to the company. Therefore, three days is typically the absolute minimum time required to complete a full quality system audit.

Does Parkinson’s Law apply to audit duration?

Parkinson’s Law certainly applies to the audit duration. If the lead auditor assigns a team member to review the CAPA process, the task is unlikely to be completed in 30 minutes, and most auditors would struggle to appear busy for more than three hours. You need enough notes to provide objective evidence of conformity for your audit report, but if you finish too quickly, then others may perceive that you were not thorough. Therefore, most auditors will begin any process audit by asking for a copy of the procedure and a log of the records available. The auditor will quickly review the procedure’s revision history to determine when the last revision was made and if there have been any significant revisions since the last audit. Next, the auditor will review the log to estimate how many records should be sampled. The auditor will then estimate how much time is needed to review the sampled records. Finally, a quick mental calculation is made to determine how much time remains for procedure review before the auditor must move on to interview the next subject matter expert.

Why are auditors always behind schedule?

An auditor begins with small, close-ended questions that are designed to put the auditee at ease. The auditor may even comment on unrelated subjects to build rapport first. Records may not be readily available, but auditors almost always have to wait for record retrieval. The request is recorded, copies are made, and the subject matter expert may need a little time to review before handing the auditor the requested record. Auditors will ask clarifying questions, and auditees will need a few moments to check their facts. Any one of these delays is insignificant by itself, but collectively there may be two-and-half minutes of delay cumulatively for each record requested if you sample five records, which represents a combined delay 12.5 minutes. If you average only seven minutes to review each record, then a sampling of five records will require 47.5 minutes. This will leave you only 12.5 minutes for introductions, review of the procedure, and conclusions. If you want to interview any of the people that investigated root-cause, then you will need more than an hour to complete your audit, and you will not finish in the one hour scheduled.

Why is it so hard to complete a full quality system audit in three days?

Most of your process audits require a few more minutes than you expected, but you will also need time to walk to the next subject matter expert, or you will be waiting for the next subject matter expert to enter the conference room. If the quality system consists of only the minimum twenty-eight required procedures, your full quality system audit will require more than 28 hours to complete. If there are additional regulatory requirements for CE Marking or MDSAP certification, you will need even more time to audit every process. You should also expect certain processes to require more time to properly sample records, such as technical documentation and design controls. Even the most experienced auditors struggle to review a technical file and/or design history file in less than two hours.

What happens to an auditor after auditing all day?

As a Notified Body auditor, I used to leave my home in Vermont on Sunday afternoon and drive two hours to the nearest major airport. Then I would be gone all week conducting audits. On Friday, I would drive home and arrive in the middle of the night. Each day audits would begin early in the morning, and I would complete the day after 8.5 to 9 hours of work. Jet lag, sleep deprivation, too little exercise, and constantly eating at restaurants took its toll. I would consult my Google calendar to learn what city I was in each morning, and to remember what company I was on my schedule for the day. I would purposely try to do as much walking around during the day just to keep my blood flowing and to help stay awake. I would read documents while pacing back-and-forth in conference rooms, and I would always make sure that we had to audit the most remote area of a facility after lunch to make sure that I didn’t fall asleep. I will tell stories and jokes to entertain my hosts, but it was necessary to break up the monotony of auditing quality systems seven days a week. I would make sure I drank at least six liters of water each day for health, but this also gave me an excuse to go to take frequent bathroom breaks. Somehow I managed to survive that lifestyle for more than three years. Each day my feet, legs, back, and neck were in severe pain. I had constant headaches, and I know the quality of my work gradually declined throughout each day. The most valuable lesson I learned was, you need to move frequently, or you will die.

unnamed Why remote audit duration should never exceed 90 minutes

What happens when you sit in front of a computer for eight hours?

I can sit in front of a computer longer than almost anyone I know. When I focus on work, four hours can elapse without me getting up from a chair even once. I might pick up my empty coffee mug four or five times to take a sip before I am conscious of the need to get another cup. On days where my schedule consists primarily of Zoom meetings, I may sit through as many as six consecutive meetings before I take the time to get up and go to the bathroom and get a drink of water. Clients may perceive that I have tremendous endurance, but there are negative consequences to this work pattern. My wrist becomes sore, and I need to switch my mouse pad and the style of the mouse I am using every day. I change computers, switch microphones, and take a short walk. My neck, back, and legs will hurt worse than any of the audits during my years as a Notified Body auditor. Sitting at a computer all day has resulted in mild symptoms of restless legs syndrome. Sitting at a computer continuously for the audit duration is physically exhausting and tedious. If you must complete a remote audit on a continuous eight-hour day, you can, but it is not healthy or productive. The negative health consequences and negative impact on productivity are equally applicable to auditees.

What can you do to reduce audit fatigue during a remote audit?

The most straightforward strategy for reducing fatigue is to take breaks. Instead of auditing for eight hours continuously, try auditing in two or three 90-minutes segments each day. If you are auditing someone in a different time zone, you may only be able to accommodate an audit duration of one 90-minute session per day without working through the night. Taking breaks will allow you to leave your computer, eat food, and even go to the bathroom. You can recharge your headset during a break too. You should consider taking a walk outside. It is incredible how much better you feel when you get some exercise, stretch, and experience a little natural light instead of the unnatural glow of your computer’s monitor. The person you are auditing will appreciate the breaks, but they will also enjoy the improvement in your overall demeanor. A simple smile after a 30-minute break has a tremendous positive impact.

How can we utilize breaks more effectively during remote audits?

Auditors need documents and records to review as objective evidence. The most obvious way to make use of breaks is for the auditor to give the auditee a list of documents and records to gather during the break. This will give the auditee an excuse to go and get the documents and records if they are stored in another location. The auditee might also scan records during a break. A break also gives subject matter experts time to re-familiarize themselves with the documents and records before resuming the audit. Auditees and auditors will need to recharge batteries, but the auditor might take time to convert their notes into a summary for the final audit report. The auditor might also review the audit criteria one more time before writing a nonconformity. The auditee might take advantage of the break to initiate a new CAPA and write a draft of the corrective action plan. Then when the audit resumes, the auditee can review the draft plan with the auditor to ensure that the plan is appropriate and nothing was accidentally omitted from the CAPA plan.

unnamed 1 Why remote audit duration should never exceed 90 minutes

Why are 90 minutes a magical audit duration?

Auditing one process in a single 45-60 minute session is ok, but if you audit two processes in a single 90-minute session, you can reduce the time spend starting and stopping the audit session by half. Adding a third process to a single session will have a smaller impact, and the meeting will need to be so long that most participants will begin to lose concentration, and fatigue becomes a significant factor. Ninety minutes is not quite long enough to audit two processes effectively. Still, an auditor can request procedures in advance of the session or spend time after the session reviewing procedures. Therefore, by paying an additional 30 minutes reviewing two procedures “off-line,” the auditor can dedicate 100% of the “on-line” time to reviewing records and interviewing subject matter experts. The result is a fast-paced, 90-minute session where each subject matter expert typically is only needed for 45 minutes. Alternatively, if you are auditing more complex records like a design history file, you can spend all 90 minutes discussing that area.

Posted in: Auditing, ISO Auditing, Remote Auditing

Leave a Comment (0) →

How to apply a risk-based auditing approach to audits and remote audits

In this article, you will learn what risk-based auditing is, and how to apply a risk-based approach to auditing and remote supplier audits.

animal nature reptile animal world How to apply a risk based auditing approach to audits and remote audits

Risk-based auditing considers the risks of failing to achieve audit objectives and the opportunities created by choosing various audit methods and strategies. For example, if you are conducting your first internal audit for a new quality system, a desktop audit of procedures might be appropriate. Alternatively, if you are auditing a mature quality system where very few changes to procedures have been made in the past year, a desktop audit would be a waste of time, and using the element approach to auditing is unlikely to add much value. Audits are meant to be a sampling. Therefore, you should focus on areas of importance, areas where previous nonconformities were identified, any new products or processes, and anything that changed significantly. 

The risk-based auditing approach is the most significant change in ISO 19011:2018

One of the main differences between ISO 19011:2018 and the previous 2011 version is the addition of a “risk-based approach” to the principles of auditing. Specifically, clause 4(g) of the guidelines for auditing management systems is, “The risk-based approach should substantively influence the planning, conducting and reporting of audits to ensure that audits are focused on matters that are significant for the audit client, and for achieving the audit program objectives.” A lot of people are unsure of what is meant by a risk-based approach, but the key to understanding this is to focus on the definition of risk. From a product perspective, the risk is the “combination of the probability of occurrence of harm and the severity of that harm.” From a process perspective, the risk is the “effect of uncertainty on an expected result” (ISO 9001:2015, clause 3.09). Therefore, auditors should emphasize medical devices that present the highest severity of harm and any devices that have a high probability of hazards or hazardous situations occurring. When an auditor is focusing on a process, rather than a specific medical device, auditors should emphasize any processes that are not under control and any recent process changes.

Auditor selection should also be risk-based

If you are conducting a supplier audit as part of your initial supplier qualification for a critical component supplier or contract manufacturer, you should consider doing a team audit with a multi-disciplinary team. This is a risk-based approach to the supplier qualification process, which ensures that you have subject matter experts evaluating each process instead of auditors with a general quality assurance background. This approach also forces more of your personnel to introduce themselves to the new supplier, and the audit will develop more reliable communication channels between your two companies. Alternatively, if you are conducting a routine internal audit of a production process, you might select a new lead auditor to conduct the audit. You don’t expect any significant findings in a routine internal audit of an established production process. In your role as an audit program manager, you need to match the new lead auditor to a process that will force them to look at all aspects of the process approach to auditing. Specifically, process validation, calibration, maintenance, and process monitoring are areas that may not apply to other administrative process areas such as purchasing.

Risk-based auditing should influence your auditing schedule.

The frequency of auditing suppliers and internal process areas should reflect the risks associated. Therefore, when you create or update your auditing schedule, you should consider the risk level of products being audited and the process being audited. Production processes with a moderate or high level of non-conforming products may need to be audited more than once per year. Still, a supplier with an excellent track record of extremely high quality and on-time delivery may be audited on alternating years. If you previously scheduled a remote audit, you may want to alternate to conducting an on-site audit the next time.

The duration of your audits should not always be the same either. If one production process makes one product in low-volume, and another production process makes multiple products in high-volume, you should not schedule a two-hour internal audit for both processes every year. The low-volume production process may only need a one-hour audit once per year. In contrast, the high-volume process may require a four-hour internal audit or multiple audits each year.

Risk-based auditing applied to remote supplier auditing.

The risk-based auditing approach was added to ISO 19011:2018 as the seventh principle of auditing. This represents the most significant change to that standard, but how does it apply to remote auditing? Despite the opportunities created by remote supplier auditing, there are also risks associated with auditing suppliers remotely. Most people worry about auditees hiding hazardous situations or unacceptable environmental conditions such as filth or disrepair. However, unacceptable cleanliness and maintenance practices don’t happen overnight. Therefore, you should expect a clean and well-maintained facility to remain that way. One approach is to alternate between remote and on-site audits to verify the overall condition of a supplier’s facility. Therefore, the risk of auditees hiding objective evidence is more an issue of trust than a highly probable occurrence.

The more probable risks associated with remote auditing are related to the potential lack of availability of records. This is especially important for paper-based quality systems. Most people try to address this risk by scanning paper documents and records, but scanning documents have limited value. Scanning paper documents is more efficiently performed in a large batch by an automated or semi-automated process. Also, auditors and inspectors typically focus on the most recent records, and auditors and inspectors rarely sample 100% of the records. Therefore, the best risk controls include:

  • Ask a guide to send a digital picture of the record.
  • Use a tripod-mounted HD webcam focused on a music stand or similar surface.
  • Ask the auditee to read the document while you take notes.

In our experience, you will probably rely on all three risk controls, but it is unlikely to delay the audit. However, in response to the limited physical access to medical device facilities and personnel, certification bodies are sending out questionnaires to assess the risk of being unable to achieve audit objectives or cover the required scope of surveillance and recertification audits. As the audit program manager, you can reduce these risks by working with supply chain managers to develop new supplier questionnaires that specifically ask questions about the capability of supporting audits remotely. In particular, it would be important to obtain facility maps to identify areas with inadequate cellular coverage and identification of records that are only available in hardcopy format.

 

Posted in: Auditing, Remote Auditing

Leave a Comment (0) →

Auditing Technical Files

This article explains what to look at and what to look for when you are auditing technical files to the new Regulation (EU) 2017/745 for medical devices.

Auditing Technical Files what to look at and what to look for 1024x681 Auditing Technical Files

Next week, August 8th @ Noon EDT, you will have the opportunity to watch a live webinar teaching you what to look at and what to look for when you are auditing technical files. Technical files are are the technical documentation required for CE Marking of medical devices. Most quality system auditors are trained on how to audit to ISO 13485:2016 (or an earlier version of that standard), but very few quality system auditors have the training necessary to audit technical files.

Why you are not qualified for auditing technical files

If you are a lead auditor, you are probably a quality manager or quality engineer. You have experience performing verification testing and validation testing, but you have not prepared a complete technical file yourself. You certainly can’t describe yourself as a regulatory expert. You are a quality system expert. A couple of webinars on the new European regulations is not enough to feel confident about exactly what the content and format of a technical file for CE marking should be.

Creating an auditing checklist

Most auditors attempt to prepare for auditing the new EU medical device regulations by creating a checklist. The auditor copies each section of the regulation into the left column of a table. Then the auditor plans to fill in the right-hand columns of the table (i.e., the audit checklist), with the records they looked at and what they looked for in the records. Unfortunately, if you never created an Essential Requirements Checklist (ERC) before, you can only write in your audit notes that the checklist was completed and what the revision date is. How would you know if the ERC was correctly completed?

In addition to the ERC, now called the Essential Performance and Safety Requirements (i.e., Annex I of new EU regulations), you also need to audit all the Technical Documentation requirements (i.e., Annex II), all the Technical Documentation on Post-Market Surveillance (i.e., Annex III), and the Declaration of Conformity (i.e., Annex IV). These four annexes are 19 pages long. If you try to copy-and-paste each section into an audit checklist, you will have a 25-page checklist with more than 400 things to check. The end result will be a bunch of checkboxes marked “Yes,” and your audit will add no value.

Audits are just samples

Every auditor is trained that audits are just samples. You can’t review 100% of the records during an audit. You can only sample the records as a “spot check.” The average technical file is more than 1,000 pages long, and most medical device manufacturers have multiple technical files. A small company might have four technical files. A medium-sized company might have 20 technical files, and a large device company might have over 100 files. (…and you thought the 177-page regulation was long.)

Instead of checking a lot of boxes, “Yes,” you should be looking for specific things in the records you audit. You also need a plan for what records to audit. Your plan should focus on the most important records and any problem areas that were identified during previous audits. You should always start with a list of the previous problem areas because there should be corrective actions that were implemented, and the effectiveness of corrective actions needs to be verified.

Which records are most valuable when auditing technical files?

I recommend selecting 5-7 records to sample. My choices would be: 1) the ERC checklist, 2) the Declaration of Conformity, 3) labeling, 4) the risk management file, 5) the clinical evaluation report, and 6) post-market surveillance reports, and 7) design verification and validation testing for the most recent design changes. You could argue that my choices are arbitrary, but an auditor can always ask the person they are planning to audit if these records would be the records that the company is most concerned about. If the person has other suggestions, you can change which records your sample. However, you should try not to sample the same records every year. Try mixing it up each year by dropping the records that looked great the previous year, and adding a few new records to your list this year.

What to look for when auditing technical files

The first thing to look for when you audit records: has the record been updated as required? Some records have a required frequency for updating, while other records only need to be updated when there is a change. If the record is more than three years old, it is probably out of date. For clinical evaluation reports and post-market surveillance reports, the new EU regulations require updating these reports annually for implantable devices. For lower-risk devices, these reports should be updated every other year or once every three years at a minimum.

Design verification and design validation report typically only require revisions when a design change is made, but a device seldom goes three years without a single change–especially devices containing software. However, any EO sterilized product requires re-validation of the EO sterilization process at least once every two years. You also need to consider any process changes, supplier changes, labeling changes, and changes to any applicable harmonized standards.

Finally, if there have been any complaints or adverse events, then the risk management file probably required updates to reflect new information related to the risk analysis.

Which record should you audit first?

The ERC, or Essential Performance and Safety Requirements checklist, is the record you should audit first. First, you should verify that the checklist is organized for the most current regulations. If the general requirements end with section 6a, then the checklist has not been updated from the MDD to the new regulations–which contains nine sections in the general requirements. Second, you should make sure that the harmonized standards listed are the most current versions of standards. Third, you should make sure that the most current verification and validation reports are listed–rather than an obsolete report.

How to learn more…

If you want to learn more about how to audit technical files, please register for our webinar on auditing technical files–August 8th @ Noon EDT. We also provide a new audit report template specifically written for your next technical file audit.

Posted in: Auditing, Technical Files

Leave a Comment (0) →

ISO 19011 – Guidelines for Auditing Quality Management Systems

The blog reviews additions and changes to the guidance for auditing quality management systems, ISO 19011, including guidance on remote auditing.

If you have ever taken a lead auditor course, one of the critical handouts for the class should have been ISO 19011. The title is “Guidelines for Auditing Quality Management Systems.” In November 2011, this Standard was updated, and the changes were not superficial.

ISO 19011 covers the topic of quality management system auditing. This Standard provides guidance on managing audit programs, conducting both internal and external audits, and how to determine auditor competency. Improvements to the 2011 Version of the Standard include:

  1. Broadening the scope to all management systems
  2. Clarifying the relationship between ISO 17021 and ISO 19011
  3. Introduction of remote audit methods
  4. Introduction of risk as an auditing concept
  5. Confidentiality is a “new” principle
  6. Clause 5, Managing an audit program, was reorganized
  7. Clause 6, Performing an audit, was reorganized
  8. Clause 7, Competence and evaluation of auditors, was reorganized & strengthened
  9. Annex B is new, and the contents of the help boxes were moved to this Annex
  10. Annex A now includes examples of discipline-specific knowledge and skills

One of the most common points of confusion in the lead auditor course is the difference between first, second, and third-party audits. In the previous revision of this Standard, this was just a note at the bottom of page one and the top of page two. The note was not very clear either. The new version of 19011, in Table 1 (reproduced below), the difference between these three types of auditing is crystal clear:

19011 table 11 ISO 19011   Guidelines for Auditing Quality Management Systems

The above table is just an example of the improvements made to ISO 19011, and of course, there is a little value-add to clarifying a definition. Figure 1 from the new version, “Process flow for the management of an audit program”, is a better example of a “value-add.” This vertical flow chart is reminiscent of Figure 1 from ISO 14971:2007. It categorizes the various stages of audit program management into the Plan-Do-Check-Act (PDCA) cycle. I highly recommend this style for presenting any process in your internal procedures as an example of best practices in writing an SOP. The flow chart even references each of the clauses in the Standard. Unfortunately, Figure 2, “Typical audit activities,” does not categorize the stages of audit activities (Clauses 6.2 – 6.7 of the revised Standard) into the PDCA cycle. I guess they needed to leave some improvement for the next revision.

The new version retained the opening meeting checklist that was in the previous revision (Clause 6.4.2), and Clause 6.4.9 has a brief closing meeting checklist. Figure 3, “Overview of the process of collecting and verifying information,” is a poor example of a flow chart. Should I make a better one? (Send me an email if you think I should.)

The most valuable changes in this revision are Clause 5.3.2, “Competence of the person managing the audit program,” and all of Clause 7. Most of the audit procedures I read neglect to define the qualifications and methods for determining the competency of the audit program manager. Clause 5.3.2 tells you how. Put it in your own procedure. Most of the procedures I read include qualifications for a “Lead Auditor,” but I seldom see anything regarding competency. Unfortunately, this Standard only specifically addresses the “Lead Auditor” competency in a two-sentence paragraph—Clause 7.2.5. When I teach people how to be a lead auditor, I spend more than an hour on this topic alone. 

ISO 19011 Standard

The Standard would be more effective by providing an example of how third-party auditors become qualified as a Lead Auditor. Third-party accreditation requires the auditor to be an “acting lead” for audit preparation, opening meetings, conducting the audit, closing meetings, and final preparation/distribution of the audit report. This must be performed for 15 certification audits (i.e., – Stage 2 certification or re-certification), and another qualified lead auditor must evaluate you and provide feedback.

The last significant additions to this Standard were the Appendices. Annex A provides examples of discipline-specific knowledge and skills of auditors. This section is a little on the dull side. I prefer to tell a story about the internal auditor that was auditing incoming inspection—but they had no idea how to check for calibration, or how to measure components.

Appendix B, the finale, has a table (Table B.1) that provides some guidance on how to conduct remote audits (i.e. desktop audits). I was pleased to see that conducting interviews is a significant part of remote auditing in this table. Section B.7 provides some suggestions concerning conducting interviews. Still, if you exhibit all 13 of the professional behavior traits found in Clause 7.2.2, then you don’t need any advice on how to speak with people. For the rest of us mortals, we could use a five-day course on interviewing alone.

Posted in: Auditing

Leave a Comment (0) →