Training

Seven ways to improve quality auditor training

2 Comments / ISO Auditing / By / , ,

A five-day lead auditor course is never enough. Effective quality auditor training must include practical feedback from an expert.

What is required for quality auditor training?

The key to training auditors to audit is consistent follow-up over a long period of time (1-2 years, depending upon the frequency of audits). I recommend following the same training process that accredited auditors must complete. I have adapted that process and developed seven (7) specific recommendations.

Training the trainer

One of my clients asked me to create a training course on how to train operators. I could have taught the operators myself, but so many people needed training that we felt it would be more cost-effective to train the trainers. Usually, I have multiple presentations archived that I can draw upon, but this time I had nothing. I had never trained engineers on how to be trainers before—at least not formally. I thought about the problems other quality managers have had in training internal auditors and how I have helped the auditors improve. The one theme I recognized was that effective quality auditor training needs to include practical feedback from an experienced auditor. An expert auditor that is training new auditors needs to identify systematic ways to provide feedback, and setting a benchmark for the number of times feedback will be provided is really helpful.

Improve by observing yourself and other quality auditors

Observing someone else is a great way to learn when you are learning any new skill. Interns often do this, which is also a technique used to train new auditors. This technique is called shadowing. You can learn by watching, but eventually, you need to try to do tasks that are beyond your comfort level, and it is best to practice auditing with an expert watching you.

Practice team member audit preparation

Many of the internal auditing procedures we see require new auditors to conduct three audits as team members before they can audit independently. In contrast, notified body auditors join as team members for 10-20 audits before they can act as lead auditors. During the training period, auditors in training observe multiple lead auditors and multiple quality systems. Each audit allows auditors in training to write nonconformities and receive feedback from a lead auditor. At the beginning of quality auditor training, the focus must be on audit preparation. What are the areas of importance, what are the results of previous audits, are there any previous audit findings to close, etc? This preparation can even be done as practice for a hypothetical audit.

During quality auditor training, practice the opening and closing meetings

Opening and closing meetings are one of the first things to teach a new lead auditor. Have new lead auditors rehearse their first few opening and closing meetings with you in private before conducting the opening and closing meetings. Ensure the lead auditor has an opening/closing meeting checklist to help them. Recording practice sessions is enormously helpful because the trainee can watch and observe their mistakes. As trainees get more experience, the opening and closing meetings should have time limits. Finally, you might ask members of top management to challenge the lead auditor with questions. The lead auditor needs to be comfortable with their decisions and the grading of the audit findings.

How to practice audit team leadership

Have new lead auditors conduct team audits with another qualified lead auditor for 10-20 audits before you allow them to conduct an audit alone. Leading the opening and closing meetings is usually the first area new lead auditors master. The most complicated area to learn is managing a team of auditors. Team members will fall behind schedule during audits, or someone will forget to audit a process. As a lead auditor, you must complete the audits for your assigned processes and communicate with the entire team to ensure everyone else is on schedule. As an observer, you must let lead auditors make mistakes and help them realize them. Initially, a trainer will encourage new lead auditors to give themselves more than enough time. As their training progresses, the timing needs to be shorter and more challenging. Ultimately, you have to push the team beyond its capability to teach new lead auditors to recognize problem signs and teach them how to fix the problems.

Shadow auditors virtually with recordings

Live shadowing is challenging for experts and trainees because you are distracted by listening to the auditee and observing the auditor. However, if an audit is recorded, the person shadowing can watch the recording. The audit is already completed, and there is little need to concentrate on the auditee. A recording allows the observer to focus on the auditor. If a new auditor is conducting their first audit, an expert should shadow the trainee for 100% of the audit. Gradually the observation can decrease with each subsequent audit.

Practice note-taking with recorded audits

Taking detailed notes is something that experts take for granted, but I learned a lot by watching FDA inspectors take notes during an inspection. Have a new auditor observe a few audits before they are allowed to participate. Make sure they take notes and explain what you are doing and why they are observing as you conduct audits. Review the notes of new auditors periodically throughout the audit to provide suggestions for improvement and identify missing information. You can also record a supplier audit or internal audit and let a new trainee take notes on the pre-recorded webinar. This eliminates the need to coordinate schedules to involve the trainee.

Quality auditor training should include practicing audit agenda creation

Have new lead auditors submit a draft audit agenda to you before sending it to the supplier or department manager. Usually, the first audit agenda will need revision and possibly multiple revisions. Make sure you train the person to include enough detail in the agenda, and using a checklist or template is recommended. The agenda creation will be part of the audit preparation, and it can be done without time pressure.

How do you audit the auditing process?

Most quality managers are experienced and have little trouble planning an audit schedule. The next step is to conduct the audit. The problem is that there is very little objective oversight of the auditing process. The ISO 13485 standard for medical devices requires that “Auditors shall not audit their own work.” Therefore, most companies will opt for one of two solutions for auditing the internal audit process: 1) hire a consultant or 2) ask the Director of Regulatory Affairs to audit the internal auditing process.

Both of the above strategies for auditing the internal audit process meet the requirements of ISO 13485, but neither approach helps to improve an internal auditor’s performance. I have interviewed hundreds of audit program managers over the years, and the most common feedback audit program managers give is “Change the wording of this finding” or “You forgot to close this previous finding.” This type of feedback is related to the report-writing phase of the audit process. I rarely hear program managers explain how they help auditors improve at the other parts of the process.

When auditors are first being trained, we typically provide examples of best practices for audit preparation, checklists, interviewing techniques, AND reports. After auditors are “shadowed” by the audit program manager for an arbitrary three times, the auditors are now miraculously “trained.” Let’s see if I can draw an analogy to make my point.

That kind of sounds like watching your 16-year-old drive the family car three times and then giving them a license.

About the Author

Rob Packard 150x150 Seven ways to improve quality auditor trainingRobert Packard is a regulatory consultant with 25+ years of experience in the medical device, pharmaceutical, and biotechnology industries. He is a graduate of UConn in Chemical Engineering. Robert was a senior manager at several medical device companies—including the President/CEO of a laparoscopic imaging company. His Quality Management System expertise covers all aspects of developing, training, implementing, and maintaining ISO 13485 and ISO 14971 certifications. 2009-2012, he was a lead auditor and instructor for one of the largest Notified Bodies. Robert’s specialty is regulatory submissions for high-risk medical devices, such as implants and drug/device combination products for CE marking applications, Canadian medical device applications, and 510(k) submissions. The most favorite part of his job is training others. He can be reached via phone at 802.258.1881 or by email. You can also follow him on LinkedIn, Twitter, and YouTube.

Seven ways to improve quality auditor training Read More »

How do you demonstrate training competence?

7 Comments / Education, ISO 14971:2019 (Risk Management) / By / ,

Anyone can sign and date training records, but how do you demonstrate the effectiveness of training and competence?
%name How do you demonstrate training competence?

What are the requirements for training?

The requirements for training are found in ISO 13485:2016, Clause 6.2 (i.e., Human Resources). Auditors and inspectors usually only ask for training records, but the requirement for records is the last item in the clause (i.e., “6.2(e) – maintain appropriate records of education, training, skills, and experience (see 4.2.5).”). The first four items in Clause 6.2 all include one word, but it’s not “records.” The most critical word in the requirements is “competence.”

What is the difference between training requirements, training records, training effectiveness, and training competence?

  • Training requirements define the “appropriate education, training, skills, and experience.” Specifically, what degree(s) is required, if any? What training curriculum needs to be completed? What skills are needed to perform the work? And how many years of experience are needed?
  • Training records are documented evidence that all training requirements have been met. Records may include a job application, resume, individual training records, group training records, or a training certificate. Records may also include the results of any quizzes to demonstrate training effectiveness and any evaluation of training competence.
  • Training effectiveness is how well a person understands the information communicated during training. Using terms from human factors engineering (i.e., the PCA Process), the person must have the correct perception and cognition. Cognition can be evaluated by giving people quizzes or asking them questions. Written and verbal exams may also include pictures and/or video. Performing these types of tasks on a quiz or exam are “knowledge tasks.”
  • Training competence is possessing the skill or ability to perform tasks. To demonstrate competence, someone that is already competent (i.e., a subject matter expert) must observe the trainee performing tasks. A person that once possessed the skill or ability can also judge competency, and a person that does not possess the skill or ability can be trained to evaluate skill and ability (e.g., an Olympic Judge or referee). Skills and abilities can be physical or cognitive, but there are also social skills. Examples of each are provided in the table below:

List of abilities and skills 1024x495 How do you demonstrate training competence?

What keeps me awake at night? (a story from April 2, 2011)

I am in Canada, it’s almost midnight, and I can’t sleep. I’m here to teach a Canadian client about ISO 14971—the ISO Standard for Risk Management of medical devices. Most companies requesting this training are doing so for one of two reasons: 1) some design engineers have no risk management training, or 2) the engineers have previous training on risk management, but the training is out-of-date. This Canadian company falls into the second category, and engineers with previous training ask the most challenging questions. This group of engineers forced me to re-read the Standard several times and reflect on the nuances of almost every single phrase. While teaching this risk management course, I learned more about risk management than I ever knew.

The four levels of the Learning Pyramid

The image at the beginning of this blog is a learning model that explains my experiences training Canadian design engineers. I call this model the “Learning Pyramid.” At the base of the pyramid, there are “Newbies.”

This is the first of four levels. At the base, students read policies and procedures, hoping to understand.

The student is now asked to watch someone else demonstrate proper procedures in the second level of the pyramid. One of my former colleagues has a saying that explains the purpose of this process well, “A picture tells a thousand words, but a demonstration is like a thousand pictures.” Our children call this “sharing time,” but everyone over 50 remembers this as “show and tell.”

The student is now asked to perform their tasks in the third level of the pyramid. This is described as “doing,” but in my auditing courses, I refer to this process as “shadowing.” Trainees will first read the procedures for Internal Auditing (level 1). Next, trainees will shadow the trainer during an audit to demonstrate the proper technique (level 2). During subsequent audits, the trainees will audit, and the trainer will shadow the trainee (level 3). During this “doing” phase, the trainer must watch, listen, and wait for what I call the “teachable moment.” This is a moment when the trainee makes a mistake, and you can use this mistake as an opportunity to demonstrate a complex subject.

Finally, in the fourth level of the Learning Pyramid, we now allow the trainee to become a trainer. This is where I am at. I am an instructor, but I am still learning. I am learning what I don’t know.

Teaching forces you back to the bottom of the Learning Pyramid

After teaching, the next step in the learning process is to return to the first level. I re-read the Standards and procedures until I understood the nuances I was unaware of. Then, I search for examples in the real world that demonstrate the complex concepts I am learning. After searching for examples, I tested my knowledge by applying the newly acquired knowledge to an FDA 510(k) or De Novo submission for a medical device client. Finally, I prepared to teach again. This reiterative process reminds me of the game Chutes and Ladders, but one key difference is that we never really reach the level of “Guru.” We continue to improve but never reach our goal of perfection.

Where is training competence in the Learning Pyramid?

Most people feel that a person is competent in a position when they can consistently perform a task. However, the ISO 13485 standard uses the phrase “necessary competence” to suggest that competency for any given task might not be the same for all people. For example, in some cases, it may be sufficient to perform a task with written instructions in front of you, while an instructor might be expected to perform the task without written instructions. The speed at which a person performs a task might also differ. For example, a secretary might be required to touch type at a speed of 60+ wpm with a QWERTY keyboard, while a stenographer must be able to use a STENO keyboard and write at 225 wpm. The accuracy requirements may differ for those two positions as well. Therefore, your company may decide that the training competence requirement for a design engineer is that they can pass an exam on risk management. However, the training competency requirement for the design project lead or Engineering Manager might include teaching inexperienced design engineers to apply the basic risk management principles. Demonstrating the ability to teach inexperienced design engineers might be demonstrated by an auditor interviewing members of your design team.

How do you demonstrate training competence? Read More »

The Audit Program Manager: 4 Areas of Auditor Competency

2 Comments / ISO Auditing / By / , ,

rookie The Audit Program Manager: 4 Areas of Auditor Competency

Passing a webinar on auditing does not make you competent.

This blog reviews an audit program manager’s four areas of auditor competency; experience, skills, training, and education.

Does your company ask incoming inspectors to update CAD drawings when there is a design change? Of course not. Your company has engineers that are trained to use SolidWorks, and it takes a new engineer awhile to become proficient with the software. Auditing is a skill that you learn—just like SolidWorks.

I’ve never met a manager that wondered where the value was in having an engineer update a drawing, but many managers view internal and supplier audits as a necessary evil. Instead of asking the expert how few audit days you can get away with, ask the expert: “What is the purpose of auditing?”

The purpose of internal auditing is to confirm that the management system is effective and identify opportunities for improvement. The purpose of supplier auditing is to verify that a supplier is capable of meeting your needs and identify opportunities for improvement. Therefore, if an auditor has no nonconformities and no opportunities for improvement were identified—what a waste of time!

To receive value from auditing, you need auditors that are competent. In clause 6.2.1 of the ISO 13485 Standard, it states, “Personnel performing work affecting product quality shall be competent based on appropriate education, training, skills, and experience.” As the audit program manager, ensure you recruit people that demonstrate auditing competency.

Education

First, educational background is important for auditors. You cannot expect someone who has never taken a microbiology course in their life to be an effective auditor of sterilization validation. Likewise, someone that has never taken a course in electricity and magnetism will not be effective as an auditor for active implantable devices. Therefore, determine what types of processes the auditor will be auditing. Then ensure that the person you hire to be an auditor has the necessary education to understand the processes they will be auditing.

Training

Second, an auditor needs to be trained before they can audit. The auditor needs training in three different aspects: 1) the process they will be auditing, 2) the standard that is the basis for assessing conformity, and 3) auditing techniques. If you are going to be auditing Printed Circuit Board (PCB) manufacturers with Surface-Mount Technology (SMT), then you need to learn about the types of components used to make PCBs, and how these components are soldered to a raw board. I know first-hand that anyone can learn how SMT works, but it took me a few months of studying.

If your company is only selling medical devices in the United States, then you will need to learn 21 CFR 820 (i.e., – the QSR). However, if your company also sells devices in Europe or Canada, you will need to learn ISO 13485, the Medical Device Directive (MDD) (93/42/EEC as modified by 2007/47/EC), and the Canadian Medical Device Regulations (CMDR). I learned about ISO 13485 in a four-and-a-half day lead auditor course in Florida,  MDD in a three-day CE Marking Course in Virginia, and the CMDR in a two-day course taught by Health Canada in Ontario. A 50-minute webinar on each regulation is not sufficient for auditing.

Finally, you need training in the techniques of auditing. A two-day course is typically needed. I took a 50-minute webinar and passed a quiz before conducting my first internal audit, but I had not developed my skills at that point. 

Skills

Third, an auditor needs communication, organizational, and analytical skills to be useful as an auditor. Communications skills must include the ability to read and write exceptionally well, and the auditor needs to be able to verbally communicate with auditees during meetings and interviews. The most difficult challenge for auditors is covering all items on their agenda in the time available. The auditor rarely has more time than the need to audit any topic, and audit team leaders must be able to manage their own time, as well as simultaneously managing the time of several other auditors. 

Experience

Last, but indeed not the least important aspect of auditor competency, is experience. This is why third-party auditors are required to act as team members under the guidance of a more experienced auditor before they are allowed to perform audits on their own. This is required, regardless of how many internal or supplier audits, the person may have conducted in the past. More experienced auditors are also required to observe new auditors and recommend modifications in their technique. Once a new auditor has completed a sufficient number of audits as a team member, the auditor is then allowed to practice leading audits while being observed. After six to nine months, a new auditor is finally ready to be a lead auditor on their own. An internal auditor does not need the same degree of experience as a third-party auditor, but being shadowed two-three times is not sufficient experience for an auditor (first or second-party). For more information about this topic, please read my blog posting on auditor shadowing.

The Audit Program Manager: 4 Areas of Auditor Competency Read More »

Instructor Effectiveness and the Power of a SNICKERS

1 Comment / ISO Auditing / By / , ,

The author discusses his personal experience attending a training class, instructor effectiveness, and reasons why he learned so much there.

I guess there are still some instructors out there that need to be reminded that we can all read the regulations on our own. We don’t need to pay $1,000+ per day to have someone read stuff for us. If that’s what you want, my 10-year old son is a fantastic reader. He’ll record anything you want, in any media format, for a much smaller dollar figure. If you want to learn something that is worth at least as much as your investment of time and money, then you need to find an instructor that can teach effectively.

Four Prerequisites for a Great Instructor:

1. The instructor must be an expert

2. The instructor must inspire participation

3. The instructor must provide practical examples for each student

4. The instructor must get everyone’s attention–and keep it

The most important determining factor of training effectiveness, however, occurs after the course is over When you are teaching quality assurance and regulatory affairs, you must develop your ability to inspire and engage students to Olympic medalist proportions. “Blah, blah, blah…” and “Death by PowerPoint” will get you fired. Don’t read your slides, don’t turn your back on the audience (or they’ll attack) and PLEASE don’t ever ask someone to read the definition of nonconformity out loud to the rest of the group. When I teach a class, you demand my best. I’m six-foot, six inches tall, and I have a loud booming voice. My mother has red hair, and she was an opera singer. I’ve got the voice to fill any auditorium and stage presence to match. But if you even start to nod off in class, I may just have to throw a Snickers bar at you.

snickers Instructor Effectiveness and the Power of a SNICKERS
This is an essential tool for any instructor. It functions as a tool to prod sleeping students awake, is small enough to cause minimal injury when thrown, serves as an emergency food supply, and is gluten-free.

If legal counsel recommends against using projectiles to encourage class participation, you might also consider one of my all-time genius ideas. I was scheduled for a two-day course in Ottawa, but the day before, I needed to perform an audit in Pennsylvania. Therefore, my flight was the last flight into Ottawa–arriving at approximately 1 a.m. My flight was delayed for more than an hour, and the person in front of me was trying to smuggle an extra carton of smokes into the country. Just before 4 a.m., my taxi arrived at the Albert at Bay Suite Hotel. The class started at 8 a.m. I made it to class on time, and excessive consumption of several pots of black coffee helped get me to lunch. Then my legs started getting a little shaky. Fortunately, there was a convenience store next door that sold my favorite chocolate–the Dark Aero bar! After four of these monstrous doses of cacao, and another pot of coffee, I could have listened to the lecture on the Canadian Medical Device Regulations all night.

aero bar Instructor Effectiveness and the Power of a SNICKERS
Hershey’s copied them, but the result was a mere shadow of Nestle’s greatness. Canadians know how to make junk food, tell a joke, and play hockey!

Lessons Learned

Despite the physical handicap of sleep deprivation, I still learned a ton from my course in Canada. Here’s why:

1. The instructors were experts. Both instructors were regulatory experts and Canadian. Both instructors taught this course twice a year for multiple years, and one of the instructors actually worked for Health Canada.

2. The instructors were blessed with the perfect audience that was hyper-motivated to pass the course. Everyone in the class worked for a Notified Body that had sponsored them to take the course. In order to stay employed and get a raise, I needed to pass that course. If I failed the exam, I had to absorb the cost to travel back to Ottawa and retake the course in February (BRRRR!).

3. Everyone has different experiences, and therefore not every example makes sense to us. Therefore, instructors need to use practical examples that are actionable. In this course, the instructors brought more than a dozen medical devices to the class. We studied the labeling and intended use of each device. Even students from Japan, Europe, and Australia were familiar with some of the products. This was critical because we all needed to be able to identify incorrect Canadian labeling.

4. The greatest asset of all was the humor of the instructor from Health Canada. He was hilarious. He had everyone laughing at his jokes for the entire course. Most of the jokes were not funny enough for a stand-up routine, but this was a mandatory regulatory course on Canadian regulations. Who would even expect a chuckle? Despite the strengths of these instructors, there is only one reason why I know the Canadian Medical Device Regulations (CMDR), as well as I do. I use them every single week.

Some Examples of How I Used the CMDR:

First, I had to audit 162 days for BSI in 2011. Ninety percent of those 162 days were for companies that required a Canadian Medical Device License. Therefore, I started auditing companies to the Canadian regulations immediately after the course. Second, I was also consulting for companies at the same time I was auditing for BSI. Consulting clients hired me to prepare and submit the Canadian Medical Device License Applications for them. I also had to revise and create new procedures specific to Canadian regulations. I spent another 60+ days in 2011 doing consulting. Finally, I was one of BSI’s instructors that taught the regulatory comparison course, which compared the regulations of the USA, Canada, Europe, Australia, and Japan.

Therefore, at least once a month, I had a classroom of 6-20 people asking me challenging questions about how to interpret and apply regulations from each of these countries to their products. I used every bit of knowledge I learned in that course in Ottawa, and I started using that knowledge immediately after the course. I had peers, superiors, clients, and students challenging my knowledge of these topics every day. This is what makes you a subject matter expert. If you need to learn something about Quality Assurance or Regulatory Affairs, a one-hour webinar, reading a blog, taking a five-day, or shadowing another more experienced person is not enough. In the end, all of the above will get you to the level of barely competent!  If you want to learn, you need a great instructor. Then you need to use everything you learned at every opportunity for several years. Some say, “If you can’t do, teach.” I say, “Bring a SNICKERS bar and throw it at them for faking it.”

Instructor Effectiveness and the Power of a SNICKERS Read More »

Scroll to Top