Author name: Robert Packard

Auditor shadowing as an effective auditor training technique

2 Comments / ISO Auditing / By /

This article reviews auditor shadowing as an effective auditor training technique, but we also identify five common auditor shadowing mistakes.

How do you evaluate auditor competency?

Somewhere in your procedure for quality audits, I’ll bet there is a section on auditor competency. Most companies require that the auditor has completed either a course for an internal auditor or a lead auditor course. If the course had an exam, you might even have evidence of training effectiveness. Demonstrating training competence is much more challenging. One way is to review internal audit reports, but writing reports is part of what an auditor does. How can you evaluate an auditor’s ability to interview people, take notes, follow audit trails, and manage their time? The most common solution is to require the auditor “shadow” a more experienced auditor several times, and then the trainer will “shadow” the trainee.

auditor with clip board 203x300 Auditor shadowing as an effective auditor training technique
If you are shadowing, you are taking notes, so you can discuss your observations with the person you are shadowing later. 

Auditor shadowing in 1st party audits

ISO 19011:2018 defines first-party audits as internal audits. When first-party auditors are being shadowed by a trainer or vice versa, there are many opportunities for training. The key to the successful training of auditors is to recognize teachable moments.

When the trainer is auditing, the trainer should look for opportunities to ask the trainee, “What should I do now?” or “What information do I need to record?” In these situations, the trainer asks the trainee what they should do BEFORE doing it. If the trainee is unsure, the trainer should immediately explain what, why, and how with real examples.

When the trainer is shadowing, the trainer should watch and wait for a missed opportunity to gather important information. In these situations, the trainer must resist guiding the trainee until after the trainee appears to be done. When it happens, sometimes the best tool is simply asking, “Are you sure you got all the information you came for?”

Here are five (5) mistakes that I observed trainers made when they were shadowing:

1. Splitting up, instead of staying together, is one of the more common mistakes I have observed. This happens when people are more interested in completing an audit rather than taking advantage of training opportunities. The trainee may be capable of auditing independently, but this is unfair to the trainee because they need feedback on their auditing technique. This is also unfair to the auditee because it is challenging to support multiple auditors simultaneously. When it is unplanned, trainers may not be available for both auditors. If an audit is running behind schedule, this is the perfect time to teach a trainee how to recover sometime in their schedule. Time management is, after all, one of the most challenging skills for auditors to master.

2. Staying in the conference room instead of going to where the work is done is a common criticism of auditors. If the information you need to audit can be found in a conference room, you could have completed the audit remotely. This type of audit only teaches new auditors how to take notes. These are necessary skills that auditors should master in a classroom before shadowing.

3. Choosing an administrative process is a mistake because administrative processes limit the number of aspects of the process approach that an auditor-in-training can practice. Administrative processes rarely have equipment that requires validation or calibration, and the process inputs and outputs consist only of paperwork, forms, or computer records. With raw materials and finished goods to process, the auditor’s job is more challenging because there is more to be aware of.

4. Not providing honest feedback is a huge mistake. Auditors need to be thick-skinned, or they don’t belong in a role where they will criticize others. Before you begin telling others how to improve, you must self-reflect and identify your strengths and weaknesses. Understanding your perspective, strengths, weaknesses, and prejudices is critical to being a practical assessor. As a trainer, it is your job to help new auditors to self-reflect and accurately rate their performance against objective standards.

5. “Silent Shadowing” has no value at all. By this, I mean shadowing another auditor without asking questions. You should mentally pretend you are doing the audit if you are a trainee. Whenever the trainer does something different from how you would do things, you should make a note to ask, “Why did you do that?” If you are the trainer, you should also mentally pretend you are doing the audit. It is not enough to be present. Your job is to identify opportunities for the trainee to improve. The better the trainee, the more challenging it becomes to identify areas for improvement.  This is why training other auditors have helped me improve my auditing skills.

Auditor shadowing in 2nd party audits

supply chain weakest link Auditor shadowing as an effective auditor training technique

Auditors responsible for supplier auditing are critical to supplier selection, supplier evaluation, re-evaluation, and the investigation of the root cause for any non-conformities related to a supplier. Auditor shadowing is a great tool to teach supplier auditors and other people responsible for supply-chain management what to look at and what to look for when they audit a supplier. If you are developing a new supplier quality engineer responsible for performing supplier audits, observing the auditor during some actual supplier audits is recommended. Supplier audits are defined as second-party audits in the ISO 19011 Standard. The purpose of these audits is not to verify conformity to all the aspects of ISO 13485. Instead, the primary purpose of these audits is to verify that the supplier has adequate controls to manufacture conforming products for your company consistently. Therefore, processes such as Management Review (Clause 5.6) and Internal Auditing (Clause 8.2.2) are not typically sampled during a second-party audit.

The two most valuable processes for a second-party auditor to sample are 1) incoming inspection and 2) production controls. Using the process approach to auditing, the second-party auditor will have an opportunity to verify that the supplier has adequate controls for documents and records for both of these processes. Training records for personnel performing these activities can be sampled. The adequacy of raw material storage can be evaluated by following the flow of accepted raw materials, leaving the incoming inspection area. Calibration records can be sampled by gathering equipment numbers from calibrated equipment used by both processes. Even process validation procedures can be assessed by comparing the actual process parameters being used in manufacturing with the documented process parameters in the most recent validation or re-validation reports.

I recommend having the trainee shadow the trainer during the process audit of the incoming inspection process and for the trainer to shadow the trainee during the process audit of production processes. The trainee should ask questions between the two process audits to help them fully understand the process approach to auditing. Supplier auditors should also be coached on techniques for overcoming resistance to observing processes involving trade secrets or where competitor products may also be present. During the audit of production processes, the trainer may periodically prompt the trainee to gather the information that will be needed for following audit trails to calibration records, document control, or for comparison with the validated process parameters. The “teachable moment” is immediately after the trainee misses an opportunity, but while the trainee is still close enough to go back and capture the missing details.

Are you allowed to shadow a 3rd party auditor or FDA inspector?

qsit inspection Auditor shadowing as an effective auditor training technique

Consider using 3rd party audits and inspections as an opportunity to shadow experienced auditors to learn what they are looking at and what they look for. In addition to shadowing an expert within your own company or an auditor/consultant you hire for an internal audit, you can also shadow a 3rd party auditor or an FDA inspector. This concept was the subject of a discussion thread I ran across on Elsmar Cove from 2005. The comments in the discussion thread supported the idea of shadowing a 3rd party auditor. The process owner (i.e., the manager responsible for that process) should be the guide for whichever process is being audited, and the process owner is responsible for addressing any non-conformities found in the area., The process owner should be present during interviews, but the process owner should refrain from commenting. The 3rd party auditor and the process owner need to know if the person being interviewed was effectively trained and if they can explain the process under the pressure of an audit or FDA inspection. If you are interested in implementing this idea, I recommend using one of two strategies (or both):

  1. Consider having the internal auditor that audited each process shadow the certification body auditor for the processes they audited during their internal audit. This approach will teach your internal auditor what they might have missed, and they will learn what the 3rd party auditors look for to simulate a 3rd party audit more effectively when conducting internal audits.
  2. Consider having the internal auditor that is assigned to conduct the next process audit of each process shadow the certification body auditor for that process. This approach will ensure that any nonconformities observed during the 3rd party audit are checked for the effectiveness of corrective actions during the next internal auditor. Your internal auditor will know precisely how the original nonconformity was identified and the context of the finding.

Auditor shadowing as an effective auditor training technique Read More »

Seven ways to improve quality auditor training

2 Comments / ISO Auditing / By / , ,

A five-day lead auditor course is never enough. Effective quality auditor training must include practical feedback from an expert.

What is required for quality auditor training?

The key to training auditors to audit is consistent follow-up over a long period of time (1-2 years, depending upon the frequency of audits). I recommend following the same training process that accredited auditors must complete. I have adapted that process and developed seven (7) specific recommendations.

Training the trainer

One of my clients asked me to create a training course on how to train operators. I could have taught the operators myself, but so many people needed training that we felt it would be more cost-effective to train the trainers. Usually, I have multiple presentations archived that I can draw upon, but this time I had nothing. I had never trained engineers on how to be trainers before—at least not formally. I thought about the problems other quality managers have had in training internal auditors and how I have helped the auditors improve. The one theme I recognized was that effective quality auditor training needs to include practical feedback from an experienced auditor. An expert auditor that is training new auditors needs to identify systematic ways to provide feedback, and setting a benchmark for the number of times feedback will be provided is really helpful.

Improve by observing yourself and other quality auditors

Observing someone else is a great way to learn when you are learning any new skill. Interns often do this, which is also a technique used to train new auditors. This technique is called shadowing. You can learn by watching, but eventually, you need to try to do tasks that are beyond your comfort level, and it is best to practice auditing with an expert watching you.

Practice team member audit preparation

Many of the internal auditing procedures we see require new auditors to conduct three audits as team members before they can audit independently. In contrast, notified body auditors join as team members for 10-20 audits before they can act as lead auditors. During the training period, auditors in training observe multiple lead auditors and multiple quality systems. Each audit allows auditors in training to write nonconformities and receive feedback from a lead auditor. At the beginning of quality auditor training, the focus must be on audit preparation. What are the areas of importance, what are the results of previous audits, are there any previous audit findings to close, etc? This preparation can even be done as practice for a hypothetical audit.

During quality auditor training, practice the opening and closing meetings

Opening and closing meetings are one of the first things to teach a new lead auditor. Have new lead auditors rehearse their first few opening and closing meetings with you in private before conducting the opening and closing meetings. Ensure the lead auditor has an opening/closing meeting checklist to help them. Recording practice sessions is enormously helpful because the trainee can watch and observe their mistakes. As trainees get more experience, the opening and closing meetings should have time limits. Finally, you might ask members of top management to challenge the lead auditor with questions. The lead auditor needs to be comfortable with their decisions and the grading of the audit findings.

How to practice audit team leadership

Have new lead auditors conduct team audits with another qualified lead auditor for 10-20 audits before you allow them to conduct an audit alone. Leading the opening and closing meetings is usually the first area new lead auditors master. The most complicated area to learn is managing a team of auditors. Team members will fall behind schedule during audits, or someone will forget to audit a process. As a lead auditor, you must complete the audits for your assigned processes and communicate with the entire team to ensure everyone else is on schedule. As an observer, you must let lead auditors make mistakes and help them realize them. Initially, a trainer will encourage new lead auditors to give themselves more than enough time. As their training progresses, the timing needs to be shorter and more challenging. Ultimately, you have to push the team beyond its capability to teach new lead auditors to recognize problem signs and teach them how to fix the problems.

Shadow auditors virtually with recordings

Live shadowing is challenging for experts and trainees because you are distracted by listening to the auditee and observing the auditor. However, if an audit is recorded, the person shadowing can watch the recording. The audit is already completed, and there is little need to concentrate on the auditee. A recording allows the observer to focus on the auditor. If a new auditor is conducting their first audit, an expert should shadow the trainee for 100% of the audit. Gradually the observation can decrease with each subsequent audit.

Practice note-taking with recorded audits

Taking detailed notes is something that experts take for granted, but I learned a lot by watching FDA inspectors take notes during an inspection. Have a new auditor observe a few audits before they are allowed to participate. Make sure they take notes and explain what you are doing and why they are observing as you conduct audits. Review the notes of new auditors periodically throughout the audit to provide suggestions for improvement and identify missing information. You can also record a supplier audit or internal audit and let a new trainee take notes on the pre-recorded webinar. This eliminates the need to coordinate schedules to involve the trainee.

Quality auditor training should include practicing audit agenda creation

Have new lead auditors submit a draft audit agenda to you before sending it to the supplier or department manager. Usually, the first audit agenda will need revision and possibly multiple revisions. Make sure you train the person to include enough detail in the agenda, and using a checklist or template is recommended. The agenda creation will be part of the audit preparation, and it can be done without time pressure.

How do you audit the auditing process?

Most quality managers are experienced and have little trouble planning an audit schedule. The next step is to conduct the audit. The problem is that there is very little objective oversight of the auditing process. The ISO 13485 standard for medical devices requires that “Auditors shall not audit their own work.” Therefore, most companies will opt for one of two solutions for auditing the internal audit process: 1) hire a consultant or 2) ask the Director of Regulatory Affairs to audit the internal auditing process.

Both of the above strategies for auditing the internal audit process meet the requirements of ISO 13485, but neither approach helps to improve an internal auditor’s performance. I have interviewed hundreds of audit program managers over the years, and the most common feedback audit program managers give is “Change the wording of this finding” or “You forgot to close this previous finding.” This type of feedback is related to the report-writing phase of the audit process. I rarely hear program managers explain how they help auditors improve at the other parts of the process.

When auditors are first being trained, we typically provide examples of best practices for audit preparation, checklists, interviewing techniques, AND reports. After auditors are “shadowed” by the audit program manager for an arbitrary three times, the auditors are now miraculously “trained.” Let’s see if I can draw an analogy to make my point.

That kind of sounds like watching your 16-year-old drive the family car three times and then giving them a license.

About the Author

Rob Packard 150x150 Seven ways to improve quality auditor trainingRobert Packard is a regulatory consultant with 25+ years of experience in the medical device, pharmaceutical, and biotechnology industries. He is a graduate of UConn in Chemical Engineering. Robert was a senior manager at several medical device companies—including the President/CEO of a laparoscopic imaging company. His Quality Management System expertise covers all aspects of developing, training, implementing, and maintaining ISO 13485 and ISO 14971 certifications. 2009-2012, he was a lead auditor and instructor for one of the largest Notified Bodies. Robert’s specialty is regulatory submissions for high-risk medical devices, such as implants and drug/device combination products for CE marking applications, Canadian medical device applications, and 510(k) submissions. The most favorite part of his job is training others. He can be reached via phone at 802.258.1881 or by email. You can also follow him on LinkedIn, Twitter, and YouTube.

Seven ways to improve quality auditor training Read More »

CAPA – Corrective/Preventative Action

6 Comments / CAPA / By / ,

What is a CAPA? How do you evaluate the need to open a new CAPA, and who should be assigned to work on it when you do?

What is a CAPA?

“CAPA” is the acronym for corrective action and preventive action. It’s a systematic process for identifying the root cause of quality problems and identifying actions for containment, correction, and corrective action. In the special case of preventive actions, the actions taken prevent quality problems from ever happening, while the corrective actions prevent quality problems from happening again. The US FDA requires a CAPA procedure, and an inadequate CAPA process is the most common reason for FDA 483 inspection observations and warning letters. When I teach courses on the CAPA process, 100% of the people can tell me what the acronym CAPA stands for. If everyone understands what a CAPA is, why is the CAPA process the most common source of FDA 483 inspection observations and auditor nonconformities?

Most of the 483 inspection observations identify one of the following seven problems:

  1. the procedure is inadequate
  2. records are incomplete
  3. actions planned did not include corrections
  4. actions planned did not include corrective actions
  5. actions planned were not taken or delayed
  6. training is inadequate
  7. actions taken were not effective

CAPA Resources – Procedures, Forms, and Training

Medical device companies are required to have a CAPA procedure. Medical Device Academy offers a CAPA procedure for sale as an individual procedure or as part of our turnkey quality systems. Purchase of the procedure includes a form for your CAPA records and a CAPA log for monitoring and measuring the CAPA process effectiveness. You can also purchase our risk-based CAPA webinar, which the turnkey quality system includes.

What’s special about preventive action?

I completed hundreds of audits of CAPA processes over the years. Surprisingly, this seems to be a process with more variation from company to company than almost any other process I review. This also seems to be a significant source of non-conformities. In the ISO 13485 Standard, clauses 8.5.2 (Corrective Action) and 8.5.3 (Preventive Action) have almost identical requirements. Third-party auditors, however, emphasize that these are two separate clauses. I like to refer to certification body auditors as purists. Although certification body auditors acknowledge that companies may implement preventive actions as an extension of corrective action, they also expect to see examples of strictly preventive actions.

You may be confused between corrective actions and preventive actions, but there is an easy way to avoid confusion. Ask yourself one question: “Why did you initiate the CAPA?” If the reason was: 1) a complaint, 2) audit non-conformity, or 3) rejected components—then your actions are corrective. You can always extend your actions to include other products, equipment, or suppliers that were not involved if they triggered the CAPA. However, for a CAPA to be purely preventive in nature, you need to initiate the CAPA before complaints, non-conformities and rejects occur.

How do you evaluate the need to open a CAPA?

If the estimated risk is low and the probability of occurrence is known, then alert limits and action limits can be statistically derived. These quality issues are candidates for continued trend analysis—although the alert or action limits may be modified in response to an investigation. If the trend analysis results in identifying events that require action, then that is the time when a formal CAPA should be opened. No formal CAPA is needed if the trend remains below your alert limit.

If the estimated risk is moderate or the probability of occurrence is unknown, then a formal CAPA should be considered. Ideally, you can establish a baseline for the occurrence and demonstrate that frequency decreases upon implementing corrective actions. If you can demonstrate a significant drop in frequency, this verifies the effectiveness of actions taken. If you need statistics to show a difference, then your actions are not effective.

A quality improvement plan may be more appropriate if the estimated risk is high or multiple causes require multiple corrective actions. Two clauses in the Standard apply. Clause 5.4.2 addresses the planning of changes to the Quality Management System. For example, if you correct problems with your incoming inspection process—this addresses 5.4.2. Clause 7.1 addresses the planning of product realization. For example, if you correct problems with a component specification where the incoming inspection process is not effective, this addresses 7.1. The plan could be longer or shorter Depending on the number of contributing causes and the complexity of implementing solutions. If implementing corrective action takes more than 90 days, you might consider the following approach.

Step 1 – open a CAPA

Step 2 – identify the initiation of a quality plan as one of your corrective actions

Step 3 – close the CAPA when your quality plan is initiated (i.e., – documented and approved)

Step 4 –verify effectiveness by reviewing the progress of the quality plan in management reviews and other meeting forums…you can cross-reference the CAPA with the appropriate management review meeting minutes in your effectiveness section

If the corrective action required is installing and validating new equipment, the CAPA can be closed as soon as a validation plan is created. The effectiveness of the CAPA is verified when the validation protocol is successfully implemented, and a positive conclusion is reached. The same approach also works for implementing software solutions to manage processes better. The basic strategy is to start long-term improvement projects with the CAPA system but monitor the status of these projects outside the CAPA system.

Best practices would be implementing six-sigma projects with formal charters for each long-term improvement project.

NOTE: I recommend closing CAPAs when actions are implemented and tracking the effectiveness checks for CAPAs as a separate quality system metric. If closure takes over 90 days, the CAPA should probably be converted to a Quality Plan. This is NOT intended to be a “workaround” to give companies a way to extend CAPAs that are not making progress on time.

Who should be assigned to work on a CAPA?

Personnel in quality assurance are usually assigned to CAPAs, while managers in other departments are less frequently assigned to CAPAs.  This is a mistake. Each process should have a process owner, who should be assigned to the root cause investigation, develop a CAPA plan, and manage the planned actions. If the manager is not adequately trained, someone from the quality assurance department should use this as an opportunity to conduct on-the-job training to help them with the CAPA–not do the work for them. This will increase the number of people in the company with CAPA competency. This will also ensure that the process owner takes a leadership role in revising and updating procedures and training on the processes that need improvement. Finally, the process will teach the process owner the importance of using monitoring and measuring the process to identify when the process is out of control or needs improvement. The best practice is to establish a CAPA Board to monitor the CAPA process, expedite actions when needed, and ensure that adequate resources are made available.

What is a root cause investigation?

If you are investigating the root cause of a complaint, people will sample additional records to estimate the frequency of the quality issue. I describe this as investigating the depth of a problem. The FDA emphasizes the need to review other product lines, or processes, to determine if a similar problem exists. I describe this as investigating the breadth of a problem. Most companies describe actions taken on other product lines and/or processes as “preventive actions.” This is not always accurate. If a problem is found elsewhere, actions taken are corrective. If potential problems are found elsewhere, actions taken are preventive. You could have both types of actions, but most people incorrectly identify corrective actions as preventive actions.

Another common mistake is to characterize corrections as corrective actions.

The most striking difference between companies seems to be the number of CAPAs they initiate. There are many reasons, but the primary reason is the failure to use a risk-based approach to CAPAs. Not every quality issue should result in the initiation of a formal CAPA. The first step is to investigate the root cause of a quality issue. The FDA requires that the root cause investigation is documented, but if you already have an open CAPA for the same root cause…DO NOT OPEN A NEW CAPA!!!

What should you do if you do not have a CAPA open for the root cause you identify?

The image below gives you my basic philosophy.

death by capa CAPA Corrective/Preventative ActionMost CAPA investigations document the estimated probability of occurrence of a quality issue. This is only half of the necessary risk analysis I describe below. Another aspect of an investigation is documenting the severity of potential harm resulting from the quality issue. If a quality issue affects customer satisfaction, safety, or efficacy, the severity is significant. Risk is the product of severity and probability of occurrence.

How much detail is needed in your CAPAs?

One of the most common reasons for an FDA 483 inspection observation related to CAPAs is the lack of detail. You may be doing all the planned tasks but must document your activity. Investigations will often include a lot of detail identifying how the root cause was identified, but you need an equal level of detail for planned containment, corrections, corrective actions, and effectiveness checks. Who is responsible, when will it be completed, how will it be done, what will the records be, and how will you monitor progress? Make sure you include copies of records in the CAPA file as well because this eliminates the need for inspectors and auditors to request additional records that are related to the CAPA. Ideally, the person reviewing the CAPA file will not need to request any additional records. For example, a copy of the revised process procedure, a copy of training records, and a copy of graphed metrics for the process are frequently missing from a CAPA file, but auditors will request this information to verify all actions were completed and that the CAPA is effective.

What is the difference between corrections and corrective actions?

Every nonconformity identified in the original finding requires correction. By reviewing records, FDA inspectors and auditors will verify that each correction was completed. In addition, several new nonconformities may be identified during the investigation of the root cause. Corrections must be documented for the newly found nonconformities as well. Corrective actions are actions you take to prevent new nonconformities from occurring. Examples of the most common corrective actions include: revising procedures, revising forms, retraining personnel, and creating new process metrics to monitor and measure the effectiveness of a process. Firing someone who did not follow a procedure is not a corrective action. Better employee recruiting, onboarding, and management oversight should prevent employees from making serious mistakes. The goal is to have a near-perfect process that identifies human error rather than a near-perfect employee that has to compensate for weak processes.

Implementing timely corrective actions

Every correction and corrective action in your CAPA plan should include a target completion date, and a specific person should be assigned to each task. Once your plan is approved, you need a mechanism for monitoring the on-time completion of each task. There should be top management or a CAPA board this is responsible for reviewing and expediting CAPAs. If CAPAs are being completed on-schedule, regular meetings are short. If CAPAs are behind schedule, management or the CAPA board needs authority and responsibility to expedite actions and make additional resources available when needed. Identifying lead and lag metrics is essential to manage the CAPA process successfully–and all other quality system processes.

What is an effectiveness check?

Implementation of actions and effectiveness of actions is frequently confused. An action was implemented when the action you planned was completed. Usually, this is documented with the approval of revised documents and training records. The effectiveness of actions is more challenging to demonstrate, and therefore it is critical to identify lead and lag metrics for each process. The lead metrics are metrics that measure the routine activities that are necessary for a process, while the lag metrics measure the results of activities. For example, monitoring the frequency of cleaning in a controlled environment is a lead metric, while monitoring the bioburden and particulates is a lag metric. Therefore, effectiveness checks should be quantitative whenever possible. Your effectiveness is weak if you need to use statistics to show a statistical difference before and after implementing your CAPA plan. If a graph of the process metrics is noticeably improved after implementing your CAPA plan, then the effectiveness is strong.

CAPA – Corrective/Preventative Action Read More »

How do you demonstrate training competence?

7 Comments / Education, ISO 14971:2019 (Risk Management) / By / ,

Anyone can sign and date training records, but how do you demonstrate the effectiveness of training and competence?
%name How do you demonstrate training competence?

What are the requirements for training?

The requirements for training are found in ISO 13485:2016, Clause 6.2 (i.e., Human Resources). Auditors and inspectors usually only ask for training records, but the requirement for records is the last item in the clause (i.e., “6.2(e) – maintain appropriate records of education, training, skills, and experience (see 4.2.5).”). The first four items in Clause 6.2 all include one word, but it’s not “records.” The most critical word in the requirements is “competence.”

What is the difference between training requirements, training records, training effectiveness, and training competence?

  • Training requirements define the “appropriate education, training, skills, and experience.” Specifically, what degree(s) is required, if any? What training curriculum needs to be completed? What skills are needed to perform the work? And how many years of experience are needed?
  • Training records are documented evidence that all training requirements have been met. Records may include a job application, resume, individual training records, group training records, or a training certificate. Records may also include the results of any quizzes to demonstrate training effectiveness and any evaluation of training competence.
  • Training effectiveness is how well a person understands the information communicated during training. Using terms from human factors engineering (i.e., the PCA Process), the person must have the correct perception and cognition. Cognition can be evaluated by giving people quizzes or asking them questions. Written and verbal exams may also include pictures and/or video. Performing these types of tasks on a quiz or exam are “knowledge tasks.”
  • Training competence is possessing the skill or ability to perform tasks. To demonstrate competence, someone that is already competent (i.e., a subject matter expert) must observe the trainee performing tasks. A person that once possessed the skill or ability can also judge competency, and a person that does not possess the skill or ability can be trained to evaluate skill and ability (e.g., an Olympic Judge or referee). Skills and abilities can be physical or cognitive, but there are also social skills. Examples of each are provided in the table below:

List of abilities and skills 1024x495 How do you demonstrate training competence?

What keeps me awake at night? (a story from April 2, 2011)

I am in Canada, it’s almost midnight, and I can’t sleep. I’m here to teach a Canadian client about ISO 14971—the ISO Standard for Risk Management of medical devices. Most companies requesting this training are doing so for one of two reasons: 1) some design engineers have no risk management training, or 2) the engineers have previous training on risk management, but the training is out-of-date. This Canadian company falls into the second category, and engineers with previous training ask the most challenging questions. This group of engineers forced me to re-read the Standard several times and reflect on the nuances of almost every single phrase. While teaching this risk management course, I learned more about risk management than I ever knew.

The four levels of the Learning Pyramid

The image at the beginning of this blog is a learning model that explains my experiences training Canadian design engineers. I call this model the “Learning Pyramid.” At the base of the pyramid, there are “Newbies.”

This is the first of four levels. At the base, students read policies and procedures, hoping to understand.

The student is now asked to watch someone else demonstrate proper procedures in the second level of the pyramid. One of my former colleagues has a saying that explains the purpose of this process well, “A picture tells a thousand words, but a demonstration is like a thousand pictures.” Our children call this “sharing time,” but everyone over 50 remembers this as “show and tell.”

The student is now asked to perform their tasks in the third level of the pyramid. This is described as “doing,” but in my auditing courses, I refer to this process as “shadowing.” Trainees will first read the procedures for Internal Auditing (level 1). Next, trainees will shadow the trainer during an audit to demonstrate the proper technique (level 2). During subsequent audits, the trainees will audit, and the trainer will shadow the trainee (level 3). During this “doing” phase, the trainer must watch, listen, and wait for what I call the “teachable moment.” This is a moment when the trainee makes a mistake, and you can use this mistake as an opportunity to demonstrate a complex subject.

Finally, in the fourth level of the Learning Pyramid, we now allow the trainee to become a trainer. This is where I am at. I am an instructor, but I am still learning. I am learning what I don’t know.

Teaching forces you back to the bottom of the Learning Pyramid

After teaching, the next step in the learning process is to return to the first level. I re-read the Standards and procedures until I understood the nuances I was unaware of. Then, I search for examples in the real world that demonstrate the complex concepts I am learning. After searching for examples, I tested my knowledge by applying the newly acquired knowledge to an FDA 510(k) or De Novo submission for a medical device client. Finally, I prepared to teach again. This reiterative process reminds me of the game Chutes and Ladders, but one key difference is that we never really reach the level of “Guru.” We continue to improve but never reach our goal of perfection.

Where is training competence in the Learning Pyramid?

Most people feel that a person is competent in a position when they can consistently perform a task. However, the ISO 13485 standard uses the phrase “necessary competence” to suggest that competency for any given task might not be the same for all people. For example, in some cases, it may be sufficient to perform a task with written instructions in front of you, while an instructor might be expected to perform the task without written instructions. The speed at which a person performs a task might also differ. For example, a secretary might be required to touch type at a speed of 60+ wpm with a QWERTY keyboard, while a stenographer must be able to use a STENO keyboard and write at 225 wpm. The accuracy requirements may differ for those two positions as well. Therefore, your company may decide that the training competence requirement for a design engineer is that they can pass an exam on risk management. However, the training competency requirement for the design project lead or Engineering Manager might include teaching inexperienced design engineers to apply the basic risk management principles. Demonstrating the ability to teach inexperienced design engineers might be demonstrated by an auditor interviewing members of your design team.

How do you demonstrate training competence? Read More »

Audit schedule and an audit agenda, what’s the difference?

8 Comments / ISO Auditing / By / , ,

Internal audit and supplier audit programs both require an audit schedule and an audit agenda, but what’s the difference between them?

What is an audit schedule?

An “audit schedule” is not a formal definition in ISO 19011:2018. However, section 5.1 of that standard states that your audit program should include nine different requirements. Item “d” is “d) schedule (number/duration/frequency) of the audits.” Typically, the audit program manager will maintain an annual audit schedule with a date indicating the date the schedule was last revised. The most common example in lead auditor training is a matrix like the one shown below. The left-hand column will list all of the individual processes that are identified in the company’s process interaction diagram, and the top row of the matrix will indicate the month when each process audit will be conducted. Typically, the expectation is to complete the audit sometime during that month, but some quality auditing procedures specify that the audit may be completed the month before or the month after to accommodate the process owner. The regulations only require that you document and maintain an audit schedule, and the standard is only considered guidance.

%name Audit schedule and an audit agenda, what’s the difference?
Example of an Internal Audit Schedule

I use a slide in lead auditor courses that gives the example of an internal auditing schedule provided above. On the surface, this example seems like a good audit schedule. Twelve auditors perform two audits each year. If each audit requires approximately two days, each auditor spends less than two percent of their work year auditing. Unfortunately, a two-percent allocation of time is insufficient to become or remain proficient at auditing. An improvement to the auditing schedule would be to assign fewer auditors so each auditor gets more experience. There is no perfect number, but assigning a few specialists will improve the chances of becoming and remaining proficient at auditing.

What is an audit agenda?

An “audit agenda” is not a formal definition in ISO 19011:2018 either. In fact, the word “agenda” is not even used in ISO 19011. Instead, section 5.5.5 of ISO 19011 states that “The assignment [of the individual audit] should be made in sufficient time before the scheduled date of the audit, in order to ensure the effective planning of the audit.” The audit plan must also be part of the records [i.e., Clause 5.5.7(b)]. Therefore, “agenda” and “plan” may be used interchangeably. Details of audit planning are provided in 6.3.2 of ISO 19011.

6 Steps to Creating an Audit Schedule

There are six steps to creating an audit schedule:

  1. What were the results of previous audits?
  2. Which processes are the most important to audit?
  3. Who should conduct your internal audit?
  4. How long should your internal audit be?
  5. Should you conduct one full quality system audit or several audits?
  6. Is a remote audit good enough?

We will address each of the six steps below.

How do the results of previous audits impact your audit schedule?

The results of an audit include nonconformities, observations for improvement (OFI), and a conclusion regarding whether the quality system is effective or not. Usually, most processes are effective, and there are no nonconformities or OFIs. Therefore, any processes that had a nonconformity or OFI should be prioritized in the audit schedule and audit planning for the future. For internal and supplier audits, a best practice is for the auditor and the process owner to discuss the corrective actions planned and determine the appropriate timeline or implementation of actions planned. Then the auditor can indicate a timeframe for re-auditing the nonconforming process after corrective actions are implemented. This strategy allows the auditor to be part of the effectiveness check. This approach is appropriate for individual process audits but not for a full-quality system audit.

Which processes are the most important to audit?

The primary element impacting the importance of processes is the risk to product quality associated with the process. Usually, support processes are of lower importance because they do not directly impact product quality. In contrast, core processes directly involved in a device’s design, manufacture, and distribution are critical. Most auditors and audit program managers emphasize design controls and production process controls as important areas to audit. However, the distribution area is often neglected. Other core processes are purchasing, sales, customer service, and servicing. Not every process is equally important when comparing two companies. For example, device manufacturers that only make software as a medical device (SaMD) often have very limited purchasing and incoming inspection activities to audit.

Who should the audit program manager assign to each internal audit?

%name Audit schedule and an audit agenda, what’s the difference?The example of a revised audit schedule provided above identifies the departments where each of the auditors works with color coding. This is done to ensure that auditors are not assigned to audit processes where they might have a conflict of interest (i.e., they would be auditing their own work). This is the most important aspect of assigning auditors. The second most important aspect is to make sure the auditor has the technical knowledge to audit the process. It is challenging to conduct an audit of manufacturing if you have not spent any time in manufacturing before. If auditors are new and their training is in progress, then the audit program manager may assign the auditor to a process specifically to give them more experience with that type of process. Inexperienced auditors often are assigned less important processes that have not changed recently. However, a better approach to training auditors is to give them a challenge with support. Having the new auditor prepare a detailed sampling plan and list of questions before the audit can prepare them for auditing a more challenging, important process that is likely to have one or more nonconformities. Auditing processes that have nonconformities is also the best way to teach a new auditor how to write the audit findings.

What should be the duration of each internal audit in your schedule?

The duration of an audit should be based on the results of previous audits, but other important factors include: 1) the number of personnel involved in the process, 2) the complexity of the process, and 3) the risk to product quality associated with the process. The MDSAP program uses a procedure for audit time determination (i.e., MDSAP AU P0008.007: Audit Time Determination Procedure), and the MDSAP audit approach document (i.e., MDSAP AU P0002.008 Audit Approach) classifies processes as having either a “direct” or “indirect” impact upon product quality based upon the applicable clause of the process (i.e., Clauses 0-6.3 are indirect, and Clauses 6.4-8.5.3 are direct). For example, the production processes and design and development processes both involve a large number of people in most organizations, the processes are complex, and both processes directly impact product quality. Therefore, I typically allocate 3-4 hours to each of those processes during an audit. In comparison, incoming inspection often involves one or two people, and the process often involves only one procedure. Incoming inspection is a “direct” process, but less time (e.g., 1 hour) should be allocated to auditing incoming inspection than the other two processes–unless there was a nonconformity in the incoming inspection process during a recent audit or unless the process was recently changed.

Should you conduct one full quality system audit or several audits?

Both approaches have strengths and weaknesses, but there is not a single best way. If I am using employees to conduct an audit, then I typically restrict the scope of the audit to a single process. Alternatively, when I use a consultant to conduct an audit, I typically conduct a full-quality system audit to minimize travel costs. Another strategy I have recommended is to identify the processes that are most important to audit first (e.g., processes with recent changes and/or nonconformities), and these processes are scheduled for individual process audits during the first half of the audit schedule. Then I schedule a full-quality system audit in the second half of the audit schedule. The strategy ensures that all important processes will be audited twice in one year, but every process will be audited at least once.

Remote audits vs On-site audits

Prior to the Covid-19 pandemic, remote audits were rare in the medical device industry. Many NBs insisted that remote audits were not permitted or effective. The pandemic forced the entire industry to create policies for remote auditing and to use remote auditing whenever possible. Now that the pandemic has ended, many companies continue to conduct remote audits to save money. Even NBs are conducting more remote audits for Stage 1 readiness audits during the ISO 13485 certification process. ISO 19011 has a section in the Appendices outlining the differences between remote and on-site audits. However, there is a minimal advantage to conducting an on-site audit of a process where the auditor is expected to spend all of their time in a conference room during the audit. If the audit is going to be done in a conference room, then why not conduct it remotely? The one exception is when most records are paper-based and unavailable electronically. Alternatively, an on-site audit is generally more effective if the process involves observing inspection activities or assembly operations. Remote audits of inspection activities and assembly operations should be reserved for re-auditing or when a process has been audited on-site in the past, but an on-site audit would still be more effective for those processes.

How many times should a process be audited annually?

Many notified bodies will expect companies to audit all processes at least once during the year. However, it doesn’t expressly state this as a requirement in the regulations, and some companies justify skipping processes that are functioning well and have not changed in the past year. Our team is seeing this more frequently as the number of lead auditors worldwide has become scarce due to the requirements of MDSAP, the MDR/IVDR implementation, and unannounced audits. However, I almost never see the opposite justification (i.e., auditing a process more than once a year). If a process has been changed significantly, or there were nonconformities, then re-auditing the process may be used to verify the effectiveness of corrective actions or to verify that personnel are compliant with the revised process.

How to take advantage of the process approach to auditing

Another improvement that can be made to the revised example of an audit schedule is to use the process approach to auditing. Instead of performing an independent document control and training audit, these two clauses/procedures can be incorporated into every audit. The same is true of maintenance and calibration support processes. Wherever maintenance and calibration are relevant, these clauses should be investigated as part of auditing that area. For example, when the incoming inspection process is audited, it makes sense to look for evidence of calibration for any devices used to perform measurements in that area. When production process controls are being audited, maintenance records of production equipment should also be sampled.

If the concept of process auditing is fully implemented, the following ISO 13485 clauses can easily be audited in the regular course of reviewing other processes:

  • 4.2.1), Quality System Documentation;
  • 4.2.3), Document Control;
  • 4.2.4), Record Control;
  • 5.3), Quality Policy;
  • 5.4.1), Quality Objectives;
  • 6.2.2), Training;
  • 6.3), Maintenance;
  • 6.4), Work Environment;
  • 7.1), Planning of Product Realization & Risk Management
  • 7.6), Calibration;
  • 8.2.3), Monitoring & Measurement of Processes
  • 8.5.2), Corrective Action; and
  • 8.5.3) Preventive Action.

This strategy reduces the number of process audits needed by more than half.

Internal Auditing: Upstream/Downstream Examples

Another way to embrace the process approach to auditing is to assign auditors to upstream or downstream processes in the product realization process from their own area. For example, Manufacturing can audit Customer Service to understand better how customer requirements are confirmed during the order confirmation process. This is an example of auditing upstream because Manufacturing receives the orders from Customer Service—often indirectly through an MRP system. Using this approach allows someone from Manufacturing to identify opportunities for miscommunication between the two departments. If Regulatory Affairs audits the engineering process, this is an example of auditing downstream. Regulatory Affairs is often defining the requirements for the Technical Files and Design History Files that Engineering creates. If someone from Regulatory Affairs audits these processes, the auditor will realize what aspects of technical documentation are poorly understood by Engineering and quickly identify retraining opportunities.

Audit schedule and an audit agenda, what’s the difference? Read More »

FDA Pre-Submission Format and Content Requirements

5 Comments / PreSTAR / By

The format and content requirements for an FDA pre-submission have not changed, but the launch of the FDA PreSTAR has changed everything.

What is an FDA pre-submission?

An FDA pre-submission aims to get answers to questions you have about a future FDA submission. The pre-submission may consist of one large PDF document or multiple PDF documents. In your pre-submission, you must select either an email response or an email response with a teleconference. One advantage of choosing a teleconference is that you can ask clarifying questions during a one-hour teleconference with the FDA. Still, you are responsible for submitting draft meeting minutes to the FDA within 15 days of the teleconference. If you select an email response, you do not need to provide meeting minutes to the FDA.

Our new 4-part FDA pre-submission webinar series is available on-demand. You can download it and watch it as many times as you want. This will be the Ultimate FDA pre-submission training. Do not miss it.

Register Now for 299 300x129 FDA Pre Submission Format and Content Requirements

Everyone asks us for examples, so we will show you how to complete the entire FDA PreSTAR for a device in this webinar series. If you would like to vote on which device we should use as an example (i.e., Option 1 = Infrared Thermometer or Option 2 = Antimicrobial Gauze), please place your vote on our LinkedIn page.

What is the difference between an FDA pre-submission and a Q-submission?

Every FDA pre-submission is a Q-submission, but not all Q-submissions are pre-submissions. The new PreSTAR template is currently limited to an FDA pre-submission, but the template will be expanded to other types of Q-subs later. The FDA pre-submission template (i.e., PreSTAR) beta version 0.1 is unnecessary for responses to interactive review questions from the FDA. Just email the Lead Reviewer (file size limit is 25 MB for email).

No SIRs 1024x342 FDA Pre Submission Format and Content Requirements

Unfortunately, the beta version 0.1 is also not ready for Submission-in-Review (SIR) meetings or responses to IDE during an interactive review.No SIRs or IDEs 1024x390 FDA Pre Submission Format and Content Requirements

13 other types of submissions might benefit from Q-submissions:

  1. Submission Issue Requests (SIRs)
  2. Study Risk Determinations
  3. Informational Meetings
  4. Breakthrough Device Designation Requests
  5. Informational Meetings
  6. PMA Day 100 meetings
  7. Agreement and Determination meetings
  8. Submissions associated with the STeP program
  9. Accessory classification requests
  10. Requests for FDA feedback on specific questions or cross-cutting policy matters
  11. Requests for recognition of publicly accessible genetic variant databases
  12. Combination product agreement meetings (CPAM), and
  13. Feedback on FDA 483 inspection observations.

We expect the PreSTAR template to eventually be available for a 513(g) request in the future because it was already validated for that purpose.

What is the Q-submission number?

All Q-submissions are assigned a document number beginning with “Q” upon receipt (i.e., Qyyxxxx). The format of the number consists of 2-digits (i.e., “yy”) for the year of submission (e.g., “23” for 2023) and 4-digits (i.e., “xxxx”) that are the following sequential number assigned by the FDA for that calendar year. Therefore, the first Q-submission received by the FDA in January 2023 is Q230001, and between 3,500 and 4,000 new submissions are usually received each year. If the subject device was submitted in a previous Q-submission, the original document number is re-used, and a supplement number is added (i.e., Qyyxxx/S001, Qyyxxx/S002, etc.). Q-submission numbering is explained in more detail in the 2023 FDA guidance.

Does the FDA charge for Q-submissions?

FDA pre-submissions do not require paying an FDA User Fee (i.e., $0).

How long does an FDA pre-submission take?

The days of squishing timelines are gone. The timeline is 70-75 calendar days. On October 5, 2022, MDUFA V was approved. As one of the MDUFA V decision targets, the FDA is tasked with reducing the timeline for responding to pre-sub questions within 70 days for 90% of pre-sub requests. The FDA is tasked with achieving this goal by March 2024. If they are successful, the FDA will receive an increase of 59 headcounts to their budget in 2024. This is approximately a $19 million incentive to respond to your pre-submission meeting questions within 70 days. To reflect these new MDUFA V decision targets, the FDA updated the Q-Sub guidance document to reflect the target date of 70 days for the email response and 75 days for teleconference meetings. The FDA also updated the Customer Collaboration Portal (CCP) to facilitate tracking of FDA pre-submission deadlines.

What is an FDA PreSTAR?

In the past, you had to create your document(s) for an FDA pre-submission. Some people create one large PDF document divided into sections, while others create separate PDF documents for each requirement of the FDA pre-submission guidance. On August 14, 2023, the FDA released an updated beta version (i.e., version 0.2) of a new PDF template (i.e., FDA PreSTAR). This new PreSTAR template provides multiple benefits to the FDA:

  1. every company uses the same format,
  2. the template automatically verifies that the pre-submission includes all required elements, and
  3. Including optional elements will encourage companies to provide more device details than they might otherwise provide.

The PreSTAR also benefits submitters:

  1. you will never forget the required elements of the FDA pre-submission,
  2. you never have to validate an FDA eCopy, and
  3. the similar format and user interface will train you to use the FDA eSTAR.

Note: October 1, 2023, was the FDA eSTAR implementation deadline. 

Do you have to use the PreSTAR template?

Nope. The PreSTAR version 0.2 is a beta version and 100% optional. However, I like it better than my templates. Your design team can still have individual documents for the user manual, device description, and testing plan. We attach the document using the button that says “Add Attachment” (see screen capture below).

Device Description screen capture for prestar 1024x576 FDA Pre Submission Format and Content Requirements

The PreSTAR template was built by Patrick Macatangga, a Tools & Templates Engineer working at the FDA on the Lifecycle Tools and Templates Team. To help with where to direct questions about the template, he suggested:

  • If you have questions or feedback regarding the voluntary use of the eSTAR for medical devices regulated by CDRH, or if you have general questions about medical devices, contact the Division of Industry and Consumer Education (DICE).
  • If you find malfunctions or errors in the eSTAR template for medical devices regulated by CDRH, contact eSubPilot@fda.hhs.gov.
  • If you have questions regarding 510(k)s, De Novo requests, or Early Submission Requests for medical devices regulated by CDRH, contact OPEQSubmissionSupport@fda.hhs.gov.

How do you submit an eCopy?

You can submit an FDA eCopy on electronic media (e.g., USB flash drive) and send it via FedEx to the FDA Document Center at the following address: Food and Drug Administration, Center for Devices and Radiological Health, Document Mail Center, 10903 New Hampshire Ave., Bldg. 66, rm. G609, Silver Spring, MD 20993-0002. However, you can also submit an FDA eCopy via a web browser (i.e., CCP…see next section on submitting a PreSTAR).

If you are submitting an eCopy through the CCP instead of an FDA PreSTAR

How do you submit a PreSTAR?

You have two options for delivery of an FDA pre-submission:

  1. save the pre-sub on electronic media (e.g., USB flash drive) and send it via FedEx to the FDA, and
  2. upload the pre-sub to the new FDA Customer Collaboration Portal (CCP).

As you can guess from the video above, we only use option 2 for FDA pre-submissions. For option 2, you can upload an eCopy (saved as a zip file) or a PreSTAR (in the native PDF format). The image below shows you how this is done, but the uploading process usually takes about one minute–depending on your file size and bandwidth. You can register for your own CCP account in seconds.

estar and eCopy upload 1024x485 FDA Pre Submission Format and Content Requirements

What is the pre-submission process?

Preparing and uploading your FDA pre-submission meeting request is only the first step of the process. You will receive an automated email confirming that your pre-submission was successfully uploaded. Then, you will receive an automated letter via email that gives you the assigned Q-sub number. You will also receive an automated email notifying you that the pre-submission was accepted, or the FDA reviewer will contact you if changes are needed. The FDA reviewer assigned will usually contact you by email within the first three weeks to schedule a teleconference if you request one. Still, the date/time offered usually does not match the availability of the FDA team, and alternate dates/times may be offered.

You will receive an email response from the FDA for each of your questions within 70 days of receipt by the FDA. If you requested a teleconference, it would typically be about 75 after receipt of the FDA pre-submission meeting request. Your team needs to prepare a detailed discussion plan for the one-hour teleconference. A slide deck is highly recommended to facilitate communication but is not required. If you provide a slide deck, you should email it to the reviewer before the meeting. You must also provide a copy of the slide deck with your meeting minutes. At the beginning of the teleconference, someone from your team must commit to submitting draft meeting minutes to the FDA within 15 days. The FDA will reply with acceptance of your meeting minutes, or they will provide an edited version. It is also common to submit a supplement FDA pre-submission with detailed protocols and new questions for the FDA.

Reuqest a call with Lindsey 1024x240 FDA Pre Submission Format and Content Requirements

FDA Pre-Submission Format and Content Requirements Read More »

Iterative design is real, waterfalls are illusions

2 Comments / Design Control / By

The Waterfall Diagram was copied by the FDA from Health Canada and ISO 9001:1994, but everyone actually uses an iterative design process.

Iterative Design – What is it?

The FDA first mandated that medical device manufacturers implement design controls in 1996. Unfortunately, in 1996 the design process was described as a linear process. In reality, the development of almost every product, especially medical devices, involves an iterative design process. The V-diagram from IEC 62304 is closer to the real design control process, but even that process is oversimplified.

Software Validation and Verification 1 Iterative design is real, waterfalls are illusions

What is the design control process?

The design control process is the collection of methods used by a team of people and companies to ensure that a new medical device will meet the requirements of customers, regulators, recognized standards, and stakeholders. With so many required inputs, it is highly unlikely that a new medical device could ever be developed in a linear process. The design control process must also integrate risk management and human factors disciplines. ISO 14971:2019, the international risk management standard, requires conducting optional control analysis. Option control analysis requires evaluating multiple risk control options and selecting the best combination of risk controls for implementation. The human factors process involves formative testing where you evaluate different solutions to user interfaces, directions for use, and training. This always requires multiple revisions before the user specifications are ready to be validated in summative usability testing. Process success is verified by conducting verification and validation testing. The process ends when the team agrees that all design transfer activities are completed, and your regulatory approval is received.

waterfall fda Iterative design is real, waterfalls are illusions

Where did design controls come from?

The diagram above is called the “Application of Design Controls to Waterfall Design Process.” The FDA introduced this diagram in 1997 in the design controls guidance document. However, the original source of the diagram was Health Canada.

This diagram is one of the first slides I use for every design control course that I teach because the diagram visually displays the design control process. The design controls process, defined by Health Canada and the US FDA, is equivalent to the design and development section found in ISO 13485 and ISO 9001 (i.e., – Clause 7.3). Seven sub-clauses comprise the requirements of these ISO Standards:

  • 7.3.1 – Design Planning
  • 7.3.2 – Design Inputs
  • 7.3.3 – Design Outputs
  • 7.3.4 – Design Reviews
  • 7.3.5 – Design Verification
  • 7.3.6 – Design Validation
  • 7.3.7 – Design Changes

In addition to the seven sub-clauses found in these ISO Standards, the FDA Quality System Regulation (QSR) also includes additional requirements in the following sub-sections of 21 CFR 820.30: a) general, h) design transfer, and J) Design History File (DHF). If you need a procedure(s) to comply with design controls, we offer two procedures:

  1. Design Controls (SYS-008)
  2. Change Control (SYS-006)

The change control process was separated from the design controls process because it is specific to changes that occur after a device is released to the market. We also have a training on Change Control.

Free Download – Overview of the Design & Development Process

What are the phases of the Design Control Process?

Normally, we finish the design control process by launching your device in the USA because this is when you should close your Design History File (DHF). However, if you are going to expand to different markets, there is a specific order we recommend. We recommend the US market first because no quality system certification is required, and the FDA 510(k) process is easier than the CE Marking process due to the implementation of the MDR and the IVDR. The Canadian market is the second market we recommend because the Canadian Device License Application process for Health Canada is even easier than the 510(k) process for Class II devices. The Canadian market is not recommended as the first country to launch because Health Canada requires MDSAP certification for your quality system, and the Canadian market is 10% of the US market size. The European market should probably be your last market because of the high cost and long timeline for obtaining CE Marking. Each of the phases of design and development is outlined in the first column of our free download, “Overview of Regulatory Process, Medical Device Development & Quality System Planning for Start-ups.”

Which regulatory filings are required during each phase of design and development?

The second column of our free download lists the regulatory filings required by the FDA for each phase. Generally, we see companies waiting too long to have their first pre-submission meeting with the FDA or skipping the pre-submission meeting altogether. This is a strategic mistake. Pre-submission meetings are free to submit to the FDA, and the purpose of the meeting is to answer your questions. Even if you are 100% confident of the regulatory pathway, you know precisely which predicate you plan to use, and you know which verification tests you need to complete, you still have intelligent questions that you can ask the FDA. Critical questions fall into three categories: 1) selection of your test articles, 2) sample size justification, and 3) acceptance criteria. Even if you don’t have a complete testing protocol prepared for the FDA to review, you can propose a rationale for your test article (e.g., the smallest size in your product family). You can also provide a paragraph explaining the statistical justification for your sample size. You might also present a paragraph explaining the data analysis method you plan to use.

The following example illustrates how discussing the details of your testing plan with the FDA can help you avoid requests for additional information and retesting. Many of the surgical mask companies that submitted devices during the Covid-19 pandemic found that the FDA had changed the sample size requirements, and they were now requiring three non-consecutive lots with a 4% AQL sample size calculation. If the company made a lot of 50,000 masks, they would be forced to sample a large number of masks, while a lot size of 250 masks allowed the company to sample the minimum sample size of 32 masks.

If the regulatory pathway for your device is unclear, you might start with the submission of a 513(g) submission during the first phase of design and development. After you have written confirmation of the correct regulatory pathway from the FDA, you can submit a pre-submission meeting request to the FDA. If the pathway for your device is a De Novo Classification Request, you might have a preliminary pre-submission meeting to get an agreement with the FDA regarding which recognized standards should be applied as Special Controls for your device. While waiting 70+ days for your pre-submission meeting with the FDA, you can obtain quotes from testing labs and prepare draft testing protocols. After the presubmission meeting, you can submit a pre-submission supplement that includes detailed testing protocols–including your rationale for the selection of test articles, sample size justification, and the acceptance criteria.

What quality assurance documentation is required during each phase?

In addition to testing reports for your verification and validation testing, you will find many other supporting documents you also need to prepare. We refer to these supporting documents generically as “quality assurance documentation” because the documents verify that you meet specific customer and regulatory requirements. However, the documents should be prepared by the person or people responsible for that portion of your design project. For example, every device will need a user manual and draft labeling. Even though you will need someone from quality to complete a regulatory checklist to ensure that all of the required symbols and general label content are included, you will also need an electrical engineer to prepare sections of the manual with EMC labeling requirements from the FDA’s EMC guidance document. In your submission’s non-clinical performance testing section, you must include human factors documentation in addition to your summative usability testing report. For example, you will need a use specification, results of your systematic search of adverse events for use errors, a task analysis, and a Use-Related Risk Analysis (URRA). Software and cybersecurity documentation includes several documents beyond the testing reports as well.

Which procedures do you need to implement during each development phase?

The last column of our free download lists the procedures we recommend implementing during each phase of the design process. In Canada and Europe, you must complete the implementation of your entire quality system before submitting a Canadian License Application or CE Marking application. In the USA, however, you can finish implementing your quality system during the FDA review of your 510k submission or even after 510k clearance is received. The quality system requirement is that your quality system is fully implemented when you register your establishment with the FDA and begin distribution.

Iterative design is real, waterfalls are illusions Read More »

FDA US Agent – What do they do?

3 Comments / FDA, Registration / By

Medical device companies exporting devices into the USA must have a US agent to register, but what does an FDA US agent do?

What does an FDA US agent do?

Every medical device company outside the USA that distributes devices in the USA must have an FDA US agent. This includes manufacturers, contract manufacturers, and specifications developers outside the USA. The US agent assists the FDA in communication with the device company. The most common communications concern questions about devices exported to the US and scheduling FDA inspections. The role of the US agent is very similar to a European Authorized Representative, a UK Responsible Person, or a Swiss Authorised Representative. Unlike an EC Representative, you do not include US agents in your device labeling. The US agent’s name and contact information only appear on your FDA Establishment Registration record on the FDA website. 

Is there any certification or contract required for a US agent?

FDA US agents have no certification process, but you should have a formal signed agreement or contract with your agent. I have never seen the FDA request a copy of the contract or a letter from a US agent or the company that is registered. However, since the agent has a legal role and responsibility, you should ensure an agreement or contract is in place. The agreement or contract should include the following elements:

  • Scope of service
  • Commitment to perform US agent services promptly
  • Duration of service (i.e., specific start and end dates)
  • Termination provisions
  • Consulting Fees for US agent services (typically an annual fee ranging from $250-$1,500)
  • Any additional consulting fees if the FDA contacts your agent
  • Who is responsible for payment of FDA User Fees ($7,653 for FY 2024 FDA User Fee)
  • Commitment to communicating complaints, especially for potential risks to public health, serious injuries, or death, directly to your company
  • Confidentiality clause or reference to a separate confidentiality agreement (Note: The agent may be compelled to disclose information they have to the FDA, but they should notify your company first if this happens.)
  • Non-solicitation of your customers or suppliers and no solicitation of employees
  • Force Majeure clause
  • Identification of the agent’s name, address, phone, and email
  • Identification of the company name, address, phone, DUNs Number
  • Identification of the company contact’s name, title, address, phone, and email
  • Identification of who will be the “Official Correspondent” in the FDA Registration Database
  • Signature and Date

The US Agent is not required to be a legal entity, but you will need to enter a “Company Name.” There is no place to enter an EIN, and DUNS number is optional. Here’s a screen capture of the account creation form below.

FURLS Account Set up 1024x811 FDA US Agent What do they do?

You should also consider adding your agent to your Approved Supplier List (i.e., LST-003). If you do not already have a procedure for Supplier Quality Management (i.e., SYS-011), Medical Device Academy has a procedure available for purchase that includes a template for review and approval of new suppliers (i.e., FRM-005) and a template for an Approved Supplier List (i.e., LST-003). The FDA US agent doesn’t need a quality system, but they should be able to demonstrate competency in US FDA device regulations with their resume and/or training records. Specifically, competency should include 21 CFR 820, 803, 806, 830, and 807. In the future, your US agent must also be competent in ISO 13485:2016. FDA inspectors are expected to request evidence of an agreement between your company and the US agent. The inspector will also review your records for qualification, approval, and ongoing evaluation of the US agent as a supplier during FDA inspections. Ideally, your agent has been directly involved in previous FDA inspections, and they can prepare you by conducting a mock-FDA inspection.

What does the FDA do to qualify US agents?

The FDA does very little to qualify a US agent. The only thing the FDA “does” is to send an automated email to the FDA US agent when you submit your initial establishment registration or renew your FDA registration. The email subject line is “ACTION REQUIRED: U.S. Agent Assignment Notification.” The email is sent from “reglist@cdrh.fda.gov.” Your agent must ensure their email client has identified this email as a “safe sender” to prevent the email from ending up in a spam folder. For medical devices, there is no requirement for the US agent to submit any other proof to the FDA.

What is an “Action Required” email?

Below is an example of the “Action Required” email that the FDA sends to FDA US agents immediately after your registration and listing is completed by a foreign firm.

Action Required Email 1024x606 FDA US Agent What do they do?

Your FDA US agent will receive an automated email from the FDA seconds after you complete your registration for an initial FDA establishment registration or the renewal of your FDA establishment registration. The agent then has ten (10) days to log in to their FURLS account and confirm that they are willing and able to serve as your company’s US agent. The email notifying your US agent includes the following language:

“If you are the U.S. Agent for this establishment, select “Yes”, and click “Submit”. If you are not the U.S. Agent for this establishment, select “No”, and click “Submit”. You must confirm you are the U.S. Agent within 10 business days. If you do not confirm that you are the U.S. Agent within 10 days, the system will automatically cancel your Receipt Code and remove the U.S. Agent information associated with the foreign establishment.”

Suppose the agent does not confirm their role within ten business days. In that case, the FDA will automatically email your company that the agent did not confirm their role. If you select a more reliable US agent, you must resubmit the request for the same person or a new person.

If you have additional questions or need a US agent, please contact Medical Device Academy.

FDA US Agent – What do they do? Read More »

Incoming Inspection – How to perform a single process audit

1 Comment / Auditing, Quality Management System / By

The incoming inspection process is my favorite process to audit, and it is the best process for teaching new auditors.

The above video demonstrates how to use a turtle diagram to conduct a process audit of the receiving inspection process. However, this article goes into more detail. You will learn what to look at and what to look for in each part of the audit process.

Preparation for your audit of incoming inspection

If you are conducting an audit of an incoming inspection, you will need a copy of the procedure (i.e., Receiving Inspection Procedure, SYS-033).

Receiving Inspection Procedure Image Incoming Inspection How to perform a single process audit

Do you need an opening meeting?

Opening meetings are not required for first-party (i.e., internal) and second-party (i.e., supplier) audits. Only third-party auditors are required to have a formal opening meeting. Having an opening meeting is always a good idea, but keep it brief and use a checklist. Try to set the tone for the audit with your opening meeting. This will be your second impression because you already had a conversation with the process owner in preparation for the meeting. However, you want to give everyone present for the opening meeting that you exhibit all the personality characteristics of a good auditor as defined by ISO 19011:2018. Professionalism, organization, and integrity should be obvious to everyone in the room. However, don’t forget to smile and be polite because your auditee might be very nervous. FDA inspectors seem to have an unwritten rule book (i.e., in addition to QSIT) that encourages them to intimidate the companies they inspect.

Step 1 – “Briefly, please describe the incoming inspection process.”

The purpose of this section is not to duplicate the level of detail found in the procedure. It is meant to provide a brief description of the process. Ideally, you want to write a single sentence for the incoming inspection process’s what, where, when, who, and how. A maximum of five sentences is needed to answer those five questions. The process owner should provide the description, and there is no need for them to go into extreme detail because you have at least six more questions to ask (see steps 2-7 below). If you are doing a supplier audit or an audit of a company you don’t work for, you might want to have a few “ice breaker” questions that precede this question. For example, you might ask the person’s name, title, and the number of years they have worked for the company. You might also consider stealing my favorite auditor disclaimer, “If you see me writing furiously, don’t worry. I’m required to write down objective evidence supporting conformity with requirements. If I start asking the same question three different ways, and I’m not writing any notes, that means I am having trouble finding evidence of conformity, and I need your help.”  

Step 2 – “What are the inputs that trigger incoming inspection?”

Inputs and outputs of any process refer to both information and physical items. For 100% administrative processes, you may not have any physical items. Incoming inspection, however, has physical goods you receive from suppliers and inspecting. Therefore, the process inputs you are looking for are physical goods and quality system records associated with those goods. For example, if a bunch of titanium round bars were ordered by a buyer in your purchasing department, the physical goods are the titanium bars. The purchase order is one of the quality system records. Other input records that are usually requested to be shipped with the titanium include a packing slip, a certification of analysis, and a dimensional inspection report. It is common to see the incoming inspection activity be delayed because the records are not included with the shipment from the supplier. One recommendation for a process improvement is to require the supplier to send records electronically at the time of shipment instead of sending hardcopies with the product. Statistical inspection sampling plans and work instructions are often confused with input records. These documents are needed to start the incoming inspection, but these are documents that belong in step six of the turtle diagram.

Step 3 – “What are the outputs of the incoming inspection process?”

After incoming inspection is completed there is a requirement to identify the status of the physical product (i.e., accepted or rejected). Usually, a green tag will be used to identify the product as accepted. The tag will also identify the part number, lot, and quantity of product accepted. If the product is titanium, each bar will get a tag. The product will then be transferred to a designated storage area. If you are conducting an audit of a supplier, or a full quality system audit, auditing the warehouse for storage and handling processes is a logical next process. The auditor should look for whether product is segregated in designated locations for specific types of product or if the storage locations are “random” but identified electronically in a material resource planning (MRP) system. The quality system records output from the incoming inspection process will be inspection records and either a green release tag or red rejection tag. If the product is rejected, the product shall be transferred to a quarantine area for nonconforming product and a nonconforming material record (i.e., NCMR) is initiated. Therefore, the process for controlling nonconforming material is another process that could be a logical next process to audit.

Step 4 – “What resources are needed for this process?”

This part of the process approach to auditing is one of the most neglected parts of the quality system. Resources include the facility infrastructure, manufacturing equipment, measurement devices used for inspection, and quality system software used to maintain records of incoming inspection. In this part of the process audit the auditor must be observant. Maintenance records might be located on the side of equipment and they can be reviewed as the auditor walks through the area. This would be an opportunity to interview personnel to make sure they can explain the maintenance process and the equipment maintenance is being performed as planned. The auditor should also determine if equipment validation is required. If the equipment is automated (e.g., automated optical inspection), then an installation qualification (i.e., IQ) should be requested as a quality system record to review at the end of the process or as part of the process for process validation. If the inspection area includes a metrology lab, then the environment may be temperature and humidity controlled. In these types of environments, records of environmental monitoring and trending of environmental conditions should be verified. Lighting, magnification, and particulate filtration could be other environmental requirements for the inspection area. Pest control should be verified in the receiving area, inspection area, and storage areas. The receiving area and warehouse storage are common areas to find pests. Calibration identification should be recorded as a potential follow-up trail for any measurement devices used in the inspection area, and if software is used you will want to verify that quality system software tool validation has been performed.

Step 5 – “Who performs this process?”

A combination of three different roles and responsibilities are typical for this process: 1) department manager, 2) receiving personnel, and 3) inspection personnel. Sometimes one or more of these roles will be combined into one job. The activities sometimes are only performed for a few hours each day, and the personnel that perform the incoming inspection process are assigned to other roles, such as warehouse storage, handling, and shipping. Auditors should always try to interview one or more of the people doing the receiving and inspection activities instead of limiting the interviews to the process owner. Often I will ask the personnel to demonstrate the receiving process and the inspection process. In order to make sure this is possible, you will need to communicate that you want to observe these activities prior to the audit or during the opening meeting. If you don’t, the receiving and inspection activities may already be completed before you start to interview the personnel. Any personnel that are unable to explain the tasks they perform may be targets for verification of training records, effectiveness of training, and competency.

Step 6 – “How is this process performed?”

If an auditor interviews personnel, most people will describe the process in a very haphazard way and steps will be missed. This is why asking people to demonstrate the process is better. The best method is for the person to access the current, approved work instruction or procedure for the process. Then the person should follow the work instruction step-by-step. This allows the person to use the work instruction or procedure as a “crutch” and reduces their nervousness. This also eliminates the skipping steps if the procedures and work instructions are sufficiently detailed. Any blank forms used and statistical inspection standards are also considered quality system documents that define how the process is performed. Sometimes the process owner will provide these documents during their interview, and other times this documents are provided as audit preparation documents. If the documents are not provided in advance the auditor should make sure that they review the documents during observation of activities being performed. This is where an auditor may identify the use of obsolete quality documents, missing details in the documents, and details that are inconsistently followed by personnel.

Step 7 – “What metrics are important for this process?”

Whenever I ask, “What metrics are important in this process?” I typically get a blank stare. Hundreds of business management leaders subscribe to the concept of “what gets managed gets done.” You are also required to establish metrics for your quality system processes in accordance with Clause 8.2.5. Therefore, you need to establish at least one metric, if not more than one. Auditing can help identify opportunities for improvement (OFI), but metrics are the best source of OFIs for a quality system. 

Do you need a closing meeting?

You should always conduct a closing meeting for your audits. However, it is also a best practice to summarize your findings for the process owner before you move on to the next process. If some records remain to be reviewed, ensure the process owner knows that the audit results are pending an outcome of reviewing the remaining records. Consider adopting the “sandwich” approach to presenting your findings: 1) something positive, 2) any nonconformities, and 3) something positive. The approach sandwiches the “bad news” between two pieces of “good news.” If you are working as part of a team, the lead auditor should always be aware of the results of your audit. The manager responsible for the process (i.e., the process owner) should also be aware of the results. Do everything you can to prevent unpleasant surprises at the end of the audit.

When you describe any nonconformities, make sure that you include all of the following information:

  1. the grading of the finding (i.e., MDSAP scoring or Major/Minor)
  2. a single sentence stating the finding
  3. the requirement, including a reference to the applicable regulation or standard
  4. objective evidence from your notes

Whenever possible, email a draft of the wording for your nonconformities to the process owner so they can be prepared with clarification questions during the closing meeting. Make sure you agree with your lead auditor before sending the wording of the finding, and copy them on the email communication. If the process owner has initiated immediate corrective action(s), make sure you note this in your report.

Finalizing your audit report

If you are conducting a supplier audit, you need to give the supplier formal feedback from the audit. You will need an audit report for your quality system records, but you are not required to give the supplier the full report. You might provide a summary of the audit for the supplier instead. If you do this, you should include a copy of that communication in your quality system record (e.g., an appendix to your audit report). If you are going to provide a summary of findings, the content should include at least the following:

  1. positive findings (i.e., strengths)
  2. negative findings (i.e., weaknesses)
  3. nonconformities (if any)
  4. required actions (e.g., supplier corrective action plan)
  5. due date(s) for objective evidence of containment, corrections, and corrective actions
  6. recommendations for follow-up (e.g., next audit)

If you prepare an internal audit report, all of the above content should be included. However, the report should have additional details:

  1. audit purpose
  2. audit scope
  3. audit date(s)
  4. audit criteria
  5. name of participants
  6. date of report
  7. closure of previous audit non-conformities
  8. reference to the audit agenda
  9. deviations, if any, from the agenda
  10. summary of the audit, including any obstructions
  11. objective evidence sampled (i.e., what you looked at and what you looked for)
  12. opportunities for improvement (if any)

Incoming Inspection – How to perform a single process audit Read More »

OpenAI and Elsmar never trust their help with regulatory questions?

5 Comments / FDA, Quality Management System, Registration, Regulatory Consultant / By / ,

Everyone has a favorite resource they use to answer regulatory questions, but can you trust OpenAI or Elsmar to answer correctly?

Screenshot 2023 04 01 10.07.10 AM e1680445207847 1024x787 OpenAI and Elsmar never trust their help with regulatory questions?

If you are deathly afraid of trying new technology, the image above is a screen capture from OpenAI describing “itself.” OpenAI is artificial intelligence (AI), but it is not self-aware yet. The image below is a screen capture from the “About” webpage for Elsmar Cove. This article was the oldest post on the Medical Device Academy, and it described how to use Elsmar Cove as a resource for quality systems and regulatory questions. To update that blog, we are comparing the use of OpenAI with Elsmar Cove. Just in case you were wondering, Elsmar Cove is #6 on our list of favorite search tools, and OpenAI is #5:

Screenshot 2023 04 01 10.14.49 AM e1680445245168 1024x548 OpenAI and Elsmar never trust their help with regulatory questions?

Are the answers provided by OpenAI and Elsmar Cove accurate?

To test the accuracy of a common regulatory question, we chose a question we weren’t 100% sure about when a client asked last month. I asked my team, but nobody was 100% certain. Basil Systems is limited to submission and post-market surveillance data. I searched FDA.gov, but it was not clear. Google gave us a link to the FDA website. I asked a couple of ex-FDA consultants, but they gave me outdated information. On Thursday, March 30, 2023, I asked Lisa King during an AAMI course I was co-teaching. Lisa is a Consumer Safety Officer at the FDA responsible for reviewing device entries into the FDA. She is also in very high demand for public training courses. She said the contract manufacturers used to be exempt from registration if they shipped to a legal manufacturer first. The regulations changed, and now 100% of contract manufacturers making a finished device must register with the FDA. She also clarified that the FDA doesn’t use the term “legal manufacturer.”

Screenshot 2023 04 01 11.00.46 AM e1680445277287 1024x676 OpenAI and Elsmar never trust their help with regulatory questions?

As you can see from the above answer provided by OpenAI, the ChatGPT engine [i.e., Model: Default (GPT-3.5)] effectively produces the correct answer. Using the same wording for the regulatory question, “Does a foreign contract manufacturer need to register with the FDA if they are shipping the medical device to the legal manufacturer first before the device is exported to the USA?” there were no results from Elsmar Cove. After several attempts, I found what I was looking for using the following search terms, “FDA registration of contract manufacturers.” There were multiple related search results, but the most useful discussion threads in the Elsmar Cove discussion forum were:

The most succinct correct answer in the forum is copied below.

Screenshot 2023 04 01 11.39.43 AM e1680445334745 1024x332 OpenAI and Elsmar never trust their help with regulatory questions?

Can you trust OpenAI and Elsmar Cove to answer your regulatory questions?

OpenAI is only as effective as the data used to train it. This is constantly evolving, but we have identified search results that were 100% accurate, results that were outdated, and results that were scary wrong. The same is true of discussion forums. Elsmar Cove is one of the best discussion forums for the medical device industry, but people also use ASQ, RAPS, and AAMI. The quality of the information provided depends upon the knowledge and experience of the people participating in the forum, but it also depends upon the forum’s moderation. Elsmar Cove has some experienced moderators with decades of experience. There is always the chance that the most experienced person in the world could answer your regulatory question incorrectly. This usually creates a problem because everyone else in the forum hesitates to challenge a recognized expert. Therefore, regardless of which resource(s) you use, always try to get a reference to the trustworthy source of the applicable regulation. Even Lisa King could make a mistake, but she immediately said, you can find the regulations in the US Code of Federal Regulations (i.e., 21 CFR 807). The bottom line is, always do your fact-checking and reference your source(s).

OpenAI and Elsmar never trust their help with regulatory questions? Read More »

Scroll to Top