Author name: Robert Packard

How do you demonstrate training competence?

7 Comments / Education, ISO 14971:2019 (Risk Management) / By / ,

Anyone can sign and date training records, but how do you demonstrate the effectiveness of training and competence?
%name How do you demonstrate training competence?

What are the requirements for training?

The requirements for training are found in ISO 13485:2016, Clause 6.2 (i.e., Human Resources). Auditors and inspectors usually only ask for training records, but the requirement for records is the last item in the clause (i.e., “6.2(e) – maintain appropriate records of education, training, skills, and experience (see 4.2.5).”). The first four items in Clause 6.2 all include one word, but it’s not “records.” The most critical word in the requirements is “competence.”

What is the difference between training requirements, training records, training effectiveness, and training competence?

  • Training requirements define the “appropriate education, training, skills, and experience.” Specifically, what degree(s) is required, if any? What training curriculum needs to be completed? What skills are needed to perform the work? And how many years of experience are needed?
  • Training records are documented evidence that all training requirements have been met. Records may include a job application, resume, individual training records, group training records, or a training certificate. Records may also include the results of any quizzes to demonstrate training effectiveness and any evaluation of training competence.
  • Training effectiveness is how well a person understands the information communicated during training. Using terms from human factors engineering (i.e., the PCA Process), the person must have the correct perception and cognition. Cognition can be evaluated by giving people quizzes or asking them questions. Written and verbal exams may also include pictures and/or video. Performing these types of tasks on a quiz or exam are “knowledge tasks.”
  • Training competence is possessing the skill or ability to perform tasks. To demonstrate competence, someone that is already competent (i.e., a subject matter expert) must observe the trainee performing tasks. A person that once possessed the skill or ability can also judge competency, and a person that does not possess the skill or ability can be trained to evaluate skill and ability (e.g., an Olympic Judge or referee). Skills and abilities can be physical or cognitive, but there are also social skills. Examples of each are provided in the table below:

List of abilities and skills 1024x495 How do you demonstrate training competence?

What keeps me awake at night? (a story from April 2, 2011)

I am in Canada, it’s almost midnight, and I can’t sleep. I’m here to teach a Canadian client about ISO 14971—the ISO Standard for Risk Management of medical devices. Most companies requesting this training are doing so for one of two reasons: 1) some design engineers have no risk management training, or 2) the engineers have previous training on risk management, but the training is out-of-date. This Canadian company falls into the second category, and engineers with previous training ask the most challenging questions. This group of engineers forced me to re-read the Standard several times and reflect on the nuances of almost every single phrase. While teaching this risk management course, I learned more about risk management than I ever knew.

The four levels of the Learning Pyramid

The image at the beginning of this blog is a learning model that explains my experiences training Canadian design engineers. I call this model the “Learning Pyramid.” At the base of the pyramid, there are “Newbies.”

This is the first of four levels. At the base, students read policies and procedures, hoping to understand.

The student is now asked to watch someone else demonstrate proper procedures in the second level of the pyramid. One of my former colleagues has a saying that explains the purpose of this process well, “A picture tells a thousand words, but a demonstration is like a thousand pictures.” Our children call this “sharing time,” but everyone over 50 remembers this as “show and tell.”

The student is now asked to perform their tasks in the third level of the pyramid. This is described as “doing,” but in my auditing courses, I refer to this process as “shadowing.” Trainees will first read the procedures for Internal Auditing (level 1). Next, trainees will shadow the trainer during an audit to demonstrate the proper technique (level 2). During subsequent audits, the trainees will audit, and the trainer will shadow the trainee (level 3). During this “doing” phase, the trainer must watch, listen, and wait for what I call the “teachable moment.” This is a moment when the trainee makes a mistake, and you can use this mistake as an opportunity to demonstrate a complex subject.

Finally, in the fourth level of the Learning Pyramid, we now allow the trainee to become a trainer. This is where I am at. I am an instructor, but I am still learning. I am learning what I don’t know.

Teaching forces you back to the bottom of the Learning Pyramid

After teaching, the next step in the learning process is to return to the first level. I re-read the Standards and procedures until I understood the nuances I was unaware of. Then, I search for examples in the real world that demonstrate the complex concepts I am learning. After searching for examples, I tested my knowledge by applying the newly acquired knowledge to an FDA 510(k) or De Novo submission for a medical device client. Finally, I prepared to teach again. This reiterative process reminds me of the game Chutes and Ladders, but one key difference is that we never really reach the level of “Guru.” We continue to improve but never reach our goal of perfection.

Where is training competence in the Learning Pyramid?

Most people feel that a person is competent in a position when they can consistently perform a task. However, the ISO 13485 standard uses the phrase “necessary competence” to suggest that competency for any given task might not be the same for all people. For example, in some cases, it may be sufficient to perform a task with written instructions in front of you, while an instructor might be expected to perform the task without written instructions. The speed at which a person performs a task might also differ. For example, a secretary might be required to touch type at a speed of 60+ wpm with a QWERTY keyboard, while a stenographer must be able to use a STENO keyboard and write at 225 wpm. The accuracy requirements may differ for those two positions as well. Therefore, your company may decide that the training competence requirement for a design engineer is that they can pass an exam on risk management. However, the training competency requirement for the design project lead or Engineering Manager might include teaching inexperienced design engineers to apply the basic risk management principles. Demonstrating the ability to teach inexperienced design engineers might be demonstrated by an auditor interviewing members of your design team.

How do you demonstrate training competence? Read More »

Audit schedule and an audit agenda, what’s the difference?

8 Comments / ISO Auditing / By / , ,

Internal audit and supplier audit programs both require an audit schedule and an audit agenda, but what’s the difference between them?

What is an audit schedule?

An “audit schedule” is not a formal definition in ISO 19011:2018. However, section 5.1 of that standard states that your audit program should include nine different requirements. Item “d” is “d) schedule (number/duration/frequency) of the audits.” Typically, the audit program manager will maintain an annual audit schedule with a date indicating the date the schedule was last revised. The most common example in lead auditor training is a matrix like the one shown below. The left-hand column will list all of the individual processes that are identified in the company’s process interaction diagram, and the top row of the matrix will indicate the month when each process audit will be conducted. Typically, the expectation is to complete the audit sometime during that month, but some quality auditing procedures specify that the audit may be completed the month before or the month after to accommodate the process owner. The regulations only require that you document and maintain an audit schedule, and the standard is only considered guidance.

%name Audit schedule and an audit agenda, what’s the difference?
Example of an Internal Audit Schedule

I use a slide in lead auditor courses that gives the example of an internal auditing schedule provided above. On the surface, this example seems like a good audit schedule. Twelve auditors perform two audits each year. If each audit requires approximately two days, each auditor spends less than two percent of their work year auditing. Unfortunately, a two-percent allocation of time is insufficient to become or remain proficient at auditing. An improvement to the auditing schedule would be to assign fewer auditors so each auditor gets more experience. There is no perfect number, but assigning a few specialists will improve the chances of becoming and remaining proficient at auditing.

What is an audit agenda?

An “audit agenda” is not a formal definition in ISO 19011:2018 either. In fact, the word “agenda” is not even used in ISO 19011. Instead, section 5.5.5 of ISO 19011 states that “The assignment [of the individual audit] should be made in sufficient time before the scheduled date of the audit, in order to ensure the effective planning of the audit.” The audit plan must also be part of the records [i.e., Clause 5.5.7(b)]. Therefore, “agenda” and “plan” may be used interchangeably. Details of audit planning are provided in 6.3.2 of ISO 19011.

6 Steps to Creating an Audit Schedule

There are six steps to creating an audit schedule:

  1. What were the results of previous audits?
  2. Which processes are the most important to audit?
  3. Who should conduct your internal audit?
  4. How long should your internal audit be?
  5. Should you conduct one full quality system audit or several audits?
  6. Is a remote audit good enough?

We will address each of the six steps below.

How do the results of previous audits impact your audit schedule?

The results of an audit include nonconformities, observations for improvement (OFI), and a conclusion regarding whether the quality system is effective or not. Usually, most processes are effective, and there are no nonconformities or OFIs. Therefore, any processes that had a nonconformity or OFI should be prioritized in the audit schedule and audit planning for the future. For internal and supplier audits, a best practice is for the auditor and the process owner to discuss the corrective actions planned and determine the appropriate timeline or implementation of actions planned. Then the auditor can indicate a timeframe for re-auditing the nonconforming process after corrective actions are implemented. This strategy allows the auditor to be part of the effectiveness check. This approach is appropriate for individual process audits but not for a full-quality system audit.

Which processes are the most important to audit?

The primary element impacting the importance of processes is the risk to product quality associated with the process. Usually, support processes are of lower importance because they do not directly impact product quality. In contrast, core processes directly involved in a device’s design, manufacture, and distribution are critical. Most auditors and audit program managers emphasize design controls and production process controls as important areas to audit. However, the distribution area is often neglected. Other core processes are purchasing, sales, customer service, and servicing. Not every process is equally important when comparing two companies. For example, device manufacturers that only make software as a medical device (SaMD) often have very limited purchasing and incoming inspection activities to audit.

Who should the audit program manager assign to each internal audit?

%name Audit schedule and an audit agenda, what’s the difference?The example of a revised audit schedule provided above identifies the departments where each of the auditors works with color coding. This is done to ensure that auditors are not assigned to audit processes where they might have a conflict of interest (i.e., they would be auditing their own work). This is the most important aspect of assigning auditors. The second most important aspect is to make sure the auditor has the technical knowledge to audit the process. It is challenging to conduct an audit of manufacturing if you have not spent any time in manufacturing before. If auditors are new and their training is in progress, then the audit program manager may assign the auditor to a process specifically to give them more experience with that type of process. Inexperienced auditors often are assigned less important processes that have not changed recently. However, a better approach to training auditors is to give them a challenge with support. Having the new auditor prepare a detailed sampling plan and list of questions before the audit can prepare them for auditing a more challenging, important process that is likely to have one or more nonconformities. Auditing processes that have nonconformities is also the best way to teach a new auditor how to write the audit findings.

What should be the duration of each internal audit in your schedule?

The duration of an audit should be based on the results of previous audits, but other important factors include: 1) the number of personnel involved in the process, 2) the complexity of the process, and 3) the risk to product quality associated with the process. The MDSAP program uses a procedure for audit time determination (i.e., MDSAP AU P0008.007: Audit Time Determination Procedure), and the MDSAP audit approach document (i.e., MDSAP AU P0002.008 Audit Approach) classifies processes as having either a “direct” or “indirect” impact upon product quality based upon the applicable clause of the process (i.e., Clauses 0-6.3 are indirect, and Clauses 6.4-8.5.3 are direct). For example, the production processes and design and development processes both involve a large number of people in most organizations, the processes are complex, and both processes directly impact product quality. Therefore, I typically allocate 3-4 hours to each of those processes during an audit. In comparison, incoming inspection often involves one or two people, and the process often involves only one procedure. Incoming inspection is a “direct” process, but less time (e.g., 1 hour) should be allocated to auditing incoming inspection than the other two processes–unless there was a nonconformity in the incoming inspection process during a recent audit or unless the process was recently changed.

Should you conduct one full quality system audit or several audits?

Both approaches have strengths and weaknesses, but there is not a single best way. If I am using employees to conduct an audit, then I typically restrict the scope of the audit to a single process. Alternatively, when I use a consultant to conduct an audit, I typically conduct a full-quality system audit to minimize travel costs. Another strategy I have recommended is to identify the processes that are most important to audit first (e.g., processes with recent changes and/or nonconformities), and these processes are scheduled for individual process audits during the first half of the audit schedule. Then I schedule a full-quality system audit in the second half of the audit schedule. The strategy ensures that all important processes will be audited twice in one year, but every process will be audited at least once.

Remote audits vs On-site audits

Prior to the Covid-19 pandemic, remote audits were rare in the medical device industry. Many NBs insisted that remote audits were not permitted or effective. The pandemic forced the entire industry to create policies for remote auditing and to use remote auditing whenever possible. Now that the pandemic has ended, many companies continue to conduct remote audits to save money. Even NBs are conducting more remote audits for Stage 1 readiness audits during the ISO 13485 certification process. ISO 19011 has a section in the Appendices outlining the differences between remote and on-site audits. However, there is a minimal advantage to conducting an on-site audit of a process where the auditor is expected to spend all of their time in a conference room during the audit. If the audit is going to be done in a conference room, then why not conduct it remotely? The one exception is when most records are paper-based and unavailable electronically. Alternatively, an on-site audit is generally more effective if the process involves observing inspection activities or assembly operations. Remote audits of inspection activities and assembly operations should be reserved for re-auditing or when a process has been audited on-site in the past, but an on-site audit would still be more effective for those processes.

How many times should a process be audited annually?

Many notified bodies will expect companies to audit all processes at least once during the year. However, it doesn’t expressly state this as a requirement in the regulations, and some companies justify skipping processes that are functioning well and have not changed in the past year. Our team is seeing this more frequently as the number of lead auditors worldwide has become scarce due to the requirements of MDSAP, the MDR/IVDR implementation, and unannounced audits. However, I almost never see the opposite justification (i.e., auditing a process more than once a year). If a process has been changed significantly, or there were nonconformities, then re-auditing the process may be used to verify the effectiveness of corrective actions or to verify that personnel are compliant with the revised process.

How to take advantage of the process approach to auditing

Another improvement that can be made to the revised example of an audit schedule is to use the process approach to auditing. Instead of performing an independent document control and training audit, these two clauses/procedures can be incorporated into every audit. The same is true of maintenance and calibration support processes. Wherever maintenance and calibration are relevant, these clauses should be investigated as part of auditing that area. For example, when the incoming inspection process is audited, it makes sense to look for evidence of calibration for any devices used to perform measurements in that area. When production process controls are being audited, maintenance records of production equipment should also be sampled.

If the concept of process auditing is fully implemented, the following ISO 13485 clauses can easily be audited in the regular course of reviewing other processes:

  • 4.2.1), Quality System Documentation;
  • 4.2.3), Document Control;
  • 4.2.4), Record Control;
  • 5.3), Quality Policy;
  • 5.4.1), Quality Objectives;
  • 6.2.2), Training;
  • 6.3), Maintenance;
  • 6.4), Work Environment;
  • 7.1), Planning of Product Realization & Risk Management
  • 7.6), Calibration;
  • 8.2.3), Monitoring & Measurement of Processes
  • 8.5.2), Corrective Action; and
  • 8.5.3) Preventive Action.

This strategy reduces the number of process audits needed by more than half.

Internal Auditing: Upstream/Downstream Examples

Another way to embrace the process approach to auditing is to assign auditors to upstream or downstream processes in the product realization process from their own area. For example, Manufacturing can audit Customer Service to understand better how customer requirements are confirmed during the order confirmation process. This is an example of auditing upstream because Manufacturing receives the orders from Customer Service—often indirectly through an MRP system. Using this approach allows someone from Manufacturing to identify opportunities for miscommunication between the two departments. If Regulatory Affairs audits the engineering process, this is an example of auditing downstream. Regulatory Affairs is often defining the requirements for the Technical Files and Design History Files that Engineering creates. If someone from Regulatory Affairs audits these processes, the auditor will realize what aspects of technical documentation are poorly understood by Engineering and quickly identify retraining opportunities.

Audit schedule and an audit agenda, what’s the difference? Read More »

FDA Pre-Submission Format and Content Requirements

5 Comments / PreSTAR / By

The format and content requirements for an FDA pre-submission have not changed, but the launch of the FDA PreSTAR has changed everything.

What is an FDA pre-submission?

An FDA pre-submission aims to get answers to questions you have about a future FDA submission. The pre-submission may consist of one large PDF document or multiple PDF documents. In your pre-submission, you must select either an email response or an email response with a teleconference. One advantage of choosing a teleconference is that you can ask clarifying questions during a one-hour teleconference with the FDA. Still, you are responsible for submitting draft meeting minutes to the FDA within 15 days of the teleconference. If you select an email response, you do not need to provide meeting minutes to the FDA.

Our new 4-part FDA pre-submission webinar series is available on-demand. You can download it and watch it as many times as you want. This will be the Ultimate FDA pre-submission training. Do not miss it.

Register Now for 299 300x129 FDA Pre Submission Format and Content Requirements

Everyone asks us for examples, so we will show you how to complete the entire FDA PreSTAR for a device in this webinar series. If you would like to vote on which device we should use as an example (i.e., Option 1 = Infrared Thermometer or Option 2 = Antimicrobial Gauze), please place your vote on our LinkedIn page.

What is the difference between an FDA pre-submission and a Q-submission?

Every FDA pre-submission is a Q-submission, but not all Q-submissions are pre-submissions. The new PreSTAR template is currently limited to an FDA pre-submission, but the template will be expanded to other types of Q-subs later. The FDA pre-submission template (i.e., PreSTAR) beta version 0.1 is unnecessary for responses to interactive review questions from the FDA. Just email the Lead Reviewer (file size limit is 25 MB for email).

No SIRs 1024x342 FDA Pre Submission Format and Content Requirements

Unfortunately, the beta version 0.1 is also not ready for Submission-in-Review (SIR) meetings or responses to IDE during an interactive review.No SIRs or IDEs 1024x390 FDA Pre Submission Format and Content Requirements

13 other types of submissions might benefit from Q-submissions:

  1. Submission Issue Requests (SIRs)
  2. Study Risk Determinations
  3. Informational Meetings
  4. Breakthrough Device Designation Requests
  5. Informational Meetings
  6. PMA Day 100 meetings
  7. Agreement and Determination meetings
  8. Submissions associated with the STeP program
  9. Accessory classification requests
  10. Requests for FDA feedback on specific questions or cross-cutting policy matters
  11. Requests for recognition of publicly accessible genetic variant databases
  12. Combination product agreement meetings (CPAM), and
  13. Feedback on FDA 483 inspection observations.

We expect the PreSTAR template to eventually be available for a 513(g) request in the future because it was already validated for that purpose.

What is the Q-submission number?

All Q-submissions are assigned a document number beginning with “Q” upon receipt (i.e., Qyyxxxx). The format of the number consists of 2-digits (i.e., “yy”) for the year of submission (e.g., “23” for 2023) and 4-digits (i.e., “xxxx”) that are the following sequential number assigned by the FDA for that calendar year. Therefore, the first Q-submission received by the FDA in January 2023 is Q230001, and between 3,500 and 4,000 new submissions are usually received each year. If the subject device was submitted in a previous Q-submission, the original document number is re-used, and a supplement number is added (i.e., Qyyxxx/S001, Qyyxxx/S002, etc.). Q-submission numbering is explained in more detail in the 2023 FDA guidance.

Does the FDA charge for Q-submissions?

FDA pre-submissions do not require paying an FDA User Fee (i.e., $0).

How long does an FDA pre-submission take?

The days of squishing timelines are gone. The timeline is 70-75 calendar days. On October 5, 2022, MDUFA V was approved. As one of the MDUFA V decision targets, the FDA is tasked with reducing the timeline for responding to pre-sub questions within 70 days for 90% of pre-sub requests. The FDA is tasked with achieving this goal by March 2024. If they are successful, the FDA will receive an increase of 59 headcounts to their budget in 2024. This is approximately a $19 million incentive to respond to your pre-submission meeting questions within 70 days. To reflect these new MDUFA V decision targets, the FDA updated the Q-Sub guidance document to reflect the target date of 70 days for the email response and 75 days for teleconference meetings. The FDA also updated the Customer Collaboration Portal (CCP) to facilitate tracking of FDA pre-submission deadlines.

What is an FDA PreSTAR?

In the past, you had to create your document(s) for an FDA pre-submission. Some people create one large PDF document divided into sections, while others create separate PDF documents for each requirement of the FDA pre-submission guidance. On August 14, 2023, the FDA released an updated beta version (i.e., version 0.2) of a new PDF template (i.e., FDA PreSTAR). This new PreSTAR template provides multiple benefits to the FDA:

  1. every company uses the same format,
  2. the template automatically verifies that the pre-submission includes all required elements, and
  3. Including optional elements will encourage companies to provide more device details than they might otherwise provide.

The PreSTAR also benefits submitters:

  1. you will never forget the required elements of the FDA pre-submission,
  2. you never have to validate an FDA eCopy, and
  3. the similar format and user interface will train you to use the FDA eSTAR.

Note: October 1, 2023, was the FDA eSTAR implementation deadline. 

Do you have to use the PreSTAR template?

Nope. The PreSTAR version 0.2 is a beta version and 100% optional. However, I like it better than my templates. Your design team can still have individual documents for the user manual, device description, and testing plan. We attach the document using the button that says “Add Attachment” (see screen capture below).

Device Description screen capture for prestar 1024x576 FDA Pre Submission Format and Content Requirements

The PreSTAR template was built by Patrick Macatangga, a Tools & Templates Engineer working at the FDA on the Lifecycle Tools and Templates Team. To help with where to direct questions about the template, he suggested:

  • If you have questions or feedback regarding the voluntary use of the eSTAR for medical devices regulated by CDRH, or if you have general questions about medical devices, contact the Division of Industry and Consumer Education (DICE).
  • If you find malfunctions or errors in the eSTAR template for medical devices regulated by CDRH, contact eSubPilot@fda.hhs.gov.
  • If you have questions regarding 510(k)s, De Novo requests, or Early Submission Requests for medical devices regulated by CDRH, contact OPEQSubmissionSupport@fda.hhs.gov.

How do you submit an eCopy?

You can submit an FDA eCopy on electronic media (e.g., USB flash drive) and send it via FedEx to the FDA Document Center at the following address: Food and Drug Administration, Center for Devices and Radiological Health, Document Mail Center, 10903 New Hampshire Ave., Bldg. 66, rm. G609, Silver Spring, MD 20993-0002. However, you can also submit an FDA eCopy via a web browser (i.e., CCP…see next section on submitting a PreSTAR).

If you are submitting an eCopy through the CCP instead of an FDA PreSTAR

How do you submit a PreSTAR?

You have two options for delivery of an FDA pre-submission:

  1. save the pre-sub on electronic media (e.g., USB flash drive) and send it via FedEx to the FDA, and
  2. upload the pre-sub to the new FDA Customer Collaboration Portal (CCP).

As you can guess from the video above, we only use option 2 for FDA pre-submissions. For option 2, you can upload an eCopy (saved as a zip file) or a PreSTAR (in the native PDF format). The image below shows you how this is done, but the uploading process usually takes about one minute–depending on your file size and bandwidth. You can register for your own CCP account in seconds.

estar and eCopy upload 1024x485 FDA Pre Submission Format and Content Requirements

What is the pre-submission process?

Preparing and uploading your FDA pre-submission meeting request is only the first step of the process. You will receive an automated email confirming that your pre-submission was successfully uploaded. Then, you will receive an automated letter via email that gives you the assigned Q-sub number. You will also receive an automated email notifying you that the pre-submission was accepted, or the FDA reviewer will contact you if changes are needed. The FDA reviewer assigned will usually contact you by email within the first three weeks to schedule a teleconference if you request one. Still, the date/time offered usually does not match the availability of the FDA team, and alternate dates/times may be offered.

You will receive an email response from the FDA for each of your questions within 70 days of receipt by the FDA. If you requested a teleconference, it would typically be about 75 after receipt of the FDA pre-submission meeting request. Your team needs to prepare a detailed discussion plan for the one-hour teleconference. A slide deck is highly recommended to facilitate communication but is not required. If you provide a slide deck, you should email it to the reviewer before the meeting. You must also provide a copy of the slide deck with your meeting minutes. At the beginning of the teleconference, someone from your team must commit to submitting draft meeting minutes to the FDA within 15 days. The FDA will reply with acceptance of your meeting minutes, or they will provide an edited version. It is also common to submit a supplement FDA pre-submission with detailed protocols and new questions for the FDA.

Reuqest a call with Lindsey 1024x240 FDA Pre Submission Format and Content Requirements

FDA Pre-Submission Format and Content Requirements Read More »

Iterative design is real, waterfalls are illusions

2 Comments / Design Control / By

The Waterfall Diagram was copied by the FDA from Health Canada and ISO 9001:1994, but everyone actually uses an iterative design process.

Iterative Design – What is it?

The FDA first mandated that medical device manufacturers implement design controls in 1996. Unfortunately, in 1996 the design process was described as a linear process. In reality, the development of almost every product, especially medical devices, involves an iterative design process. The V-diagram from IEC 62304 is closer to the real design control process, but even that process is oversimplified.

Software Validation and Verification 1 Iterative design is real, waterfalls are illusions

What is the design control process?

The design control process is the collection of methods used by a team of people and companies to ensure that a new medical device will meet the requirements of customers, regulators, recognized standards, and stakeholders. With so many required inputs, it is highly unlikely that a new medical device could ever be developed in a linear process. The design control process must also integrate risk management and human factors disciplines. ISO 14971:2019, the international risk management standard, requires conducting optional control analysis. Option control analysis requires evaluating multiple risk control options and selecting the best combination of risk controls for implementation. The human factors process involves formative testing where you evaluate different solutions to user interfaces, directions for use, and training. This always requires multiple revisions before the user specifications are ready to be validated in summative usability testing. Process success is verified by conducting verification and validation testing. The process ends when the team agrees that all design transfer activities are completed, and your regulatory approval is received.

waterfall fda Iterative design is real, waterfalls are illusions

Where did design controls come from?

The diagram above is called the “Application of Design Controls to Waterfall Design Process.” The FDA introduced this diagram in 1997 in the design controls guidance document. However, the original source of the diagram was Health Canada.

This diagram is one of the first slides I use for every design control course that I teach because the diagram visually displays the design control process. The design controls process, defined by Health Canada and the US FDA, is equivalent to the design and development section found in ISO 13485 and ISO 9001 (i.e., – Clause 7.3). Seven sub-clauses comprise the requirements of these ISO Standards:

  • 7.3.1 – Design Planning
  • 7.3.2 – Design Inputs
  • 7.3.3 – Design Outputs
  • 7.3.4 – Design Reviews
  • 7.3.5 – Design Verification
  • 7.3.6 – Design Validation
  • 7.3.7 – Design Changes

In addition to the seven sub-clauses found in these ISO Standards, the FDA Quality System Regulation (QSR) also includes additional requirements in the following sub-sections of 21 CFR 820.30: a) general, h) design transfer, and J) Design History File (DHF). If you need a procedure(s) to comply with design controls, we offer two procedures:

  1. Design Controls (SYS-008)
  2. Change Control (SYS-006)

The change control process was separated from the design controls process because it is specific to changes that occur after a device is released to the market. We also have a training on Change Control.

Free Download – Overview of the Design & Development Process

What are the phases of the Design Control Process?

Normally, we finish the design control process by launching your device in the USA because this is when you should close your Design History File (DHF). However, if you are going to expand to different markets, there is a specific order we recommend. We recommend the US market first because no quality system certification is required, and the FDA 510(k) process is easier than the CE Marking process due to the implementation of the MDR and the IVDR. The Canadian market is the second market we recommend because the Canadian Device License Application process for Health Canada is even easier than the 510(k) process for Class II devices. The Canadian market is not recommended as the first country to launch because Health Canada requires MDSAP certification for your quality system, and the Canadian market is 10% of the US market size. The European market should probably be your last market because of the high cost and long timeline for obtaining CE Marking. Each of the phases of design and development is outlined in the first column of our free download, “Overview of Regulatory Process, Medical Device Development & Quality System Planning for Start-ups.”

Which regulatory filings are required during each phase of design and development?

The second column of our free download lists the regulatory filings required by the FDA for each phase. Generally, we see companies waiting too long to have their first pre-submission meeting with the FDA or skipping the pre-submission meeting altogether. This is a strategic mistake. Pre-submission meetings are free to submit to the FDA, and the purpose of the meeting is to answer your questions. Even if you are 100% confident of the regulatory pathway, you know precisely which predicate you plan to use, and you know which verification tests you need to complete, you still have intelligent questions that you can ask the FDA. Critical questions fall into three categories: 1) selection of your test articles, 2) sample size justification, and 3) acceptance criteria. Even if you don’t have a complete testing protocol prepared for the FDA to review, you can propose a rationale for your test article (e.g., the smallest size in your product family). You can also provide a paragraph explaining the statistical justification for your sample size. You might also present a paragraph explaining the data analysis method you plan to use.

The following example illustrates how discussing the details of your testing plan with the FDA can help you avoid requests for additional information and retesting. Many of the surgical mask companies that submitted devices during the Covid-19 pandemic found that the FDA had changed the sample size requirements, and they were now requiring three non-consecutive lots with a 4% AQL sample size calculation. If the company made a lot of 50,000 masks, they would be forced to sample a large number of masks, while a lot size of 250 masks allowed the company to sample the minimum sample size of 32 masks.

If the regulatory pathway for your device is unclear, you might start with the submission of a 513(g) submission during the first phase of design and development. After you have written confirmation of the correct regulatory pathway from the FDA, you can submit a pre-submission meeting request to the FDA. If the pathway for your device is a De Novo Classification Request, you might have a preliminary pre-submission meeting to get an agreement with the FDA regarding which recognized standards should be applied as Special Controls for your device. While waiting 70+ days for your pre-submission meeting with the FDA, you can obtain quotes from testing labs and prepare draft testing protocols. After the presubmission meeting, you can submit a pre-submission supplement that includes detailed testing protocols–including your rationale for the selection of test articles, sample size justification, and the acceptance criteria.

What quality assurance documentation is required during each phase?

In addition to testing reports for your verification and validation testing, you will find many other supporting documents you also need to prepare. We refer to these supporting documents generically as “quality assurance documentation” because the documents verify that you meet specific customer and regulatory requirements. However, the documents should be prepared by the person or people responsible for that portion of your design project. For example, every device will need a user manual and draft labeling. Even though you will need someone from quality to complete a regulatory checklist to ensure that all of the required symbols and general label content are included, you will also need an electrical engineer to prepare sections of the manual with EMC labeling requirements from the FDA’s EMC guidance document. In your submission’s non-clinical performance testing section, you must include human factors documentation in addition to your summative usability testing report. For example, you will need a use specification, results of your systematic search of adverse events for use errors, a task analysis, and a Use-Related Risk Analysis (URRA). Software and cybersecurity documentation includes several documents beyond the testing reports as well.

Which procedures do you need to implement during each development phase?

The last column of our free download lists the procedures we recommend implementing during each phase of the design process. In Canada and Europe, you must complete the implementation of your entire quality system before submitting a Canadian License Application or CE Marking application. In the USA, however, you can finish implementing your quality system during the FDA review of your 510k submission or even after 510k clearance is received. The quality system requirement is that your quality system is fully implemented when you register your establishment with the FDA and begin distribution.

Iterative design is real, waterfalls are illusions Read More »

FDA US Agent – What do they do?

3 Comments / FDA, Registration / By

Medical device companies exporting devices into the USA must have a US agent to register, but what does an FDA US agent do?

What does an FDA US agent do?

Every medical device company outside the USA that distributes devices in the USA must have an FDA US agent. This includes manufacturers, contract manufacturers, and specifications developers outside the USA. The US agent assists the FDA in communication with the device company. The most common communications concern questions about devices exported to the US and scheduling FDA inspections. The role of the US agent is very similar to a European Authorized Representative, a UK Responsible Person, or a Swiss Authorised Representative. Unlike an EC Representative, you do not include US agents in your device labeling. The US agent’s name and contact information only appear on your FDA Establishment Registration record on the FDA website. 

Is there any certification or contract required for a US agent?

FDA US agents have no certification process, but you should have a formal signed agreement or contract with your agent. I have never seen the FDA request a copy of the contract or a letter from a US agent or the company that is registered. However, since the agent has a legal role and responsibility, you should ensure an agreement or contract is in place. The agreement or contract should include the following elements:

  • Scope of service
  • Commitment to perform US agent services promptly
  • Duration of service (i.e., specific start and end dates)
  • Termination provisions
  • Consulting Fees for US agent services (typically an annual fee ranging from $250-$1,500)
  • Any additional consulting fees if the FDA contacts your agent
  • Who is responsible for payment of FDA User Fees ($7,653 for FY 2024 FDA User Fee)
  • Commitment to communicating complaints, especially for potential risks to public health, serious injuries, or death, directly to your company
  • Confidentiality clause or reference to a separate confidentiality agreement (Note: The agent may be compelled to disclose information they have to the FDA, but they should notify your company first if this happens.)
  • Non-solicitation of your customers or suppliers and no solicitation of employees
  • Force Majeure clause
  • Identification of the agent’s name, address, phone, and email
  • Identification of the company name, address, phone, DUNs Number
  • Identification of the company contact’s name, title, address, phone, and email
  • Identification of who will be the “Official Correspondent” in the FDA Registration Database
  • Signature and Date

The US Agent is not required to be a legal entity, but you will need to enter a “Company Name.” There is no place to enter an EIN, and DUNS number is optional. Here’s a screen capture of the account creation form below.

FURLS Account Set up 1024x811 FDA US Agent What do they do?

You should also consider adding your agent to your Approved Supplier List (i.e., LST-003). If you do not already have a procedure for Supplier Quality Management (i.e., SYS-011), Medical Device Academy has a procedure available for purchase that includes a template for review and approval of new suppliers (i.e., FRM-005) and a template for an Approved Supplier List (i.e., LST-003). The FDA US agent doesn’t need a quality system, but they should be able to demonstrate competency in US FDA device regulations with their resume and/or training records. Specifically, competency should include 21 CFR 820, 803, 806, 830, and 807. In the future, your US agent must also be competent in ISO 13485:2016. FDA inspectors are expected to request evidence of an agreement between your company and the US agent. The inspector will also review your records for qualification, approval, and ongoing evaluation of the US agent as a supplier during FDA inspections. Ideally, your agent has been directly involved in previous FDA inspections, and they can prepare you by conducting a mock-FDA inspection.

What does the FDA do to qualify US agents?

The FDA does very little to qualify a US agent. The only thing the FDA “does” is to send an automated email to the FDA US agent when you submit your initial establishment registration or renew your FDA registration. The email subject line is “ACTION REQUIRED: U.S. Agent Assignment Notification.” The email is sent from “reglist@cdrh.fda.gov.” Your agent must ensure their email client has identified this email as a “safe sender” to prevent the email from ending up in a spam folder. For medical devices, there is no requirement for the US agent to submit any other proof to the FDA.

What is an “Action Required” email?

Below is an example of the “Action Required” email that the FDA sends to FDA US agents immediately after your registration and listing is completed by a foreign firm.

Action Required Email 1024x606 FDA US Agent What do they do?

Your FDA US agent will receive an automated email from the FDA seconds after you complete your registration for an initial FDA establishment registration or the renewal of your FDA establishment registration. The agent then has ten (10) days to log in to their FURLS account and confirm that they are willing and able to serve as your company’s US agent. The email notifying your US agent includes the following language:

“If you are the U.S. Agent for this establishment, select “Yes”, and click “Submit”. If you are not the U.S. Agent for this establishment, select “No”, and click “Submit”. You must confirm you are the U.S. Agent within 10 business days. If you do not confirm that you are the U.S. Agent within 10 days, the system will automatically cancel your Receipt Code and remove the U.S. Agent information associated with the foreign establishment.”

Suppose the agent does not confirm their role within ten business days. In that case, the FDA will automatically email your company that the agent did not confirm their role. If you select a more reliable US agent, you must resubmit the request for the same person or a new person.

If you have additional questions or need a US agent, please contact Medical Device Academy.

FDA US Agent – What do they do? Read More »

Incoming Inspection – How to perform a single process audit

1 Comment / Auditing, Quality Management System / By

The incoming inspection process is my favorite process to audit, and it is the best process for teaching new auditors.

The above video demonstrates how to use a turtle diagram to conduct a process audit of the receiving inspection process. However, this article goes into more detail. You will learn what to look at and what to look for in each part of the audit process.

Preparation for your audit of incoming inspection

If you are conducting an audit of an incoming inspection, you will need a copy of the procedure (i.e., Receiving Inspection Procedure, SYS-033).

Receiving Inspection Procedure Image Incoming Inspection How to perform a single process audit

Do you need an opening meeting?

Opening meetings are not required for first-party (i.e., internal) and second-party (i.e., supplier) audits. Only third-party auditors are required to have a formal opening meeting. Having an opening meeting is always a good idea, but keep it brief and use a checklist. Try to set the tone for the audit with your opening meeting. This will be your second impression because you already had a conversation with the process owner in preparation for the meeting. However, you want to give everyone present for the opening meeting that you exhibit all the personality characteristics of a good auditor as defined by ISO 19011:2018. Professionalism, organization, and integrity should be obvious to everyone in the room. However, don’t forget to smile and be polite because your auditee might be very nervous. FDA inspectors seem to have an unwritten rule book (i.e., in addition to QSIT) that encourages them to intimidate the companies they inspect.

Step 1 – “Briefly, please describe the incoming inspection process.”

The purpose of this section is not to duplicate the level of detail found in the procedure. It is meant to provide a brief description of the process. Ideally, you want to write a single sentence for the incoming inspection process’s what, where, when, who, and how. A maximum of five sentences is needed to answer those five questions. The process owner should provide the description, and there is no need for them to go into extreme detail because you have at least six more questions to ask (see steps 2-7 below). If you are doing a supplier audit or an audit of a company you don’t work for, you might want to have a few “ice breaker” questions that precede this question. For example, you might ask the person’s name, title, and the number of years they have worked for the company. You might also consider stealing my favorite auditor disclaimer, “If you see me writing furiously, don’t worry. I’m required to write down objective evidence supporting conformity with requirements. If I start asking the same question three different ways, and I’m not writing any notes, that means I am having trouble finding evidence of conformity, and I need your help.”  

Step 2 – “What are the inputs that trigger incoming inspection?”

Inputs and outputs of any process refer to both information and physical items. For 100% administrative processes, you may not have any physical items. Incoming inspection, however, has physical goods you receive from suppliers and inspecting. Therefore, the process inputs you are looking for are physical goods and quality system records associated with those goods. For example, if a bunch of titanium round bars were ordered by a buyer in your purchasing department, the physical goods are the titanium bars. The purchase order is one of the quality system records. Other input records that are usually requested to be shipped with the titanium include a packing slip, a certification of analysis, and a dimensional inspection report. It is common to see the incoming inspection activity be delayed because the records are not included with the shipment from the supplier. One recommendation for a process improvement is to require the supplier to send records electronically at the time of shipment instead of sending hardcopies with the product. Statistical inspection sampling plans and work instructions are often confused with input records. These documents are needed to start the incoming inspection, but these are documents that belong in step six of the turtle diagram.

Step 3 – “What are the outputs of the incoming inspection process?”

After incoming inspection is completed there is a requirement to identify the status of the physical product (i.e., accepted or rejected). Usually, a green tag will be used to identify the product as accepted. The tag will also identify the part number, lot, and quantity of product accepted. If the product is titanium, each bar will get a tag. The product will then be transferred to a designated storage area. If you are conducting an audit of a supplier, or a full quality system audit, auditing the warehouse for storage and handling processes is a logical next process. The auditor should look for whether product is segregated in designated locations for specific types of product or if the storage locations are “random” but identified electronically in a material resource planning (MRP) system. The quality system records output from the incoming inspection process will be inspection records and either a green release tag or red rejection tag. If the product is rejected, the product shall be transferred to a quarantine area for nonconforming product and a nonconforming material record (i.e., NCMR) is initiated. Therefore, the process for controlling nonconforming material is another process that could be a logical next process to audit.

Step 4 – “What resources are needed for this process?”

This part of the process approach to auditing is one of the most neglected parts of the quality system. Resources include the facility infrastructure, manufacturing equipment, measurement devices used for inspection, and quality system software used to maintain records of incoming inspection. In this part of the process audit the auditor must be observant. Maintenance records might be located on the side of equipment and they can be reviewed as the auditor walks through the area. This would be an opportunity to interview personnel to make sure they can explain the maintenance process and the equipment maintenance is being performed as planned. The auditor should also determine if equipment validation is required. If the equipment is automated (e.g., automated optical inspection), then an installation qualification (i.e., IQ) should be requested as a quality system record to review at the end of the process or as part of the process for process validation. If the inspection area includes a metrology lab, then the environment may be temperature and humidity controlled. In these types of environments, records of environmental monitoring and trending of environmental conditions should be verified. Lighting, magnification, and particulate filtration could be other environmental requirements for the inspection area. Pest control should be verified in the receiving area, inspection area, and storage areas. The receiving area and warehouse storage are common areas to find pests. Calibration identification should be recorded as a potential follow-up trail for any measurement devices used in the inspection area, and if software is used you will want to verify that quality system software tool validation has been performed.

Step 5 – “Who performs this process?”

A combination of three different roles and responsibilities are typical for this process: 1) department manager, 2) receiving personnel, and 3) inspection personnel. Sometimes one or more of these roles will be combined into one job. The activities sometimes are only performed for a few hours each day, and the personnel that perform the incoming inspection process are assigned to other roles, such as warehouse storage, handling, and shipping. Auditors should always try to interview one or more of the people doing the receiving and inspection activities instead of limiting the interviews to the process owner. Often I will ask the personnel to demonstrate the receiving process and the inspection process. In order to make sure this is possible, you will need to communicate that you want to observe these activities prior to the audit or during the opening meeting. If you don’t, the receiving and inspection activities may already be completed before you start to interview the personnel. Any personnel that are unable to explain the tasks they perform may be targets for verification of training records, effectiveness of training, and competency.

Step 6 – “How is this process performed?”

If an auditor interviews personnel, most people will describe the process in a very haphazard way and steps will be missed. This is why asking people to demonstrate the process is better. The best method is for the person to access the current, approved work instruction or procedure for the process. Then the person should follow the work instruction step-by-step. This allows the person to use the work instruction or procedure as a “crutch” and reduces their nervousness. This also eliminates the skipping steps if the procedures and work instructions are sufficiently detailed. Any blank forms used and statistical inspection standards are also considered quality system documents that define how the process is performed. Sometimes the process owner will provide these documents during their interview, and other times this documents are provided as audit preparation documents. If the documents are not provided in advance the auditor should make sure that they review the documents during observation of activities being performed. This is where an auditor may identify the use of obsolete quality documents, missing details in the documents, and details that are inconsistently followed by personnel.

Step 7 – “What metrics are important for this process?”

Whenever I ask, “What metrics are important in this process?” I typically get a blank stare. Hundreds of business management leaders subscribe to the concept of “what gets managed gets done.” You are also required to establish metrics for your quality system processes in accordance with Clause 8.2.5. Therefore, you need to establish at least one metric, if not more than one. Auditing can help identify opportunities for improvement (OFI), but metrics are the best source of OFIs for a quality system. 

Do you need a closing meeting?

You should always conduct a closing meeting for your audits. However, it is also a best practice to summarize your findings for the process owner before you move on to the next process. If some records remain to be reviewed, ensure the process owner knows that the audit results are pending an outcome of reviewing the remaining records. Consider adopting the “sandwich” approach to presenting your findings: 1) something positive, 2) any nonconformities, and 3) something positive. The approach sandwiches the “bad news” between two pieces of “good news.” If you are working as part of a team, the lead auditor should always be aware of the results of your audit. The manager responsible for the process (i.e., the process owner) should also be aware of the results. Do everything you can to prevent unpleasant surprises at the end of the audit.

When you describe any nonconformities, make sure that you include all of the following information:

  1. the grading of the finding (i.e., MDSAP scoring or Major/Minor)
  2. a single sentence stating the finding
  3. the requirement, including a reference to the applicable regulation or standard
  4. objective evidence from your notes

Whenever possible, email a draft of the wording for your nonconformities to the process owner so they can be prepared with clarification questions during the closing meeting. Make sure you agree with your lead auditor before sending the wording of the finding, and copy them on the email communication. If the process owner has initiated immediate corrective action(s), make sure you note this in your report.

Finalizing your audit report

If you are conducting a supplier audit, you need to give the supplier formal feedback from the audit. You will need an audit report for your quality system records, but you are not required to give the supplier the full report. You might provide a summary of the audit for the supplier instead. If you do this, you should include a copy of that communication in your quality system record (e.g., an appendix to your audit report). If you are going to provide a summary of findings, the content should include at least the following:

  1. positive findings (i.e., strengths)
  2. negative findings (i.e., weaknesses)
  3. nonconformities (if any)
  4. required actions (e.g., supplier corrective action plan)
  5. due date(s) for objective evidence of containment, corrections, and corrective actions
  6. recommendations for follow-up (e.g., next audit)

If you prepare an internal audit report, all of the above content should be included. However, the report should have additional details:

  1. audit purpose
  2. audit scope
  3. audit date(s)
  4. audit criteria
  5. name of participants
  6. date of report
  7. closure of previous audit non-conformities
  8. reference to the audit agenda
  9. deviations, if any, from the agenda
  10. summary of the audit, including any obstructions
  11. objective evidence sampled (i.e., what you looked at and what you looked for)
  12. opportunities for improvement (if any)

Incoming Inspection – How to perform a single process audit Read More »

Artificial Intelligence and Machine Learning Medical Devices

3 Comments / 510(k), Artificial Intelligence (AI) & Machine Learning (ML), Design Change Control / By

The FDA released a new draft guidance document about artificial intelligence and machine learning (AI/ML) functions in medical devices.

What is a predetermined change control plan for artificial intelligence (AI) software?

The new FDA guidance is specific to predetermined change control plans for marketing submissions. The guidance was released on March 30, 2023, but the document is dated April 3, 2023. The draft guidance applies to artificial intelligence (AI) or Machine Learning-Enabled Device Software Functions (ML-DSF), including modifications automatically implemented by the software and modifications to the models implemented manually.

New Artificial Intelligence PCCP Guidance Document 1024x857 Artificial Intelligence and Machine Learning Medical Devices

A PCCP must be authorized through 510k, De Novo, or PMA pathways, as appropriate. The purpose of including a PCCP in a marketing submission is to seek premarket authorization for these intended device modifications without necessitating additional marketing submissions for each change described in the PCCP.

How do you determine if a 510k is required for a device modification, and how would a PCCP affect this?

Currently, there are three guidance documents relating to the evaluation of changes and determination if a new premarket submission is required:

These guidance documents will still be the first steps in evaluating changes. Only changes specific to artificial intelligence (AI) or ML-DSF that would result in a new pre-market submission could be subject to a PCCP.

Examples of Employing AI/ML-DSF PCCPs

  • Retraining a model with more data to improve device performance while maintaining or increasing sensitivity. If this type of change is pre-approved in the PCCP, the labeling can be updated to reflect the improved performance once the change has been implemented. 
  • Extending the scope of compatible hardware with a device system. For example, if the algorithm was initially trained using one specific camera, ultrasound, defined parameter, etc., then a PCCP could add additional cameras/ultrasounds/modified parameters. 
  • Retraining a model to optimize site-specific performance for a specific subset of patients with a particular condition for whom sufficient data was unavailable. The PCCP could expand the indications once such data were available.

What is the difference between a locked vs. adaptive algorithm?

A locked algorithm is a software function involving human input, action, review, and/or decision-making before implementation. Once the algorithm is designed and implemented, it cannot be changed without modifying the source code.

Locked algorithms contrast with adaptive/automatic algorithms, where the software will implement changes without human intervention. The adaptive/automatic algorithms are designed to adjust according to changing input conditions. The adaptive/automatic algorithm is designed to recognize patterns in the input data and adjust its processing accordingly.

Typically locked algorithms apply to fixed functions such as a decision tree, static look-up table, or complex classifier. For AI/ML-DSF, manually implemented algorithms may involve training the algorithm on a new dataset or serving a new function. Once the training is complete, the algorithm will be implemented into the software. Adaptive algorithms are programmed such that their behavior changes over time as it is run based on the information it processes.

As it relates to a PCCP, the detailed description of the intended modifications needs to specify which algorithm type is being modified.

What is included in a PCCP for artificial intelligence (AI) software?

A PCCP should consist of:

  • Detailed Description of Intended Modifications
  • Modification Protocol describing the verification and validation activities, including pre-defined acceptance criteria
  • Impact Assessment identifying the benefits and risks introduced by the changes

The detailed description of the intended modifications should list each proposed device modification and the rationale for each change. If changes require labeling modifications, that should also be described. It should also be clearly stated whether or not the proposed change is intended to be implemented automatically or manually. The description should describe whether the change will be implemented globally across all devices on the market or locally, specific to different devices based on the unique characteristics of the device’s patient or clinical site.

The types of modifications that are appropriate for a PCCP include modifications related to quantitative measurements of ML-DSF performance specifications, changes related to device inputs, and limited modifications relating to the device’s use and performance. The draft guidance provides some examples of each of those modification types. 

The content of the modification protocol section requires a description of planned data management practices relating to the reference standard and annotation process, a description of re-training practices and processing steps, performance evaluation methods and acceptance criteria, and internal procedures for implementing updates. 

The impact assessment is the documentation of the evaluation of the benefits and risks of implementing the PCCP for the software. Any controls or mitigations of the risks should be described in this section. 

Appendix A of the draft guidance includes example elements of modification protocol components for ML-DSFs. Appendix B includes examples of ML-DSF scenarios employing PCCPs.

If, at some point, the manufacturer wants to make changes to the content of the PCCP relating to either the modifications described or the methods used to validate those changes, that generally would require a new marketing submission for the device. 

Utilizing a PCCP in your QMS Change Control System

When evaluating and implementing changes, the manufacturer shall do so in accordance with their Quality Management System change control processes. This should require a review of planned modifications against the FDA guidance documents for evaluating changes and the PCCP. For the change to be acceptable under the PCCP, it must be specified in the Description of Modifications and implemented in conformance with the methods and specifications described in the Modification Protocol. A new premarket submission is required if it does not meet those requirements.

Artificial Intelligence and Machine Learning Medical Devices Read More »

OpenAI and Elsmar never trust their help with regulatory questions?

5 Comments / FDA, Quality Management System, Registration, Regulatory Consultant / By / ,

Everyone has a favorite resource they use to answer regulatory questions, but can you trust OpenAI or Elsmar to answer correctly?

Screenshot 2023 04 01 10.07.10 AM e1680445207847 1024x787 OpenAI and Elsmar never trust their help with regulatory questions?

If you are deathly afraid of trying new technology, the image above is a screen capture from OpenAI describing “itself.” OpenAI is artificial intelligence (AI), but it is not self-aware yet. The image below is a screen capture from the “About” webpage for Elsmar Cove. This article was the oldest post on the Medical Device Academy, and it described how to use Elsmar Cove as a resource for quality systems and regulatory questions. To update that blog, we are comparing the use of OpenAI with Elsmar Cove. Just in case you were wondering, Elsmar Cove is #6 on our list of favorite search tools, and OpenAI is #5:

Screenshot 2023 04 01 10.14.49 AM e1680445245168 1024x548 OpenAI and Elsmar never trust their help with regulatory questions?

Are the answers provided by OpenAI and Elsmar Cove accurate?

To test the accuracy of a common regulatory question, we chose a question we weren’t 100% sure about when a client asked last month. I asked my team, but nobody was 100% certain. Basil Systems is limited to submission and post-market surveillance data. I searched FDA.gov, but it was not clear. Google gave us a link to the FDA website. I asked a couple of ex-FDA consultants, but they gave me outdated information. On Thursday, March 30, 2023, I asked Lisa King during an AAMI course I was co-teaching. Lisa is a Consumer Safety Officer at the FDA responsible for reviewing device entries into the FDA. She is also in very high demand for public training courses. She said the contract manufacturers used to be exempt from registration if they shipped to a legal manufacturer first. The regulations changed, and now 100% of contract manufacturers making a finished device must register with the FDA. She also clarified that the FDA doesn’t use the term “legal manufacturer.”

Screenshot 2023 04 01 11.00.46 AM e1680445277287 1024x676 OpenAI and Elsmar never trust their help with regulatory questions?

As you can see from the above answer provided by OpenAI, the ChatGPT engine [i.e., Model: Default (GPT-3.5)] effectively produces the correct answer. Using the same wording for the regulatory question, “Does a foreign contract manufacturer need to register with the FDA if they are shipping the medical device to the legal manufacturer first before the device is exported to the USA?” there were no results from Elsmar Cove. After several attempts, I found what I was looking for using the following search terms, “FDA registration of contract manufacturers.” There were multiple related search results, but the most useful discussion threads in the Elsmar Cove discussion forum were:

The most succinct correct answer in the forum is copied below.

Screenshot 2023 04 01 11.39.43 AM e1680445334745 1024x332 OpenAI and Elsmar never trust their help with regulatory questions?

Can you trust OpenAI and Elsmar Cove to answer your regulatory questions?

OpenAI is only as effective as the data used to train it. This is constantly evolving, but we have identified search results that were 100% accurate, results that were outdated, and results that were scary wrong. The same is true of discussion forums. Elsmar Cove is one of the best discussion forums for the medical device industry, but people also use ASQ, RAPS, and AAMI. The quality of the information provided depends upon the knowledge and experience of the people participating in the forum, but it also depends upon the forum’s moderation. Elsmar Cove has some experienced moderators with decades of experience. There is always the chance that the most experienced person in the world could answer your regulatory question incorrectly. This usually creates a problem because everyone else in the forum hesitates to challenge a recognized expert. Therefore, regardless of which resource(s) you use, always try to get a reference to the trustworthy source of the applicable regulation. Even Lisa King could make a mistake, but she immediately said, you can find the regulations in the US Code of Federal Regulations (i.e., 21 CFR 807). The bottom line is, always do your fact-checking and reference your source(s).

OpenAI and Elsmar never trust their help with regulatory questions? Read More »

How quickly will RTA policy take effect for cybersecurity devices?

2 Comments / 510(k), Cybersecurity / By

Breaking news! The FDA just released new guidance on the refusal to accept (RTA) policy for cybersecurity devices.

Picture of new FDA guidance on RTA policy for cybersecurity devices 838x1024 How quickly will RTA policy take effect for cybersecurity devices?

Where can I find the new cybersecurity devices guidance?

The new guidance is titled “Cybersecurity in Medical Devices: Refuse to Accept Policy for Cyber Devices and Related Systems Under Section 524B of the FD&C Act,” and you can download a copy of the PDF directly from our website. This is the first time the FDA has created a definition for a “cyber device,” but this guidance is specific to the refusal to accept policy (RTA) rather than guidance for the format and content of pre-market notification (i.e., 510k) If you want to learn about new guidance documents as they are released, we recommend that you sign up for FDA email notifications. If you want to be notified of when our new blogs are posted, subscribe to our blog email notification list on this page.

What is a “cyber device” in the context of this cybersecurity devices guidance and submissions?

This new guidance defines “cyber device” using the following language:

  1. includes software validated, installed, or authorized by the sponsor as a device or in a device;
  2. has the ability to connect to the internet; and
  3. contains any such technological characteristics validated, installed, or authorized by the sponsor that could be vulnerable to cybersecurity threats.

What does “refusal to accept” (RTA) mean?

“Refusal to accept” or (RTA) is a policy that the FDA implemented for pre-market notification submissions (i.e., 510k) in 2012. The process occurs during the first 15 calendar days of the FDA review process. The FDA assigns a preliminary reviewer to perform the RTA screening of the submission, and the person completes an RTA checklist. The FDA substitutes an RTA screening with a technical screening for FDA eSTAR templates, and this is one of the reasons why Medical Device Academy uses the FDA eSTAR templates for all 510k submissions and De Novo classification requests instead of using the older 510k format and content requirements with 20 sections.

When will the FDA begin rejecting submissions during the RTA processes?

The FDA states directly in the guidance document that they will not reject submissions for cybersecurity for the balance of FY 2023 (i.e., before October 1, 2023). The wording used by the FDA is: “The FDA generally intends not to issue “refuse to accept” (RTA) decisions for premarket submissions for cyber devices that are submitted before October 1, 2023, based solely on information required by section 524B of the FD&C Act. Instead, the FDA will work collaboratively with sponsors of such premarket submissions as part of the interactive and/or deficiency review process.” We believe the FDA will update the eSTAR template to include requirements for cybersecurity on October 1, 2023. It will not be possible to submit a 510k that does not include the cybersecurity requirements in future eSTAR templates, because the eSTAR automatically verifies the completion of each section in the template.

Will there be another cybersecurity guidance released soon?

The FDA announced last October that a new cybersecurity guidance would be replacing the 2014 final guidance for cybersecurity. A draft was released in 2018, and an updated draft was released in 2022. The final updated guidance is included in the A-list of FDA priorities for final guidance documents, but the updated final version has not been released yet. The FDA webpage for cybersecurity was updated to include this new guidance on RTA policy for cybersecurity devices. We believe this indicates that the updated final version will be released soon. When it is released, we will publish a new blog about that guidance.

How quickly will RTA policy take effect for cybersecurity devices? Read More »

Nine easy ways to organize and improve quality system procedures

2 Comments / ISO 13485:2016, ISO Certification, Quality Management System / By / , ,

Would you like to learn nine ways to improve your quality system procedures? One method is precisely the opposite of our advice from 2011.

During a CAPA course I taught on Friday, January 28, 2011, one of the attendees asked if we teach a course on “How to write better quality system procedures.” Unfortunately, we could only offer material from a course about “Training the trainer.” That “Training the trainer” course focused on visual communication. Several books related to Lean Manufacturing explain how to use visual communication to replace text (i.e., “a picture says a thousand words”). During my ride home, however, I thought of a few other ideas that might help anyone writing or re-writing a procedure. The article was updated and posted as a new blog on Tuesday, March 28, 2023.

1. Use a standardized template for your procedures

In 2013 we published a blog about using a procedure template where we described our 12-part procedure template (i.e., TMP-001).  You don’t have to mimic our template, but using a template will accelerate the speed of your writing when you create procedures, and it makes sure you don’t forget any of the essential elements. In addition, using templates ensures a consistent format that makes it easier for everyone to find the information they are looking for. Just make sure that your document control procedure allows flexibility to deviate from the template. The ISO 13485:2016 standard does require a “mandatory” format. Referring to your template as “suggested formatting” will avoid unnecessary nonconformities.

2. Create a process “turtle diagram” for each quality procedure

All of the procedures that Medical Device Academy created have a flow chart at the beginning of the procedure showing the procedures and forms associated with processes that are inputs to that procedure and outputs from that procedure. To systematically improve our procedures, we will be systematically replacing those flow charts with turtle diagrams for each process. This will give more detail than our current flow charts, and internal and external auditors can use the turtle diagrams to understand process interactions.

3. Avoid making unnecessary references to regulations and standards

If you are writing a procedure on risk management—it makes sense to reference ISO 14971. It does not make sense to reference all the other risk analysis standards unless you specifically use them to perform risk analysis. ISO 14971:2019, Clause 4.1, also states that you “shall establish, implement, document, and maintain an ongoing process for” risk management activities. However, the ISO 14971 standard is not directly linked to other procedures. Therefore, ISO 14971 should only be referenced in another if you are using it in that procedure or referencing it directly. For example, the Quality Manual (i.e., POL-001) explicitly references ISO 14971. In contrast, the design control procedure (i.e., SYS-008) references the risk management procedure (SYS-010) but doesn’t reference ISO 14971.
Concerning regulations, you should only reference regulations if the procedure meets a specific requirement. Color coding with symbols should demonstrate traceability to requirements (see method #5 below for further explanation). Rather than adding a reference to regulations in a procedure where there is no requirement, a better approach is to indicate in the Quality Manual that only procedures that have specific requirements will reference the regulations, such as 21 CFR 820 or Part 1 of the Canadian MDR.

4. Track standards, regulations, and the version used in your procedures

In the original 2011 version of this article, we advised quality managers to “avoid including the revision of a standard” because “this is just another opportunity for unnecessary nonconformities.” However, we find that our team has trouble identifying every procedure that a change in regulation or a standard might impact. A systematic process is needed to identify every procedure referencing a regulation or standard. Therefore, we will reference all impacted procedures next to the regulation or standard in our Master Document List (i.e., LST-001). References to the regulations will be added to the main tab for policies, procedures, and work instructions (i.e., [POL, SYS, and WI]). References to the standards will be added to the tab for documents of external origin (i.e., [Doc Ext Origin]).

Many people feel that you should not reference the version of a standard in a procedure because adding the version of the standard increases the number of documents that need to be updated when a standard changes. However, if you are only referencing standards in procedures when it is necessary, then that procedure should be reviewed and updated for the need to be changed. Updating the version of the Standard referenced is the best way to document that a gap analysis against the new version has been completed and the necessary updates were made to the procedure.

5. Use color coding and symbols in your quality system procedures

Example of Cluase Cross references in quality system procedures 1024x517 Nine easy ways to organize and improve quality system procedures

Matthew Walker, Medical Device Academy’s manager of the human factors team, has systematically updated many of our procedures to the EU Medical Device Regulations 2017/745 and the In Vitro Diagnostic Regulations 2017/746. When he updates our procedures, he references the regulations and applicable ISO 13485:2016 clauses. During certification audits, certification body auditors sometimes have difficulty finding where specific requirements are located in the procedures. Therefore, Matthew added color-coded clause references for our clients and auditors as a corrective action. To make the procedures inclusive for people that are color-blind, Matthew added symbols to supplement the color coding. The extra addition of symbols has proven invaluable because now anyone can search the documents electronically for a symbol to find where all the references are located.

6. Indicate the process owner and training requirements associated with each procedure

Identifying the process owner and training requirements in every procedure makes it easier to define who is responsible for reviewing and revising procedures. For the training requirements, the process owner should specify who needs to be trained on the process. Why? They know the procedure best. If there is a “grey area,” this should be resolved with the department manager for the job function. In addition, retraining requirements should be specified. The training section should also clarify if retraining is required when revising a procedure. If the revision is minor, training should only be necessary for people not trained on a previous revision.

7. Adopt the Plan-Do-Check-Act (PDCA) model for the structure of quality system procedures

For the “Plan” portion, the procedure should explain how to prepare to do something. This planning activity can apply to anything from planning to perform an audit to planning to inspect incoming raw materials. The “Do” portion is what most people refer to as the “Procedure” section. The “Check” portion of the procedure is a great place to specify the monitoring and measurement requirements for the process (see Section 8.1 of the Standard). Finally, the “Act” portion of the procedure should indicate what to do when target metrics are unmet. For example, what should be done when an alert limit is reached? What should be done when an action limit is reached?

8. Include the revision history of quality system procedures

It’s helpful to know which Document Change Notice (DCN) approved the document revision, why the changes were made, the nature of the changes, whether there is a related corrective action, and when the change was made. This will also tell auditors whether there is anything new to audit since the previous internal or external audit. This section is usually near the beginning of our procedures, but it doesn’t matter if the revision history is at the end or the beginning. However, it does help to be consistent.

9. Identify the form number, location, and retention period for each record

We have a section about quality system records near the end of every procedure. This section lists each quality system record that is associated with the procedure. The relevant form is referenced, but we recommend storing these records in electronic or paper folders labeled with the form number. If the files are digital, a hyperlink should be included. If the files are paper, then you should list the physical location of storage. The retention period can be listed in each procedure. Still, it will be essential to ensure that this information matches the regulatory requirements and record retention requirements in your “Control of Records” procedure (i.e., SYS-002).

 

Nine easy ways to organize and improve quality system procedures Read More »

Scroll to Top