Everyone has a favorite resource they use to answer regulatory questions, but can you trust OpenAI or Elsmar to answer correctly?
If you are deathly afraid of trying new technology, the image above is a screen capture from OpenAI describing “itself.” OpenAI is artificial intelligence (AI), but it is not self-aware yet. The image below is a screen capture from the “About” webpage for Elsmar Cove. This article was the oldest post on the Medical Device Academy, and it described how to use Elsmar Cove as a resource for quality systems and regulatory questions. To update that blog, we are comparing the use of OpenAI with Elsmar Cove. Just in case you were wondering, Elsmar Cove is #6 on our list of favorite search tools, and OpenAI is #5:
Search the FDA.gov device databases – 3rd place, you can email DICE or call 301.796.7100
Search Google (world’s #1 search engine) – 4th place, ask any 5-year-old for help
Are the answers provided by OpenAI and Elsmar Cove accurate?
To test the accuracy of a common regulatory question, we chose a question we weren’t 100% sure about when a client asked last month. I asked my team, but nobody was 100% certain. Basil Systems is limited to submission and post-market surveillance data. I searched FDA.gov, but it was not clear. Google gave us a link to the FDA website. I asked a couple of ex-FDA consultants, but they gave me outdated information. On Thursday, March 30, 2023, I asked Lisa King during an AAMI course I was co-teaching. Lisa is a Consumer Safety Officer at the FDA responsible for reviewing device entries into the FDA. She is also in very high demand for public training courses. She said the contract manufacturers used to be exempt from registration if they shipped to a legal manufacturer first. The regulations changed, and now 100% of contract manufacturers making a finished device must register with the FDA. She also clarified that the FDA doesn’t use the term “legal manufacturer.”
As you can see from the above answer provided by OpenAI, the ChatGPT engine [i.e., Model: Default (GPT-3.5)] effectively produces the correct answer. Using the same wording for the regulatory question, “Does a foreign contract manufacturer need to register with the FDA if they are shipping the medical device to the legal manufacturer first before the device is exported to the USA?” there were no results from Elsmar Cove. After several attempts, I found what I was looking for using the following search terms, “FDA registration of contract manufacturers.” There were multiple related search results, but the most useful discussion threads in the Elsmar Cove discussion forum were:
The most succinct correct answer in the forum is copied below.
Can you trust OpenAI and Elsmar Cove to answer your regulatory questions?
OpenAI is only as effective as the data used to train it. This is constantly evolving, but we have identified search results that were 100% accurate, results that were outdated, and results that were scary wrong. The same is true of discussion forums. Elsmar Cove is one of the best discussion forums for the medical device industry, but people also use ASQ, RAPS, and AAMI. The quality of the information provided depends upon the knowledge and experience of the people participating in the forum, but it also depends upon the forum’s moderation. Elsmar Cove has some experienced moderators with decades of experience. There is always the chance that the most experienced person in the world could answer your regulatory question incorrectly. This usually creates a problem because everyone else in the forum hesitates to challenge a recognized expert. Therefore, regardless of which resource(s) you use, always try to get a reference to the trustworthy source of the applicable regulation. Even Lisa King could make a mistake, but she immediately said, you can find the regulations in the US Code of Federal Regulations (i.e., 21 CFR 807). The bottom line is, always do your fact-checking and reference your source(s).
Would you like to learn nine ways to improve your quality system procedures? One method is precisely the opposite of our advice from 2011.
During a CAPA course I taught on Friday, January 28, 2011, one of the attendees asked if we teach a course on “How to write better quality system procedures.” Unfortunately, we could only offer material from a course about “Training the trainer.” That “Training the trainer” course focused on visual communication. Several books related to Lean Manufacturing explain how to use visual communication to replace text (i.e., “a picture says a thousand words”). During my ride home, however, I thought of a few other ideas that might help anyone writing or re-writing a procedure. The article was updated and posted as a new blog on Tuesday, March 28, 2023.
1. Use a standardized template for your procedures
In 2013 we published a blog about using a procedure template where we described our 12-part procedure template (i.e., TMP-001). You don’t have to mimic our template, but using a template will accelerate the speed of your writing when you create procedures, and it makes sure you don’t forget any of the essential elements. In addition, using templates ensures a consistent format that makes it easier for everyone to find the information they are looking for. Just make sure that your document control procedure allows flexibility to deviate from the template. The ISO 13485:2016 standard does require a “mandatory” format. Referring to your template as “suggested formatting” will avoid unnecessary nonconformities.
2. Create a process “turtle diagram” for each quality procedure
All of the procedures that Medical Device Academy created have a flow chart at the beginning of the procedure showing the procedures and forms associated with processes that are inputs to that procedure and outputs from that procedure. To systematically improve our procedures, we will be systematically replacing those flow charts with turtle diagrams for each process. This will give more detail than our current flow charts, and internal and external auditors can use the turtle diagrams to understand process interactions.
3. Avoid making unnecessary references to regulations and standards
If you are writing a procedure on risk management—it makes sense to reference ISO 14971. It does not make sense to reference all the other risk analysis standards unless you specifically use them to perform risk analysis. ISO 14971:2019, Clause 4.1, also states that you “shall establish, implement, document, and maintain an ongoing process for” risk management activities. However, the ISO 14971 standard is not directly linked to other procedures. Therefore, ISO 14971 should only be referenced in another if you are using it in that procedure or referencing it directly. For example, the Quality Manual (i.e., POL-001) explicitly references ISO 14971. In contrast, the design control procedure (i.e., SYS-008) references the risk management procedure (SYS-010) but doesn’t reference ISO 14971.
Concerning regulations, you should only reference regulations if the procedure meets a specific requirement. Color coding with symbols should demonstrate traceability to requirements (see method #5 below for further explanation). Rather than adding a reference to regulations in a procedure where there is no requirement, a better approach is to indicate in the Quality Manual that only procedures that have specific requirements will reference the regulations, such as 21 CFR 820 or Part 1 of the Canadian MDR.
4. Track standards, regulations, and the version used in your procedures
In the original 2011 version of this article, we advised quality managers to “avoid including the revision of a standard” because “this is just another opportunity for unnecessary nonconformities.” However, we find that our team has trouble identifying every procedure that a change in regulation or a standard might impact. A systematic process is needed to identify every procedure referencing a regulation or standard. Therefore, we will reference all impacted procedures next to the regulation or standard in our Master Document List (i.e., LST-001). References to the regulations will be added to the main tab for policies, procedures, and work instructions (i.e., [POL, SYS, and WI]). References to the standards will be added to the tab for documents of external origin (i.e., [Doc Ext Origin]).
Many people feel that you should not reference the version of a standard in a procedure because adding the version of the standard increases the number of documents that need to be updated when a standard changes. However, if you are only referencing standards in procedures when it is necessary, then that procedure should be reviewed and updated for the need to be changed. Updating the version of the Standard referenced is the best way to document that a gap analysis against the new version has been completed and the necessary updates were made to the procedure.
5. Use color coding and symbols in your quality system procedures
Matthew Walker, Medical Device Academy’s manager of the human factors team, has systematically updated many of our procedures to the EU Medical Device Regulations 2017/745 and the In Vitro Diagnostic Regulations 2017/746. When he updates our procedures, he references the regulations and applicable ISO 13485:2016 clauses. During certification audits, certification body auditors sometimes have difficulty finding where specific requirements are located in the procedures. Therefore, Matthew added color-coded clause references for our clients and auditors as a corrective action. To make the procedures inclusive for people that are color-blind, Matthew added symbols to supplement the color coding. The extra addition of symbols has proven invaluable because now anyone can search the documents electronically for a symbol to find where all the references are located.
6. Indicate the process owner and training requirements associated with each procedure
Identifying the process owner and training requirements in every procedure makes it easier to define who is responsible for reviewing and revising procedures. For the training requirements, the process owner should specify who needs to be trained on the process. Why? They know the procedure best. If there is a “grey area,” this should be resolved with the department manager for the job function. In addition, retraining requirements should be specified. The training section should also clarify if retraining is required when revising a procedure. If the revision is minor, training should only be necessary for people not trained on a previous revision.
7. Adopt thePlan-Do-Check-Act (PDCA) model for the structure of quality system procedures
For the “Plan” portion, the procedure should explain how to prepare to do something. This planning activity can apply to anything from planning to perform an audit to planning to inspect incoming raw materials. The “Do” portion is what most people refer to as the “Procedure” section. The “Check” portion of the procedure is a great place to specify the monitoring and measurement requirements for the process (see Section 8.1 of the Standard). Finally, the “Act” portion of the procedure should indicate what to do when target metrics are unmet. For example, what should be done when an alert limit is reached? What should be done when an action limit is reached?
8. Include therevision history of quality system procedures
It’s helpful to know which Document Change Notice (DCN) approved the document revision, why the changes were made, the nature of the changes, whether there is a related corrective action, and when the change was made. This will also tell auditors whether there is anything new to audit since the previous internal or external audit. This section is usually near the beginning of our procedures, but it doesn’t matter if the revision history is at the end or the beginning. However, it does help to be consistent.
9. Identify the form number, location, and retention period for each record
We have a section about quality system records near the end of every procedure. This section lists each quality system record that is associated with the procedure. The relevant form is referenced, but we recommend storing these records in electronic or paper folders labeled with the form number. If the files are digital, a hyperlink should be included. If the files are paper, then you should list the physical location of storage. The retention period can be listed in each procedure. Still, it will be essential to ensure that this information matches the regulatory requirements and record retention requirements in your “Control of Records” procedure (i.e., SYS-002).
Process monitoring is required but do you know whether monitoring every procedure is required by the FDA QSR or ISO 13485?
One of the elements that Medical Device Academy has incorporated into each procedure we created in our turnkey quality system is a section titled, “monitoring and measurement.” The purpose of this section is to force each process owner to identify a process metric for monitoring every procedure. In some cases, we suggest a metric that would be appropriate for most companies establishing a new quality system. In other procedures, we use the following default text:
Where are the requirements for process monitoring in 21 CFR 820?
Some of the companies that have purchased our turnkey quality system have asked, “Is it required to monitor and measure something in every procedure?” In general, it is not a specific requirement to have a metric specified in each procedure. In fact, if your quality system is not ISO 13485 certified, there are actually only a few places where the US FDA requires monitoring. The FDA does not have a section specific to monitoring and measurement of processes, but there is a section of the regulations specific to statistical techniques (i.e. 21 CFR 820.250). However, it does not state in the QSR that statistical analysis is required for all processes. In fact, there are only six instances where the word “statistical” is used:
21 CFR 820.100(a)(1) – “Analyzing processes, work operations, concessions, quality audit reports, quality records, service records, complaints, returned product, and other sources of quality data to identify existing and potential causes of nonconforming product, or other quality problems. Appropriate statistical methodology shall be employed where necessary to detect recurring quality problems;”
21 CFR 820.200(b) – “Each manufacturer shall analyze service reports with appropriate statistical methodology in accordance with § 820.100.”
21 CFR 820.250 – “(a) Where appropriate, each manufacturer shall establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics. (b) Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.” Note: the other two instances are the title of 21 CFR 820.250.
The word “monitoring” is equally rare (i.e. 4x) in the QSR:
21 CFR 820.70(a) – “Each manufacturer shall develop, conduct, control, and monitor production processes to ensure that a device conforms to its specifications…Where process controls are needed…(2) Monitoring and control of process parameters and component and device characteristics during production.”
21 CFR 820.75(b) – “Each manufacturer shall establish and maintain procedures for monitoring and control of process parameters for validated processes to ensure that the specified requirements continue to be met…(2) For validated processes, the monitoring and control methods and data, the date performed, and, where appropriate, the individual(s) performing the process or the major equipment used shall be documented.”
Where are the requirements for process monitoring in ISO 13485:2016?
ISO 13485:2016 has a section specific to monitoring and measurement of processes (i.e. Clause 8.2.5). In addition, the word “monitoring” occurs 52 times in the standard and there are 60 incidents of some variant or the exact word. , but there is a section of the regulations specific to statistical techniques (i.e. 21 CFR 820.250). However, it does not state in the QSR that statistical analysis is required for all processes. In fact, there are only six instances where the word “statistical” is used. There are four Clause headings that actually include the word monitoring:
Clause 7.6, Control of monitoring and measuring equipment
Clause 8.2, Monitoring and measurement
Clause 8.2.5, Monitoring and measurement of processes
Clause 8.2.6, Monitoring and measurement of product
In Clause 1, Scope, and Clause 4.1.5, the Standard states that any outsourced processes remain the responsibility of the company and must be accounted for in the quality system by monitoring, maintaining, and controlling the processes.
Monitoring of risk is included in the definition of “risk management” in the Standard (i.e. Clause 3.18).
Clause 4.1.3 states that the organization shall, “b) ensure the availability of resources and information necessary to support the operation and monitoring of these processes…d) monitor, measure as appropriate, and analyze these processes.”
Clause 4.2.3 states that the contents of the Medical Device File (i.e. MDR or TF), shall include, “d) procedures for measuring and monitoring.”
Monitoring and measurement of processes and product are required inputs to the Management Review in Clauses 5.6.2e) and f).
Clause 6.4.1 requires a procedure for monitoring the work environment if it can have an effect on product quality.
Clause 7.1 requires the company to consider including monitoring in product realization planning.
Clause 7.4.1 requires a plan for monitoring suppliers.
Clause 7.5.1 requires monitoring production and service, including the monitoring of process parameters and product characteristics.
Clause 7.5.6 requires monitoring of validated process parameters.
Clause 7.5.8 requires identification of status with regard to product monitoring and measurement (i.e. inspection status).
Clause 7.6 requires monitoring and measurement of calibrated devices and validation of any computer software used to monitor calibrated devices.
Clause 8.1 states that companies shall plan and implement monitoring and measurement of processes.
Clause 8.2 is titled, “Monitoring and measurement.”
Clause 8.2.1 requires monitoring of customer feedback.
Clause 8.2.5 requires monitoring of processes to ensure planned results are achieved.
Clause 8.2.6 requires monitoring of products to ensure product requirements have been met.
Clause 8.4 requires data analysis of monitoring data from at least six different processes:
Feedback
Conformity to product requirements
Characteristics and trends of processes and products, including opportunities for improvement
Suppliers
Audits
Service reports, as appropriate
In summary, while not every single clause that requires a procedure includes a requirement for monitoring, there are a number of processes where the requirement to monitor the process is explicitly stated.
Why do all of our procedures include the requirement for metrics?
Medical Device Academy expanded the requirement for monitoring to all procedures for five reasons:
Quality objectives must be “established at relevant functions and levels within the organization.” Therefore, establishing monitoring requirements for each procedure ensures that top management has metrics for every process and a lack of data is never an excuse for not establishing a new quality objective when improvement is needed.
If every procedure has a requirement for monitoring, then employees don’t have to remember which processes require monitoring and which processes do not explicitly require monitoring.
The process approach to auditing includes metrics of the process as one of the seven items that are included in every process turtle diagram, and therefore, including metrics for each procedure facilitates the process approach to auditing.
If a company does not have a process metric already established, it is often difficult to perform an investigation of the root cause of quality issues. If a metric is already being monitored for the process, this facilitates the investigation of the root cause and you can use the baseline monitoring data to help you establish effectiveness criteria for the corrective action.
Finally, most companies struggle to identify preventive actions as required by Clause 8.5.3, and we have found that data analysis of monitoring data is the best source of identifying new preventive actions.
What are the disadvantages when you monitor and measure something in every procedure?
The primary reason for resistance to identifying a metric for monitoring in every procedure is that it will increase the workload for the employees responsible for that process. However, monitoring of data does not always increase workload. In fact, when process data is recorded in real-time on a run chart it is often possible to identify a trend much earlier than when data is simply recorded and subjected to monitoring.
Example #1: The automatic tracking of toner in a printer tells HP when to ship you a new toner cartridge before you need it. This ensures that there is no loss in productivity because you never run out of ink or the ability to print documents.
Example #2: Companies will use project management software (e.g. Monday.com) to monitor labor utilization. This will help identify when a specific resource is nearing capacity. When this occurs, the project manager can add time buffers to prerequisite steps and adjust the starting date of the resource-limited tasks to an earlier starting date. This ensures that more time is available to finish the task or to take advantage of resource availability at an earlier date.
Example #3: Monitoring the revision date for procedures helps the document control process owner identify procedures that should be evaluated for the need to be revised and updated. Often this is articulated as a quality objective of reviewing and updating all procedures within 2 years. This also ensures that procedures remain current and compliant with regulatory requirements.
What are the advantages of monitoring every procedure?
The phrase “what gets measured gets managed” is a popular business philosophy that implies measuring employee activity increases the likelihood that employees will complete a task or perform it well. In contrast, if a process is not monitored, employees may assume that it is not important and the tasks may be skipped or completely forgotten. Setting quantitative goals is also sometimes integrated with economic incentives or bonuses that are granted to individuals and teams.
FDA transition from QSR to ISO 13485
The US FDA is planning its transition from 21 CFR 820 to ISO 13485 as the quality system criteria. This will force companies to make adjustments to their quality systems and increase the amount of process monitoring performed. My general advice is to work with employees that are performing tasks to identify streamlined methods for monitoring those tasks without being overly burdensome. Then you and the employees you manage can analyze the data together and identify opportunities for improvement. When you do this, experiment with manual methods using whiteboards and paper charts that are visible in public areas first. Only implement automated solutions after you have optimized the data being collected and the frequency of data collection, and remember that not every process will benefit from automated statistical process control. Sometimes the simple approach is best.
This article describes what a gap analysis is in the context of managing your quality system when standards and regulations are updated.
Compliance Assessment/Gap Analysis
What is a Gap Analysis? An introductory look.
Well, that depends on the context. The dictionary definition is “A technique that businesses use to determine what steps need to be taken in order to move from its current state to its desired, future state. Also called need-gap analysis, needs analysis, and needs assessment.”
For the most part, this is correct, but we need to tweak it just a little bit to fit better into our regulatory affairs niche, specifically medical device manufacturers. A gap analysis for financial investment or an advertising firm will be very different than one for a medical device distributor. It might even be better served to be called a compliance assessment/gap analysis, but I am sure someone else has thought of that long before me.
For our purposes, the gap analysis is a formal comparative review of an internal process or procedure against a standard, good practice, law, regulation, etc. This blog article will be an introductory look into that process. We also created a procedure case study that shows how a gap analysis can be used to review your management procedure against the requirements in ISO 13485:2016.
What are the two BIG goals of a Gap Analysis?
It sounds like a simple exercise, but the gap analysis or “GA” for short can have two very different but complementary functions. Rather than simply hunting for areas of non-compliance, the first goal is to find and demonstrate areas of compliance.
The second more obvious goal is to find the gaps between the process and the regulatory requirements they are being compared against.
Why is demonstrating compliance important?
Because this is a formal documented review, a gap analysis provides documentation in a traceable manner of meeting the requirements that have been laid out. That traceability is important because it allows anyone to read the report, see the requirement, and locate the area of the procedure that demonstrates conformity with that requirement.
The report itself is an objective tool, not something that is meant to be a witch hunt. The gap analysis will compare document contents. If you want to verify that the entire process is fully compliant, you will need to dig deeper and observe if the activities laid out within the procedure are being performed per the procedure instructions. It is possible to draft procedures that are compliant with text requirements but non-compliant in the manner that the actions are being performed and documented.
What about gaps?
The gaps, or areas of non-compliance highlight opportunities for improvement, if there are any. A gap assessment may not find any gaps and present a report that clearly and neatly outlines and explains how each regulatory requirement is being met.
If there are any gaps identified, that does not mean that there is cause for concern. This should be viewed instead as an opportunity for improvement. Standards and procedures change over time, and, naturally, procedures and processes will have to change with them.
The very act of the gap analysis shows that there is a documented effort towards continual improvement as long as the gaps are addressed.
Addressing the Gaps
The report is ideally the first and last step, and you have a wonderful piece of paper to show that someone checked, and all of the required areas are being met. However, this is not always the case. When there are gaps, they must be filled.
Addressing a gap should happen in a traceable manner, one that shows it was identified, acknowledged, and then how it was fixed. Something that might be addressed through your CAPA process, but that is a topic for a different time.
In Closing
The compliance assessment/gap analysis is a singular tool used in the overall maintenance of a quality system. Its actions and performance are similar to a simplified type of audit, but the gap analysis itself is not going to replace your regularly scheduled audit activities. However, it will help you monitor and keep your fingers on the overall pulse of your quality system. This is also especially helpful in situations where standards and regulations are updated, and your quality system needs to be evaluated and updated accordingly.
For more in-depth education in specialized areas of the assessment, look into our training on Technical File Auditing for MDR compliance against Regulation (EU) 2017/745 at the link below.
