Archive for Regulatory Consultant

OpenAI and Elsmar never trust their help with regulatory questions?

Everyone has a favorite resource they use to answer regulatory questions, but can you trust OpenAI or Elsmar to answer correctly?

Screenshot 2023 04 01 10.07.10 AM e1680445207847 1024x787 OpenAI and Elsmar never trust their help with regulatory questions?

If you are deathly afraid of trying new technology, the image above is a screen capture from OpenAI describing “itself.” OpenAI is artificial intelligence (AI), but it is not self-aware yet. The image below is a screen capture from the “About” webpage for Elsmar Cove. This article was the oldest post on the Medical Device Academy, and it described how to use Elsmar Cove as a resource for quality systems and regulatory questions. To update that blog, we are comparing the use of OpenAI with Elsmar Cove. Just in case you were wondering, Elsmar Cove is #6 on our list of favorite search tools, and OpenAI is #5:

Screenshot 2023 04 01 10.14.49 AM e1680445245168 1024x548 OpenAI and Elsmar never trust their help with regulatory questions?

Are the answers provided by OpenAI and Elsmar Cove accurate?

To test the accuracy of a common regulatory question, we chose a question we weren’t 100% sure about when a client asked last month. I asked my team, but nobody was 100% certain. Basil Systems is limited to submission and post-market surveillance data. I searched, but it was not clear. Google gave us a link to the FDA website. I asked a couple of ex-FDA consultants, but they gave me outdated information. On Thursday, March 30, 2023, I asked Lisa King during an AAMI course I was co-teaching. Lisa is a Consumer Safety Officer at the FDA responsible for reviewing device entries into the FDA. She is also in very high demand for public training courses. She said the contract manufacturers used to be exempt from registration if they shipped to a legal manufacturer first. The regulations changed, and now 100% of contract manufacturers making a finished device must register with the FDA. She also clarified that the FDA doesn’t use the term “legal manufacturer.”

Screenshot 2023 04 01 11.00.46 AM e1680445277287 1024x676 OpenAI and Elsmar never trust their help with regulatory questions?

As you can see from the above answer provided by OpenAI, the ChatGPT engine [i.e., Model: Default (GPT-3.5)] effectively produces the correct answer. Using the same wording for the regulatory question, “Does a foreign contract manufacturer need to register with the FDA if they are shipping the medical device to the legal manufacturer first before the device is exported to the USA?” there were no results from Elsmar Cove. After several attempts, I found what I was looking for using the following search terms, “FDA registration of contract manufacturers.” There were multiple related search results, but the most useful discussion threads in the Elsmar Cove discussion forum were:

The most succinct correct answer in the forum is copied below.

Screenshot 2023 04 01 11.39.43 AM e1680445334745 1024x332 OpenAI and Elsmar never trust their help with regulatory questions?

Can you trust OpenAI and Elsmar Cove to answer your regulatory questions?

OpenAI is only as effective as the data used to train it. This is constantly evolving, but we have identified search results that were 100% accurate, results that were outdated, and results that were scary wrong. The same is true of discussion forums. Elsmar Cove is one of the best discussion forums for the medical device industry, but people also use ASQ, RAPS, and AAMI. The quality of the information provided depends upon the knowledge and experience of the people participating in the forum, but it also depends upon the forum’s moderation. Elsmar Cove has some experienced moderators with decades of experience. There is always the chance that the most experienced person in the world could answer your regulatory question incorrectly. This usually creates a problem because everyone else in the forum hesitates to challenge a recognized expert. Therefore, regardless of which resource(s) you use, always try to get a reference to the trustworthy source of the applicable regulation. Even Lisa King could make a mistake, but she immediately said, you can find the regulations in the US Code of Federal Regulations (i.e., 21 CFR 807). The bottom line is, always do your fact-checking and reference your source(s).

Posted in: FDA, Quality Management System, Registration, Regulatory Consultant

Leave a Comment (2) →