OpenAI and Elsmar never trust their help with regulatory questions?

Everyone has a favorite resource they use to answer regulatory questions, but can you trust OpenAI or Elsmar to answer correctly?

Screenshot 2023 04 01 10.07.10 AM e1680445207847 1024x787 OpenAI and Elsmar never trust their help with regulatory questions?

If you are deathly afraid of trying new technology, the image above is a screen capture from OpenAI describing “itself.” OpenAI is artificial intelligence (AI), but it is not self-aware yet. The image below is a screen capture from the “About” webpage for Elsmar Cove. This article was the oldest post on the Medical Device Academy, and it described how to use Elsmar Cove as a resource for quality systems and regulatory questions. To update that blog, we are comparing the use of OpenAI with Elsmar Cove. Just in case you were wondering, Elsmar Cove is #6 on our list of favorite search tools, and OpenAI is #5:

Screenshot 2023 04 01 10.14.49 AM e1680445245168 1024x548 OpenAI and Elsmar never trust their help with regulatory questions?

Are the answers provided by OpenAI and Elsmar Cove accurate?

To test the accuracy of a common regulatory question, we chose a question we weren’t 100% sure about when a client asked last month. I asked my team, but nobody was 100% certain. Basil Systems is limited to submission and post-market surveillance data. I searched FDA.gov, but it was not clear. Google gave us a link to the FDA website. I asked a couple of ex-FDA consultants, but they gave me outdated information. On Thursday, March 30, 2023, I asked Lisa King during an AAMI course I was co-teaching. Lisa is a Consumer Safety Officer at the FDA responsible for reviewing device entries into the FDA. She is also in very high demand for public training courses. She said the contract manufacturers used to be exempt from registration if they shipped to a legal manufacturer first. The regulations changed, and now 100% of contract manufacturers making a finished device must register with the FDA. She also clarified that the FDA doesn’t use the term “legal manufacturer.”

Screenshot 2023 04 01 11.00.46 AM e1680445277287 1024x676 OpenAI and Elsmar never trust their help with regulatory questions?

As you can see from the above answer provided by OpenAI, the ChatGPT engine [i.e., Model: Default (GPT-3.5)] effectively produces the correct answer. Using the same wording for the regulatory question, “Does a foreign contract manufacturer need to register with the FDA if they are shipping the medical device to the legal manufacturer first before the device is exported to the USA?” there were no results from Elsmar Cove. After several attempts, I found what I was looking for using the following search terms, “FDA registration of contract manufacturers.” There were multiple related search results, but the most useful discussion threads in the Elsmar Cove discussion forum were:

The most succinct correct answer in the forum is copied below.

Screenshot 2023 04 01 11.39.43 AM e1680445334745 1024x332 OpenAI and Elsmar never trust their help with regulatory questions?

Can you trust OpenAI and Elsmar Cove to answer your regulatory questions?

OpenAI is only as effective as the data used to train it. This is constantly evolving, but we have identified search results that were 100% accurate, results that were outdated, and results that were scary wrong. The same is true of discussion forums. Elsmar Cove is one of the best discussion forums for the medical device industry, but people also use ASQ, RAPS, and AAMI. The quality of the information provided depends upon the knowledge and experience of the people participating in the forum, but it also depends upon the forum’s moderation. Elsmar Cove has some experienced moderators with decades of experience. There is always the chance that the most experienced person in the world could answer your regulatory question incorrectly. This usually creates a problem because everyone else in the forum hesitates to challenge a recognized expert. Therefore, regardless of which resource(s) you use, always try to get a reference to the trustworthy source of the applicable regulation. Even Lisa King could make a mistake, but she immediately said, you can find the regulations in the US Code of Federal Regulations (i.e., 21 CFR 807). The bottom line is, always do your fact-checking and reference your source(s).

5 thoughts on “OpenAI and Elsmar never trust their help with regulatory questions?”

  1. First off I want to say terrific blog! I had a quick question that I’d like to ask if you don’t mind.
    I was curious to find out how you center yourself and clear your thoughts prior
    to writing. I have had a hard time clearing my mind in getting my ideas out there.
    I truly do take pleasure in writing however it just seems like the first 10 to 15 minutes are lost simply just trying to figure out
    how to begin. Any ideas or hints? Cheers!

    1. Thank you for the words of praise.

      I highly recommend reading Stephen King’s Book: “On Writing: 10th Anniversary Edition: A Memoir of the Craft”. He covers this question in detail.

      I personally never experience writer’s block. My “secret” is that I do my creative drafts at night when I’m tired. Then I do my editing in the morning. My work requires writing many pages each day, so a 500-word blog is fun and games for me.

      One of the things that many great writers do is write 2x what they intend to publish. Then they edit their work down to size. By taking this approach they can just let their work flow without worrying about technical aspects of writing.

      My next evolution will be to teach myself to write using dictation software (i.e. – Dragon Naturally Speaking). I can write at 80-100 WPM now, but that will increase my rate to 130+ WPM.

      Cheers!

  2. Pingback: OpenAI And Elsmar Never Trust Their Help With Regulatory Questions? - Plato Data Intelligence

  3. Pingback: Effective Management Skills for Managers - Medical Device Academy Medical Device Academy

  4. Pingback: Auditor shadowing as an effective auditor training technique Medical Device Academy

Leave a Comment

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top