There are thousands of companies in the healthcare industry, both established and startup organizations, touting new and innovative uses of “AI” and “AI-augmented” solutions to improve products or processes. From coding and documentation to drug discovery and clinical decision support — it seems like nearly every company is jumping on the AI bandwagon, often using a large language model (LLM) to power chatbots.

But just because AI is trendy, doesn’t mean that every organization is approaching their AI use cases in the most efficient way. If you are considering AI-based use cases for your organization, especially if they relate to clinical practice guidelines, then this article is for you.

Today, we will outline five key tips and considerations to ensure that you are getting the most out of your AI-focused healthcare solution incorporating clinical guidelines.

1. Closely Define the Scope, Use Case, and User Needs

When considering the use of LLMs with clinical practice guidelines, the first step is to clearly define the use case. Your definition will vary depending on your overall needs and goals, but three main considerations include:

  • The Scope of Guidelines (including Organizations)
    • You need to determine whether your use case is limited to a specific organization’s guidelines, if it’s meant to span an entire medical specialty, or if it will expand even further beyond that. 
  • The Scope of Specialty
    • Will you be building an application for a certain medical specialty (e.g., gastroenterologists)? Or are you building an application that is meant to be used more broadly across multiple (or all) medical specialty areas? 
  • The Scope of Content Types
    • Will your LLM-powered chatbot focus strictly on answering questions related to guidelines? Or will it need to also understand basic biology, or medication information, or billing/coding information? 

2. Consider the Application and Access Methods

Consider how users will access and interact with your application. Making that determination early on will help you narrow down some of the questions and considerations that follow later in this post. 

  • Platforms
    • For example, will this integration be used in a website or a mobile app? Will you be offering it as an API? Or will it be interfacing directly with third-party applications, such as electronic health records? You may wish to start with one of those elements, and then gradually expand into more. In that case, your workflow and processes need to account for that future expansion.
  • To Paywall, Or Not To Paywall
    • Is this a chatbot that will be used only by paying users, or free, but still registered users (e.g., members of a specific organization)? Or will both registered and anonymous users have access? 

3. Determining Which LLM To Use

The next step in crafting your approach to the use of LLMs in healthcare, and especially in clinical practice guidelines, is determining which LLM to use. There are many articles on the topic of how to choose an LLM, so we won’t go too far into this other than stating that there are four key factors to consider:

  • Cost and Usage Rights
    • It’s important to identify the cost(s) associated with each LLM, as well as the specific usage rights. Some LLMs may be freely available, but only for non-commercial purposes. So cost and usage rights go hand in hand.
  • Performance Metrics
    • At the end of the day, LLMs are only as good as the responses they provide. When choosing an LLM for guidelines, consider performance metrics, such as completeness, accuracy/hallucination rate, succinctness/answer quality, fluency, coherence, and more. 
  • Scalability
    • Depending on your anticipated user count, scalability may be one of the biggest factors to consider. Think about both the short-term and the long-term future growth needs pertaining to concurrent users and the amount of content ingested.
  • Maintenance and Update Planning
    • This component can go along with scalability, but you need to consider how you will monitor the performance (in multiple ways), and assemble a plan for continuous refinement and improvement.

4. Align On Data Delivery and Data Structure

For as good as LLMs are at parsing and understanding unstructured text, they are even better when given structured data. This aspect is even more critical when using LLMs to retrieve or understand medical guidelines and guideline recommendations. Some of the specific considerations for aligning data delivery and structure with guidelines and AI, include:

  • Data Format
    • How will you structure the data you are feeding into your LLM/machine learning tool(s)? And how will you standardize that structure and formatting across guidelines, and even across other content types? 
  • Usage of Retrieval-Augmented Generation (RAG) vs. Continuous Training
    • Both approaches have their pros and cons. If the content is moving at a slower pace, such as an organization building an AI-enabled guidelines search engine, then training may be more appropriate. For organizations wishing to incorporate guidelines from many organizations and/or content outside of guidelines, training may not be practical.
  • Metadata
    • This is pretty straightforward, but it’s important to consider the types and quantities of metadata you provide the LLM, along with the guideline content itself. For example, taking a structured approach to recommendation grading or level of evidence. But you could go further by breaking down inclusion criteria, action types, and more.
  • Handling of Algorithms, Images, and Tables
    • Guidelines have algorithms, images, and tables. LLMs are getting better at understanding the content and context of these elements, but the safest method would be to build some sort of secondary process to break this data down further. For example, using optical character recognition (OCR) on the algorithms — or even better: introducing text metadata fields to help provide the logic in a more structured manner.

5. Miscellaneous End User Display Considerations

There are numerous considerations from a UI/UX perspective, but when dealing with clinical guidelines integrating with AI and chatbots, two of the most important considerations include the handling of citations and dealing with conflicting information.

  • Handling Citations 
    • This is very important, as trust is a critical factor when using LLMs or chatbots for decision support. You need to consider how you will source the information used in the responses, and how you will make it easy for the end users to identify not only the source, but also where within that source the content comes from.
  • Handling Conflicting Recommendations
    • Not all clinical guidelines agree with one another. There are nearly a dozen differing guidelines for colorectal cancer, and nearly as many for diabetes, high cholesterol, and more. For organizations wishing to build AI applications that span multiple organizations’ guidelines, you must have a plan for dealing with conflicting information.

These tips and considerations will help you and your team think about how to leverage clinical practice guidelines with AI, LLMs, chatbots, natural language processing and/or other types of machine learning.


Feel free to drop us a line if you would like to learn more about how your organization can make the most out of pairing guidelines and AI or LLMs, or if you are interested in institutional subscriptions, computable guidelines solutions, or enterprise access to our guidelines API.

Copyright ® 2025 Guideline Central, all rights reserved.