"ChatGPT is useful, but I have no idea what is going on inside it"—many people feel this way. The term LLM (large language model), which appears in almost every explanation of AI, is precisely "what is inside ChatGPT."
Knowing roughly how an LLM works helps you understand why AI produces plausible falsehoods, why you must not enter patient information, and which tasks it suits and which it does not—in a way that actually makes sense.
This article explains what an LLM is with as little jargon as possible, how it relates to clinic work, and what to verify before adopting one.
Disclaimer: This article is general information. Specifications and service availability may change. When considering adoption, check each vendor's latest information and the current version of the applicable guidelines.
What an LLM Is—A Mechanism That Predicts the Next Word
An LLM (Large Language Model) is a program that has read an enormous amount of text and learned "which word is likely to come next after this one."
Shown the beginning of a sentence, it predicts the most likely next word. It repeats this prediction one word at a time to assemble a passage.
When ChatGPT returns an answer, what is happening internally is this continuous "next-word prediction." It appears to understand the question, look up knowledge, and compose an answer—but the fundamental operation is "keep producing a plausible continuation."
The "large" refers to the sheer volume of text it has read and the scale of its internal computation. It has learned patterns of how words connect from vast amounts of text—web pages, books, papers. That is why it can handle medical terms, legal terms, and everyday conversation with equal apparent fluency.
Why It Seems to "Know" Things
How can "predicting the next word" answer questions?
Because learning word connections at that scale causes something like knowledge to accumulate internally. Given the phrase "the first-line drug for hypertension is," the model predicts whichever words most frequently followed that phrase in its training text. To us, that looks like knowing.
Two properties matter here.
1. Knowledge has an expiry date. An LLM knows only text up to the point it was trained. Events after that, revised regulations, and newly released drugs are either unknown to it or answered with outdated information. This is called the "knowledge cutoff."
2. It is closer to "recalling" than "looking up." An LLM by itself does not consult external documents. It generates a plausible continuation from patterns absorbed during training. That is why, asked for a source, it may produce the name of a paper that does not exist.
Making it "look things up and answer" requires a separate mechanism that connects it to external data—covered in What Is AI Search and RAG? and What Is MCP?.
Why It Makes Mistakes—Hallucination
The phenomenon of an LLM confidently producing plausible but incorrect content is called "hallucination."
Once you understand the mechanism, this looks less like a bug and more like a structural property. The LLM selects the next word by "plausibility," not "correctness." Even in areas where its training data is thin, it can still generate a "likely-sounding continuation."
Examples that could arise in a clinic:
| Situation | Possible hallucination |
|---|---|
| Asking about a drug dose | Returns a non-existent dose or outdated package-insert content |
| Asking about billing requirements | Returns a non-existent add-on or a pre-revision fee |
| Requesting literature | Presents a fabricated paper with plausible authors and journal |
| Drafting a patient explanation | Mixes in statements that are true in general but not for this patient |
The countermeasure is not "making the AI smarter." It is two things: using it on the premise that a human verifies, and having it reference correct external data. The former is covered in Where to Draw the Line on Delegating Work to AI; the latter in the RAG and MCP articles above.
Tokens and Context Length—How Much It Can Handle at Once
Of the terms that appear in LLM explanations, two bear on practice.
A token is the smallest unit in which an LLM handles text. In Japanese, roughly one to two characters make one token (it varies by service). Many paid services price by token count.
Context length (context window) is the maximum amount of text an LLM can read at once. Anything beyond it is effectively "invisible." When an AI in a long conversation seems to forget your earlier instructions, this is one of the causes.
The practical implication is simple: passing only the relevant portion of a document yields better accuracy and lower cost than passing the whole thing.
Other terms are collected in Generative AI Terms Explained for Clinic Staff.
ChatGPT, Claude, and Gemini Are All "Service Versions" of LLMs
A frequent point of confusion:
| Name | What it actually is |
|---|---|
| LLM | The name of the mechanism/technology (the engine) |
| ChatGPT / Claude / Gemini | Names of services that make an LLM usable through a chat interface and more |
| GPT-x / Claude x / Gemini x | Names of individual models running inside each service |
In car terms, the LLM is the "engine," ChatGPT and the others are "models of car," and the model names are "engine part numbers."
In other words, every service runs on fundamentally the same mechanism; the differences show up in training approach, safety measures, contract terms, and integration with other systems. A comparison of the three appears in How Clinics Should Choose Between ChatGPT, Claude, and Gemini.
How It Touches Clinic Work
LLMs suit "work that handles text" in general. Here are the uses already seen in clinics, with their fit.
Well-suited tasks (drafting, converting, summarizing)
- Drafting referral letters and clinical summaries
- Organizing consultation transcripts into SOAP format (combined with speech recognition)
- Summarizing long documents and meeting notes
- Rephrasing patient explanations in plain language
- Drafting internal documents and notices
- Drafting routine email replies
Poorly-suited tasks (where accuracy is non-negotiable)
- Final decisions on drug doses and interactions
- Confirming billing requirements (uses that demand the "correct" fee or criteria)
- Diagnosis itself
The common thread: strong at "producing a first draft," weak at "producing the right answer." Respect that line and the documentation burden drops substantially.
What Providers Should Verify
Two cautions follow directly from the mechanism.
1. Is input used for training?
An LLM gets smarter by learning from text. The flip side is that some services use what you type as training data for the next version. Free and consumer plans are sometimes configured this way by default.
Because medical information is involved, whether the contract explicitly states that input is not used for training is the first thing to verify.
2. What may be entered?
Regardless of training use, the baseline is never enter patient-identifying information (name, date of birth, address, contact details, chart number) into a general-purpose external AI service. The realistic arrangement is two-tiered: limit general-purpose services to work that contains no patient information, and handle anything involving patient data under contracts and environments designed for medical information.
For the legal framing and security thinking, see Using Generative AI in Healthcare and The Three-Ministry Guidelines Explained.
Common Misconceptions
"An LLM is an evolved search engine." No. Search "finds documents that exist"; an LLM "generates plausible text." Uses that require sources must be combined with search or external data reference.
"A paid plan makes it accurate." Paid plans mainly change model capability and contract terms. Hallucination does not disappear.
"The AI understands and then answers." Mechanically, it is word prediction. It genuinely appears to understand—but skipping verification on that assumption is dangerous.
"The latest model covers the latest regulatory revisions." Revisions after the training cutoff may not be reflected. Verify regulations and fees against primary sources.
The Relationship to the EMR
In the EMR context, LLMs matter chiefly in three settings: recording, documents, and checking.
Transcribing consultation dialogue and organizing it into SOAP. Drafting referral letters and certificates. Flagging possible billing omissions from the record. All are "first draft" tasks—the LLM's strength.
The design fork is whether the LLM can see the contents of the chart. A general-purpose chat AI knows nothing about in-clinic data, so a person must copy the needed information across. If AI is built into the EMR itself, the LLM drafts from the actual chart data. The two differ greatly in accuracy and in how easily you can control where information goes.
Pottech's AI Karte is an AI-native EMR designed around AI from the outset, handling everything from recording to billing on a single foundation. For the AI-native concept, see What Is an AI-Native Electronic Medical Record?; for the broader picture, see What Is an AI EMR?.
Conclusion
- An LLM is a mechanism that predicts the next word; ChatGPT, Claude, and Gemini are services built on LLMs
- Knowledge expires at the training cutoff, and the model "recalls" rather than "looks up"
- Hallucination is structural; address it with human verification and external data reference
- Tokens are the pricing unit and context length is the per-request ceiling—pass only what is needed
- In clinics, LLMs are strong at drafting, converting, and summarizing, weak at producing the right answer
- Verify whether input is used for training and what may be entered
- In the EMR, whether the AI can see actual chart data separates accuracy and safety
For details on AI Karte or to request a demo, please contact us.