A letter arrives at a health plan. A federal auditor has selected a sample of member records and wants the medical documentation supporting every diagnosis submitted for each one.
For each record, one question. What in this documentation supports that diagnosis?
Someone opens the software that flagged the code in the first place. It reports 94 percent.
That number is not an answer to the question asked. Understanding why is a useful way into what current AI systems can and cannot do, and the gap is architectural rather than a matter of model quality.
Where the money comes from
Sickness is established through diagnosis codes drawn from clinical notes. The payment rests on a code, and the code rests on what a clinician wrote. The government reviews the notes, and when the documentation does not support the code, the payment was not owed.
That review has found something uncomfortable. The HHS Office of Inspector General has audited diagnosis codes it flags as high risk for miscoding, and describes the outcome in its own compliance guidance: it “reviewed medical records that MAOs provided as support; however, a high percentage of the associated HCCs could not be validated.”
The dominant error is not invention. It is a condition that was once real and is no longer carried forward year over year because nothing in the workflow ever takes it off. OIG names that directly. Among the conduct it lists as revealed by federal investigations is “failing to remove diagnosis codes previously submitted to CMS when chart reviews provide information that those codes were unsupported or otherwise invalid.”
Two questions that look identical
“Is this code probably right?” and “which documentation requirement does this code satisfy?” sounds like one question asked twice.
The first asks about likelihood, and a well-trained model handles it well. The second asks about a rule, and has a determinate answer with a location attached: this passage, in this note, meets this criterion.
Finding is not evaluating
The first software generation used natural language processing to find things. Feed it a note and it identifies the medical concepts, matching inconsistent human writing against a standard vocabulary. The same condition might appear a dozen ways in one document. NLP normalises them.
A note may mention a condition because the patient had it years ago, because a clinician excluded it, or because the patient explicitly does not have it. Negation detection catches a good deal of that. None of it turns this word here into this documentation is sufficient, and the second is what the auditor asked about.
Prediction gets better at the wrong thing
Then came statistical models, and more recently large language models. These do not search for terms, they estimate, predicting which codes are likely. The good ones are strikingly accurate.
Ask one to explain itself and you get a fluent paragraph that reads like expert justification. This is where it gets uncomfortable.
The paragraph is also a prediction. The model is not reporting a test it ran, it is generating text resembling the explanation that usually accompanies this conclusion. Often it is right. When it is wrong, it is harder to catch than an obvious error, precisely because it reads well.
Scaling the model improves the prose. It does not convert the prose into a record of anything. Scale buys fluency, not provability, and the two have been quietly treated as the same thing for several years.
A second consequence took longer to notice. A system estimating how likely a code is has no way to say a code is wrong. Its whole output is candidates and confidence values, with no slot for this one should come off.
Which explains the pattern in the audit findings better than any story about incentives does. Software built this way can only add. The people using it may fully intend to remove unsupported codes. The tool has no output that corresponds to doing so.
Regulators have noticed the automated version of this. OIG lists as a risk area the practice of “querying physicians via electronic medical record platforms (including prompts generated by artificial intelligence algorithms) or otherwise prompting physicians to add risk-adjusting diagnoses that patients did not have.” The objection is not to algorithms. It is to algorithms that only ever push one way.
Holding a rule
The systems that can answer the second question combine a neural component with a symbolic one.
The neural part reads the note and handles linguistic variability. The symbolic part holds the criteria in explicit form, as structured logic and relationships rather than learned weights, and tests what the neural part proposes against them.
What changes is the output. It returns which criterion was tested, whether the record met it, and where the supporting evidence sits. That is a different object from a confidence score, and it is the one the auditor asked for.
Because the criterion exists explicitly, the test can also come back false. That is what allows a risk adjustment solution built this way to flag a submitted code as unsupported rather than only proposing additions, which matters considerably in a setting where one-directional output has drawn federal attention.
The part nobody puts in the demo
There is a real cost. Someone has to build and maintain the rules. Every guideline revision and change in criteria is work, and the system is only as current as that work. A predictive model needs none of it, which is why predictive models ship faster.
That is an engineering trade, not a marketing point. More effort up front, for output someone can defend when a record gets pulled.
Why this generalises
Medicare Advantage makes a clean case study because the question is formal and the reviewer has federal authority. But the structure recurs anywhere software has to justify a decision to someone empowered to challenge it.
Worth remembering the next time a demo answers a hard question with a percentage.
