There is a particular kind of confidence that comes from a successful demo. A founder clicks through a polished Jupyter notebook, the model answers beautifully, the investors lean forward. Three months later, the same system is generating patient summaries that cite studies that don’t exist, or drafting customer emails that quote refund policies that were updated fourteen months ago. The model hasn’t changed. The confidence has.
I’ve had some version of this conversation a dozen times in the past year alone. The engineers are competent. The models are capable. What’s missing — what almost always turns out to be missing — is architecture. Not infrastructure, not compute, not even the model itself. The deliberate, principled design of how an LLM sits inside a larger system of data, verification, and feedback.