Why AI models fall apart on a second site's data

I'm Kanstantsin. I do independent technical due diligence and feasibility assessments on AI and computer-vision systems — usually for someone who is about to invest in one, acquire one, or commit a year of engineering to building one. This article is about the failure I meet more often than any other: a model that is superb in the demo and mediocre, or useless, the moment it meets data from a second site.

Let me define site up front, because this is not only a hospital problem. A site is any source of data with its own habits — a laboratory, a scanner, a production line, a city, a camera mount, a single customer. Wherever data is produced by a particular setup, that setup leaves fingerprints. The pattern that follows is the same in histology, in autonomous driving, in retail analytics and in industrial inspection.

The pattern

It goes like this. The deck reports 0.96 AUC. The demo is live, not a video, and it works — you watch it work. Then the pilot starts on your data, and the number lands somewhere between 0.70 and 0.85. The team is not evasive about it; they are genuinely surprised. Then comes the sentence I have heard in nearly every one of these conversations: "it just needs a little fine-tuning on your data."

Sometimes that is true. Often it is not, and the gap you are looking at is not a gap in tuning — it is the difference between a model that learned the task and a model that learned a particular building. The uncomfortable part is that nobody was lying. The 0.96 was real. It was simply measured in a way that could not have detected the problem.

The model didn't lie. The evaluation did.

Almost every one of these systems was validated with a random split. Pool the data, shuffle, take 80% to train and 20% to test. It is the default in every tutorial, and on a pooled multi-site dataset it is close to worthless.

Here is why. If your data comes from three labs and you shuffle it, slides from all three labs land in both the training set and the test set. The model is then free to learn something much easier than the actual task: which lab is this from, and what tends to be true at that lab? A neural network is an efficient opportunist. If the source of an image is legible in its texture — and it usually is, through stain chemistry, scanner optics, compression, illumination, even the sharpening profile of a particular device — the network will use it, because it is a cheaper route to a low loss than learning morphology.

This is what makes the failure so hard to see from the outside: the test set is contaminated not by duplicated rows, but by shared context. Test performance stops measuring generalisation and starts measuring recall of a site's quirks. The number is honest about the wrong question.

A useful diagnostic — and one I like because it takes an afternoon and settles an argument — is to train a classifier to predict the source site from the image alone, with the clinical or business label thrown away. If that model works well, site identity is plainly visible in the pixels. Which means it was always available to the main model as a shortcut, and a random split gave it every incentive to take it.

Four shifts worth separating

"Domain shift" is used as one word for at least four different problems, and they have different fixes. Separating them is most of the analytical work:

1. Acquisition shift. The instrument changed. A different scanner, stain protocol, lens, mount height, exposure, codec. The underlying object is identical; its representation is not.
2. Population shift. What is in front of the sensor changed. Different patient demographics, a different city's road furniture, a different product mix, a different season.
3. Annotation shift. The labels changed, though nobody framed it that way. New annotators, a different rubric, a different threshold for "positive". The ground truth moved, so the model is being scored against a different ruler.
4. Prevalence shift. The base rate changed. Nothing about the images or the model changed at all.

The fourth one deserves a paragraph of its own, because it is the one that ambushes commercial teams. Sensitivity and specificity are properties of the model. Precision is not — it depends on how common the positive class is. Take a model at 95% sensitivity and 95% specificity. Where the condition occurs in 10% of cases, roughly two-thirds of its alerts are correct. Move the same model, unchanged, to a screening population where the condition occurs in 1% of cases, and only about one alert in six is correct. The model did not degrade by a single point. The workflow around it just became unusable, and the people using it will — correctly — stop trusting it.

No amount of fine-tuning addresses that, because nothing is broken. It is arithmetic. This is why "what is the prevalence in the deployment population, and what was it in the test set?" belongs in the first ten questions you ask.

Histology makes it concrete

I have spent a lot of time in cell and tissue imaging, and it is the cleanest illustration I know. Two laboratories both run H&E — the same stain, the same textbook, the same century of practice. And their slides do not look alike. Stain vendors and lot numbers differ. Section thickness differs by a couple of microns. Fixation time differs because one lab is busier. The scanners differ in their optics and their colour handling. Put slides from both labs side by side and the difference in colour and contrast is obvious to the naked eye — long before you compute anything.

Now add the second problem: in histology, the ground truth itself is contested. Ask several pathologists to classify the same cells and they will not fully agree — inter-rater agreement in this kind of task is frequently far below what people outside the field assume, and for some categories it is close to a coin toss between experts. So when a vendor reports 0.96 against "the ground truth", the honest follow-up is: whose ground truth, how many of them, and what happened when they disagreed? If a single annotator produced the labels at a single lab, then a 0.96 may be measuring how faithfully the model reproduces one person's habits at one site.

Stain normalisation and colour augmentation help — genuinely, and they should be in any serious histology pipeline. But they compensate for variation; they do not manufacture the variation that was never collected. Nothing substitutes for data that is large enough and varied enough, gathered from more than one place.

It is not a medical problem

Change the vocabulary and the same structure appears everywhere:

• Mobility and ADAS. A perception stack tuned in one city meets another city's lane markings, signage, cyclists and weather. The sensor suite is repackaged, the camera sits four centimetres lower, and the calibration assumptions quietly stop holding. This is precisely why an operational design domain is written down — an ODD is a statement about which distribution the claim covers.
• Industrial inspection. The model is trained on one line under one lighting rig. The second factory has a skylight. Now defect detection has a time of day.
• Retail and security. Cameras at a different height and angle, different crowd density, different demographics, different lighting temperature.
• Documents and OCR. One client's scanner, one template, one language variant — then a customer arrives with a fax from 2003.

In each case the demo was honest and the second site was a different problem wearing the same name.

What to ask for

This is the part you can use tomorrow, whether or not you ever hire anyone to do it for you. Six requests, in order of how much they reveal per minute spent:

1. "Show me the split." Not the metric — the split. Was it grouped by site, by device, by patient, by customer? If subjects or sites appear on both sides of the line, the headline number is describing memorisation. This single question resolves a surprising share of engagements.
2. "Show me a genuinely external test set." Held out at the site level, from a place the model never saw, ideally touched exactly once. A test set that has been evaluated against forty times during development is not a test set any more; it is a slow, expensive validation set, and the model has been fitted to it through the team's own decisions.
3. "Show me per-site numbers." Never accept a pooled average across sites. A pooled 0.90 can be 0.97 at the two sites that contributed most of the data and 0.62 at the third. The average conceals precisely the thing you are buying — behaviour somewhere new.
4. "Show me calibration, not just ranking." AUC is indifferent to whether the probabilities mean anything. If the product acts on a threshold — and almost every product does — you need to know that a 0.8 still means 0.8 at the new site. Recalibration is often cheap; discovering that you needed it after deployment is not.
5. "Show me the failures." Ask for the worst cases on the external set, not the best. How a team narrates its own failure modes is the most reliable signal I know of engineering maturity. Teams that can say precisely where their model breaks have usually looked; teams that cannot usually have not.
6. "Who labelled the external set?" If the same annotator labelled train and test, annotation shift is invisible by construction — you have measured consistency, not correctness.

What good looks like

It is worth saying plainly, because this article could read as a case for suspicion, and it isn't one. Strong teams pre-empt every question above. They volunteer the site-level split before being asked. They show the external number even though it is lower, and they can tell you why it is lower and which shift caused it. They know their prevalence assumptions. They have a plan for the next site that is more specific than "fine-tuning".

A modest external number with a credible explanation is worth more than a spectacular internal one. The first is a measurement; the second is a hope with a decimal point. When I write a verdict, a team that shows me 0.84 on a held-out site and understands the gap gets a materially better read than a team showing 0.96 that cannot tell me how the data was split.

If you are the one building

The order of operations matters more than the technique:

• Collect variety before you collect volume. Ten thousand images from one scanner teach less than two thousand from five. Diversity is the axis that buys generalisation; sheer count is the axis that buys confidence intervals.
• Split by site from day one. It is nearly free at the start and expensive to retrofit once the numbers everyone has memorised turn out to be wrong.
• Treat normalisation and augmentation as compensation, not cure. They narrow the gap. They do not fabricate the distribution you never sampled.
• Budget for external validation as a line item, not as something you do if there is time. There will not be time.
• Write down the ODD — the conditions under which you claim the system works. A narrow claim you can defend is a stronger commercial asset than a broad claim you cannot.

The reason this failure is so persistent is not that the engineers are careless. It is that the random split is the default, the default produces a flattering number, and a flattering number is rarely interrogated by the people it flatters. The work of diligence is mostly the unglamorous business of asking how a number was produced, and then asking again about the part that was skipped.

None of this requires suspicion of anyone. It requires knowing which questions have load-bearing answers — and being willing to sit with an uncomfortable one for a moment instead of moving on.

If you are weighing an AI or computer-vision system right now and you are not sure whether the number in the deck is a measurement or a hope, that is exactly the kind of thing I do — an independent read, delivered as a written verdict and a recorded walkthrough. Reach out: kanstantsin@kvdd.eu. I'm happy to help.