Predictive Analytics for Population Health: Lessons Learned

Every health system holds, in its electronic health records, early signs of the illnesses it will have to treat next year: rising blood pressure readings, missed prescriptions, repeat emergency visits, a cluster of respiratory consultations in one district. Predictive analytics tries to turn those signs into action before a patient deteriorates or an outbreak spreads. The track record so far is mixed. Some programmes have improved outcomes measurably, and some widely deployed models have performed far worse than claimed. The difference usually comes down to validation, calibration and whether anyone is organised to act on the prediction.

Three kinds of population health prediction

It helps to separate three uses that are often grouped together:

• Individual risk stratification estimates each person's probability of a future outcome, such as a cardiovascular event, hospital admission or frailty, so that preventive care can be targeted.
• In-hospital deterioration prediction scores inpatients continuously to warn clinicians hours before sepsis, cardiac arrest or ICU transfer.
• Epidemiological surveillance works at population level, detecting unusual patterns in consultations, calls, laboratory results or prescriptions that may signal an outbreak or a seasonal surge.

Each needs different data, a different validation method and a different response team. A national programme needs all three.

Israel's Clalit: prediction tied to outreach

Clalit Health Services, the largest of Israel's health funds, insures more than half the country's population, about 4.5 million members according to its researchers' 2020 paper. It has long used its integrated records for proactive, risk-based care. Its COVID-19 response early in the pandemic shows the approach at work.

In early 2020 there was almost no individual-level COVID-19 outcome data to train a model on. Clalit's team, writing in Nature Communications, built a baseline model predicting severe respiratory infection or sepsis, trained on more than one million patient records, then recalibrated it to published COVID-19 case-fatality rates. On validation the model reached an area under the ROC curve of 0.942 for COVID-19 mortality, with much better calibration than the uncalibrated baseline.

What matters most is what Clalit did with the scores. According to the paper, members in the highest-risk group received phone calls from their care providers and two risk groups received text messages about social distancing. The scores were also used to prioritise PCR testing and, once infection was confirmed, to decide whether patients should be treated in hospital or followed more closely in the community. The model was useful because it was connected to an operational response.

The NHS: decades of primary-care risk scores

England's primary care records, coded consistently over many years, have supported a series of validated risk tools. QRISK3, published in the BMJ in 2017, estimates ten-year cardiovascular risk. It was derived on 7.89 million patients from 981 general practices and validated on 2.67 million patients from a separate 328 practices, and added factors such as chronic kidney disease, migraine, severe mental illness and blood-pressure variability. The electronic Frailty Index, developed and validated on more than 900,000 patient records, uses routinely coded deficits to identify older people with mild, moderate or severe frailty.

The pandemic showed what national-scale prediction can do. In February 2021, NHS Digital applied the QCovid risk model, developed at the University of Oxford and independently validated by the Office for National Statistics, across the population. It identified around 1.7 million additional people with combinations of risk factors that put them at similar risk to those already classed as clinically extremely vulnerable. They were added to the shielding list, which also moved them forward for vaccination.

Surveillance as a standing capability

For population surveillance, the UK Health Security Agency runs syndromic surveillance systems covering emergency department attendances, GP in-hours consultations and NHS 111 calls, and publishes weekly summaries. These systems are not exotic AI, but they show what any predictive surveillance layer needs underneath it: timely, routinely collected, consistently coded data from across the health system.

Kaiser Permanente: deterioration alerts with a response team

Kaiser Permanente Northern California's Advance Alert Monitor scans inpatient records hourly and flags patients at high risk of deterioration within the next 12 hours. Alerts go first to a team of specially trained remote nurses, who review the record and bring in the on-site rapid response team where needed. An evaluation of the staggered rollout across 21 hospitals, published in the New England Journal of Medicine in 2020, found lower 30-day mortality after an alert in the intervention cohort (15.8% versus 20.4% in the comparison cohort).

Pitfall one: models that do not survive external validation

A model's published performance describes the population it was built on. In 2021, researchers at Michigan Medicine reported in JAMA Internal Medicine on an external validation of a proprietary sepsis prediction model implemented at hundreds of US hospitals. Across 38,455 hospitalisations, the model's AUC was 0.63, well below the developer's reported performance. It missed 67% of sepsis cases while generating alerts on 18% of all hospitalised patients.

The lesson is not that sepsis prediction is impossible. It is that no model should go live in a new setting without local, retrospective validation on that setting's data, followed by prospective monitoring once deployed.

Pitfall two: bias hidden in the label

In 2019 Obermeyer and colleagues, writing in Science, examined a commercial algorithm widely used in the US to select patients for care-management programmes. It predicted future healthcare costs as a proxy for health need. Because less money had historically been spent on Black patients with the same level of illness, the algorithm judged them healthier than equally sick White patients. The authors estimated that correcting the disparity would raise the share of Black patients identified for extra help from 17.7% to 46.5%.

The model was technically accurate at predicting cost. The bias came from choosing the wrong target. Any national programme should ask, for every model, whether the outcome it predicts is the outcome that matters clinically, and check performance separately for regions, age groups, sexes, and urban and rural populations.

Pitfall three: alert fatigue

A model that alerts on many patients to catch a few will be ignored. In the sepsis validation above, the ratio of alerts to true cases meant clinicians would have to assess many patients for each case found. When alerts are frequent and mostly false, clinicians learn to dismiss them, including the ones that matter. Practical mitigations include:

• Setting thresholds against the capacity of the team that must respond, not only against statistical metrics.
• Sending alerts to a dedicated reviewer, as Kaiser Permanente does, rather than straight to busy ward staff.
• Suppressing repeat alerts for the same patient and documenting why an alert was overridden.
• Monitoring positive predictive value and response times as standing operational metrics.

Pitfall four: drift over time

Populations, coding practices and treatments change. A model calibrated on pre-pandemic data, or on data from before a new EHR rollout, can degrade quietly. Continuous monitoring of calibration, periodic retraining under change control, and a clear owner for each model in production are basic requirements, not optional extras.

A checklist for decision-makers

1. Define the clinical action and the team that will take it before choosing a model.
2. Validate locally, on retrospective national data, before any clinical use.
3. Report calibration and performance by subgroup, not just overall AUC.
4. Start with a silent-mode pilot, then a limited live deployment with outcome measurement.
5. Monitor alert volumes, positive predictive value and drift continuously.
6. Keep clinicians in the decision loop and record how predictions are used.

Applying these lessons in Moldova

Disease prediction is one of the five AI domains of Moldova's National Healthcare Digitalization Program (2025–2030), which covers around 2.36 million people. The programme plans EHR-based risk prediction and real-time epidemiological monitoring. Its most important design constraint is data maturity: EHR coverage is expected to grow from under 10% today to about 80% by 2030. Early models will therefore be trained on partial, uneven data, which makes local validation, subgroup analysis and drift monitoring more important, not less. As coverage grows roughly eightfold, models will need regular recalibration.

Import Medical supports the programme with sovereign, on-premise AI computing capacity operated inside Moldova, so risk models can be trained, validated and run next to national health records without that data leaving the country. Read more about our disease prediction solution and the AI infrastructure behind it. Related reading: the evidence on AI in radiology, health data sovereignty in the EU and AI for healthcare operations. To discuss a validation-first pilot, contact us.

predictive analytics population health risk prediction epidemiological surveillance EHR

Sources

1. Developing a COVID-19 mortality risk prediction model when individual-level data are not available (Barda et al., Nature Communications 2020) — PubMed Central
2. Development and validation of QRISK3 risk prediction algorithms (Hippisley-Cox et al., BMJ 2017) — PubMed Central
3. Real-time alerts associated with lower mortality — Kaiser Permanente
4. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients (Wong et al., JAMA Intern Med 2021) — PubMed
5. Dissecting racial bias in an algorithm used to manage the health of populations (Obermeyer et al., Science 2019) — PubMed
6. Syndromic surveillance — UK Health Security Agency data dashboard