External validation of the QLifetime cardiovascular risk prediction tool: population cohort study

Abstract

Background

Prediction of lifetime cardiovascular disease (CVD) risk is recommended in many clinical guidelines, but lifetime risk models are rarely externally validated. The aim of this study was to externally validate the QRiskLifetime incident CVD risk prediction tool.

Methods

Independent external validation of QRiskLifetime using Clinical Practice Research Datalink data, examining discrimination and calibration in the whole population and stratified by age, and reclassification compared to QRISK3. Since lifetime CVD risk is unobservable, performance was evaluated at 10-years’ follow-up, and lifetime performance inferred in terms of performance for in the different age-groups from which lifetime predictions are derived.

Results

One million, two hundreds sixty thousand and three hundreds twenty nine women and 1,223,265 men were included in the analysis. Discrimination was excellent in the whole population (Harrell’s-C = 0.844 in women, 0.808 in men), but moderate to poor stratified by age-group (Harrell’s C in people aged 30–44 0.714 for both men and women, in people aged 75–84 0.578 in women and 0.556 in men). Ten-year CVD risk was under-predicted in the whole population, and in all age-groups except women aged 45–64, with worse under-prediction in older age-groups. Compared to those at highest QRISK3 estimated 10-year risk, those with highest lifetime risk were younger (mean age: women 50.5 vs. 71.3 years; men 46.3 vs. 63.8 years) and had lower systolic blood pressure and prevalence of treated hypertension, but had more family history of premature CVD, and were more commonly minority ethnic. Over 10-years, the estimated number needed to treat (NNT) with a statin to prevent one CVD event in people with QRISK3 ≥ 10% was 34 in women and 37 in men, compared to 99 and 100 for those at highest lifetime risk.

Conclusions

QRiskLifetime underpredicts 10-year CVD risk in nearly all age-groups, so is likely to also underpredict lifetime risk. Treatment based on lifetime risk has considerably lower medium-term benefit than treatment based on 10-year risk.

Background

Although the incidence of cardiovascular disease(CVD) has fallen in most developed countries over the last 30 years, CVD remains one of the most common causes of morbidity and mortality worldwide. Prevention of CVD is therefore a policy priority, and a key practical question is who should be targeted for pharmacological primary prevention. In relation to initiation of statins, risk prediction tools are usually recommended by guidelines for the primary prevention of CVD to target treatment at people above a specified threshold of predicted risk. Prediction tools typically predict either over a fixed time (often ten years) or over a lifetime. Lifetime risk prediction is argued to be more appropriate in younger people who may not exceed a particular 10-year risk threshold even though they have markedly unfavourable CVD risk profiles (mitigated in the short-term by being young) and are at high risk of premature CVD beyond 10-years [1,2,3,4,5]. Lifetime risk models also appropriately account for competing mortality risk, which is ignored and a cause of over-prediction in many CVD risk prediction tools [6,7,8]. Lifetime CVD risk prediction tools are recommended to guide treatment in several international guidelines, although there is no consensus on what threshold of lifetime risk should trigger an offer of statin treatment [1]. Lifetime risk prediction is not currently recommended for CVD risk stratification by the National Institute for Health and Care Excellence (NICE) [9], but NICE have identified lifetime risk prediction as a topic to examine further in a future guideline update [10]. In the UK, the QRiskLifetime prediction tool is available as a standalone web-based tool [11] or as the risk engine underlying the Joint British Societies risk calculator (JBS3) [2] and heart age [12] tools.

External validation of CVD risk prediction tools is needed before they are widely implemented, but lifetime models are difficult to validate since observational datasets do not have lifetime follow-up. The same is also true in the datasets used to derive lifetime risk prediction, including the QRiskLifetime derivation dataset [3]. In derivation, lifetime CVD risk is therefore estimated by using shorter-term observed CVD rates at different ages to infer what would happen to someone in the future, under the assumption that age-specific incidence of CVD will not change in the meantime. The same effectively applies in validation, which can only be done over shorter time-scales [3, 5], with true lifetime performance inferred by performance in different age-groups. The aim of this paper is to externally validate the QRiskLifetime CVD prediction model in a large UK primary care dataset using a 10-year time horizon, and to explore recalibration compared to QRISK3.

Methods

The overall design of the study is an independent external validation of a risk prediction (prognosis) model, designed and reported consistent with TRIPOD guidelines [13].

Data source and population

Analysis used Clinical Practice Research Datalink (CPRD) Gold) [14], which includes linked primary care, hospital and mortality data. Patients were eligible if they: were permanently registered with a practice contributing up-to-standard data for at least one year and with linkage to Hospital Episode Statistics (HES) discharge and Office of National Statistics (ONS) mortality data, and had no prior history of CVD or statin treatment. Cohort entry was defined as the latest of 01/01/04, a patient’s 30th birthday, or contribution of up-to-standard data for at least 1 year. Cohort exit was the earliest of: first CVD event; death; prescription of a statin; deregistration from the practice; end of data collection from the practice; or end of study on 31/3/16. All outcomes and predictors were recorded as part of routine clinical care, and therefore recorded blind to the study hypothesis. No formal power calculation was done, as the study size is determined by the data available in CPRD which was considered sufficient [15].

Outcomes

A first CVD event was defined as the earliest recording of any fatal or non-fatal coronary heart disease (CHD), ischaemic stroke, or transient ischaemic attack, recorded as ICD-10 codes in HES admissions or as the underlying cause of death in ONS death registration data, or as Read codes in GP electronic health records. ICD-10 and Read codes defining outcomes are those used in QRISK3 derivation [16] (detailed in a previous paper [6]).

Prediction model

We used publicly available QRISK®-lifetime-2011 software to calculate QRiskLifetime scores for each patient to age 95 and additionally constrained to a 10-year prediction horizon (under GNU Lesser General Public Licence v3). Predictor variables including body mass index, smoking, cholesterol and blood pressure were ascertained from GP electronic health records. All predictor variables are listed in Table 1. Our cohort matched the QRiskLifetime derivation sample and methods with some exceptions, namely: (1) We used a cohort entry date of 1/1/04 rather than 1/1/98; (2) When calculating baseline values, the derivation paper included cholesterol values measured aftercohort entry, whereas we only included cholesterol values measured before cohort entry; and (3) Individual Townsend deprivation scores were not available, so we used the median of the vigintile (equal 20th ) of score that an individual lived in. Predictor codesets used and methods of data handling have been previously published [6].

Table 1 Baseline data compared to QRiskLifetime derivation cohort

Missing data

Supplementary Table S1 details the extent of missing data and how missingness was handled. Multivariate Imputation by Chained Equations [17] was used to generate five imputed datasets for missing body mass index (BMI), total cholesterol:HDL cholesterol ratio (TC:HDL), systolic blood pressure (SBP), and smoking status. Analyses of these datasets were combined using Rubin’s rules [18] to give summary point estimates with confidence limits that reflect the added uncertainty associated with imputing missing values.

Statistical methods

The lifetime (to age 95) and 10-year risk of experiencing a cardiovascular event was calculated for each patient using QRISK®-lifetime-2011 software without recalibration. The performance of the risk score was assessed by examining discrimination and calibration of the model over a 10-year time horizon, in the whole population and stratified by age-group and Charlson Comorbidity Index (CCI) at study entry [19].

Discrimination is the ability of the risk score to differentiate between patients who experience a CVD event during follow-up and patients who do not. Discrimination was evaluated using Harrell’s C-statistic (a C-statistic of 1 indicates perfect discrimination, whereas a C-statistic of 0.5 indicates discrimination no better than chance; we interpreted values > 0.8 as showing excellent discrimination, 0.6–0.79 as moderate, and 0.5–0.59 as poor), Royston and Sauerbrei’s D statistic (higher values indicate greater discrimination) and an R-squared statistic of explained variation in censored survival data [20, 21].

Calibration refers to how closely predicted risk and observed probabilities agree at group-level. This was assessed for equally-sized groups of participants ranked by predicted risk. Calibration of the risk score predictions was assessed by plotting observed proportions with an event versus predicted probabilities. Since QRiskLifetime accounts for competing mortality risk, we evaluated calibration using the Aalen-Johansen estimator of observed risk which accounts for the competing risk of non-CVD death and therefore estimates the cumulative incidence of CVD [22]. Calibration plots were generated separately by sex for all patients and for subgroups of age and modified Charlson Comorbidity Index.

Consistent with the validation of QRiskLifetime over a 10-year time horizon, we examined changes in which patients were recommended for treatment based on either QRISK3 or QRiskLifetime 10-year predicted risk of ≥ 10% (the threshold recommended by the UK National Institute for Health and Care Excellence [23]). We calculated the Net Reclassification Index (NRI) with bootstrapped 95% confidence intervals at the 10-year 10% predicted risk threshold. NRI was calculated for people experiencing a CVD event (NRI+), for people not experiencing a CVD event (NRI-) and overall (NRI). NRI examines the extent to which using QRiskLifetime is better at classifying cases who experience the event as high-risk (10-year risk ≥ 10%) and non-cases as low risk (10-year risk < 10%). Since there is no recommended threshold of lifetime risk at which to define an individual as high-risk, we also compared which patients were recommended for treatment by QRISK3 at the 10% threshold and by QRiskLifetime using a threshold defined to identify the same number of patients (i.e. if QRISK3 recommended 19.0% of patients for treatment, we selected the 19.0% of patients at highest lifetime risk). For both comparisons, we examined the characteristics of patients recommended for treatment, the observed risk of CVD at 10 years, and the number needed to treat (NNT) to prevent one new CVD event assuming all people recommended for treatment actually took a statin assuming a relative risk reduction of 25% for new CVD events. All models were fitted in R v4.1.0.

Results

There were 1,260,329 women with mean age 49.3 (SD 14.2) years and 1,223,265 men with mean age 47.6 (SD 13.0) years in the external validation cohorts. Compared to the QRiskLifetime internal validation cohort [3], there was: a larger proportion of people from minority ethnic backgrounds; fewer people with a recorded family history of premature CVD; a higher proportion of treated hypertension; and somewhat higher proportions of atrial fibrillation and chronic kidney disease (Table 1). There were higher proportions with missing data in this study than the original study, likely reflecting the use of data recorded after cohort entry date in the derivation study (Supplementary Table 1).

Median follow-up was 5.7 (interquartile range [IQR] 2.2–10.2) years in women and 5.2 (IQR 2.0-9.3) years in men, similar to the QRISK3 cohort [16]. Crude incidence of CVD was higher in men than women (7.5 vs. 5.5 CVD events/1000 person-years), and increased markedly with age (Supplementary Table S2). Non-cardiovascular death had similar incidence to CVD in women, whereas in men incident cardiovascular disease was more common in men up to age 65–69 years, with non-cardiovascular more common subsequently (Supplementary Table S3 and Figure S1).

In the entire population over 10-years, QRiskLifetime discrimination was excellent in both women (C = 0.844 in this study vs. area under receiver operating curve [AUROC] 0.842 in original study internal validation [3]) and men (C = 0.808 vs. AUROC = 0.828 in internal validation [3]) (Table 2). However, when stratified by age, discrimination was only moderate in younger age-groups and was poor in people aged 75–84 (C = 0.578 in women, 0.556 in men). Stratified by CCI, discrimination was excellent in people with low morbidity (CCI = 0 or 1) but only moderate in people with high morbidity (in women with CCI = 3+, C = 0.724; in men with CCI = 3+, C = 0.656).

Table 2 Discrimination and model fit (evaluated at 10 years follow-up)

In the whole population over 10-years, there was reasonable calibration (with some under-prediction) in the eight deciles of lowest predicted risk with QRiskLifetime, but considerable under-prediction in the two deciles of highest predicted risk (Fig. 1). Stratified by age (Fig. 2), calibration was good in people aged 45–64, with under-prediction in all other age-groups which was largest in people aged 75–84. Stratified by CCI, there was under-prediction at all levels of morbidity which was more marked at higher levels of predicted risk and at higher levels of multimorbidity (Fig. 3).

Fig. 1

Calibration in women (left hand) and men (right hand) for whole population

Fig. 2

Calibration in women (left hand) and men (right hand) stratified by age

Fig. 3

Calibration in women (left hand) and men (right hand) stratified by Charlson Comorbidity Index

In the reclassification analysis (Tables 3, 5 and 4), compared to QRISK3, QRiskLifetime classified fewer people as having 10-year risk ≥ 10%. QRISK3 classified 239,396 (19.0%) women as high-risk, compared to 194,411 (15.4%) women classified as high-risk by QRiskLifetime over 10-years. QRISK3 classified 341,962 (28.0%) men as high-risk, compared to 276,369 (22.6%) men classified as high-risk by QRiskLifetime over 10-years (Table 3). 15.1% of women were classified as high-risk (≥ 10% over 10-years) by both tools, 3.9% as only high-risk by QRISK3 and 0.3% as only high-risk by QRiskLifetime (with the remaining 80.7% <10% on both scores). 21.9% of men were classified as high-risk (≥ 10% over 10-years) by both tools, 6.1% as only high-risk by QRISK3 and 0.7% as only high-risk by QRiskLifetime.

Table 3 Reclassification as high or low risk by QRiskLifetime compared to QRISK3 with both predicting risk over 10-years

Table 4 Characteristics of people recommended for treatment

In women, compared to QRISK3, QRISKLifetime slightly improved classification in those who did not experience an event (Net Reclassification Index NRI- = 0.035, 95% CI 0.034 to 0.035), but worsened classification in those who did experience an event (NRI+ = -0.080, 95% CI -0.082 to -0.077), with overall NRI − 0.045 (95% confidence interval − 0.047 to -0.042; in other words, overall 4.5% of participants are incorrectly reclassified). In men, compared to QRISK3, QRISKLifetime slightly improved classification in those who did not experience an event (NRI- = 0.054, 95% CI 0.054 to 0.054), but worsened classification in those who did experience an event (NRI+ = -0.083, 95% CI -0.084 to -0.082), with overall NRI − 0.029 (95% confidence interval − 0.030 to -0.028).

Those recommended for treatment by QRiskLifetime based on 10-year risk were slightly older than those recommended by QRISK3, but patient characteristics were otherwise similar (Table 4). Fewer people were recommended for treatment by QRiskLifetime based on 10-year risk but the percentage experiencing an event was higher (estimated number needed to treat (NNT) from statin prescription to prevent one event in women 34 for QRISK3 vs. 30 for QRiskLifetime; for men 37 vs. 33).

By design, thresholds of predicted lifetime risk for “recommending treatment” were chosen so that exactly the same number of people at highest lifetime risk were identified as were identified by QRISK3 10-year risk ≥ 10% (Table 5). Both tools therefore “recommended” 19.0% of women and 28.0% of men for treatment. Only 5.3% of all women were identified as high-risk by both tools, with a different 13.7% identified as high-risk by one or other of the prediction tools. Similarly, 8.9% of men were identified as high-risk by both prediction tools and a different 19.1% by one or other of the tools. Compared to people identified as high-risk by QRISK3, those with highest predicted lifetime risk were much younger, had lower mean systolic blood pressure, and a lower proportion with treated hypertension, but much higher proportions with family history of premature CVD and from a minority ethnic background, and somewhat higher mean total cholesterol:HDL cholesterol ratio and BMI (Table 4). Compared to those recommended for treatment based on 10-year predicted risk, there were fewer CVD events observed in people at the highest predicted lifetime risk, and the estimated NNT to prevent one CVD event from statin treatment was 99 in women and 100 in men.

Table 5 Reclassification as high or low risk by QRiskLifetime predicting lifetime risk compared to QRISK3 predicting risk over 10-years

Discussion

Similar to the internal validation study [3], this independent evaluation of the QRiskLifetime CVD risk prediction tool finds that it has excellent discrimination in the whole population over a 10-year prediction horizon, but discrimination is poor to moderate in age and CCI subgroups. In terms of calibration over a 10-year prediction horizon, there was some under-prediction in the whole population. Stratified by age, calibration was excellent in women aged 45–64 and good in men aged 45–64, but there was considerable under-prediction in other age-groups which was larger in younger people at higher risk and in all older people.

Over a 10-year prediction horizon at the 10% risk threshold recommended by NICE [9], QRiskLifetime recommended fewer people for statin treatment (15.4% of women and 22.6% of men) than QRISK3 (19.0% of women and 28.0% of men), although the estimated NNT to prevent one CVD event over 10-years was slightly lower for QRiskLifetime.

Comparing those recommended for treatment by QRISK3 predicted 10-year risk ≥ 10% versus the same proportion at highest estimated lifetime risk by QRiskLifetime, the populations recommended for treatment were markedly different, with those at highest predicted lifetime risk being considerably younger, being much more likely to have a family history of premature CVD and be from a minority ethnic background. Treating the same number of patients at highest predicted lifetime risk as the number with QRISK3 10-year risk ≥ 10%, the estimated NNT with a statin to prevent one CVD event over 10 years was approximately three times higher compared to QRISK3 (in women 99 vs. 34; in men 100 vs. 37). Any benefit of treating those at the highest lifetime rather than the highest 10-year CVD risk is therefore considerably deferred.

Important strengths of the study are the use of population data and study design, conduct and reporting consistent with methodology recommendations [13, 24], publishing all codesets used [6], accounting for competing mortality risks, and examining performance in key subgroups. Key limitations are those common to studies using linked routine data. In the context of lifetime risk prediction, the most important of these is the relatively short follow-up of study participants although this is similar to other studies in this context. Constraining validation to events observed over ten years therefore does not allow evaluation of the potential benefit of longer-term prediction in younger people. However, even if data were available, then evaluating model performance over 20 or more years may reduce applicability to contemporary risk prediction given declining secular trends in age-standardised incident CVD. A further limitation is the high proportion of people with missing data. As with the derivation study and other studies, we used multiple imputation but the assumption that data is missing at random may be incorrect [6, 25].

Brotons et al. also found substantial differences in who was recommended for treatment by 10-year vs. lifetime risk prediction tools, but did not validate lifetime predictions. [4] Like QRiskLifetime, the LIFE-CVD risk prediction tool estimates both 10-year and lifetime CVD risk. LIFE-CVD derivation was in a US dataset, with validation in several European cohorts, with reasonable discrimination and whole population calibration at 10-years follow-up [5]. However, unlike this study, calibration was not examined stratified by age and if calibration is less good in older people, then the implication would be that lifetime estimates are also not well calibrated.

Guidelines currently only recommend lifetime CVD risk prediction as an adjunct to 10-year risk prediction [1], but without specifying any risk thresholds for action. In the absence of lifetime follow-up data and in the context of falling age-standardised rates of incidence CVD, there is no way to directly evaluate how well lifetime estimates perform, but given the observed under-prediction over 10-years in every age-group in this study, we believe that QRiskLifetime is likely to under-predict risk over a lifetime. It is unclear whether similar issues apply to other lifetime risk tools because calibration has not been examined in subgroups of age [5, 26]. More broadly, for all CVD risk prediction, excellent discrimination and calibration in the whole population does not mean that discrimination and calibration are good enough in important subgroups [27], and validation should explore subgroup performance [6].

Even if a lifetime prediction tool were well calibrated in different age-groups, lifetime risk prediction requires an assumption that future risk in younger people will be the same as the risk observed in older people now. Given large falls in CVD incidence in recent decades and continuing change in CVD risk profiles (declining smoking but increasing obesity and diabetes), this assumption is a very strong one. Furthermore, although lifetime expected benefit is greater if treatment is started at a younger age, this study finds that the expected benefit in the medium-term (over 10-years) is considerably smaller. Given the lack of direct evidence, early treatment based on predicted lifetime risk therefore requires a leap of faith by both patient and clinician that additional years of early treatment will lead to larger benefit in the distant future. In that context, careful explanation of predicted risks is needed, and patient preferences are critical to take into account [5, 28].

A key limitation in the field is that UK and other linked routine data resources used to derive and validate CVD risk prediction usually suffer from limited follow-up because patients are lost when they deregister with a participating practice or organisation. We constrained validation of performance to 10-years to allow a direct comparison with QRISK3, but even without this, follow-up is constrained by deregistration from participating practices, and very long follow-up also requires the use of very historical baseline data when data quality is poorer and CVD incidence was higher. Improvements in data linkage and increasing access to whole population data have the potential to significantly improve observability over long period of follow-up, and deriving and validating new prediction tools in these datasets which account for competing mortality risk is a priority.

More broadly, lifetime CVD risk prediction is an attempt to deal with a key problem of 10-year CVD risk prediction: that younger people at high risk of premature CVD often do not have 10-year CVD risk that exceeds current threshold for treatment. Using age-stratified 10-year risk thresholds might mitigate this problem [28], but risks large proportions of people being recommended for lifelong medication that most will not benefit from. With advances in cardiac imaging, alternative strategies include using coronary artery calcium scoring [28] or CT coronary angiography (CTCA) to screen people at increased predicted risk for asymptomatic coronary artery disease, and to treat the diseased rather than the at-risk. Early diagnosis and treatment is an attractive strategy given the problems of risk prediction over long periods of time, but while such a strategy using CTCA has been shown to be effective in people with chest pain [29], its value in a true primary prevention population is uncertain and needs to be established [30].

Conclusion

QRiskLifetime under-predicts risk over a 10-year prediction horizon in all patients except women aged 45–64, and is therefore likely to under-predict risk over a lifetime. Given limited follow-up in derivation and validation studies, any lifetime prediction in younger people requires the strong assumption that age-stratified incidence of CVD will remain stable over decades. Compared to treatment based on 10-year risk, treatment based on lifetime risk therefore requires a considerably larger leap of faith on the part of clinicians and patients.