Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Evaluation and comparison of six GRACE models for the stratification of undifferentiated chest pain in the emergency department

BMC Cardiovascular Disorders volume 20, Article number: 199 (2020) Cite this article

Abstract

Background

The Global Registry of Acute Coronary Events (GRACE) score is recommended for stratifying chest pain. However, there are six formulas used to calculate the GRACE score for different outcomes of acute coronary syndrome (ACS), including death (Dth) or composite of death and myocardial infarction (MI), while in hospital (IH), within 6 months after discharge (OH6m) or from admission to 6 months later (IH6m). We aimed to perform the first comprehensive evaluation and comparison of six GRACE models to predict 30-day major adverse cardiac events (MACEs) in patients with acute chest pain in the emergency department (ED).

Methods

Patients with acute chest pain were consecutively recruited from August 24, 2015 to September 30, 2017 from the EDs of two public hospitals in China. The 30-day MACEs included death, acute myocardial infarction (AMI), emergency revascularization, cardiac arrest and cardiogenic shock. The correlation, calibration, discrimination, reclassification and diagnostic accuracy at certain cutoff values of six GRACE models were evaluated. Comparisons with the History, ECG, Age, Risk Factors, and Troponin (HEART) and Thrombolysis in Myocardial Infarction (TIMI) scores were conducted.

Results

A total of 2886 patients were analyzed, with 590 (20.4%) patients experiencing outcomes. The GRACE (IHDthMI), GRACE (IH6mDthMI), GRACE (IHDth), GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI) showed positive linear correlations with the actual MACE rates (r ≥ 0.568, P < 0.001). All these models had good calibration (Hosmer-Lemeshow test, P ≥ 0.073) except GRACE (IHDthMI) (P < 0.001). The corresponding C-statistics were 0.83(0.81,0.84), 0.82(0.81,0.83), 0.75(0.73,0.76), 0.73(0.72,0.75), 0.72(0.70,0.73) and 0.70(0.68,0.71), respectively, first two of which were comparable to HEART (0.82, 0.80–0.83) and superior to TIMI (0.71, 0.69–0.73). With a sensitivity ≥95%, GRACE (IHDthMI) ≤81 and GRACE (IH6mDthMI) ≤79 identified 868(30%) and 821(28%) patients as low risk, respectively, which were significantly better than other GRACEs and HEART ≤3(22%). With a specificity ≥95%, GRACE (IHDthMI) > 186 and GRACE (IH6mDthMI) > 161 could recognize 12% and 11% patients as high risk, which were greater than other GRACEs, HEART ≥8(9%) and TIMI ≥5(8%).

Conclusions

In this Chinese setting, certain strengths of GRACE models beyond HEART and TIMI scores were still noteworthy for stratifying chest pain patients. The validation and reasonable application of appropriate GRACE models in the evaluation of undifferentiated chest pain should be recommended.

Peer Review reports

Background

Chest pain and related symptoms are the most common reasons for patients to present to the emergency department (ED) [1, 2], and present extremely heterogeneous with a wide spectrum of underlying conditions ranging from lethal diseases such as acute myocardial infarction (AMI) to minor acute problems such as intercostal neuralgia. Ruling in or ruling out high-risk conditions in a timely manner is of great importance and a great challenge [3,4,5,6]. Furthermore, the majority of undifferentiated acute chest pain patients are low risk and do not require further invasive tests or admission [6, 7]. Therefore, risk stratification for chest pain patients at EDs has been recommended in several guidelines [6, 8] to not only identify as many true low-risk patients as possible but also avoid missing major adverse cardiac events (MACEs).

The Global Registry of Acute Coronary Events (GRACE) score is an objective prediction tool for definite acute coronary syndrome (ACS), incorporating age, vital signs, kidney function, ECG and troponin levels [9]. This tool has been validated for risk stratification of individuals with acute chest pain [10,11,12,13,14,15,16,17]. In particular, the 0 h/3 h algorithm with the GRACE score incorporated into is recommended (Class I, Level B) for risk stratification and rule-out of AMI in patients with suspected non-ST-elevation ACS by 2015 European Society of Cardiology (ESC) guideline [6].

However, there are six formulas used to calculate the GRACE score for different outcomes, including those for predicting in-hospital death [9], in-hospital death or myocardial infarction (MI) [18], death within 6 months after discharge [19], death or MI within 6 months after discharge, death from admission to 6 months later [20], and death or MI from admission to 6 months later [20]. None of these formulas are specific for rule-out/rule-in of high-risk conditions in patients with undifferentiated chest pain presenting to the ED. The GRACE models have been compared with other scores for stratifying undifferentiated chest pain, such as the History, ECG, Age, Risk Factors, and Troponin (HEART) score and the Thrombolysis in Myocardial Infarction (TIMI) score [21, 22]. Generally, GRACE was inferior to the HEART score, and the most common GRACE score applied was the one for predicting in-hospital death [11,12,13,14,15,16]. The questions of why this model is selected and whether it is the most appropriate one remain unanswered. No study has comprehensively assessed these scores in detail in chest pain patients. Therefore, the superiority of certain GRACE scores remains unclear.

Using a range of model performance indices, we aimed to evaluate the performance of six GRACE models and compare their discrimination, reclassification and diagnostic accuracy with those of HEART and TIMI scores to rule out/rule in 30-day MACEs among acute chest pain patients presenting to the ED.

Methods

Study design

This is a secondary analysis of a previous cohort. We prospectively collected data through an observational study of acute chest pain patients from August 24, 2015 to September 30, 2017 in EDs of two public hospitals in China, the urban ED of the Qilu Hospital of Shandong University (a university-affiliated teaching hospital) and the rural ED of the People’s Hospital of Wenshang County. This study was approved by the ethics committees of the collaborating hospitals. Written informed consent was obtained from all participants.

Patient enrolment

Patients were consecutively recruited if they were aged 18 or older and had acute nontraumatic chest pain and troponin tests. Acute symptoms of myocardial ischaemia or an ischaemic equivalent, such as epigastric discomfort, dyspnoea or fatigue, were also considered as chest pain according to the American Heart Association case definitions [23].

To assess the performance of risk scores for the stratification of non-ST-elevation chest pain, patients with ST-elevation myocardial infarction (STEMI) were excluded. Other exclusion criteria for analysis included patients unable or unwilling to provide informed consent.

Data collection and measurements

Clinical information was extracted from the medical records and collected through patient interviews by research assistants using a standardized case report form (CRF); the variables were in accordance with the international standards [23]. Demographics, risk factors, previous medical history, symptom characteristics, physical examination, vital signs, troponin values, laboratory tests, triage, treatments and outcomes were covered.

Patients participated in follow-up telephone interviews at 30 days after enrolment, and information about MACEs and hospital attendances was collected.

Risk score calculations

Methods and formulas used to calculate the GRACE risk scores have been described in detail previously [24], including all 6 models for predicting in-hospital death (IHDth), in-hospital death or MI (IHDthMI), death within 6 months after discharge (OH6mDth), death or MI within 6 months after discharge (OH6mDthMI), death from admission to 6 months later (IH6mDth), and death or MI from admission to 6 months later (IH6mDthMI) (Table 1). These scores were calculated retrospectively using the prospectively obtained data. An ECG with ST depression (new horizontal or down-sloping ST depression ≥0.05 mV in two contiguous leads) or ST elevation (new ST elevation at the J point in two contiguous leads with the cut-points: ≥0.1 mV in all leads other than leads V2-V3 where the following cut points apply: ≥0.2 mV in men ≥40 years; ≥0.25 mV in men < 40 years, or ≥ 0.15 mV in women) [25] was defined as ischaemic ST deviation. Two independent cardiologists interpreted the ECGs blinded to the clinical data, troponin levels and events. Discrepancies were evaluated by a third cardiologist. The troponin results from the first blood sample arranged by emergency physicians in their daily work were used to calculate scores. The 99th percentile of the upper reference limit (URL) was used as the cutoff for determining positivity. In-hospital PCI and in-hospital CABG were assigned a score of 0 because the GRACE models were used to stratify chest pain patients immediately after arrival at ED and the PCI or CABG performed during admission were not applicable. In emergency care practice, not all GRACE predictor variables can be collected completely, especially the serum creatinine test (not routinely assessed) and the Killip class (not rated in patients without AMI). Here, we performed two kinds of assessment: one was based on the complete GRACE variables with the creatinine value and Killip class assigned as zero if absent; the other was based on the deletion of creatinine and Killip class from all observations using mini-GRACE, which has been introduced in the development of NICE guideline 94 and validated through a large MINAP registry of patients with ACS [26, 27]. The HEART and TIMI scores were calculated according to previous studies (Supplementary Table 1) [22, 28].

Table 1 Variables included in six GRACE models
Full size table

Clinical outcomes

The primary outcome was the composite endpoint of MACEs within 30 days, including death from all causes, AMI (index and subsequent), emergency revascularization, cardiac arrest and cardiogenic shock. The diagnosis of AMI was performed according to the third universal definition of MI, as a detection of the rise and/or fall in cardiac biomarkers with at least one value above the 99th percentile of URL and with symptoms or ECG changes or imaging indicative of new ischaemia [25]. Cardiogenic shock was defined as persistent (> 30 min) systolic blood pressure (SBP) of less than 90 mmHg and/or cardiac index < 2.2 L/min/m2 secondary to cardiac dysfunction, requiring intravenous inotropic or mechanical support [23]. Two senor cardiologists from the clinical events committee adjudicated the MACEs independently using all available clinical records, and discrepancies were evaluated by a third senior physician. If patients were lost to follow-up, a local death registry was used to supplement the survival status.

Statistical analysis

Continuous variables are presented as the mean (standard deviation), and categorical variables are presented as the number of cases (percentage). Baseline characteristics between the MACE and no MACE groups were compared using t tests for continuous variables and chi-square (χ2) tests for categorical variables. Pearson product-moment correlation was used as “r” to describe the direction and quantify the strength of the linear association between GRACE scores and the incidence of MACEs in individuals with chest pain. The calibration was evaluated using the Hosmer-Lemeshow goodness-of-fit test (HLT). Low HLT χ2 and P value > 0.05 illustrate agreement between observed and predicted probabilities of an event and good model fit. Discrimination of scores was assessed by the area under the curve (AUC) of receiver operating characteristic (ROC) curves. An AUC of 0.8 ~ 0.9 is considered excellent and 0.7 ~ 0.8 acceptable [29]. Taking into account the implicit correlation between the curves of these scores, we used the Delong test to compare any two AUCs [30]. Reclassification was performed to assess how well a risk score improved predictions compared with another one based on category-free net reclassification improvement (NRI) and absolute integrated discrimination improvement (IDI). Diagnostic accuracy with 95% confidence intervals (CIs) of the different scores was determined, including sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV). To guarantee the safety of discharging low-risk patients, cutoff values of each model to obtain sensitivities of at least 95%, 98% and 99% were identified. High-risk cutoff values were set at a specificity of ≥95% or ≥ 90%. The sensitivity and specificity of different strategies were compared using the McNemar test, while proportions of patients classified as low-risk and high risk, NPV and PPV were compared using the χ2 test. A P value of less than 0.05 (two-sided significance testing) was considered statistically significant in the analysis. All statistical analyses were performed using SAS V.9.4 (SAS Institute Inc., Cary, North Carolina, USA) or MedCalc V.18.11.3 (MedCalc Software, Ostend, Belgium).

Results

Study population

A total of 3536 patients with acute nontraumatic chest pain and initial cTn tests presented in the participating EDs from August 24, 2015 to September 30, 2017. Some patients were excluded due to denial of informed consent (77) and diagnosis of STEMI (472). There were 88 patients with insufficient information to calculate the GRACE scores, including 74 due to no initial ECG and 14 due to no SBP values. For 13 surviving patients, follow-up contacts were unsuccessful. Eventually, 2886 patients remained for analysis (Fig. 1). Baseline characteristics and initial evaluation between patients with and without 30-day MACEs are compared in Table 2. Patients with 30-day MACEs tended to be older, be male, have a higher burden of risk factors and have significantly higher GRACE scores than those without 30-day MACEs (P < 0.001).

Fig. 1
figure 1

Study flowchart. cTn cardiac troponin, ECG electrocardiography, STEMI ST-segment elevation myocardial infarction, SBP systolic blood pressure

Full size image
Table 2 Baseline characteristics of the study population
Full size table

Outcomes

There were 590 (20.4%) chest pain patients with adjudicated MACEs in 30 days after presentation, including 52 patients (1.8%) who died from all causes, 549 (19.0%) with index AMI, 24 (0.8%) with subsequent AMI, 10 (0.3%) who underwent emergency revascularization, 32 (1.1%) who experienced cardiac arrest and 32 (1.1%) who experienced cardiogenic shock.

Correlation between GRACE scores and actual event rates

All six GRACE models showed good positive linear correlation with the actual MACE rates in patients with undifferentiated chest pain (Fig. 2). The GRACE (IHDthMI) and GRACE (IH6mDthMI) exhibited very strong relationships, with r values of 0.913 and 0.920, respectively (P < 0.001).

Fig. 2
figure 2

Pearson product-moment correlation between GRACE scores and actual event rates. GRACE Global Registry of Acute Coronary Events, MACE major adverse cardiac events

Full size image

Agreement between observed and predicted probabilities of an event

As shown in Fig. 3, the predicted probabilities of an event were much close to the observed event rates across deciles of five GRACE models. And the HLT P values for the GRACE (IH6mDthMI), GRACE (IHDth), GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI) were 0.113, 0.446, 0.608, 0.312 and 0.073, respectively. However, the P value of GRACE (IHDthMI) was < 0.001.

Fig. 3
figure 3

Hosmer-Lemeshow goodness-of-fit tests of six GRACE models. GRACE Global Registry of Acute Coronary Events, MACE major adverse cardiac events

Full size image

Discrimination

The ROC curves of all the GRACE models, HEART and TIMI scores are depicted in Fig. 4. The AUCs of GRACE (IHDthMI), GRACE (IH6mDthMI), GRACE (IHDth), GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI) were 0.83 (0.81, 0.84), 0.82 (0.81, 0.83), 0.75 (0.73, 0.76), 0.73 (0.72, 0.75), 0.72 (0.70, 0.73) and 0.70 (0.68, 0.71), respectively. The AUCs of GRACE (IHDthMI) and GRACE (IH6mDthMI) were equal to the C-statistic of HEART score at 0.82 (0.80, 0.83) and superior to the other GRACE models and the TIMI score (0.71, 0.69–0.73) (P < 0.001). The C-statistics of the GRACE models and the HEART and TIMI scores in each participating hospital are presented in Supplementary Table 2.

Fig. 4
figure 4

Receiver operating characteristic curves of six GRACE models (a), HEART and TIMI scores (b) for the prediction of MACE within 30 days. GRACE Global Registry of Acute Coronary Events, HEART History, ECG, Age, Risk factors, Troponin, MACE major adverse cardiac events, TIMI Thrombolysis in Myocardial Infarction. * Significantly different from the HEART score. # Significantly different from the TIMI score

Full size image

Reclassification

As shown in Table 3, the GRACE (IHDthMI) improved risk classifications of chest pain patients with positive NRI and IDI over the other GRACE models and the TIMI score (P < 0.001), while it presented comparable ability to the HEART score. The same trend was seen in the GRACE (IH6mDthMI) model.

Table 3 The reclassification measurements of the GRACE (IHDthMI) and GRACE (IH6mDthMI) compared with other scores
Full size table

Rule-out and rule-in of events

For discharging low-risk patients safely with sensitivity ≥95% (as shown in Table 4), GRACE (IHDthMI) ≤81 and GRACE (IH6mDthMI) ≤79 could identify 868 (30%) and 821 (28%) patients as low risk, respectively, which were significantly better than other GRACEs and a HEART score ≤ 3 (22%). Additionally, GRACE (IHDthMI) ≤81 and GRACE (IH6mDthMI) ≤79 ruled out MACEs with an NPV of 0.967 (0.955, 0.979) and 0.965 (0.952, 0.977), respectively, exceeding other GRACEs and comparable to a HEART score ≤ 3 with an NPV of 0.970 (0.957, 0.983). If the sensitivity was set at ≥98%, the proportion of patients identified as low risk would decrease to 14% for GRACE (IHDthMI) ≤64 and GRACE (IH6mDthMI) ≤61 with no change in the superiority to HEART ≤2 (11%) and TIMI =0 (12%) (Supplementary Table 3). If the sensitivity was improved up to ≥99%, the proportions of low-risk patients would drop below 10% for GRACEs, which were still significantly greater than a HEART score ≤ 1 (3%) (Supplementary Table 3).

Table 4 Performance of different models in terms of diagnostic accuracy at cutoff values with certain sensitivity and specificity levels
Full size table

Regarding ruling in MACEs with specificity ≥95% (Table 4), GRACE (IHDthMI) > 186 and GRACE (IH6mDthMI) > 161 could recognize 12% and 11% of patients as high risk, respectively, which were more than a HEART ≥8 (9%) and a TIMI ≥5 (8%). The sensitivities of GRACE (IHDthMI) > 186 and GRACE (IH6mDthMI) > 161 were even better than those of HEART ≥8 and TIMI ≥5 without compromising specificity. As shown in Supplementary Table 4, GRACE (IHDthMI) > 168 and GRACE (IH6mDthMI) > 146 recognized 19% and 18% of patients as high risk, which were smaller than the proportion (22%) identified by the HEART score (7–10). However, the GRACE (IHDthMI) > 168 had greater specificity (0.904, 0.892–0.916) and PPV (0.591, 0.550–0.633) than HEART ≥7, with a specificity of 0.870 (0.856, 0.884) and a PPV of 0.534 (0.496, 0.573).

Performance of the mini-GRACE models

The mini-GRACE (IHDthMI), mini-GRACE (IH6mDthMI), mini-GRACE (IHDth), mini-GRACE (IH6mDth) and mini-GRACE (OH6mDth) showed positive linear correlations with the actual MACE rates (r ≥ 0.793, P < 0.001). A very strong relationship remained in the mini-GRACE (IH6mDthMI) (r = 0.917). The mini-GRACE (IHDth), mini-GRACE (IH6mDth) and mini-GRACE (OH6mDth) had good calibration (P ≥ 0.517) while the other two did not (Supplementary Table 5). The mini-GRACE (IHDthMI) and mini-GRACE (IH6mDthMI) models, with AUCs of 0.82 (0.80, 0.83) and 0.81 (0.79, 0.82), respectively, were still superior to other models in discrimination and reclassification (Supplementary Figure 1) (Supplementary Table 6).

Discussion

This study provides the first comprehensive evaluation and comparison of all six GRACE risk-prediction models in patients with undifferentiated chest pain. In the two Chinese EDs included in this study, all six GRACEs showed a positive linear correlation with actual MACE rates, and the five models had good calibration. All the C-statistics were ≥ 0.70. The GRACE (IHDthMI) and GRACE (IH6mDthMI) exhibited very strong relationships with actual MACE rates (r > 0.9) and showed excellent discriminatory capability (AUC > 0.80). Improvements in AUC, NRI and IDI indicated that GRACE (IHDthMI) and GRACE (IH6mDthMI) were comparable to the HEART score and superior to the other models.

The GRACE risk scores were developed using multivariable regression to assist cardiologists in estimating the risk of different outcomes in hospitalized patients with ACS and have been indicated to provide the most accurate stratification of risk of ACS both on admission and at discharge [31, 32]. One model is specific to one kind of outcome, including death or composite of MI and death during hospitalization, within 6 months after discharge and from admission to 6 months later. The MI referred to here is the subsequent AMI occurring after the index ACS. However, for undifferentiated chest pain, the high-risk conditions mainly present a composite endpoint of index AMI, subsequent AMI, death, emergency revascularization, cardiac arrest and cardiogenic shock within 30 days after presentation to the ED [33]. The incidence of index AMI is much greater than that of subsequent AMI, as shown in our study. Our results suggested that the GRACE models showed at least a moderate correlation with the actual incidence of MACEs in the undifferentiated chest pain cohort. In particular, very strong correlations appeared in GRACE (IHDthMI) and GRACE (IH6mDthMI). Furthermore, the predicted probabilities of an event and the observed event rates were significantly similar across deciles of five GRACE models. Therefore, there are foundations for the GRACE models to provide accurate stratification of patients with acute chest pain.

In previous studies, C-statistics for predicting 30-day MACEs in chest pain patients were merely evaluated according to the GRACE (IHDth) with AUCs of 0.60 to 0.83, which were always inferior to those of the HEART score [10,11,12,13,14,15,16]. Consistently, we found that the AUC of the GRACE (IHDth) was 0.75 (0.73, 0.76), which was actually lower than that of the HEART score in this study. However, GRACE (IHDth) was neither the only GRACE model nor the best GRACE model for stratifying chest pain. The GRACE (IHDthMI) and GRACE (IH6mDthMI) had better total discriminatory capability (AUC > 0.8) and reclassification without difference from the HEART score. Although the performance of all these models was not good in the rural hospital as in the urban hospital, the advantages of GRACE (IHDthMI), GRACE (IH6mDthMI) and HEART were consistent in both EDs. Significantly positive NRI and IDI in this study showed that the GRACE (IHDthMI) and GRACE (IH6mDthMI) could provide a higher predicted probability of an event for high-risk patients and a lower predicted probability for low-risk patients than the other four models. The possible explanation may be that events predicted by these two GRACE models referred to attacks of AMI rather than merely death, though not the index AMI. Compared with the models for events after discharge, the periods in the hospital or from admission to 6 months later were closer to the 30-day follow-up after presentation to the ED. Our results did not refute previous conclusions but complemented them by providing more complete recognition of the GRACE models.

Exact cutoff values should be determined for clinical use to identify low-risk patients for safe and early discharge without compromising the immediate treatment of high-risk chest pain. Reaney et al. found that GRACE (IHDth) 0–55 could reach a sensitivity of 95.2% and NPV of 95.8%, identifying 21.2% patients as low risk. GRACE (IHDth) ≥119 defined 16% of patients as high risk (specificity 89.8%; PPV 48.1%) [16]. Poldervaart et al. determined GRACE (IHDth) ≤72 as the cutoff and 19.1% patients were classified as low-risk (sensitivity 95%; NPV 96%) [14]. Cullen et al. chose the cutoff of GRACE (OH6mDth) ≤50 to determine low-risk (24% patients) with a sensitivity of 98.9%, and the cutoff for recognizing high risk (28% patients) was ≥100, with a specificity of 76.2% [17]. In our study, the performance of GRACE (IHDth) was relatively consistent with that of previous studies, with a value of ≤79 identifying 18% patients as low risk (sensitivity 95.1%; NPV 94.4%) and a value of > 145 defining 16% patients as high risk (specificity 90.0%; PPV 49.9%). At the same sensitivity and specificity, the GRACE (IHDthMI) and GRACE (IH6mDthMI) outperformed the GRACE (IHDth) and other GRACEs. Although there is no rigorous standard for the sensitivity of risk-stratification models for chest pain, an international survey suggested that clinicians may expect a sensitivity of 99% or higher for AMI or other MACEs [34]. If the sensitivity was set at ≥99%, GRACE (IHDthMI) and GRACE (IH6mDthMI) were still superior, but the proportions of low-risk patients would drop below 10%. A meta-analysis demonstrated that the pooled sensitivity and specificity of a HEART score ≤ 3 for predicting MACEs were 96.7% (94.0, 98.2%) and 47.0% (41.0, 53.5%), respectively [35]. HEART≤3 in our cohort had a similar sensitivity (96.8%) but a lower specificity (27.0%). The sensitivity of HEART ≤2 was higher at 98.8%(97.9, 99.7%) at the cost of a lower proportion (11%) of patients identified as low risk. In our previous report, the HEART score would not appear to provide additionally helpful risk stratification to the usual care for discharging low-risk patients [36]. Regarding the high-risk category, HEART ≥7 did not perform as well (specificity 87.0%; PPV 53.4%) as in Reaney’s study [16].

For ruling out and ruling in MACEs, the HEART score illustrated a certain advantage over the GRACE (IHDth) but not the GRACE (IHDthMI) or GRACE (IH6mDthMI). The strengths of GRACE are still noteworthy. Possible explanations might be the detailed class and objectivity of components of the GRACE beyond HEART and TIMI. Although the HEART score was directly developed for undifferentiated patients, the assignment of every variable only included three qualitative classes (i.e., 0,1,2) [21]. The classes for each component of TIMI score are even lesser (only 0 or 1) [22]. In contrast, the GRACE scores included many more quantitative variables, such as age, SBP, pulse and creatinine, which are supposed to identify subtler differences and result in more exact stratification. As highlighted by the 0 h/1 h algorithm from the ESC guideline recommendations, quantitative interpretation overcomes the qualitative interpretation of high-sensitivity troponin levels for ruling out and ruling in AMI in chest pain patients, and the cutoff levels are assay specific [6, 37]. Furthermore, some “soft” variables are included in the HEART score, such as the medical history, risk factors and symptoms. It has been shown that these variables do not have a sufficient discriminatory ability to rule in or rule out ACS in the ED [38]. The combination of symptom variables as a “history” component in the HEART score was still not clearly stated and not assessed systematically [39]. The GRACE score can avoid this situation due to the absence of subjective variables. The popularity of handheld devices has made the complexity of GRACE no longer a disadvantage.

The results from the assessment of mini-GRACE were mainly in accordance with those of the complete models. Although the correlation of the mini-GRACE (IHDthMI) and the calibration of the mini-GRACE (IH6mDthMI) were lower than the complete models, the discrimination and reclassification of these two mini scores remained excellent and significantly outperformed other models. This illustrates that the differences in model performance may be due to the disparities in weights of shared variables.

Limitations

This study had several limitations. First, the performance of different GRACE scores was assessed in chest pain patients from two hospitals in China. Although urban and rural hospitals were both included, the validation of each score in wider patients should be determined by further studies of heterogeneous groups. In particular, the cutoff levels of the GRACEs are not the same in different studies due to the disparity of inclusion and exclusion criteria and the incidence and definition of MACEs. Determination and validation of the specific cutoff values in clinical practice in certain hospitals are needed. Second, the cardiac marker used in the calculation of scores was the contemporary cTn assay arranged by emergency physicians in their daily work. The ability of scores combined with high-sensitivity cTns to stratify chest pain still needs to be evaluated in future studies. Third, all components used in the risk scores were calculated automatically through a computer algorithm. The ECG variables were based on the standard interpretation from senior cardiologists. This calculation process deviated from clinical reality. Further studies to evaluate the discrimination of scores calculated immediately by the treating physicians are needed.

Conclusions

From our evaluation and comparison of the six GRACE models in a prospective cohort of undifferentiated chest pain patients in two Chinese EDs, we found that all six GRACE models presented acceptable or excellent discriminatory capacity for predicting 30-day MACEs. In particular, the GRACE (IHDthMI) and GRACE (IH6mDthMI) were comparable to the HEART score and superior to other GRACEs and TIMI in terms of discrimination and reclassification. At a certain sensitivity and specificity, GRACE (IHDthMI) and GRACE (IH6mDthMI) could identify more patients to rule out or rule in 30-day MACEs than other models. Although cutoff levels of the GRACEs may be specific in different cohorts and validation of these levels are needed, the reasonable application of appropriate GRACE models in the evaluation of undifferentiated chest pain patients in ED should be recommended.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

ACS:

Acute coronary syndrome

AMI:

Acute myocardial infarction

AUC:

Area under the curve

CIs:

Confidence intervals

CRF:

Case report form

ED:

Emergency department

GRACE:

Global registry of acute coronary events

HEART:

History, ECG, age, risk factors, troponin

HLT:

Hosmer-Lemeshow goodness-of-fit test

IDI:

Integrated discrimination improvement

MACEs:

Major adverse cardiac events

NRI:

Net reclassification improvement

NPV:

Negative predictive value

PPV:

Positive predictive value

STEMI:

ST-elevation myocardial infarction

TIMI:

Thrombolysis in myocardial infarction

URL:

Upper reference limit

References

  1. Centers for Disease Control. National Hospital Ambulatory Medical Care Survey: 2011 Emergency Department Summary Tables. 2011. http://www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/2011_ed_web_tables.pdf.

  2. Xue J, Han Z, Wang M, et al. Main etiologies for patients presented to ER with chest pain or chest pain equivalent. Clin Med China. 2012;28(10):1042–6.

    Google Scholar 

  3. Ayerbe L, González E, Gallo V, Coleman CL, Wragg A, Robson J. Clinical assessment of patients with chest pain; a systematic review of predictive tools. BMC Cardiovasc Disord. 2016;16:18.

    Article  Google Scholar 

  4. Amsterdam EA, Wenger NK, Brindis RG, et al. 2014 AHA/ACC guideline for the Management of Patients with non-ST-elevation acute coronary syndromes: a report of the American College of Cardiology/American Heart Association task force on practice guidelines. J Am Coll Cardiol. 2014;64(24):e139–228.

    Article  Google Scholar 

  5. Pope JH, Aufderheide TP, Ruthazer R, et al. Missed diagnoses of acute cardiac ischemia in the emergency department. N Engl J Med. 2000;342(16):1163–70.

    Article  CAS  Google Scholar 

  6. Roffi M, Patrono C, Collet JP, et al. 2015 ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: task force for the Management of Acute Coronary Syndromes in patients presenting without persistent ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J. 2016;37(3):267–315.

    Article  CAS  Google Scholar 

  7. Amsterdam EA, Kirk JD, Bluemke DA, et al. Testing of low-risk patients presenting to the emergency department with chest pain: a scientific statement from the American Heart Association. Circulation. 2010;122(17):1756–76.

    Article  Google Scholar 

  8. Chew DP, Scott IA, Cullen L, et al. National Heart Foundation of Australia & Cardiac Society of Australia and new Zealand: Australian clinical guidelines for the Management of Acute Coronary Syndromes 2016. Heart Lung Circ. 2016;25(9):895–951.

    Article  Google Scholar 

  9. Granger CB, Goldberg RJ, Dabbous O, et al. Predictors of hospital mortality in the global registry of acute coronary events. Arch Intern Med. 2003;163(19):2345–53.

    Article  Google Scholar 

  10. Lyon R, Morris AC, Caesar D, Gray S, Gray A. Chest pain presenting to the emergency department--to stratify risk with GRACE or TIMI? Resuscitation. 2007;74(1):90–3.

    Article  Google Scholar 

  11. Graham C, Mo J-r, Jiang H-l, et al. Comparison of the prognostic value of TIMI, GRACE, Banach and HEART scores to evaluate patients with chest pain in a Guangzhou emergency department: Prospective observational study. Resuscitation. 2014;85:S17.

    Article  Google Scholar 

  12. Reaney PEH, Cooper J. A comparision of the HEART, TIMI and GRACE scores in the prediction of a major adverse cardiac event (MACE) in patients presenting with undifferentiated cardiac chest pain: a prospective cohort study in a UK population. Emerg Med J. 2016;33:916–7.

    Article  Google Scholar 

  13. Sakamoto JT, Liu N, Koh ZX, et al. Comparing HEART, TIMI, and GRACE scores for prediction of 30-day major adverse cardiac events in high acuity chest pain patients in the emergency department. Int J Cardiol. 2016;221:759–64.

    Article  Google Scholar 

  14. Poldervaart JM, Langedijk M, Backus BE, et al. Comparison of the GRACE, HEART and TIMI score to predict major adverse cardiac events in chest pain patients at the emergency department. Int J Cardiol. 2017;227:656–61.

    Article  CAS  Google Scholar 

  15. Ramsay G, Podogrodzka M, McClure C, Fox KA. Risk prediction in patients presenting with suspected cardiac pain: the GRACE and TIMI risk scores versus clinical evaluation. QJM. 2007;100(1):11–8.

    Article  CAS  Google Scholar 

  16. Reaney PDW, Elliott HI, Noman A, Cooper JG. Risk stratifying chest pain patients in the emergency department using HEART, GRACE and TIMI scores, with a single contemporary troponin result, to predict major adverse cardiac events. Emerg Med J. 2018;35(7):420–7.

    Article  Google Scholar 

  17. Cullen L, Greenslade J, Hammett CJ, et al. Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an Australian emergency department. Heart Lung Circ. 2013;22(10):844–51.

    Article  Google Scholar 

  18. Eagle KA, Lim MJ, Budaj A, et al. A robust prediction model for all forms of acute coronary syndromes: estimating the risk of in-hospital death and myocardial infarction in the global registry of acute coronary events registry. J Am Coll Cardiol. 2003;41(6):353–4.

    Article  Google Scholar 

  19. Eagle KA, Lim MJ, Dabbous OH, et al. A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry. J Am Med Assoc. 2004;291(22):2727–33.

    Article  CAS  Google Scholar 

  20. Fox KA, Dabbous OH, Goldberg RJ, et al. Prediction of risk of death and myocardial infarction in the six months after presentation with acute coronary syndrome: prospective multinational observational study (GRACE). BMJ. 2006;333(7578):1091.

    Article  Google Scholar 

  21. Six AJ, Backus BE, Kelder JC. Chest pain in the emergency room: value of the HEART score. Neth Heart J. 2008;16(6):191–6.

    Article  CAS  Google Scholar 

  22. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA. 2000;284(7):835–42.

    Article  CAS  Google Scholar 

  23. Cannon CP, Brindis RG, Chaitman BR, et al. 2013 ACCF/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes and coronary artery disease: a report of the American College of Cardiology Foundation/American Heart Association task force on clinical data standards (writing committee to develop acute coronary syndromes and coronary artery disease clinical data standards). Circulation. 2013;127(9):1052–89.

    Article  Google Scholar 

  24. Center for Outcomes Research UoMMS. Methods and formulas used to calculate the GRACE risk scores for patients presenting to hospital with an acute coronary syndrome. Worcester, MA: University of Massachusetts Medical School, 2011. http://www.outcomes-umassmed.org/grace/files/GRACE_RiskModel_Coefficients.pdf.

  25. Thygesen K, Alpert JS, Jaffe AS, et al. Third universal definition of myocardial infarction. Eur Heart J. 2012;33(20):2551–67.

    Article  Google Scholar 

  26. Royal College of Physicians. Unstable angina and NSTEMI: the early Management of Unstable Angina and non-ST-segment-elevation myocardial infarction: clinical guideline 94. London: National Institute for Health and Clinical Excellence; 2010.

    Google Scholar 

  27. Simms AD, Reynolds S, Pieper K, et al. Evaluation of the NICE mini-GRACE risk scores for acute myocardial infarction using the myocardial Ischaemia National Audit Project (MINAP) 2003-2009: National Institute for cardiovascular outcomes research (NICOR). Heart. 2013;99(1):35–40.

    Article  Google Scholar 

  28. Mahler SA, Riley RF, Hiestand BC, et al. The HEART pathway randomized trial: identifying emergency department patients with acute chest pain for early discharge. Circ Cardiovasc Qual Outcomes. 2015;8(2):195–203.

    Article  Google Scholar 

  29. Hosmer DW, Lemeshow S. Applied logistic regression. New York: Wiley, Inc; 2000.

    Book  Google Scholar 

  30. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.

    Article  CAS  Google Scholar 

  31. Investigators G. Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes. Am Heart J. 2001;141(2):190–9.

    Article  Google Scholar 

  32. Aragam KG, Tamhane UU, Kline-Rogers E, et al. Does simplicity compromise accuracy in ACS risk prediction? A retrospective analysis of the TIMI and GRACE risk scores. PLoS One. 2009;4(11):e7947.

    Article  Google Scholar 

  33. Hollander JE, Blomkalns AL, Brogan GX, et al. Standardized reporting guidelines for studies evaluating risk stratification of ED patients with potential acute coronary syndromes. Acad Emerg Med. 2004;11(12):1331–40.

    Article  Google Scholar 

  34. Than M, Herbert M, Flaws D, et al. What is an acceptable risk of major adverse cardiac event in chest pain patients soon after discharge from the emergency department?: a clinical survey. Int J Cardiol. 2013;166(3):752–4.

    Article  Google Scholar 

  35. Van Den Berg P, Body R. The HEART score for early rule out of acute coronary syndromes in the emergency department: a systematic review and meta-analysis. Eur Heart J Acute Cardiovasc Care. 2018;7(2):111–9.

    Article  Google Scholar 

  36. Wang G, Zheng W, Wu S, et al. Comparison of usual care and the HEART score for effectively and safely discharging patients with low-risk chest pain in the emergency department: would the score always help? Clin Cardiol. 2020;43(4):371–8.

  37. Reichlin T, Schindler C, Drexler B, et al. One-hour rule-out and rule-in of acute myocardial infarction using high-sensitivity cardiac troponin T. Arch Intern Med. 2012;172(16):1211–8.

    Article  Google Scholar 

  38. Hollander JE, Than M, Mueller C. State-of-the-art evaluation of emergency department patients presenting with potential acute coronary syndromes. Circulation. 2016;134(7):547–64.

    Article  Google Scholar 

  39. Zheng W, Ma J, Wu S, et al. Effective combination of isolated symptom variables to help stratifying acute undifferentiated chest pain in the emergency department. Clin Cardiol. 2019;42(4):467–75.

    Article  Google Scholar 

Download references

Acknowledgements

Special thanks to the research assistants in each participating hospital and all patients who agreed to take part in this study.

Funding

This study was supported by the National Key R&D Program of China (2017YFC0908700, 2017YFC0908703), National S&T Fundamental Resources Investigation Project (2018FY100600, 2018FY100602), Taishan Pandeng Scholar Program of Shandong Province (tspd20181220), Taishan Young Scholar Program of Shandong Province (tsqn20161065, tsqn201812129) and Key R&D Program of Shandong Province (2016ZDJS07A14, 2018GSF118003), which provided financial support for the collection, analysis, and interpretation of data.

Author information

Author notes

  1. Wen Zheng and Guangmei Wang contributed equally to this work.

Authors and Affiliations

  1. Department of Emergency Medicine and Chest Pain Center, Qilu Hospital of Shandong University, No.107, Wen Hua Xi Road, Jinan, 250012, Shandong, China

    Wen Zheng, Guangmei Wang, Jingjing Ma, Shuo Wu, He Zhang, Jiaqi Zheng, Feng Xu, Jiali Wang & Yuguo Chen

  2. Clinical Research Center for Emergency and Critical Care Medicine of Shandong Province, Institute of Emergency and Critical Care Medicine of Shandong University, Qilu Hospital of Shandong University, Jinan, China

    Wen Zheng, Guangmei Wang, Jingjing Ma, Shuo Wu, He Zhang, Jiaqi Zheng, Feng Xu, Jiali Wang & Yuguo Chen

  3. Key Laboratory of Emergency and Critical Care Medicine of Shandong Province, Key Laboratory of Cardiopulmonary-Cerebral Resuscitation Research of Shandong Province, Qilu Hospital of Shandong University, Jinan, China

    Wen Zheng, Guangmei Wang, Jingjing Ma, Shuo Wu, He Zhang, Jiaqi Zheng, Feng Xu, Jiali Wang & Yuguo Chen

  4. The Key Laboratory of Cardiovascular Remodeling and Function Research, Chinese Ministry of Education, The State and Shandong Province Joint Key Laboratory of Translational Cardiovascular Medicine, Chinese Ministry of Health and Chinese Academy of Medical Sciences, Qilu Hospital of Shandong University, Jinan, China

    Wen Zheng, Guangmei Wang, Jingjing Ma, Shuo Wu, He Zhang, Jiaqi Zheng, Feng Xu, Jiali Wang & Yuguo Chen

Authors
  1. Wen Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangmei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingjing Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. He Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiaqi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Feng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jiali Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yuguo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YC, JW and FX contributed to the initiation, planning and conduction of the study. WZ, GW, JM, HZ and JZ conducted the study supervision. WZ, GW and SW did the analysis and interpretation of the data. WZ and GW drafted the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Yuguo Chen.

Ethics declarations

Ethics approval and consent to participate

Ethical approval was obtained from Qilu Hospital of Shandong University Medical Ethics Committee (No. (2015)058) and People’s Hospital of Wenshang County Medical Ethics Committee. Written informed consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: Figure S1.

Receiver operating characteristic curves of mini-GRACE models to predict MACEs within 30 days. GRACE Global Registry of Acute Coronary Events, MACEs major adverse cardiac events. * Significantly different from the HEART score. # Significantly different from the TIMI score. Table S1. Components and assignments of the HEART and TIMI scores. Table S2. C-statistics of GRACE models, HEART and TIMI scores to predict MACEs within 30 days in each participating hospital. Table S3. Performance of different models in terms of diagnostic accuracy at cutoff values with certain sensitivities. Table S4. Performance of different models in terms of diagnostic accuracy at cutoff values with specificity ≥90%. Table S5. Measures of the correlation and calibration of the mini-GRACE models. Table S6. The reclassification measurements of the mini-GRACE (IHDthMI) and mini-GRACE (IH6mDthMI) compared with other scores.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, W., Wang, G., Ma, J. et al. Evaluation and comparison of six GRACE models for the stratification of undifferentiated chest pain in the emergency department. BMC Cardiovasc Disord 20, 199 (2020). https://doi.org/10.1186/s12872-020-01476-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12872-020-01476-3

Keywords

BMC Cardiovascular Disorders

ISSN: 1471-2261

Contact us