Evaluation and comparison of six GRACE models to stratify undifferentiated chest pain at the emergency department

Background The Global Registry of Acute Coronary Events (GRACE) score is recommended for stratifying chest pain. However, there are six formulas used to calculate the GRACE score for different outcomes of acute coronary syndrome (ACS), including death (Dth) or composite of death and myocardial infarction (MI), during in hospital (IH), in 6 months after discharge (OH6m) or from admission to 6 months later (IH6m). The more appropriate one for strati�cation of undifferentiated chest pain remains unclear. We aimed to provide �rstly comprehensive evaluation and comparison of six GRACE models to predict 30-day major adverse cardiac events (MACE) in acute chest pain at the emergency department (ED). Methods Patients with acute chest pain were consecutively recruited from August 24, 2015 to September 30, 2017 in EDs of two public hospitals in China. The primary outcome was MACE within 30 days, including death, acute myocardial infarction (MI), emergency revascularization, cardiac arrest and cardiogenic shock. GRACE scores were calculated retrospectively using the prospectively obtained data. Correlation, calibration, discrimination and reclassi�cation of six GRACE models were evaluated. Results A total of 2886 patients were analyzed, with 590 (20.4%) patients getting outcomes. The GRACE (IH6mDthMI), GRACE (IHDthMI), GRACE (IHDth), GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI) showed positive linear correlation with actual MACE rates (r ≥ 0.568, p<0.001), �rst two of which exerted very strong relationships (r>0.9). All these models had good calibration (Hosmer-Lemeshow goodness-of-t test, p ≥ 0.073) except GRACE (IHDthMI) (p<0.001). The corresponding c-statistics were 0.82(0.81,0.83), 0.83(0.81,0.84), 0.75(0.73,0.76), 0.73(0.72,0.75), 0.72(0.70,0.73) and 0


Background
Chest pain and related symptoms are the most common reasons for patients to present to the emergency department (ED), 1,2 and present extremely heterogeneous with a wide spectrum of conditions ranging from lethal diseases such as acute myocardial infarction (AMI) to minor acute problems such as intercostal neuralgia.4][5] Furthermore, the majority of undifferentiated acute chest pain patients are actually low risk and do not require further invasive tests or admission. 4,6 herefore, risk strati cation for chest pain patients at the EDs has been recommended in several guidelines, 4,7 to not only identify true low risk patients as many as possible but also avoid missing major adverse cardiac events (MACE).
The Global Registry of Acute Coronary Events (GRACE) score is an objective prediction tool for de nite acute coronary syndrome (ACS), incorporating age, vital signs, kidney function, ECG and troponin levels. 8is tool has been validated and recommended in guidelines for risk strati cation in acute chest pain. 3,4,7 Hoever, there are six formulas used to calculate the GRACE score for different outcomes, including the ones for predicting in-hospital death, 8 in-hospital death or myocardial infarction(MI), 9 death in 6 months after discharge, 10 death or MI in 6 months after discharge, 11 death from admission to 6 months later, 12 and death or MI from admission to 6 months later. 12.No one is speci c for rule-out of high-risk conditions in patient with undifferentiated chest pain presenting to the ED.4][15][16][17][18][19] No study has assessed them all comprehensively in details in chest pain patients.So, the superiority of certain GRACE scores still remains unclear.
Using a range of model performance indices, we aimed to evaluate the performance of six GRACE models and compare the discrimination capacity of them to predict 30-day MACE in acute chest pain presenting to the ED.

Study design
We prospectively collected data through an observational study of acute chest pain from August 24, 2015   to September 30, 2017 in EDs of two public hospitals in China, the urban ED of the Qilu Hospital of Shandong University (a university-a liated teaching hospital) and the rural ED of the People's Hospital of Wenshang County.This study has been approved by the ethics committee at collaborating hospitals.
Written informed consent was obtained from all participants.

Patients enrollment
Patients were consecutively recruited if they were aged 18 or older, with acute nontraumatic chest pain and troponin tests.Acute symptoms of myocardial ischemia or an ischemic equivalent, such as epigastric discomfort, dyspnea or fatigue, were also considered as chest pain according to the American Heart Association case de nitions. 20r assessing the performance of risk scores on strati cation of non-ST-elevation chest pain, patients with ST-elevation myocardial infarction (STEMI) were excluded.Other exclusion criteria for analysis included patients unable or unwilling to provide informed consent.

Data collection and measurements
Clinical information were extracted from the medical records and collected through patient interviews by research assistants using a standardized case report form (CRF), in which the variables were in accordance with the international standards. 20,21 emographics, risk factors, previous medical history, symptom characteristics, physical examination, vital signs, troponins values, laboratory tests, triage, treatments and outcomes were covered.
Patients received follow-up interviews through telephone at 30 days after enrolment, and information about MACE and hospital attendances were collected.

Risk scores calculations
Methods and formulas used to calculate the GRACE risk scores have been described in detail previously, including all the 6 models for predicting in-hospital death (IHDth), in-hospital death or MI (IHDthMI), death in 6 months after discharge (OH6mDth), death or MI in 6 months after discharge (OH6mDthMI), death from admission to 6 months later (IH6mDth), and death or MI from admission to 6 months later (IH6mDthMI). 11(Table 1) These scores were calculated retrospectively using the prospectively obtained data.An ECG with ST depression (new horizontal or down-sloping ST depression ≥0.05 mV in two contiguous leads) or ST elevation (new ST elevation at the J point in two contiguous leads with the cutpoints: ≥0.1 mV in all leads other than leads V2-V3 where the following cut points apply: ≥0.2 mV in men ≥40 years; ≥0.25 mV in men <40 years, or ≥0.15 mV in women) 22 was de ned as ischemic ST deviation.
Two independent cardiologists interpreted ECG blinded to the clinical data, troponin levels and events.And discrepancies were evaluated by a third cardiologist.The troponin results from the presentation blood sample arranged by emergency physicians in their daily work were used to calculate scores.And the 99th percentile of the upper reference limit (URL) was used as the cutoff for determining positive.Inhospital PCI and in-hospital CABG were assigned as 0, because the GRACE models were used to stratify chest pain patients immediately after arrival at ED and the PCI or CABG executed during admission were not applicable.In emergency care practice, not all GRACE predictor variables can be collected completely, especially the serum creatinine test (not routinely arranged) and the Killip class (not rated in majority of patients without AMI).Here, we took two kinds of assessment: one was based on the complete GRACE variables with the creatinine value and Killip class assigned as zero if absent; the other one was based on the deletion of creatinine and Killip class from all observations using mini-GRACE, which has been introduced in the development of NICE guideline 94 and validated through a large MINAP registry of patients with ACS. 23,24 nical outcomes The primary outcome was the composite endpoint of MACE within 30 days, including death from all causes, AMI (index and subsequent), emergency revascularization, cardiac arrest and cardiogenic shock.
The diagnosis of AMI was made according to the third universal de nition of MI, as an detection of the rise and/or fall of cardiac biomarkers with at least one value above the 99th percentile of URL and with symptoms or ECG changes or imaging indicative of new ischaemia. 22Cardiogenic shock was de ned as persistent (>30 min) systolic blood pressure (SBP) of less than 90 mmHg and/or cardiac index <2.2L/min/m 2 secondary to cardiac dysfunction, requiring intravenous inotropic or mechanical support. 20wo senor cardiologists from clinical events committee adjudicated the MACE independently using all available clinical records, and discrepancies were evaluated by a third senior physician.If patients were lost to contact, local death registry was used to supplement the survival status.

Statistical analysis
Continuous variables were presented as mean (standard deviation) and categorical variables as number of cases (percentage).Baseline characteristics between MACE and no MACE groups were compared using t test for continuous variables and chi-square (χ 2 ) test for categorical variables.
Pearson product-moment correlation, which is commonly abbreviated as "r", was used to describe the direction and quantify the strength of the linear association between GRACE scores and the incidence of MACE in chest pain.A coe cient of >0.9 indicates a very strong positive relationship. 25The calibration was evaluated using Hosmer-Lemeshow goodness-of-t test (HLT).Low HLT χ 2 and P value >0.05 illustrate agreement between observed and predicted probabilities of an event and a good model t. 26 Discrimination of scores was assessed by the area under the curve (AUC) of receiver operating characteristic (ROC).An AUC at ≥0.9 is considered as outstanding, 0.8~0.9excellent and 0.7~0.8acceptable. 27Taking into account the implicit correlation between the curves of these scores, we used the Delong test to compare any two AUCs. 28Reclassi cation was analyzed to assess how well a risk score improved predictions compared with another one, by category-free net reclassi cation improvement (NRI) and absolute integrated discrimination improvement (IDI).Positive NRI and IDI (measures >0 and p<0.05) illustrate the signi cantly improvement of one model's classi cation over another one. 29The performance of mini-GRACE models were also assessed.A P value of less than 0.05 (two-sided signi cance testing) was considered statistically signi cant in the analysis.All statistical analyses were performed using SAS V.9.4 (SAS Institute Inc., Cary, North Carolina, USA) or MedCalc V.18.11.3 (MedCalc Software, Ostend, Belgium).

Study population
A total of 3536 patients with acute nontraumatic chest pain and initial cTnI tests presented in the participating EDs from August 24, 2015 to September 30, 2017.Part of patients were excluded for denial of informed consent (77) and diagnosis of STEMI (472).There were 88 patients with insu cient information to calculate the GRACE scores, including 74 for no initial ECG and 14 for no SBP values.In 13 survivals, follow-up contacts were unsuccessful.Eventually, 2886 patients remained for analysis (Figure 1).Baseline characteristics and initial evaluation between patients with and without 30-day MACE are compared in table 2. Patients with 30-day MACE tend to be older, male, higher burden of risk factors and had signi cantly higher GRACE scores than those without 30-day MACE (p<0.001).
Correlation between GRACE scores and actual events rates All the six GRACE models showed good positive linear correlation with the actual MACE rates in patients with undifferentiated chest pain (Figure 2).The GRACE (IH6mDthMI) and GRACE (IHDthMI) exerted very strong relationships, with r at 0.920 and 0.913 respectively (p<0.001).

Agreement between observed and predicted probabilities of an event
As shown in Figure 3, the predicted probabilities of an event were much close to the observed events rates across deciles of ve GRACE models.And the HLT P values for the GRACE (IH6mDthMI), GRACE (IHDth), GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI) were 0.113, 0.446, 0.608, 0.312 and 0.073, respectively.But the P value of GRACE (IHDthMI) was <0.001.

Discussion
This study provides the rst comprehensive evaluation and comparison of all the six GRACE riskprediction models in patients with undifferentiated chest pain presenting to the ED.The GRACE (IH6mDthMI) model demonstrated excellent performance across a range of indices for risk strati cation.The GRACE (IH6mDthMI) and GRACE (IHDthMI) outperformed other GRACE models in terms of discriminating between high and low-risk patients.
To rule out high-risk conditions safely and rapidly in chest pain patients presenting to the ED is a big challenge for emergency physicians.If low-risk patients could be identi ed and discharged early, the patients' and health care burden would be reduced signi cantly. 30But the premise is that high-risk patients should receive timely management. 5Right decision making depends on correct risk strati cation.International cardiac guidelines recommended that risk strati cation tools should be used to assess chest pain patients presenting to the ED. 3,4,7 Te GRACE risk scores have been proved to provide the most accurate strati cation of risk both on admission and at discharge of ACS patients. 31But the performance of these models in the chest pain remain unclear.The GRACE risk prediction models were developed using multivariable regression from a multinational registry to assist cardiologists in estimating risk of different outcomes in hospitalized patients with ACS. 32One model is speci c to one kind of outcomes, including death or composite of MI and death, during in hospital, in 6 months after discharge and from admission to 6 months later.The MI referred here is the subsequent AMI occurring after the index ACS (i.e., the cause for the patient's initial presentation).But as for undifferentiated chest pain, the high-risk conditions mainly present a composite endpoint of index AMI, subsequent AMI, death, emergency revascularization, cardiac arrest and cardiogenic shock within 30 days after presentation to the ED. 33,34  the incidence of index AMI is much greater than subsequent AMI, as shown in our study.Our results presented that the GRACE models showed at least moderate correlation with the actual incidence of MACE in the undifferentiated chest pain cohort.Especially, very strong correlation appeared in GRACE (IH6mDthMI) and GRACE (IHDthMI).Furthermore, the predicted probabilities of an event and the observed events rates were signi cantly similar across deciles of GRACE (IH6mDthMI) and other four GRACE models.So, there are foundations for GRACE (IH6mDthMI) to provide accurate strati cation of patients with acute chest pain.
Consistent with the correlation, the discrimination of GRACE (IH6mDthMI) and GRACE (IHDthMI) were excellent (>0.8) and signi cantly better than other GRACE models.4][15][16][17][18][19] It is di cult to make judgement subjectively about the advantage and disadvantage of the six GRACE models.In this study, we evaluated the discriminatory accuracy of all six GRACE models and found that AUC of the GRACE (IHDth) was only 0.75, inferior to the GRACE (IH6mDthMI) and GRACE (IHDthMI).Other three models, including GRACE (IH6mDth), GRACE (OH6mDth) and GRACE (OH6mDthMI), exerted even lower but acceptable discriminatory abilities (0.70~0.73).Our results provide more complete recognition of the performance in GRACE models for discriminating patients with high or low risk of 30-day MACE.
Reclassi cation of the GRACE (IH6mDthMI) and GRACE (IHDthMI) took advantages as well.The NRI quanti es the sum of proportions of correct movements in categories-upwards (higher predicted probability) for events and downwards (lower predicted probability) for non-events.And the IDI quanti es the sum of increased average predicted probability of events in patients with MACE and the decreased average probability in patients without MACE.Signi cant positive NRI and IDI in this study showed that the GRACE (IH6mDthMI) and GRACE (IHDthMI) could give higher predicted probability of an event for high-risk patients and lower predicted probability for low-risk patients than other four models.
Results from the assessment of mini-GRACE were mainly in accordance with the complete models.
Although, the correlation of the mini-GRACE (IHDthMI) and the calibration of the mini-GRACE (IH6mDthMI) got decline compared with the complete ones, the discrimination and reclassi cation of these two mini scores remained excellent and outperformed other models signi cantly.Unlike the requirement of accurate risk prediction in de nite ACS, the aim of using the GRACE scores to stratify undifferentiated chest pain is to separate patients with high or low-risk of MACE correctly. 3,4,7 Cequently, we prefer discrimination and reclassi cation rstly when these four indices are not in full accord.
In addition, the GRACE scores all consist several variables with detailed categories to be calculated, however, the popularity of handheld devices has made the complexity no longer a disadvantage.After all, correct decision is more important than simplicity for the chest pain triage.

Limitations
This study had several limitations.Firstly, the performance of different GRACE scores was assessed in chest pain patients from two hospitals in China.Although urban and rural hospitals were both covered, the validation of each score in wider patients should be determined by further studies of heterogeneous groups.Secondly, the cardiac marker used in the calculation of scores was the contemporary cTnI assays arranged by emergency physicians in their daily work.The ability of scores combined with high-sensitivity cTn to stratify chest pain still need to be evaluated in next studies.Thirdly, all components used in the risk scores were calculated automatically through computer algorithm.And the variables about ECG were based on the standard interpretation from senior cardiologists.This calculating process was deviated from clinical reality.Further studies to evaluate the discrimination of scores calculated immediately by the treating physicians are needed.

Conclusions
From our compressive evaluation and comparison of the six GRACE models in a prospective cohort of undifferentiated chest pain patients presenting to the ED, we found that the GRACE (IH6mDthMI) and GRACE (IHDthMI) models had very strong correlation with actual events and excellent discrimination.The GRACE (IH6mDthMI) also had good calibration.Improvement in AUC, NRI and IDI yielded the same conclusion that the GRACE (IH6mDthMI) and GRACE (IHDthMI) were superior to other four models in discriminating chest pain patients with high or low risk of 30-day MACE.The reasonable application of appropriate GRACE models in the evaluation of undifferentiated chest pain patients in ED should be recommended.