In this cohort study we evaluated the ability of the CCI and ESRD-CI to predict mortality in a population of incident dialysis patients. The association between comorbidity and mortality in our dataset was not as strong as in the derivation cohorts. In addition although higher comorbidity burden using the CCI and ESRD-CI was associated with mortality after multivariable adjustment, there was not a large separation in mortality risk when evaluating incremental changes in comorbidity scores. We can speculate that the limited utility of the CCI and ESRD-CI is due to several potential underlying reasons; limited generalizability of both indices, variability in comorbidity classification and comorbidity prevalence within either cohort, limitations with respect to derivation (including statistical over-fitting) and limited utility of comorbidity indices in general.

Limited generalizability may explain the observation of only partial validity. The CCI and ESRD-CI used a relatively small population from a single geographical area and validation in this study occurred in a separate single geographical area. Differences in the determinants of health in two communities may contribute to the different outcomes of two dialysis patients with similar comorbidities. Social determinants of health, in particular, have been shown to impact mortality rates in patients with ESRD [37–39]. More recently, frailty has been shown to be an important prognostic factor for incident dialysis patients [40]. Therefore, prediction models that incorporate clinical, demographic and social factors as well as assessments of frailty may be more applicable to ESRD cohorts [41]. An attempt at further validation of this index in a national or international sample of patients from a number of different centers in different geographic regions might help to clarify its generalizability and provide a clearer picture of its clinical utility.

There were notable characteristic differences comparing patients in our dataset and the dataset of the original studies, which may explain the reduced discrimination. Our cohort was in a more recent era (2006–2013), and the relative impact of some comorbid conditions may have changed [42–45]. The mortality risk in our cohort was higher than either derivation cohort [15] however; certain comorbidities such as chronic lung disease (16.7 % vs. 27.4 %) and neoplasia (6.5 % vs. 12.2 %) were less common in our cohort [15]. Furthermore, there was a reduced prevalence of overall comorbidity compared to the derivation cohorts. This reduced burden of disease might have affected the predictive value of the individual comorbidities comprising both indices. Alternatively, under-reporting of individual health conditions in each cohort may have explained the observed differences. For example, if comorbidity in our validation cohort was under-reported, despite a “falsely low” burden of comorbidity patients would have continued to have a high mortality rate. This in turn would have reduced the discrimination in our dataset.

The derivation of the CCI or ESRD-CI may have also impacted its validation. The linear predictor from a Cox model is ideal for developing a prognostic index. The linear predictor is described as the weighted sum of the variables in the model, where the regression coefficients are the weights [25]. In the development of the CCI, (which was replicated in the derivation of the ESRD-CI to maintain consistency) scoring weights were assigned to each HR derived from the Cox model [4, 15]. This may lead to over-weighting of conditions with high hazard ratios but limited precision and marked variability around the estimate.

While there are limitations with the comorbidity indices, it is important to acknowledge that both indices did have some validity. It is not unexpected that a higher level of comorbidity would be associated with a higher risk of death among dialysis patients, however most evaluations of comorbidity scores look at short term mortality or use validation techniques that do not incorporate survival time [15, 19, 23, 24]. Discrimination was lower in our validation dataset, however, the relative hazard for death was proportional across the index and the mortality association using either index persisted despite a relatively long duration of follow-up. Furthermore, it is not uncommon for validation studies to identify some reduction in predictive value [46] In addition there are other features of these indices that make them valuable. Both are intuitive; comorbid conditions that would be expected to confer a higher hazard for mortality are weighted more heavily. The major exception to this would be HIV (a component of the original CCI) a condition with a lower contemporary mortality rate [42]. Another advantage is that both indices draw on clinical data that is often present in patient charts, making them practical tools that do not necessarily require other diagnostic testing or laboratory investigation. Finally, the association with mortality persisted in our study even after multivariable adjustment for known predictors of mortality among dialysis patients.

It is important to note that there are limitations of comorbidity indices in general that warrant consideration. Most indices do not fully take into account the stage of progression, severity or proximity of the comorbid condition in relation to dialysis initiation [1, 14–16]. Furthermore, accumulation of comorbidity that often accompanies the early period after dialysis initiation [47] is not typically included in comorbidity indices. Global scoring systems that incorporate all prior comorbidities are easier to calculate and extract from patient records and facilitate ease of clinical application at point-of-care. More novel indices such as the Adult Comorbidity Evaluation-27, which takes into account the proximity and severity of the comorbidity, have been shown to out perform the CCI at predicting mortality [48, 49] and may be better suited for dialysis cohorts. Additionally, simple prognostic models that include the “surprise question” have also been shown to accurately predict survival for patients receiving hemodialysis [50].

This study utilized a substantial Canadian cohort and a long follow up time while leveraging electronic medical records to ensure a high quality and robust dataset. Additionally validation of the index was performed utilizing stringent methodology [25]. Examining the adjusted association between the CCI and ESRD-CI and mortality was enhanced by the relatively large number of outcomes.

There are, however, limitations to this study. The classification of comorbidities for each patient was determined by clinical impression (based on documentation in paper chart and electronic records). This introduces the possibility for misclassification bias. The derivation cohort(s) would have also been affected by misclassification bias, potentially compounding the observed differences in disease prevalence and prognostic utility. Our ability to completely validate either index was limited by the level of information provided in the original studies. In particular, an assessment of calibration (the ability of the index to assign the correct event probability at any relevant follow-up time and every level of predicted risk [25]) could not be performed.