Validation of administrative database codes for acute kidney injury in kidney transplant recipients

Abstract

Background

Validation studies of acute kidney injury (AKI) diagnostic codes performed in the general population have shown poor sensitivity, but the accuracy of such codes in the kidney transplant population remains unknown.

Objective

The objective of this study is to determine the accuracy of AKI diagnostic codes in kidney transplant recipients. We hypothesized that the sensitivity of diagnostic codes would be significantly greater in the kidney transplant population since these patients are closely followed by nephrologists and are more likely to have serum creatinine measured.

Design

The design is a population-based retrospective cohort study using healthcare administrative and laboratory databases.

Setting

The setting is in Southwestern Ontario and Ottawa, Ontario, Canada, from 2003 to 2012.

Patients

We included first-time kidney transplant recipients admitted to hospital for whom serum creatinine was measured in hospital and within 6 months prior (n = 524).

Methods

Patients meeting the Acute Kidney Injury Network (AKIN) classification serum creatinine change criteria were classified as having AKI. We determined the sensitivity, specificity, and negative and positive predictive values for the ICD-10-CA code for AKI when present as an admission diagnosis, most responsible diagnosis, or any diagnosis compared to a reference standard of AKI defined by the AKIN criteria (stage 1 or greater, stage 2 or greater, or stage 3).

Results

Forty-five percent of included kidney transplant patients had a diagnosis of AKI. The most sensitive coding algorithm (reference standard AKIN stage 2 or greater, ICD-10 code present as any diagnosis) had a sensitivity of 42.1 % (95 % CI 31.7, 53.3), a specificity of 90.6 % (95 % CI 87.6, 93.0), and a positive likelihood ratio of 4.5. The median (IQR) rise in serum creatinine from baseline in patients with and without AKI codes was 104 (57 to 158) μmol/L and 16 (−3 to 41) μmol/L, respectively (Mann-Whitney test, p < 0.0001).

Limitations

The low sensitivity of the AKI code may be due to an alternative diagnosis of acute rejection being assigned in certain cases. The cause of AKI could not be determined.

Conclusions

Similar to the general population, the ICD-10 N17x code misses many kidney transplant patients with AKI during their hospitalization. This makes the code unusable for studying the incidence and consequences of AKI in hospitalized kidney transplant patients.

Abrégé

Mise en contexte

Les études de validation portant sur les codes diagnostiques pour l’identification de l’insuffisance rénale aiguë (IRA) menées dans la population générale ont révélé une piètre sensibilité de détection de ceux-ci. Toutefois, leur fiabilité chez les patients ayant subi une greffe du rein reste à démontrer.

Objectif de l’étude

Cette étude visait à évaluer la précision des codes de diagnostic de l’IRA chez les patients ayant reçu une greffe rénale. Le postulat de départ était que la sensibilité de ces codes serait nettement supérieure au sein de cette population puisque ces patients bénéficient d’un suivi par un néphrologue et sont plus susceptibles de voir leur taux de créatinine mesuré régulièrement.

Cadre et type d’étude

Il s’agit d’une étude de cohorte rétrospective, représentative de la population. Elle a été réalisée entre 2003 et 2012, à Ottawa et dans le sud-ouest de l’Ontario, à partir de bases de données de laboratoire.

Participants

Un total de 524 receveurs d’une première greffe rénale ayant été hospitalisés et dont la créatinine a été mesurée au moment de l’hospitalisation et dans les 6 mois précédant celle-ci.

Méthodologie

Les patients dont le taux de créatinine correspondait aux stades de classification de l’IRA du Acute Kidney Injury Network (AKIN) ont été considérés en insuffisance rénale aiguë. Des valeurs de spécificité et de sensibilité ainsi que des valeurs de prédiction positives et négatives pour le code ICD-10-CA en situation d’IRA ont été déterminées lorsque l’IRA constituait le diagnostic d’admission, le diagnostic le plus probable mis en cause, ou n’importe lequel des diagnostics évoqués, en comparaison avec les standards de référence définis par les critères de l’AKIN (stade 1 ou supérieur, stade 2 ou supérieur, ou stade 3).

Résultats

Au total, 45 % des patients admis dans cette étude ont reçu un diagnostic d’IRA. L’algorithme de codage le plus sensible [stade 2 en référence aux standards AKIN avec pour tout diagnostic la présence du code ICD-10-CA] présentait une sensibilité de 42,1 % [95 % IC 31,7 à 53,3], une spécificité s’élevant à 90,6 % [95 % IC 87,6 à 93,0] et un rapport de vraisemblance positif de 4,5. La hausse médiane [étendue interquartile] de créatinine sérique par rapport aux valeurs de base était de 16 [-3 à 41] μmol/L pour les patients non codés et de 104 [57 à 158] μmol/L pour les patients chez qui on décelait le code ICD-10-CA [test de Mann-Whitney, p < 0,000 1].

Limite de l’étude

Dans certains cas, la faible sensibilité du code ICD-10-CA pourrait s’expliquer par un code de rejet aigu. Les causes d’épisodes d’IRA n’ont pas pu être établies dans cette étude.

Conclusions

À l’instar de l’ensemble de la population, le code ICD-10 N17x sous-estime l’incidence de l’IRA chez les greffés rénaux lors d’une hospitalisation. Ceci rend l’utilisation de ce code peu propice à des fins d’études sur l’incidence et les conséquences de l’IRA chez les patients hospitalisés qui ont subi une greffe du rein.

What was known before

Validation studies performed in the general population demonstrate that acute kidney injury (AKI) diagnostic codes have low sensitivity but high specificity. The accuracy of AKI diagnostic codes in the kidney transplant population has not been studied.

What this adds

The ICD-10 N17x diagnostic code for AKI has a low sensitivity in the kidney transplant population, making the code unusable for studying the incidence and consequences of AKI in hospitalized kidney transplant patients. These results help inform the conduct of future studies in the kidney transplant population utilizing administrative data.

Background

Health administrative databases house an enormous amount of data that might permit the conduct of large observational studies in an efficient, relatively low-cost manner [1, 2]. However, researchers utilizing such databases must be aware of the limitations of the data and the potential for biased results [3–5]. In particular, the validity of studies for which key exposures or outcomes are identified with diagnostic or procedural codes depends upon the accuracy of such codes [5, 6]. When used for clinical research, the accuracy of diagnostic and procedural codes for the entities they are supposed to represent should be determined.

The accuracy of diagnostic codes for acute kidney injury (AKI) has been measured in the general population, showing a very low sensitivity (approximately 30 %) but a notably high specificity (generally >95 %) [7]. A recent study examined the incidence and outcomes associated with AKI in the kidney transplant population, with AKI being defined using diagnostic codes [8]. However, the accuracy of diagnostic codes for AKI has never been determined in the kidney transplant population.

This study measured the accuracy of the International Classification of Diseases, Tenth Revision (ICD-10) code N17x for AKI in kidney transplant recipients admitted to hospital. We hypothesized that the ICD-10 code would more accurately identify AKI in the kidney transplant population compared to the general population because kidney transplant patients have a higher prevalence of AKI, are more likely to have their kidney function followed closely, and are more likely to have a nephrologist involved in their care during a hospital admission [8–10].

Methods

Study design and setting

We conducted a population-based retrospective validation study in the province of Ontario, Canada, using Ontario’s population-based health administrative databases at the Institute for Clinical Evaluative Sciences (ICES) and laboratory data from Southwestern Ontario and Ottawa, Ontario. Residents of Ontario have universal access to hospital care and physician services under a single provincial payer system, resulting in a comprehensive repository of health administrative data. The availability of laboratory data was limited to Southwestern Ontario and Ottawa, Ontario. The study was conducted according to a pre-specified protocol that was approved by the Ottawa Hospital Research Ethics Board. The reporting of this study follows guidelines set out for studies assessing diagnostic accuracy (Appendix 1) [11].

Data sources

We created our study’s analytical dataset using seven databases that were linked using encrypted unique identifiers. We identified kidney transplant recipients using the Canadian Organ Replacement Register (CORR), which captures data on every kidney transplant in the province of Ontario [12]. Laboratory data were obtained from the Ottawa Hospital Data Warehouse (OHDW) for Ottawa patients and from Cerner and Gamma-Dynacare for Southwestern Ontario patients. OHDW houses inpatient and outpatient lab information for individuals who had bloodwork drawn at one of three hospitals in Ottawa, Ontario. Cerner is a hospital network in Southwestern Ontario, housing inpatient and outpatient laboratory data from 12 hospitals. Gamma-Dynacare is a laboratory service provider that contains outpatient lab information for individuals who had bloodwork drawn at any of their 148 collection sites in Ontario. Demographics and vital status information were obtained from the Ontario Registered Persons Database (which records the sex, birthdate, and death date of all Ontarians) and CORR. Diagnostic and procedural information from all hospitalizations were determined using the Canadian Institute for Health Information Discharge Abstract Database (CIHI-DAD), which captures data on every hospitalization in the province of Ontario. Information was also obtained from the Ontario Health Insurance Plan database, which contains all health claims for inpatient and outpatient physician services. We have previously used these databases to research kidney health outcomes and health services [13–15].

Study cohort

We included patients with the following characteristics: (a) first kidney-only transplant recipients; (b) hospitalized 6 months or later following kidney transplantation; (c) having at least one serum creatinine value measured during the hospital admission; (d) discharged from hospital prior to the end date of laboratory data availability; and (e) serum creatinine data available anytime between 2 weeks to 6 months prior to the admission date to determine baseline creatinine. Hospital admissions less than 6 months post transplant were excluded to eliminate as much as possible AKI secondary to post-operative complications, delayed graft function, and early acute rejection. Hospital admissions occurring between April 1, 2003 and December 31, 2012 (Ottawa) and March 31, 2012 (Southwestern Ontario) were eligible for inclusion. Hospital admissions with an admission date prior to April 1st, 2003 were excluded due to the use of ICD-9 diagnostic codes prior to this date. Originally, ICD-9 codes were included as a separate analysis, but this analysis had to be suppressed (as per ICES privacy regulations), due to the presence of too many small cells (total n = 118 patients). A look-back period of 3 years from the date of hospital admission was used to determine comorbidities. Codes used to define comorbidities of interest are outlined in Appendix 2. When multiple eligible hospital admissions were available for a patient, one was selected at random in order to avoid clustering in the analysis.

Criteria for AKI

We used the Acute Kidney Injury Network (AKIN) staging system to define AKI [16]. AKIN stage 1 is defined by an increase in serum creatinine ≥26.4 μmol/L (0.3 mg/dL) or a 1.5- to 2-fold increase from baseline. AKIN stage 2 is defined by a >2- to 3-fold increase in serum creatinine from baseline. AKIN stage 3 is defined by an increase in serum creatinine >3-fold from baseline, or a serum creainine >354 μmol/L, with an acute increase of at least 44 μmol/L (0.5 mg/dL) [16]. The urine output criterion for the AKIN staging system was not used as these data were not available from our databases. The peak creatinine during a hospital admission was used to define the presence or absence of AKI and the AKIN stage. If multiple baseline creatinine values were available, the most recent value was used, except if drawn less than 2 weeks prior to admission. Creatinine values drawn very close to admission were excluded due to a heightened chance of the patient being unwell at the time of the bloodwork; the result may therefore not reflect a true baseline value but possibly the beginning of the AKI episode.

ICD-10 code for AKI

Trained coders review all charts to record appropriate diagnosis codes and their associated attributes following a discharge from hospital. Coders follow the Canadian Coding Standards developed by CIHI [17]. According to CIHI’s guidelines, the coders are not permitted to interpret laboratory tests; however, they can record a condition based on laboratory measurements if a physician documents the condition in the patient’s chart. For hospitalization records (included in the CIHI-DAD), coders may record up to 25 conditions using ICD-10 diagnostic codes. They must also indicate the diagnosis type. A diagnosis type “M” is the main or most responsible diagnosis, which is the condition that contributed most to the hospital length of stay or used the greatest amount of resources. An admission diagnosis is any condition that existed prior to the admission and was treated during the hospital stay [17].

In our study, we tested the accuracy of the ICD-10 code N17x, which defines “acute renal failure,” when present as diagnosis type “M” or most responsible/main diagnosis, admission type diagnosis, or any diagnosis type (present in any one of the 25 potential diagnosis fields).

Statistical analysis

We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value of the ICD-10 N17x code compared to a reference standard of changes in serum creatinine using the AKIN staging system for AKI (formulas and a sample 2 × 2 table are presented in Appendix 3). We calculated 95 % confidence intervals for single proportions using the Wilson Score method [18]. We calculated the positive likelihood ratio using the sensitivity and specificity (Appendix 3). We also compared the change in serum creatinine between patients who were coded positive or negative for N17x. The difference in the distribution of change in creatinine between code negative and code positive patients was formally tested using the Mann-Whitney test. We conducted all analyses using the SAS software, version 9.4 (SAS institute Inc., Cary, NC, USA).

Results

We identified a total of 524 kidney transplant patients with eligible hospital admissions from 2003 to 2012 that met our inclusion criteria. Patient selection is outlined in Additional file 1: Figure S1. Baseline characteristics are outlined in Table 1. The mean (standard deviation, SD) age was 57.7 (12.1) years. The median (interquartile range, IQR) time from kidney transplant to the index hospital admission was 3.5 (1.5, 7.1) years. The baseline serum creatinine was measured a median (IQR) of 34 (22, 68) days prior to hospital admission. The peak creatinine was measured a median (IQR) of 1 (0, 2) day post admission. AKI (based on AKIN criteria) occurred in 45.0 % of the cohort, and 14.1 % of the cohort was coded with ICD-10 N17x. Of the patients with AKI, most (67.8 %) had mild disease (AKIN stage 1).

Table 1 Baseline characteristics

The diagnostic performance of the various coding algorithms is presented in Table 2. The diagnostic code type of “all diagnoses” performed the best. Compared to a reference standard of AKIN stage 1 or greater, the ICD-10 N17x code for AKI showed a sensitivity of 28.0 % (95 % CI 22.6, 34.0) and specificity of 97.2 % (95 % CI 94.6, 98.6). Compared to a reference standard of AKIN stage 2 or greater, the ICD-10 code showed a sensitivity of 42.1 % (95 % CI 31.7, 53.3) and a specificity of 90.6 % (95 % CI 87.6, 93.0). Overall, specificity was high, >90 % for most code types and definitions of AKI. The positive predictive value decreased significantly with increasing severity of AKI: AKIN stage 1 or greater 89.2 % (95 % CI 80.1, 94.4); AKIN stage 2 or greater 43.2 % (95 % CI 32.6, 54.6); and AKIN stage 3 29.7 % (95 % CI 20.5, 40.9) (Table 2).

Table 2 Diagnostic performance characteristics of three different algorithms for ICD-10 code N17x using the AKIN staging system for AKI as the reference standard

The absolute and relative changes in serum creatinine for patients that were coded positive and negative for AKI are presented in Table 3 and Additional file 1: Figures S2 and 3. In patients who were code positive and code negative for AKI, the median (IQR) absolute rise in serum creatinine from baseline was 104 (57 to 158) μmol/L and 16 (−3 to 41) μmol/L, respectively. The median (IQR) percent relative change was 56.9 (35 to 111) and 12.9 (−2.2 to 31), for code positive and code negative patients, respectively. The difference in the distribution of the absolute and relative changes in serum creatinine between code negative and code positive patients was statistically significant when the distributions were compared using the Mann-Whitney test (p < 0.0001).

Table 3 Change in serum creatinine from baseline in all patients with and without an ICD-10 N17x code for AKI (referred to as code positive or code negative)

Discussion

This study measured the accuracy of the ICD-10 N17x code for the diagnosis of AKI in the kidney transplant population. The best performing code type was “all diagnoses” (i.e., present in any diagnostic field during a hospital admission). All code types, when compared to all definitions of AKI, showed a low to moderate sensitivity but high specificity. For AKI AKIN stage 1 or greater (any type of AKI), the positive predictive value of the code was high at almost 90 %. This suggests that the code would be reasonable for cohort selection if only kidney transplant patients with AKI were of interest. However, one would have to be satisfied that there was no differential misclassification, given the low sensitivity of the code, resulting in the exclusion of many patients who truly have AKI. Also, if only patients with more severe forms of AKI (AKIN stage 2 or 3) were of interest, the code would not be appropriate for cohort creation given the low prevalence of this disease stage with a resultant low positive predictive value.

The code performed poorly when a code type of “main or most responsible diagnosis” was used. This could be due to the fact that AKI often occurs in the setting of another illness, such as an infection [9, 19], which may be coded as the main diagnosis as opposed to AKI. The positive predictive value of the code was quite variable depending on the reference standard used. A low positive predictive value for severe AKI (AKIN stage 3) was found with all code types. This is likely due to the very low prevalence of AKIN stage 3 (10.5 %) in our cohort. The positive predictive value of a test (in this case the code) is known to vary significantly depending on the prevalence of the disease [20]. Specificity was quite high for all code types; however, it was slightly lower for code types with a higher sensitivity. Most code types were more sensitive (less false negatives) when a higher stage of AKI was used as the reference standard. This is to be expected because severe AKI is more clinically apparent and therefore more likely to be recorded in the chart.

The sensitivity of the code was lower than expected. When examined in healthy elderly and elderly patients with chronic kidney disease (CKD) (both at higher risk for AKI [21–23], similar to kidney transplant patients), the ICD-10 code for AKI had a sensitivity of 62 and 76 % respectively, compared to a reference of at least a doubling in serum creatinine [24]. One possible reason for the lower than expected sensitivity of the code is that acute rejection (a cause of AKI), has its own diagnostic code. Although we could not verify the true incidence of acute rejection in our study, it should be very low given that the median time from transplant to index admission was 3.5 years, and acute rejection is very infrequent beyond the first year [25]. Nonetheless, a small proportion of AKI episodes (determined based on the reference standard of a rise in creatinine) were likely assigned a code for acute rejection as opposed to AKI. The specificity of the code in the kidney transplant population was slightly lower than in the general or elderly population [7], suggesting that kidney transplant patients are more likely to have a code assigned for AKI when their kidney function is actually stable. Overall, the code demonstrated limited sensitivity; however, a high positive predictive value for any AKI was found. The code was also able to distinguish two populations with significantly different rises in serum creatinine.

To our knowledge, this is the first study to measure the accuracy of the ICD-10 diagnostic code for AKI in the kidney transplant population. Prior studies have examined the accuracy of AKI codes, mostly ICD-9, in the general, elderly, and elderly CKD populations [7, 24, 26]. We studied transplant patients from two healthcare regions in the province of Ontario, making the sample more representative and thus generalizable. We had serum creatinine values available, making it possible to compare the administrative diagnostic code to the gold standard for diagnosing AKI, as opposed to relying on chart re-abstraction.

Limitations to our study should be noted. First, for the reference standard used to define AKI, we adapted the creatinine-based component of the AKIN classification system, which defines AKI using both serum creatinine and urine output measurements [16]. It is also recommended that the AKIN classification system be applied only after a patient has achieved an optimal state of hydration. Unfortunately, urine output measurements and clinical data, such as hydration status and the administration of intravenous fluids, were not available in the administrative datasets that we used for this study. However, even if urine output data were available, urine output measurements are difficult to obtain and are poorly documented outside the intensive care setting. In addition, the sole use of serum creatinine is a commonly accepted method of defining AKI, both clinically and for research purposes [9, 27, 28]. Second, there is no consensus definition for AKI that has been validated in the kidney transplant population [29, 30]. However, all established classification systems apply similar serum creatinine and urine output criteria [30, 31]; AKI is defined similarly for transplant and non-transplant patients in the clinical setting; and the AKIN staging system was used to define AKI and correlated with poor outcomes in a prior study of kidney transplant patients [9]. Third, we did not specify a timeline of <48 h within which AKI had to occur (as specified in the AKIN criteria). The use of stringent timelines would likely diminish the accuracy of the AKI code since these timelines are not applied in the clinical setting. Finally, data on the cause of AKI was not available. The accuracy of administrative coding for AKI may differ depending on the cause, especially in transplantation, where a diagnosis of acute rejection may be coded preferentially over a diagnosis of AKI.

Conclusions

In summary, our study demonstrates that identifying AKI in kidney transplant patients using administrative diagnostic codes will result in an underestimation of the true incidence and misclassification of patients with AKI. This limitation makes the code ineffectual for determining the incidence and consequences of AKI in hospitalized kidney transplant patients.

Appendices

Appendix 1

Table 4 STARD (STAndards for the Reporting of Diagnostic accuracy studies) checklist

Appendix 2

Table 5 Coding definitions for demographic, comorbid conditions, and AKI characteristics

Appendix 3

Table 6 Sample 2 by 2 table for assessing diagnostic performance characteristics (sensitivity, specificity, positive predictive value, and negative predictive value) for ICD-10 code N17x