Utilizing electronic health records to predict acute kidney injury risk and outcomes: workgroup statements from the 15^th ADQI Consensus Conference

Abstract

The data contained within the electronic health record (EHR) is “big” from the standpoint of volume, velocity, and variety. These circumstances and the pervasive trend towards EHR adoption have sparked interest in applying big data predictive analytic techniques to EHR data. Acute kidney injury (AKI) is a condition well suited to prediction and risk forecasting; not only does the consensus definition for AKI allow temporal anchoring of events, but no treatments exist once AKI develops, underscoring the importance of early identification and prevention. The Acute Dialysis Quality Initiative (ADQI) convened a group of key opinion leaders and stakeholders to consider how best to approach AKI research and care in the “Big Data” era. This manuscript addresses the core elements of AKI risk prediction and outlines potential pathways and processes. We describe AKI prediction targets, feature selection, model development, and data display.

Abrégé

Les données figurant dans les dossiers médicaux électroniques (DMÉ) sont considérables, tant au point de vue du volume que du débit ou de la variété. Ces trois caractéristiques et la tendance générale à adopter les DMÉ ont soulevé un intérêt pour appliquer les techniques d’analyse prédictive des mégadonnées aux données contenues dans les dossiers médicaux électroniques. L’insuffisance rénale aiguë (IRA) est une maladie qui convient parfaitement à une méthode de prévision et de prévention des risques: non seulement la définition acceptée de cette affection permet-elle un ancrage temporel des événements ; mais il n’existe aucun traitement une fois que la maladie est déclarée, ce qui montre l’importance d’une détection précoce. L’Acute Dialysis Quality Initiative (ADQI) a convoqué un groupe de travail constitué de leaders d’opinion et autres intervenants du milieu pour se pencher sur la meilleure façon d’approcher la recherche et les soins offerts aux patients atteints d’IRA en cette ère de mégadonnées. Le présent article traite des éléments centraux de la prévention des risques et en expose les procédures potentielles. Nous y décrivons les cibles de prévention de l’IRA, la sélection des paramètres, l’élaboration des modèles et l’affichage des données.

Background

The term “big data” has traditionally been used to describe extraordinarily large and complex datasets. For many medical practitioners, this concept was initially epitomized by genomics – the colossal amount of discrete data generated by high throughput sequencing techniques required analytic methods ranging far beyond standard statistical approaches [1]. However, “omics” are now ubiquitous and “big data” has become vernacular in medicine [2, 3]. Clinical researchers are beginning to employ innovative, high-content analytic techniques capable of integrating and exploring the exceedingly large and diverse datasets contained within the electronic health record (EHR).

EHR data, which are generated through the routine provision of clinical care, are “big” from the standpoint of volume (number of discrete data points available), velocity (rate at which new data accumulates), and variety (myriad of data elements available for interrogation) [3, 4]. These aspects, along with its singular clinical relevance, make EHR data ideal for disease prediction and risk forecasting. In particular, acute kidney injury (AKI) is a syndrome which lends itself well to predictive modeling and early risk stratification (Fig. 1). The presence of a standard, consensus definition allows accurate and efficient AKI diagnosis [5]; temporal anchoring of the AKI event creates a distinct pre-disease dataset to which high-content, high-throughput predictive techniques can be applied (Fig. 1). Additionally, although AKI has been associated with poor short and long term outcomes in both adults and children, no treatments exist to mitigate or cure AKI once it has developed [6–13]. The ability to predict AKI in hospitalized patients would provide the opportunity to modify care pathways and implement interventions. This, in turn, could prevent AKI events, thereby reducing mortality, shortening length of stay, averting the development of chronic kidney disease, and potentially creating novel quality of care indicators [13, 14]. In this manuscript, we present evidence informed, consensus driven statements regarding the concepts of primary relevance when considering the capacity of EHR data to be used in AKI prediction applications.

Fig. 1

Signal Identification for AKI Development and Progression. Current consensus AKI definitions allow AKI events to be precisely anchored from a temporal standpoint, clearly defining a pre-disease state. As the patient progresses from “No AKI” to “AKI,” the pattern of data generated within the EHR changes, creating an “AKI signal” which can be identified through advanced analytic techniques. This signal can be translated into a prediction model which is capable of identifying patients at high risk for AKI development. Reproduced with permission from ADQI

Methods

This consensus meeting following the established ADQI process, as previously described [15]. The broad objective of ADQI is to provide expert-based statements and interpretation of current knowledge for use by clinicians according to professional judgment and identify evidence care gaps to establish research priorities. The 15th ADQI Consensus Conference Chairs convened a diverse panel representing relevant disciplines from five countries from North America and Europe around the theme of “Acute Kidney Injury in the Era of Big Data” for a 2-day consensus conference in Banff, Canada on September 6–8, 2015. During the pre-conference phase of the meeting, each work group performed a comprehensive literature search to summarize areas where broad consensus exists, categorize knowledge gaps, and identify future priorities for research. Specifically for the AKI prediction workgroup, the literature search was conducted using the terms “acute kidney injury prediction”, “acute renal failure prediction”, and “AKI prediction” in MEDLINE using PUBMED as the search engine. This search yielded a total of 679 articles for review. Studies were limited to articles published in 2010–2015 to reflect more recent harmonized AKI definitions. Studies were included if they discussed a prediction model and did not isolate the analysis to identification of independent risk factors. Studies were excluded if the focus of the prediction model was novel biomarkers due to practical issues in using these markers in current clinical practice. Thirty-four articles were selected in the initial review. Upon reviewing the articles, there was a consensus amongst work group members to include seven additional articles published prior to 2010; these articles used earlier consensus definitions for AKI, laid the groundwork for the subsequently developed models, and were archetype models when published [16–22]. Four core questions/concepts were crafted for presentation to the entire ADQI consensus group during the conference (Table 1). During the conference our work group developed consensus positions, and plenary sessions involving all ADQI contributors were used to present, debate, and refine these positions. Following the conference this summary report was generated, revised, and approved by all members of the work group.

Table 1 Core Questions for ADQI Consensus Group

Results

Question 1: Across the spectrum of AKI, which event or events should be targeted for prediction?

Prior to developing a model, it is important to carefully choose the target for prediction. From the outset, the consensus group believed it was imperative that, for the purposes of prediction, AKI be diagnosed and identified according to the generally accepted consensus definition and classification scheme, the KDIGO criteria [5]. This is the most current consensus definition, it harmonizes the previously proposed AKI criteria (RIFLE, pRIFLE, and AKIN), and is applicable to both adults and children [5, 23–25]. In order to build the strongest and most useful predictive model, we would recommend forecasting AKI events with a horizon of 48–72 h. While it would be advantageous to identify AKI events as early as possible, lengthening the event horizon reduces the accuracy of the model; we believe the suggested horizon gives practitioners adequate time to modify practice, optimize hemodynamics, and mitigate potential injury without sacrificing predictive power. The group additionally believed that rather than targeting all AKI, it would be initially advantageous to predict “moderate/severe” AKI as defined as KDIGO stage 2 or 3. While this recommendation is based on evidence-informed opinion, there are rational justifications for making it. First, this is consistent with the initial ADQI consensus statement which described the RIFLE criteria; operationally, Stage 1 KDIGO-defined AKI correlates with RIFLE stage “Risk” [24]. Treating KDIGO-defined Stage 1 AKI as “AKI risk,” allows it to become a subsequent predictor for moderate/severe AKI. Second, AKI predictors or risk factors have traditionally been more strongly associated with higher severity AKI [26, 27]. The greater strength of association will likely result in more powerful predictive modeling by reducing confounding; the development of robust models is of paramount importance for these initial big data attempts at predictive AKI analytics. Finally, while “mild” Stage 1 AKI has been associated with poorer outcomes, the association with these outcomes is significantly stronger for Stages 2/3 [6, 11, 27–31]. This ability to strongly link AKI with outcomes has an additional benefit as it will allow the models to predict not only AKI, but AKI-related outcomes as well. In one potential scenario proposed by the workgroup, a model would provide predictive AKI risk up until the occurrence of AKI then, at the inflection point of AKI development, it would provide a one-time predictive risk for patient-centered, clinically important outcomes. The workgroup acknowledges that if only Stage 2 and 3 AKI are targeted for prediction, early simulative subanalysis should be performed to evaluate the suitability of this approach.

Consensus Statement

For the purpose of developing AKI prediction models using the data contained within the EHR, the prototype should predict risk both for developing KDIGO-defined Stage 2/3 AKI as well as patient-centered and clinically important AKI-related outcomes.

Question 2: For the purposes of predictive modelling, what paradigm should be used for variable identification and selection?

Prior to applying “big data” analytics to AKI prediction, the consensus group believed it was important to appraise the AKI prediction models which had been developed to date. Based upon our predictive goals outlined in the prior section, model variables of particular interest would be causally and/or temporally associated both with the development of AKI and with AKI-related outcomes.

A number of investigators have approached AKI prediction using standard multivariable regression methodology [17–22, 32, 33]. Models have been developed for a variety of patient populations with a particular emphasis on cardiac surgery patients [34, 35]; notably, less work has been performed in general critical care populations despite the fact that they are also at high risk for AKI [36–38]. Even less established are prediction models in non-critically ill patients. However, given the ultimate goal of preventing AKI, we also need to consider predictive modeling in these populations in order to identify high-risk patients as early as possible [39, 40]. A fairly comprehensive list of studies and variables are shown in Table 2. Variables from patient-specific models are often constrained to the clinical care specific to that population; for example, models for cardiac surgery patients include cardiopulmonary bypass time and number of bypass grafts. However, a number of variables commonly appear across many of the existing models (i.e., age, baseline renal function, medications, diabetes, hypertension, etc.); these variables may be better suited for a generalized model. Most models had modest predictive success with area under the receiver operating curves (AUC) approximating 0.75; a few models reached AUCs as high as 0.9, although the sample sizes were smaller and there was a pre-selection of high-risk patients [41–44]. Regardless of their ultimate utility in defining predictive variables, these models give us a minimum AUC threshold to target for successful model development.

Table 2 Selected list of Predictive Models Currently Available in the Literature

As stated, ideal variables would be associated with both the development of AKI and patient centered, clinically important outcomes following AKI. Notably, many of the same risk factors described in Table 2 as predicting AKI occurrence have also been shown to predict AKI-associated mortality [36, 45–51]. In addition to these factors, positive fluid balance has been associated with increased mortality in both pediatric and adult patients with AKI [52–56]. Receipt of renal replacement therapy (RRT) is another outcome worth forecasting after AKI has occurred. Although most of the published clinical scores predicting receipt of RRT have focused on post-cardiac surgery patients, they have identified many of the same predictors for AKI occurrence in broader populations [17, 19, 32, 34]. AKI is known to be associated with the development of CKD and ESRD, therefore, prediction of these long-term outcomes among AKI survivors should also be targeted; archetype variables associated with these outcomes are shown in Table 2 [8, 57–68].

While the group believed it was imperative that previously identified AKI predictors be reviewed, to truly harness the power of the EHR a de novo approach which considers the entirety of the dataset is required (Fig. 2). There are a number of potential data-mining, machine learning approaches to this sort of feature selection which could be used alone or in combination (Table 3). One method, neural networks, employ non-linear models that feed a set of predictors (inputs) into a hidden layer of units which then use a non-linear transformation to send a value to an output; the prediction is a summation of all the values coming from the hidden layer [69]. This non-linear technique is sometimes described as a “black box” method since it’s difficult to determine the form of the final predictive model. However, this is of little concern in feature selection, as identifying the group of the most influential variables is of interest. A second potential method is that of random forests which is an extension of the binary split classification tree approach [70, 71]. Multiple trees are created by allowing a random number of the predictor variables to be considered at each split of each tree. This results in trees that cover a larger solution space, potentially increasing accuracy. Another set of methods is cluster analysis, a group of unsupervised learning techniques. Observations are grouped according to their similarities in a multidimensional space, based on a distance measure [72]. The resultant clusters can then be further explored to see which ones have a very high or a very low incidence of the outcome measure. A similar method is a technique known as self-organizing maps, in which unsupervised neural networks map a highly dimensional space onto a two-dimensional map [73, 74]. Principal components analysis and support vector machines are two similar feature selection methods which could be employed in this space [75, 76]. Although this is not a comprehensive list of the methods that might be considered, these are excellent exemplar techniques which can be used to identifying novel features and transform known risk factors.

Fig. 2

Development of AKI Prediction Algorithm. The first step in the development of an AKI prediction model is feature selection. This process would evaluate known risk factors identified from the literature and would use machine learning techniques to identify novel risk factors from amongst the EHR dataset. All appropriate features would be considered for inclusion in the actual prediction model which would weight individual variables to create a generalizable model. This model would be validated using a different (or subset of existing) dataset. Once validated, the model could then be integrated directly into the EHR to allow real time AKI alerting. Reproduced with permission from ADQI

Table 3 Big data modeling techniques

In summary, the suggested approach highlights our belief that accurate prediction of AKI takes precedence over finding putative variables, though the suggested approaches do not preclude discovery of new risk factors for AKI. Additionally, while it is useful to review previously established variables associated with AKI from existing studies, application of high content, machine learning techniques to the complete EHR dataset will be the driving force behind variable selection. The ability to dynamically identify and integrate variables from amongst innumerable patient-level data elements represents a marked departure from classically developed model building approaches.

Consensus Statement

Variables included in prototype AKI prediction models should be identified using a hybrid approach; risk factors which are well established in the literature should be considered along with novel risk factors identified via machine learning techniques. Application of these unsupervised approaches should take precedence as it allows feature selection to be dynamic, thereby generating the strongest prediction from existing data elements.

Question 3: What is the optimal approach for model building and EHR integration?

Once the aforementioned hybrid variable selection process was complete, previously identified risk factors and potential predictors discovered via big data techniques could be considered for inclusion in a model. Inclusion criteria could include:

• 1. Evidence over multiple studies that the risk factor was a powerful predictor of AKI

• 2. Identification by machine learning techniques to be predictive of AKI and outcomes

• 3. Available discretely within the EHR to allow easy integration

• 4. Reliably/accurately recorded within the EHR

Variables need not necessarily be universal. For example, pediatric or ICU specific variables could be considered; the model could be dynamic with certain features active/inactive in certain locations/populations. Additionally, it is possible that effect modification of the variables could vary between patients or populations; the presence or absence of certain variables might change the weighting of the residual variables.

While we advocate for a big data approach to identify novel predictive features, initially we would recommend that the predictive model itself be built through more standard statistical modelling. This is primarily due to the inherent limitations of current EHR architecture. EHRs are built to optimize patient level data review and display; they are not necessarily organized to optimize cohort level analysis [77]. This makes implementation of a resource-intense machine learning algorithm into the EHR itself technically and operationally problematic. Therefore, once the variables were identified by literature search and machine learning methodology, it is likely that a logistic regression model, discriminant analysis, or decision tree algorithm would be employed to predict the development of AKI [71, 78, 79]. Data could accumulate on a “rolling window” concept and a prediction could be generated at a pre-specified interval (hourly, every two hours, every shift); alternatively, the model could generate a score in real time as each new data value is received. One conceptual approach would allow this model to generate a risk score ranging from 0 to 100; low scores would be indicative of minimal AKI risk and high scores would be indicative of significant AKI risk. Scoring on a continuous scale would allow both low and high thresholds to be set. In many ways, the ability to identify patients at negligible AKI risk could be as valuable as identifying patients at great AKI risk. An algorithm such as this could be active up until the time the patient develops AKI. At that inflection point, a final, one-time score could be generated which would be reflective of the patients AKI-related outcome risk, thereby allowing practitioners to identify patients at great risk for poorer outcomes.

It is important to note that while the EHR has operational and structural limitations to the application of big data techniques, alternatives should be available in the future. For example, many clinical data warehouse (CDW) solutions have become available for analytic purposes [80–83]. These CDWs represent “shadow” EHRs in which data has been manipulated, linked, and stored in a fashion conducive to high-content, high-throughput analytics [82, 83]. Once such CDWs become as ubiquitous as EHRs, big data approaches could be applied directly to the CDW environment. However, to truly exploit the full capacity of the EHR and EHR data, a more progressive approach is necessary. The EHR has transcended its original purpose; although it is currently a care monitoring and delivery tool, it has the potential to revolutionize clinical care paradigms. To achieve this, data architecture must become as important as data entry and analytics must be prioritized. The creation of a true “learning EHR” could be the key to higher quality, lower cost care delivered with greater efficacy and efficiency.

Consensus Statement

While machine learning techniques should be used to identify novel AKI risk factors, prototype AKI prediction models should be built using more standard statistical weighing techniques to allow effective EHR integration. However, analytics should attain higher priority and the operational limitations of the EHR should be addressed. Consequently, subsequent predictive iterations should progress towards full EHR-integration of high content analytic techniques.

Question 4: What is the optimal output of an archetype predictive model?

After the rigorous steps undertaken to select variables and develop a predictive model, we propose that any prototypes be directly integrated into the EHR for automated real time usage. The increasingly widespread use of EHRs across hospitals has substantially increased the amount of data available to providers [84]. However, while EHRs purportedly improve patient outcomes, studies that have validated these benefits are lacking [85–87]. Several potential EHR-related barriers to improving outcomes have been identified and include information overload, ineffective data display, and poor implementation processes [88–90] Therefore, it is imperative that an AKI prediction model not only harness the power of the EHR data set, but also that it effectively conform to the strengths and limitations of EHR processes. Ideally, AKI risk prediction tools should directly extract relevant data predictors in real-time, deliver a relevant “renal risk score,” and provide feedback to practitioners regarding potential actionable items. One potential a concept would be to create a “renal dashboard” (Fig. 3a and b).

Fig. 3

a and b Renal Dashboard. Once the risk prediction model is developed and validated, it is important to determine how to deliver the information to providers. One possible output might be a “Renal Dashboard” (a). The display would visually display the time trend of AKI as well as a numeric value (with confidence intervals) for the current risk. For any patients who develop AKI, information about outcome risk would be provided; in this example, the outcomes of interest are need for RRT, mortality, development of ESRD, and likelihood of renal recovery. The dashboard could be dynamic, allowing providers to drill into the risk score. In the patient level display (b), information would be available about how the risk had trended over the past 24 h as well as what factors were affecting the current risk score most significantly. In this example, AKI risk information is provided in a visually stimulating manner with a dynamic component capable of driving care modification. Reproduced with permission from ADQI

The main objective of the renal dashboard would be to provide feedback on the absolute risk of developing moderate to severe AKI within the next 48–72 h as well as to present information about the clinical features contributing to these risks. The electronic dashboard format could be tailored for a particular provider, service, or unit. Each patient could have a risk score (in percentage) with an accompanying confidence interval (Fig. 3a); a confidence interval component would give practitioners an idea of how certain the AKI risk was at any given time. In addition to absolute risk scores, the dashboard could be configured to display time trends in risk scores which might give a better sense of evolving AKI risk. Time trends should be displayed in a visually stimulating fashion (i.e., sparklines) to demonstrate the dynamic nature of real-time AKI-risk. A fully optimized dashboard might allow providers to “drill into” the risk score (Fig. 3b), revealing a magnified view as well as more detailed data on the most recent predictors that contributed to a significant increase in risk score. The identification of specific vital sign indicators, laboratory parameters, medication administration data, or other clinical factors that contributed directly to a rise in AKI risk will help guide providers toward implementing risk reduction actions.

A secondary objective of the dashboard might be to provide updated feedback on the risk of adverse outcomes associated with AKI once it actually develops. Early iterations of this sort of prototype may be limited to one-time scores for AKI-related outcomes. However, at the inflection of AKI development, separate risk scores for mortaltiy, receipt of RRT, CKD, and renal recovery could be provided. As an example, the ability to predict receipt of RRT may help providers plan for appropriate patient disposition (i.e., transfer to ICU for CRRT) and timely procedures (i.e., placement of dialysis catheter). Prediction of long-term renal and cardiovascular outcomes could be especially useful at the time of discharge, facilitating appropriate referrals, vascular access planning, and long-term care goal discussions.

We anticipate that a renal dashboard such as this could be displayed either directly within the system or independently from the EHR platform. Although information would be directly fed to the prediction model from up-to-date EHR data, each healthcare system, service, or unit may tailor the physical setting of the dashboard display to fit their workflows. For example, in an ICU setting where incidence of AKI may be as high as 40 %, the renal dashboard may be displayed on computerized workstations on wheels so that providers can incorporate the real-time information and feedback provided by the renal dashboard into their multi-disciplinary rounds [31]. For other services and locations where incidence of AKI is much lower - for example, the labor and delivery unit - the renal dashboard may serve in a more adjunctive role, to be monitored by a specialized “renal response” team (akin to traditional “rapid response” teams).

The consensus group acknowledges that numerous such dashboards could be created for similar medical conditions to assist with risk stratification. The approach described in this manuscript is designed to underscore the utility of a dashboard scheme. We realize that developing multiple dashboards for individualized diseases is unlikely to be efficient or effective in the long run. Operationally, a superior approach would be to seamlessly integrate a renal dashboard component into existing dashboard which is used to evaluate a range of quality and performance indicators.

Consensus Statement

The output from predictive models should be delivered to practitioners in a fashion that is cognizant of EHR limitations and strengths, minimizes workflow inefficiency, and maximizes utility.

Conclusion

The EHR dataset is a massive collection of clinically relevant data elements generated through the routine provision of patient care. Its size and complexity lend themselves to “big data” techniques; these in turn offer the potential to use the entire EHR dataset to predict AKI and AKI related outcomes. Variable selection should employ high-content, unsupervised analytic techniques. Developing predictive models should focus on EHR integration and optimize the output for clinical utility.