Needs assessment: towards a more responsive Canadian Society of Nephrology Annual General Meeting (CSN AGM) program

The EC-CSN is in the process of using the results of the above described needs assessment in the planning of the 2016 CSN AGM. As mentioned, the issues with the highest identified perceived and/or unperceived learning needs include topics related to glomerulonephritis and to vascular access on dialysis.

To our knowledge, this is the first published example of a needs assessment for a nephrology CME event that considers both perceived and unperceived needs. That being said, there are numerous published examples of needs assessments for CME. Most of these are perceived needs assessments that utilize survey questionnaires, including Curran et al. in 2007 (family medicine CME in Canada) [11], Turner et al. in 2006 (multiple sclerosis in the USA) [12], Turner et al. in 2004 (occupational medicine) [13], and Purdy in 2002 (migraine) [14]. Dupuis et al. conducted a perceived needs assessment related to Crohn’s disease that utilized a triangulated, mixed-method approach—questionnaire plus semi-structured interview [15].

There are also examples of unperceived needs assessments for CME: Aeschilmann et al. utilized a multiple choice quiz to assess learning needs for rheumatology CME [16]. Klein et al. utilized an environmental scan, in the form of an electronic clinical practice audit, to conduct their needs assessment for family medicine CME in Canada [17]. Finally, Laidlaw et al. conducted a needs assessment of general practitioners for malignant melanoma that utilized multiple methods and considered both perceived and unperceived needs—surveys of general practitioners and specialists (perceived needs), plus literature review and advisory group recommendation review (unperceived needs) [18].

There are a number of accepted techniques for conducting perceived needs assessments for CME. The tool most commonly used is the survey questionnaire, which elicits written responses to multiple questions. Strengths of this technique include efficiency in terms of material and human resources, their ability to address a wide range of topics, and the fact that the information can be returned in a standardized fashion (if, for example, multiple choice questions are used). Conversely, problems can arise when response rates are low (sometimes necessitating expensive follow-up efforts) and if the surveys themselves are poorly constructed [19]. The EC-CSN selected this method for its perceived needs assessment because of these indicated strengths.

Additional published strategies for perceived needs assessments include interviews and focus groups (“group interviews”). These methods acquire personalized, in-depth information; in focus groups, members draw from one another to enhance the information provided. These, however, are very time-consuming and resource-intensive to conduct [5]. For these latter reasons, the EC-CSN did not utilize these techniques.

There are also a number of accepted strategies for conducting unperceived needs assessments for CME. A popular one is the environmental scan. For this technique, existing information/documents are scanned unobtrusively for learning needs. These documents could include minutes of meetings, chart audits, attendance data, national databases, published guidelines of care, and literature searches, among others. A major strength of this technique lies in its economical sources of data, i.e., documents have already been produced for another purpose, and so no new expenditures are required. Further, a wide spectrum of data is available, the data are often generated iteratively, and it can be scanned unobtrusively. On the other hand, weaknesses include data that may be too broadly defined, too time-consuming and costly to analyze, and the fact that there can be political interference with accessing/analyzing the data [5, 20]. The CORR database was deemed an ideal selection for data as it was freely offered for review by its governing board (in a de-politicized manner), and no costs were attached to its review.

Other published strategies for unperceived needs assessments include chart audits and pre-course testing. These provide objective information but can be very time-consuming to conduct [5, 16]. For these reasons, the EC-CSN did not utilize these techniques.

Finally, the published literature suggests that the validity of the needs assessment is increased if multiple techniques are used, and the results of these techniques are compared—a process known as triangulation [5]. Our study utilizes such a technique which we feel strengthens our results in terms of validity.

There are a number of strengths to the present study. It represents the first published study of a needs assessment for a nephrology CME event that considers both perceived and unperceived needs. It therefore demonstrates that such work can be done in order to enhance the delivery of nephrology CME. Second, for the perceived needs assessment, a reasonably representative sample of Canadian nephrologists—171 individuals—responded to the survey, enhancing its validity. Third, for the unperceived needs assessment, a robust database that represents dialysis practice throughout Canada was available for analysis. Fourth, triangulation of the different methods utilized was performed, further enhancing the validity of the work.

We acknowledge that there are some limitations to the study. For the perceived needs assessment, the topics for the survey were determined by the EC-CSN after much discussion; that said, some of the topics were much more general (e.g., “Dialysis Outcomes”), and others were very specific (e.g., “Radiographic Interventions and the Kidney”). This could have led to some topics being selected more than others by the respondents.

Perhaps the greatest limitation of the unperceived needs assessment is the lack of a standard, validated, published approach for conducting it. The environmental scan, in a general sense, is a validated method, particularly for qualitative data [5]; however, we could identify no standard approach for handling this kind of data. That said, we feel that the method we devised was reasonable—we believe that the panel of experts was able to assess the factors that detract from perfect guideline target achievement and to determine if those factors were nephrologist-related or not. Second, the results were dependent on the availability of CORR data. CORR does not generate data for the non-dialysis-related topics, so the survey results for these topics could not be triangulated with any known reference. Third, the authors concede that the level of evidence cited within published guidelines can be poor; for example, KDOQI’s minimum acceptable target for urea clearance (a single pool Kt/V of 1.2 for thrice weekly dialysis) reflects grade 1B level evidence [9].

A third possible weakness in the unperceived needs assessment is that the targets set by the panel of experts were chosen because they were what the panel felt to be reasonable but were not correlated with patient outcomes. Indeed, the literature does include a number of studies that look at the extent to which achievement of guideline targets predicts outcome. Djukanovic et al. demonstrated that the failure to achieve KDOQI targets for Kt/V, hemoglobin, and PTH was associated with an increased relative risk of time to death in a Serbian hemodialysis population [21]. Anton-Perez et al. demonstrated an inability to meet KDOQI vascular access targets in a Spanish population and showed that this was independently associated with high mortality [22]. On the other hand, Tangri et al. not only demonstrated variable guideline target achievement for various mineral metabolism factors (calcium, phosphate, PTH) in a British renal registry (hemodialysis plus peritoneal dialysis) but also showed that those patients who achieved guideline targets did not have a survival advantage [23]. As such, in our view, there is insufficient evidence in the literature to support the use of outcome-based targets for our needs assessment—not all guidelines are addressed, and those that are addressed arrive at inconsistent conclusions.

For part C, triangulation, as for the unperceived needs assessment, there is no validated, published approach for triangulating this kind of data. Most of the published literature that describes triangulation methods for needs assessments involves qualitative data and utilizes analysis of themes derived from such data sources as interviews and focus groups [15, 18]. We do feel, however, that our method of comparing the rank order of the various topics identified by the two assessment methods is reasonable. Second, the triangulation was performed on data that was not quite contemporaneous: The unperceived needs assessment was conducted on data obtained as of December 2013; the survey questionnaire (perceived needs) was administered in April 2015. This was done in order to use the most current data possible when planning the meeting. The likelihood that this introduced significant error is low because (a) the datasets differ only by 16 months and (b) the results of the unperceived needs assessment have not changed very much over the 5 years that this assessment has been conducted.