Effectiveness of automated notification and customer service call centers for timely and accurate reporting of critical values: a Laboratory Medicine Best Practices systematic review and meta-analysis.


Guideline Developer(s)

Laboratory Medicine Best Practices
Centers for Disease Control and Prevention

Date Released

2012 Sep

Evidence Supporting the Recommendations

Type of Evidence Supporting the Recommendations

The type of supporting evidence is identified and graded for the recommendation (see the "Major Recommendations" field).

Implementation of the Guideline

Description of Implementation Strategy

Feasibility of Implementation

For an automated notification system to have a reasonable chance of succeeding, health system administrators must assure policies and procedures are in place that mandate two-way communication of required acknowledgment/confirmation of receipt. Policies concerning routing and escalation after unsuccessful notification attempts must be in place, the staff must remain proficient in the use of manual procedures in the event of a technology failure and/or when escalation protocols require that laboratory staff revert to manual contacts.

Implementation Tools

Resources

Benefits/Harms of Implementing the Guideline Recommendations

Potential Benefits
  • The attention directed towards improvements for critical value notification is driven by the assumption that timely reporting will lead to timely clinical interventions and corresponding secondary prevention of co-morbidities and more effective treatment outcomes.
  • Although the available evidence neither supports nor rejects automatic notification systems, it seems likely that health care enterprises will increasingly seek integrated technology systems to manage patient processes, including automated critical value notification. The electronic audit trail captured by the automated notification system can play an important role in performance monitoring and evaluation, including targeted interventions for clinicians who do not attend to critical results. In addition, it has been observed that the development of automated notification systems can productively lead to a re-examination of critical value policies and thresholds and the development of interpretative reporting support, particularly for critical results in areas such as coagulation disorders; hemoglobin and anemia evaluations; autoimmune disorders; serum protein analysis; immunophenotyping analysis; genetic and molecular diagnostics; endocrinology; toxicology; and other new tests with which clinicians may be less familiar. For the use of call centers, the principal additional benefit appears to come from freeing laboratory workers from the time consuming activity of locating the responsible caregiver.
Potential Harms
  • Automated notification systems may have unintended disadvantages, such as disrupting usual lines of communication, and providing too much/too frequent information. The risk of losing back-up contact information must be properly anticipated. There are also risks for patient privacy violations, with protected health information being misdirected and/or mobile communication devices being accessible to unauthorized users.
  • The use of call centers may require additional communications with laboratory staff when a responsible caregiver requires additional information that call center staff are unable to provide. This may result in a delay of treatment while the responsible laboratorian is located. No information is available about the frequency of this occurrence, but it may undermine the convenience for assigning critical value communication responsibilities to the call center.

Rating Scheme for the Strength of the Recommendations

Recommendation Categories

The recommendation rating categories are consistent with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) group findings (2008) and reflect the extent to which the available evidence gives one confidence that a practice will do more good than harm.

Recommend: The practice should be identified as a best practice for implementation in appropriate care settings, taking into account variations in implementation and/or care settings. This recommendation results from consistent and high or moderate overall evidence of effectiveness strength rating of desirable impacts.

No recommendation for or against: The potentially favorable impact on care outcomes and/or error reduction is not sufficient, or not sufficiently supported by evidence to indicate that it should be identified as a best practice for implementation in appropriate care settings. Additional studies may be warranted to strengthen the relevant evidence base. This recommendation results from insufficient evidence to determine effectiveness.

Recommend against: The practice should not be identified as a best practice for implementation because of consistent evidence of adverse effects.

Qualifying Statements

Qualifying Statements

The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry (CDC/ATSDR).

Limitations

The Laboratory Medicine Best Practices Initiative (LMBP) systematic review methods are consistent with practice standards for systematic reviews but all of these methods are imperfect and include subjective assessments at multiple points that may produce bias. In particular, rating study quality depends on consensus assessments that may be affected by such things as rater experience and the criteria used. This systematic review may also be subject to publication bias, although unlike most systematic reviews this review includes unpublished studies which may mitigate that bias. Nonetheless, unpublished studies may be subject to a more general reporting bias in which institutions may have been more likely to share large and desirable effect sizes. The restriction to English language studies to satisfy the requirement of multiple reviewers for each study may also introduce bias.

Methodology

Methods Used to Collect/Select the Evidence

Hand-searches of Published Literature (Primary Sources)
Hand-searches of Published Literature (Secondary Sources)
Searches of Electronic Databases
Searches of Unpublished Data

Description of Methods Used to Collect/Select the Evidence

The question answered by this evidence review is: What practices are effective for communicating laboratory critical value results in an inpatient healthcare setting in a timely fashion to the licensed caregiver who can act on them? This review question is addressed in the context of an analytic framework for the quality issue that is timely and accurately reporting critical values. The relevant PICO elements, which establish the scope and content of the review are:

  • Population: all patients in healthcare settings with laboratory results that include a critical value
  • Intervention(s): automated notification systems and call centers for communicating critical values
  • Comparison practice/intervention(s): manual critical values notification systems
  • Outcomes: timeliness and accuracy of reporting or receipt of critical value information, or timeliness of treatment based on critical value information.

The literature search strategy was developed with the assistance of a medical librarian and included a systematic search in September 2011 of three electronic databases (PubMed, EMBASE and CINAHL) for English language articles from 1995 to 2011. The search contained the following Medical Subject Headings: cellular phone; clinical laboratory information system; computers, handheld; critical care; and hospital communication systems as well as these keywords: alerting system; automated alerting system; call center; critical value; and notification process. The search strategy also included hand searching of bibliographies from relevant information sources, consultation with and references from experts in the field and the solicitation of unpublished quality improvement studies resulting in direct submissions to the Laboratory Medicine Best Practices Initiative. The screening, abstraction and evaluation of individual studies were conducted by at least two independent reviewers.

Abstracts returned by the search were screened for potential eligibility using explicit exclusion criteria. Studies were excluded if: (1) the study was off-topic (i.e., did not assess the effectiveness of a rapid communication practice in an inpatient setting); or (2) was commentary or opinion. All other studies were retrieved. Once retrieved, documents were also excluded if: (1) it was not a practice of interest (i.e., the study did not discuss a rapid communication practice); (2) was not a study (i.e., the study did not examine the effectiveness of a rapid communication practice relative to a comparison practice); or (3) did not report effectiveness results for an outcome of interest (i.e., did not measure timeliness or accuracy of test reporting). All other studies contributed to the review, although not all studies provided sufficient data to contribute to the meta-analysis.

Number of Source Documents

The search procedures yielded 123 separate bibliographic records that were screened for eligibility to contribute evidence of critical value communication. An additional 79 records were identified through hand searching, and unpublished submissions (see Fig. 3 in the original guideline document). Of the 196 unique references identified, 155 were excluded as off topic in screening and 41 studies were retrieved for full text review. An annotated bibliography for these studies is provided in Appendix C (see the original guideline document). Of the 41 studies subjected to full text review, 12 were excluded for not being a study comparing a rapid communication practice to an alternative practice, 13 provided a comparison, but did not test a rapid communication practice, while 2 did not provide effectiveness evidence on an outcome of interest to the review.

The full text review and evaluation of the 11 eligible studies resulted in excluding 2 studies for poor study quality. Four studies provided valid estimates of the impact of automated notification and five provided valid estimates on call centers for improving the communication of critical values.

Methods Used to Assess the Quality and Strength of the Evidence

Weighting According to a Rating Scheme (Scheme Given)

Rating Scheme for the Strength of the Evidence

Overall Strength of Evidence Ratings

The revised definitions for these categories, modeled after the US Preventive Services Task Force (2008) are as follows:

High: An adequate volume of evidence is available and includes consistent evidence of substantial healthcare quality changes from studies without major limitations.

Moderate: Some evidence is available and includes consistent evidence of substantial healthcare quality changes from without major limitations.

Suggestive: Limited evidence is available and includes consistent evidence of moderate healthcare quality changes from a small number of studies without major limitations; OR the quality of some of the studies' design and/or conduct is limited.

Insufficient: Any estimate of an effect is very uncertain. Available evidence of effectiveness is:

  • Inconsistent or weak; OR
  • Consistent but with a minimal effect; OR
  • Contained in an inadequate volume to determine effectiveness
Methods Used to Analyze the Evidence

Meta-Analysis
Systematic Review with Evidence Tables

Description of the Methods Used to Analyze the Evidence

Although not required by Laboratory Medicine Best Practices Initiative (LMBP) criteria for making a practice recommendation, when possible, results were standardized to a common metric. Results based on mean differences were standardized using Cohen's d, which is calculated as the difference in means divided by their pooled standard deviation while criterion referenced results (e.g., percent of calls completed within 30 min) were standardized using odds ratios. A d-score of "0" indicates that the two practices are equally successful and observed differences are quantified according to their location along a standardized normal distribution, while the odds ratio centers on 1 (i.e., an odds ratio [OR] of 1=no difference), and observed differences are distributed along a logarithmic scale between 0 and N. To assist with the judgments of impact and consistency, standardized results were plotted on common graphs and a random effects inverse variance weighted grand mean* of the effectiveness of each practice was calculated. Standardizing findings and synthesizing results provide a valuable tool for estimating precision and assessing the consistency and patterns of results across studies.

The key criteria for including studies in the meta-analyses were: sufficient data to calculate an effect size, a "good" or "fair" study quality rating (estimating the extent to which each study yielded an unbiased estimate of the result of the practice), and use of an outcome that was similar enough to the other studies being summarized. When outcomes are similar and the study's effect size is attributable to the intervention or practice, then the grand mean estimate and its confidence interval are likely a more accurate representation of the results of a practice than that obtained from individual studies. Occasionally, studies will meet these criteria, but are sufficiently different in implementation or population to be excluded from the meta-analysis. By convention, all meta-analysis results are presented in tabular forest plots and are generated using Comprehensive Meta-analysis software (Statistical Solutions, v. 2.2.064).

Appendix D provides the summary tables for abstracted and standardized information for each eligible study reviewed, each study's quality rating on 4 dimensions and rationale for each quality rating (see the original guideline document).

*Random-effects model assumes there is no common population effect size for the included studies and the studies' effect size variation follows a distribution with the studies representing a random sample. This is in contrast to the fixed-effects model which assumes a single population effect size for all studies and that observed differences reflect random variation.

Methods Used to Formulate the Recommendations

Expert Consensus

Description of Methods Used to Formulate the Recommendations

This evidence review followed the Centers for Disease Control and Prevention (CDC)'s Laboratory Medicine Best Practices Initiative (LMBP)'s "A-6 Cycle" systematic review methods for evaluating quality improvement practices (see the "Availability of Companion Documents" field). This approach is derived from previously validated methods, and is designed to produce transparent systematic review results of practice effectiveness to support evidence-based best practice recommendations. The LMBP review topic selection is conducted by the LMBP workgroup following criteria, which require that they consider the existence of: (1) a measurable quality gap; (2) outcome measure(s) of broad stakeholder interest addressing at least one of the Institute of Medicine healthcare quality aims: safe, effective, patient-centered, timely, efficient and equitable; and (3) quality improvement practices available for implementation.

A review team, including a review coordinator and staff specifically trained to apply the LMBP methods, independently conducts the systematic review with guidance from a multi-disciplinary expert panel which includes individuals selected for their diverse perspectives and relevant expertise in the topic area, laboratory management, and evidence review methods. The process begins with an initial screening of bibliographic search results and ends with a full-text review, abstraction and evaluation of each eligible study using the LMBP methods. All evidence identified for the review is judged by the use of the same criteria and standards, thus unpublished evidence is subject to the same quality standards as published studies. To reduce subjectivity and the potential for bias, all screening, abstraction and evaluation are conducted by at least two independent reviewers and all differences are resolved through review team consensus after consultation with the expert panel, statisticians, and research methodologists as appropriate.

Recommendations are formulated in accordance with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group findings to reflect the extent to which one can be confident that following the recommendations will do more good than harm.

Cost Analysis

Economic Evaluation

Only one study provided any data related to an economic evaluation. That study reported that 230 hours of Information Technology staff time were required over a 5-month period to develop the automated notification system. No other practice-specific economic evaluations (cost, cost-effectiveness, or cost-benefit analyses) were found in the search results for call centers. It may be observed, however, that call center-based critical value notification requires that the healthcare facility has sufficient call volume and is adequately staffed to communicate the calls. Call center agents must be properly trained with automated policy and procedure manuals incorporated into the 'help screens' used by the call agents. When the combined volume of critical value calls and other activities is not sufficient, it may not be economical to operate a call center solely for the purpose of reporting critical laboratory test values.

Method of Guideline Validation

Not stated

Description of Method of Guideline Validation

Not applicable

Identifying Information and Availability

Bibliographic Source(s)

Liebow EB, Derzon JH, Fontanesi J, Favoretto AM, Baetz RA, Shaw C, Thompson P, Mass D, Christenson R, Epner P, Snyder SR. Effectiveness of automated notification and customer service call centers for timely and accurate reporting of critical values: a laboratory medicine best practices systematic review and meta-analysis. Clin Biochem. 2012 Sep;45(13-14):979-87. [45 references] PubMed

Adaptation

Not applicable: the guideline was not adapted from another source.

Source(s) of Funding

This systematic review is supported by contract CB-11-214 from the Centers for Disease Control and Prevention. Battelle Memorial Institute provided administrative, research and technical support for this review along with input from a panel of subject matter experts in laboratory medicine and systematic reviews.

Guideline Committee

Laboratory Medicine Best Practices Critical Values Reporting Expert Panel

Laboratory Medicine Best Practices Workgroup

Composition of Group That Authored the Guideline

Authors: Edward B. Liebow, Battelle Centers for Public Health Research and Evaluation, United States; James H. Derzon, Battelle Centers for Public Health Research and Evaluation, United States; John Fontanesi, Department of Medicine & Center for Management Science in Health, Department of General Internal Medicine, University of California-San Diego, United States; Alessandra M. Favoretto, Battelle Centers for Public Health Research and Evaluation, United States; Rich Ann Baetz, Battelle Centers for Public Health Research and Evaluation, United States; Colleen Shaw, Centers for Disease Control and Prevention, United States; Pamela Thompson, Centers for Disease Control and Prevention, United States; Diana Mass, University of Maryland School of Medicine, United States; Robert Christenson, Associated Laboratory Consultants, United States; Paul Epner, Paul Epner, LLC, United States; Susan R. Snyder, Battelle Centers for Public Health Research and Evaluation, United States

Laboratory Medicine Best Practices (LMBP) Critical Value Reporting Expert Panel Members: Robert Christenson, PhD (University of Maryland School of Medicine); Corinne Fantz, PhD (Emory University School of Medicine); Dana Grzybicki, MD, PhD (University of Colorado Cancer Center); Lee Hilborne, MD, MPH* (Quest Diagnostics, UCLA Medical School); Kent Lewandrowski, MD (Massachusetts General Hospital); Mary Nix, MT(ASCP)* (Agency for Healthcare Research and Quality); Rick Panning, MBA (Allina Hospitals and Clinics)

*LMBP Workgroup member

LMBP Workgroup, 2008–2011: Raj Behal, MD, MPH (Rush University Medical Center); Robert H. Christenson, PhD, DABCC, FACB (University of Maryland Medical Center); John Fontanesi, PhD (University of California, San Diego); Julie Gayken, MT (ASCP) (Regions Hospital); Cyril (Kim) Hetsko, MD, FACP (University of Wisconsin-Madison, Chief Medical Officer, COLA and Trustee, American Medical Association); Lee H. Hilborne, MD, MPH (UCLA David Geffen School of Medicine, President, American Society for Clinical Pathology, RAND, Quest Diagnostics); James H. Nichols, PhD (Baystate Medical Center); Mary Nix, MS, MT(ASCP)SBB (Agency for Healthcare Research and Quality); Stephen S. Raab, MD (University of Colorado Denver School of Medicine); Milenko Tanasijevic, MD, MBA (Brigham and Women's Hospital, Dana Farber Cancer Institute, Harvard Medical School); Ann M. Vannier, MD (Kaiser Permanente Southern California Regional Reference Laboratories); Ann Watt, MBA, RHIA (The Joint Commission)

Ex-officio Members: Sousan S. Altaie, PhD (Food and Drug Administration); James Cometa (Centers for Medicaid and Medicare Services)

Financial Disclosures/Conflicts of Interest

Members of the Laboratory Medicine Best Practices (LMBP™) systematic review expert panels and Workgroup are asked to provide information about financial, professional or other associations that may represent or appear to be a potential conflict related to the conduct of the LMBP™ systematic evidence reviews. The Centers for Disease Control and Prevention (CDC) LMBP™ staff reviews the conflict of interest disclosures. If a conflict arises in connection with an LMBP™ systematic review or publication, the appropriate disclosure is provided.

Guideline Status

This is the current release of the guideline.

This guideline meets NGC's 2013 (revised) inclusion criteria.

Guideline Availability

Electronic copies: Available from the PubMed Central (PMC) Web site.

Availability of Companion Documents

The following are available:

  • Laboratory Medicine Best Practices (LMBP). Effective practices for the timely and accurate reporting of laboratory testing critical values. Atlanta (GA): Centers for Disease Control and Prevention (CDC); 2012 Feb. 4 p. Electronic copies: Available from the Centers for Disease Control and Prevention (CDC) Web site.
  • Effectiveness of automated notification and customer service call centers for timely and accurate reporting of critical values: a Laboratory Medicine Best Practices systematic review and meta-analysis. Appendices. Electronic copies: Available from the PubMed Central (PMC) Web site.
  • Effectiveness of automated notification and customer service call centers for timely and accurate reporting of critical values: a Laboratory Medicine Best Practices systematic review and meta-analysis. Search strategy. 2011 Sep. 1 p. Electronic copies: Available from the CDC Web site.
  • Christenson RH, Snyder SR, Shaw CS, Derzon JH, Black RS, Mass D, Epner P, Favoretto AM, Liebow EB. Laboratory Medicine Best Practices: systematic evidence review and evaluation methods for quality improvement. Clin Chem 2011;57:816-25. Electronic copies: Available from the Clinical Chemistry Web site.
  • Snyder S, Liebow E, Shaw C, Black R, Christenson R, Derzon J, Epner R, Favoretto A, John L, Mass D, Patta A, Rose S, Washington M. Laboratory Medicine Best Practices: developing systematic evidence review and evaluation methods for quality improvement phase 3 final technical report. Atlanta (GA): Centers for Disease Control and Prevention (CDC); 2010 May 27. 159 p. Electronic copies: Available from the CDC Web site.
Patient Resources

None available

NGC Status

This NGC summary was completed by ECRI Institute on May 28, 2015. The information was verified by the guideline developer on August 13, 2015.

Copyright Statement

No copyright restrictions apply.

Scope

Disease/Condition(s)

Any condition or disease under evaluation that may result in a critical laboratory value

Guideline Category

Diagnosis
Evaluation
Prevention
Technology Assessment

Clinical Specialty

Critical Care
Emergency Medicine
Family Practice
Internal Medicine
Pathology
Preventive Medicine

Intended Users

Advanced Practice Nurses
Allied Health Personnel
Clinical Laboratory Personnel
Health Care Providers
Hospitals
Nurses
Physician Assistants
Physicians

Guideline Objective(s)
  • To conduct a systematic review of the evidence available in support of automated notification methods and call centers
  • To acknowledge other considerations in making evidence-based recommendations for best practices in improving the timeliness and accuracy of critical value reporting
Target Population

All patients in healthcare settings with laboratory results that include a critical value

Interventions and Practices Considered

Automated notification systems and call centers for communicating critical values

Major Outcomes Considered
  • Timeliness and accuracy of reporting or receipt of critical value information
  • Timeliness of treatment based on critical value information

Recommendations

Major Recommendations

Definitions for the ratings of overall strength of evidence and recommendation categories are provided at the end of the "Major Recommendations" field.

Conclusion and Recommendation

No recommendation is made for or against the use of automated notification systems in communicating critical values to responsible licensed healthcare providers for inpatients in hospital settings. Although multiple studies of automated notification systems provided evidence of substantial improvement in the timeliness of critical value notification, only one study was judged to be of "good" quality. Given Laboratory Medicine Best Practices Initiative (LMBP) criteria that multiple good studies are necessary to recommend a practice, the overall strength of evidence for automated notification systems is rated "suggestive."

On the basis of moderate overall strength of evidence of effectiveness, call centers are recommended as a best practice to improve critical value notification in inpatient care settings. The moderate overall strength of evidence rating is due to sufficient evidence of practice effectiveness from 5 studies; two "good" and three "fair" studies reporting "substantial" improvement in the timeliness of communicating critical value information.

Definitions

Overall Strength of Evidence Ratings

The revised definitions for these categories, modeled after the US Preventive Services Task Force (2008) are as follows:

High: An adequate volume of evidence is available and includes consistent evidence of substantial healthcare quality changes from studies without major limitations.

Moderate: Some evidence is available and includes consistent evidence of substantial healthcare quality changes from without major limitations.

Suggestive: Limited evidence is available and includes consistent evidence of moderate healthcare quality changes from a small number of studies without major limitations; OR the quality of some of the studies' design and/or conduct is limited.

Insufficient: Any estimate of an effect is very uncertain. Available evidence of effectiveness is:

  • Inconsistent or weak; OR
  • Consistent but with a minimal effect; OR
  • Contained in an inadequate volume to determine effectiveness

Recommendation Categories

The recommendation rating categories are consistent with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) group findings (2008) and reflect the extent to which the available evidence gives one confidence that a practice will do more good than harm.

Recommend: The practice should be identified as a best practice for implementation in appropriate care settings, taking into account variations in implementation and/or care settings. This recommendation results from consistent and high or moderate overall evidence of effectiveness strength rating of desirable impacts.

No recommendation for or against: The potentially favorable impact on care outcomes and/or error reduction is not sufficient, or not sufficiently supported by evidence to indicate that it should be identified as a best practice for implementation in appropriate care settings. Additional studies may be warranted to strengthen the relevant evidence base. This recommendation results from insufficient evidence to determine effectiveness.

Recommend against: The practice should not be identified as a best practice for implementation because of consistent evidence of adverse effects.

Clinical Algorithm(s)

None provided

Institute of Medicine (IOM) National Healthcare Quality Report Categories

IOM Care Need

Getting Better
Living with Illness
Staying Healthy

IOM Domain

Effectiveness
Patient-centeredness
Timeliness

Disclaimer

NGC Disclaimer

The National Guideline Clearinghouse™ (NGC) does not develop, produce, approve, or endorse the guidelines represented on this site.

All guidelines summarized by NGC and hosted on our site are produced under the auspices of medical specialty societies, relevant professional associations, public or private organizations, other government agencies, health care organizations or plans, and similar entities.

Guidelines represented on the NGC Web site are submitted by guideline developers, and are screened solely to determine that they meet the NGC Inclusion Criteria which may be found at http://www.guideline.gov/about/inclusion-criteria.aspx.

NGC, AHRQ, and its contractor ECRI Institute make no warranties concerning the content or clinical efficacy or effectiveness of the clinical practice guidelines and related materials represented on this site. Moreover, the views and opinions of developers or authors of guidelines represented on this site do not necessarily state or reflect those of NGC, AHRQ, or its contractor ECRI Institute, and inclusion or hosting of guidelines in NGC may not be used for advertising or commercial endorsement purposes.

Readers with questions regarding guideline content are directed to contact the guideline developer.

To get started, log in or create your free account Create Account