Big Brother

CASE 20

Big Brother Is Watching:

Utilizing Clinical Decision Support as a Tool to Limit Adverse Drug Events

AARON ROBERTS

A CRISIS IN THE EMERGENCY ROOM

It was nearing 3 p.m. in the emergency department and the charge nurse was rapidly losing her daily battle to make the hands of the clock move more quickly. The day had proven to be somewhat busy for a Friday afternoon, and she couldn’t wait for a chance to rest her weary feet. The most disappointing case of the day involved a kind, elderly gentleman that the attending physician was only barely able to stabilize. She remembered taking a peek at his chart and being stunned at how quickly a manageable situation had gotten out of hand. A little past 8 a.m., “Miller, Richard” had arrived in her unit after his routine morning finger prick had alerted him to an increased blood glucose level—high enough, he thought, to warrant some medical attention. The 78-year-old patient had recounted his current medication to the receptionist and the staff had checked multiple times to determine whether he had any drug allergies. After a brief stint in the waiting area and a decently short time in one of the emergency beds, he was out the door. In hand were two new prescriptions and an extra dose of comfort. However, 4 hours later, he was back in the emergency department, this time in critical condition with clear signs of an adverse drug reaction. The charge nurse could only wonder whether direct hospital error or some other cause had contributed to Mr. Miller’s condition.

MEDICATION ERRORS

In the United States, medication errors account for 44,000 deaths and millions of hospitalizations and clinical visits a year.1 These statistics are the consequences of adverse drug events (ADEs), which are any unexpected side effects attributed to the use of a combination of drugs. Researchers have estimated that 95% of adverse drug reactions go unreported.2 However, of those ADEs that are recognized, a substantial proportion can be attributed directly to mistakes made by hospital staff. Many health providers have turned to new health information technology as a way to reduce this burgeoning problem. In particular, the development of computer-decision support systems is a potential step toward reducing this problem and a step toward preventing people like Richard Miller from spending his night in the emergency department.

Costs

In 2003, U.S. spending on prescription drugs soared to over $200 billion, with spending for 40 popular drugs reaching $1 billion apiece.3 The prevalence of suboptimal prescribing practices has been estimated to be as high as 30%.4 Every year, $177.4 billion is wasted to cover morbidity and mortality caused by adverse drug events.5 A drug reaction may lead to hospitalization and outpatient treatment, which can account for over a third of hospital visits for the aging population.6 When combined, the direct costs for hospitalization for ADEs, the costs of poor prescribing practices, and calculated indirect costs amount to a sum that would reinvigorate an ailing U.S. healthcare system.

Health Disparities

Mr. Miller, as an elderly diabetic, belongs to a subgroup that experiences a disproportionate number of adverse drug events. Although increased age has been cited as a key risk factor for adverse drug reactions, other measures, such as the existence of comorbidities, better typify this issue.7 Among older patients, the drugs insulin, warfarin, and digoxin contribute to a third of all adverse drug events.8 Patients with chronic conditions, especially those that affect drug distribution, experience more adverse drug events.9 Individuals taking anticoagulants or medications that interfere with the central nervous system are also at increased risk.10 Additionally, whites and those with private insurance are more likely to face adverse drug reactions as their increased access to health services causes them to be at risk more often.11

Limited Research

The rush of new and increasingly popular drugs to the market has led to controversy regarding the effectiveness and appropriateness of certain treatment options. For example, statins, a class of drugs designed to reduce levels of low-density lipoprotein cholesterol, are widely prescribed. Every year, articles show strong evidence that these drugs are extremely effective at producing better health outcomes.12 They are countered by equally striking evidence showing this class of drugs is largely ineffective.13 The uncertainty demonstrated by scientific research hasn’t slowed the increase in prescribing rates, with statins quadrupling in total use among ambulatory patients over 10 years and occupying 90% of the lipid-lowering market in 2002.14 Researchers are unlikely to use older patients as subjects in trials to test new drugs.15 So, one possibility in Mr. Miller’s case is that the hospital staff may have administered the correctly indicated drug, but the research that contributed to its approval may not have been comprehensive. Additionally, physicians don’t generally wait for the final verdict on effectiveness before choosing to prescribe new drugs, making adverse drug events and suboptimal prescribing a possibility.

FIGURE 20-1 Medication appropriateness index.

Source: Hanlon JT, Schmader KE, Samsa GP, et al. A method for assessing drug therapy appropriateness. J Clin Epidemiol. 1992;45:1045–1051.

Prescribing Errors

On the other hand, the physician in charge of Mr. Miller’s case may have prescribed an incorrect medication—either one that negatively impacted his concurrent ailments or one that interacts poorly with other medications. Patients often see a variety of different specialists, who aren’t always given accurate information about patient medication history.16 Changes to drug regimens made by one provider may not always be relayed to other physicians caring for a shared patient.17 Even when such prescribing information is shared, physicians and pharmacists may have limited knowledge about adverse drug events because they are not up to date on current findings. To limit prescribing errors, physicians should be using tools like the Beers criteria for potentially inappropriate medication use in the elderly18 and the Medication Appropriateness Index (see Figure 20-1) to limit prescribing errors.

Mistakes can occur elsewhere as well. Appropriately prescribed medication may not reach the patient due to poor handwriting or overuse of abbreviations. Telephone orders may be easily confused.19 When healthcare providers overcome these barriers, patients are able to get the correct prescriptions and will hopefully be placed along the path toward improved health.

Medication Adherence

It’s possible that Mr. Miller may have taken an incorrect dosage or combined his new prescription with some synergistic medium such as alcohol. Medication adherence requires proper understanding of proper dosage and when to take medications as well as a willingness to stick to drug regimens. Simple things, like remembering to take medication, also become more difficult as people age or succumb to disorders like dementia. The costs associated with many popular pharmaceuticals can also hinder medication adherence, as patients sometimes choose not to fill prescriptions in order to save money.

Strong provider-patient relationships, in which patients have their concerns answered, contribute to higher rates of adherence.14 Patients take the medications they feel were prescribed correctly and will achieve the desired results. Assistance from physicians and pharmacists with prescription refills and medication schedules helps to improve patient comprehension and increase medication adherence. The minimal time available to hurried emergency physicians and nurses may not leave sufficient time to ensure patients like Mr. Miller understand their medication instructions.20

Challenges in Long-Term Care

Nursing homes continue to care for a substantial fraction of the population over 65. Some would argue Mr. Miller may need more directed care if he cannot manage his medication needs. Unfortunately, the choice to move an elderly patient to managed care doesn’t completely protect him from adverse drug events. Studies have shown that nursing home patients take, on average, 8.8 medications daily, making nursing homes breeding grounds for medication error.21 Patients have been found to receive the incorrect medication, to receive medication at the wrong time of day, or miss a treatment altogether. Nursing home workers have a high level of job strain, have to cope with poor staffing levels, and often feel guilty about not being able to adequately meet the needs of residents.22 Within such a high-demand work environment, mistakes are not surprising. In a third of cases, however, medication errors are made repeatedly.23 Decreasing rates of adverse drug events not only provides better health outcomes among nursing home residents; it also helps to lower the already stifling costs of institutionalized care.

Polypharmacy as a Risk Factor

Mr. Miller’s earlier trip to the emergency department and his current condition could be completely coincidental. It is possible his body had not withstood the effects of polypharmacy. Polypharmacy, the excessive prescription of medication, has begun to take a heavy toll on the elderly population. By 2000, 37 million doctor visits by people over the age of 65 (one quarter of all such visits) ended with patients being prescribed at least five different medications.24 Excessive prescription can refer simply to the number of medications (usually five or more) or to cases in which drugs are inappropriately or unnecessarily prescribed. Patients taking more than five medications are more likely to have combinations that lead to adverse drug events.14 The reaction that brought Mr. Miller back into the hospital could have been caused by one of his other combinations and may have had nothing to do with his new prescriptions. Nonetheless, the most effective approach to prevent his unfortunate situation would have been if the emergency department staff had taken steps to reassess his drug use.

ADVERSE DRUG EVENTS AS A PUBLIC HEALTH CONCERN

Adverse drug events are a concern for individuals as well as communities. Driving has been an area of particular concern for older adults. Many medications, especially antidepressants, alter the ability of an individual to stay focused behind the wheel. Very little research has been conducted on the effect of multiple medications on driving ability.25 Considering the general decline in driving ability and reaction time associated with specific classes of drugs, this should be an area of concern for policy makers. It would have been most unfortunate if Mr. Miller had experienced his reaction while behind the wheel.

Falls are also associated with adverse drug events. Individuals prescribed numerous prescriptions are at high risk for falls and consequent debilitating injuries. Falls are a serious concern for the aging population as the associated injuries can often lead to a loss of independence or hasten individual decline toward death. Interventions designed to reduce the incidence of falls often aim to reduce adverse combinations of drugs among patients.26 Patients who take fewer medications have fewer drug reactions that interfere with their daily activities.

HEALTH INFORMATION TECHNOLOGY—A SOLUTION?

New technologies are always emerging in health-related fields as older instruments are updated or a new product is designed to increase productivity. Electronic medical records (EMRs) emerged as a way to improve quality of care and increase efficiency. An EMR serves as a replacement for paper medical records by electronically storing critical patient information in a central location.27 EMRs provide enhanced documentation and allow physicians to pool patient information. They may offer certain clinical tools, like medication ordering or patient assessments. Equipped with evidence for the effectiveness of this technology, 31% of emergency departments and 17% of physicians’ offices had switched to EMRs in 2003.28 Currently, 11% of hospitals have fully implemented systems, while 66% more have partially implemented systems.29 Although a skeptical Mr. Miller may question the safety of health information being stored electronically, his physicians are more thankful for the clarity these systems can provide. Early support systems were programmed to provide reminders to hospital staff, to catch errors before they occur, or to perform basic diagnostic tasks.30 These more progressive programs, termed clinical decision support (CDS) systems, have been utilized in attempts to reduce the occurrence of adverse drug events.

Three examples that follow describe research into the impact of clinical decision support programs. Could these systems have changed the circumstance at hand and prevented Mr. Miller’s critical health crisis?

Case 1: CDS in Prescribing Practices

Researchers in the Pacific Northwest took notice of the high prevalence of adverse drug events nationwide and the thousands of deaths that can be attributed to preventable mistakes. Their study design attempted to reduce ADEs through the use of CDS. All 450,000 members of a health maintenance organization group were included in the study in an attempt to best represent the general population of this region. The primary care physicians, nurse practitioners, and physician’s assistants recruited for the study were already acquainted with electronic medical record systems, allowing them to enter patient orders (for lab tests, medication needs, and treatment options) electronically. The quasiexperimental intervention began with a 12-month observational period designed to identify provider prescribing practices, followed by a 27-month intervention period. Using a computerized decision support system, researchers were able to alert providers of a preferred alternative when they prescribed a nonpreferred drug. The criterion for a nonpreferred drug was established as those that were not indicated for use in older patients, including a class of long-acting benzodiazepines and tertiary amine tricyclic antidepressants. This technology alerted physicians when they prescribed one of the suboptimal drugs. It is important to note that even though these drugs were acceptable in younger patients, the technology raised alerts for all patients. Thus, alerts were drug specific and did not vary based on patient characteristics or any aspect specific to a patient’s case. After receiving an alert, providers then had the option to change the medication or to ignore the alert. The number of medication alerts and data on dispensing rates were used to determine the effectiveness of the intervention.31

Patients whose cases evoked alerts were most likely to be older (22.9 alerts per 10,000 for patients over age 65 versus 8.2 alerts in patients under 65) and female (69.4% of elderly women but only 56.2% of elderly men triggered alerts). A 22% decrease from the initial prescribing rate of nonpreferred medications was seen after the first month of the intervention; this lower rate of prescription of 16.1 dispenses per 10,000 held for the duration of the trial. The use of preferred medications also spiked 20% after the first month of the intervention and then experienced a slight but steady increase throughout the course of the experiment. The most dramatic changes in prescribing pattern occurred in the elderly population. However, prescription of preferred medications also increased among nonelderly patients, an unexpected result for researchers. The subclasses of drugs that experienced the largest overall change in dispensing rates in favor of preferred drug classes were those for which physicians could see clear evidence for clinical equivalency and for which there was a consensus on drug effectiveness.31

Overall, the use of this new technology helped to limit the frequency of nonpreferred medications prescribed to elderly patients. The system appeared to be widely accepted by clinicians, as some were more likely to order the preferred medication after receiving an alert because the system automatically filled important parts of the prescription, thus saving the physician more time to allot to adequately treating patients.31

Could a system like this have helped improve Mr. Miller’s situation?

Case 2: CDS in Medication Adherence

Respiratory therapies represent an area in which patients do not always employ the recommended treatment options. One approach designed to improve treatment utilization is to train more knowledgeable healthcare providers who can better communicate care management strategies to patients. One randomized intervention study was carried by Indiana University’s medical group in an attempt to increase treatment adherence among respiratory patients. This intervention targeted patients seen by physicians across four hospitals with shared medical records. Seven hundred and six patients were initially enrolled in the study and about two thirds completed the final survey.32

Patients were randomized into four different intervention groups according to the physicians they saw, and were additionally randomized to obtain medication from a single pharmacist. In the first group, both the physician and the pharmacist received the intervention, while only the provider or the pharmacist received the intervention in the second and third. A final comparison group received no intervention. Patient-specific care suggestions were generated by a panel of expert clinicians and programmed into each physician’s electronic workstations, with explanations and references. Care suggestions fit into several key categories that can be found in Table 20-1. As a member of the intervention group, Mr. Miller’s physician would have been presented with a care suggestion on her workstation when she wanted to order new care options or review patient information. Members of the intervention groups were required to view all suggestions, with an option to order or omit the new care options. They were also presented with information being used in concurrent studies, such as medication warning alerts32 (similar to the alerts from the Pacific Northwest intervention).

Researchers were left with somewhat less promising results than they would have liked. At the conclusion of the trial, there was no significant difference in patient adherence or satisfaction with providers across the experimental groups. There was an isolated improvement in emotional quality of life for those who received medication from a pharmacist in the intervention group. In this provider-centered approach, designed to create improvements in adherence, physicians expressed mixed opinions about the care suggestions. Some felt the suggest ions were helpful and educational, while others believed they were too rigorous and infringed on physician autonomy. Physicians in the intervention groups also experienced significantly higher healthcare costs.32

TABLE 20-1 Potential Suggestions for Providers Used in Case 2—CDS in Medication Adherence

•  Performing pulmonary function tests

•  Giving influenza and pneumococcal vaccinations

•  Prescribing inhaled steroid preparations in patients with frequent symptoms of dyspnea

•  Prescribing inhaled anticholinergic agents in patients with chronic obstructive pulmonary disease

•  Escalating doses of inhaled b-adrenergic agonists for all patients with persistent symptoms

•  Prescribing theophylline for patients with chronic obstructive pulmonary disease and continued symptoms despite aggressive use of inhaled anticholinergic agents, b-agonists, and steroids

•  Encouraging smoking cessation

Source: Data from Tierney W, Overhage M, Murray M, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005;40(2): 477–498.

Would a system like this have helped Mr. Miller’s geriatrician convey health information more effectively, and thereby enhance Mr. Miller’s medication adherence?

Case 3: North Carolina Initiative

The North Carolina Long-Term Care Polypharmacy Initiative used CDS to improve patient health outcomes in the nursing home setting. If fate did push Mr. Miller into a long-term care setting, his care would be greatly impacted by the changes enacted by this initiative. This intervention was developed by a Medicaid case management firm, Community Care of North Carolina, and involved both physicians and pharmacists. The study originated from a desire to lower medication costs for long-term care residents, but evolved into an effort to reduce polypharmacy and adverse drug events. This intervention built upon two previous pilot studies and began with an assessment of detailed baseline demographic and health information for over 8000 participants.33

Lengthy computer algorithms were used to determine five categories of medication profiles that should be brought to the attention of the pharmacists. The alert categories, listed in Table 20-2, were formed based on drug interactions, costs, and effectiveness. Using the alert system and Medicaid claims information, patient medication records were reviewed and flagged for further pharmacy services. The pharmacists would then make recommendations to providers, which, based on the discretion of the provider, could lead to a change in drug regimens. A prospective aspect was also added to the intervention, allowing pharmacists to take action when they received new medication orders that warranted concern. This two-pronged approach was designed to end a cycle of bad practice, by protecting new and existing patients from adverse medication combinations. An initial 90-day baseline period was followed by a 90-day intervention period and a 90-day postintervention period. Hospitalizations, drug alerts, and costs were tracked. The targeted group was compared with other Medicaid patients not enrolled in this program in order to provide a viable comparison group.31

For analysis, subjects were divided into 10 groups based on the services they received and whether those were retroactive or prospective actions. The most significant results were found among patients who received both prospective and retrospective services that eventually led to drug changes. On average, patients saved about $21 per month, which could translate to annual savings of about $2 million for everyone in the initiative. Participants were immediately less likely to experience hospitalizations, and the reduction of drug alerts suggests better medical outcomes over longer periods of time. The total number of medications taken remained mostly constant throughout the study.33

TABLE 20-2 Description of Drug Alert Categories Used in Case 3: North Carolina Initiative

Source: Data from Trygstad TK, Christensen DB, Wegner SE, et al. Analysis of the North Carolina Long-Term Care Polypharmacy Initiative: a multiple-cohort approach using propensity-score matching for both evaluation and targeting. Clin Ther. 2009;31(9):2018–2037.

If this system had been in place in the pharmacy where Mr. Miller had filled his prescriptions, what reviews of his existing and new medications might have been performed? To what effect?

A BRIGHTER TOMORROW WITH BIG BROTHER

All three of these studies illustrate interventions that attempt to address the myriad possible underlying causes of Mr. Miller’s adverse drug event. The results found in the Pacific Northwest intervention are not atypical. Decision support systems in the clinical field tend to be met with little resistance by clinicians and produce moderate results. However, research also demonstrates that it’s more challenging to deal with patients already receiving inappropriate medications. While CDS can often result in fewer prescribing errors, it may not reach those patients who are on stable medication regimens.

Although case 2 could ultimately prove to be a beneficial model, computer-decision support was unable to produce positive results in this attempt to address adherence with respiratory medications. Perhaps certain aspects of the case design or target population prevented researchers from reaching the target goals. The North Carolina Initiative produced much more promising results, yet it still had a few downsides to overcome.

It is also helpful to consider what happens to prescribing practices once intervention programs end. Once support systems are removed, have providers learned to order the correct drugs or will they simply revert to old mistakes? Making successful interventions sustainable is an important next step.

As healthcare systems implement these sorts of clinical decision support tools, the innovations will be largely invisible to the patients. Mr. Miller would never directly interact with the health information technology and would not be aware that Big Brother was watching. All he would experience is better health.

About the Author

Aaron Roberts grew up in Rochester, New York, and recently graduated from Brown University with a double concentration in human biology and community health. He developed his love for medicine at quite a young age. After working as a surgical technician in a women’s care unit, he developed an interest in the health of vulnerable populations. Currently, his career focus is on outreach to urban youths and producing sustainable elder care while he works his way toward medical school and an eventual career in pediatrics.

REFERENCES

1. Institute of Medicine. To Err Is Human: Building a Safer Health System. Kohn LT, Corrigan JM, Donaldson MS, eds. Washington, DC: National Academy Press; 2000.

2. Hazell L, Shakir SA. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29(5):385–396.

3. Aitken M, Berndt E, Cutler D. Prescription drug spending trends in the United States: looking beyond the turning point. Health Aff. 2009;28(1):w151–w160.

4. Cannon KT, Choi MM, Zuniga MA. Potentially inappropriate medication use in elderly patients receiving home health care: a retrospective data analysis. Am J Geriatr Pharmacother. 2006;4(2):134–143.

5. Ernst FR, Grizzle AJ. Drug-related morbidity and mortality: updating the cost-of-illness model. J Am Pharm Assoc (Wash). 2001;41:192–199.

6. Page RL II, Ruscin F. The risk of adverse drug events and hospital-related morbidity and mortality among older adults with potentially inappropriate medication use. Am J Geriatr Pharmacother. 2006;4(4):297–305.

7. Beyth R, Shorr R. Epidemiology of adverse drug reactions in the elderly by drug class. Drugs Aging. 1999;14(3):231–239.

8. Budnitz D, Pollock D, Weidenbach K, Mendelsohn A, Schroeder T, Annest J. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006;296(15):1858–1866.

9. Chester M, Chen L, Kaski JC. Identification of patients at high risk for adverse coronary events while awaiting routine coronary angioplasty. Br Heart J. 1995;73:216–222.

10. Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes. Am J Med. 2000;109:87–94.

11. Eric T, David S, Burstin H, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Medical Care. 2000;38(3):261–271.

12. Wilt TJ, Bloomfield HE, MacDonald R, et al. Effectiveness of statin therapy in adults with coronary heart disease. Arch Intern Med. 2004;164:1427–1436.

13. van der Harst P, Voors AA, van Gilst WH, et al. Statins in the treatment of chronic heart failure: biological and clinical considerations. Cardiovasc Res. 2006;71(3): 443–454.

14. Ma J, Sehgal NL, Ayanian JZ, et al. (2005) National trends in statin use by coronary heart disease risk category. PLoS Med. 2005;2(5):e123.

15. Hutchins F, Unger J, Crowley J, Coltman C, Albain K. Underrepresentation of patients age 65 years and older in cancer-treatment trials. N Engl J Med. 1999;341:2061–2067.

16. Gurwitz J, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289:1107–1116.

17. Staroselsky M, Volk LA, Tsurikova R, et al. (2008). An effort to improve electronic health record list accuracy between visits: patients’ and physicians’ response. Int J Med Inf. 2008;77(3):153–160.

18. Fick DM, Cooper JW, Wade WE, et al. Updating the Beers criteria for potentially inappropriate medication use in older adults: results of a U.S. consensus panel of experts. Arch Intern Med. 2003;163:2716–2724. [Published correction appears in Arch Intern Med. 2004;164:298].

19. Chase BA, Bennett NL. Medication safety, systems and communications. http://www.massmed.org/Content/NavigationMenu2/MedicationSafetySystemsandCommunication/CourseInformation/default.htm. Published 2006. Accessed September 25, 2010.

20. Williams AF, Manias E, Walker R. Adherence to multiple, prescribed medications in diabetic kidney disease: a qualitative study of consumers’ and health professionals’ perspectives. Int J Nurs Stud. 2008;45(12):1742–1756.

21. Handler SM, Nace DA, Studenski SA, Fridsma DB. Medication error reporting in long term care. Am J Geriatr Pharmacother. 2004;2:190–196.

22. Morgan DG, Semchuk KM, Stewart NJ, D’Arcy C. Job strain among staff of rural nursing homes: a comparison of nurses, aides, and activity workers. J Nurs Adm. 2002;32(3):152.

23. Crespin DJ, Modi AV, Wei D, et al. Repeat medication errors in nursing homes: contributing factors and their association with patient harm. Am J Geriatr Pharmacother. 2010;8(3):258–270.

24. Aparasu RR, Mort JR, Brandt H. Polypharmacy trends in office visits by the elderly in the United States, 1990 and 2000. Res Soc Adm Pharm. 2005;1(3):446–459.

25. Lococo KH, Staplin L. Literature Review of Polypharmacy and Older Drivers: Identifying Strategies to Collect Drug Usage and Driving Functioning among Older Drivers. Washington, DC: U.S. Department of Transportation, National Highway Traffic Safety Administration; 2006. Report no. DOT HS 810 558.

26. Moylan KC, Binder EF. Falls in older adults: risk assessment, management and prevention. Am J Med. 2007;120(6):493.e1–493.e6.

27. Allen D, Fenwick M. Continued progress: hospital use of information technology, American Hospital Association. www.aha.org/aha/research-and-trends/AHA-policy-research/2007.html. Published 2007. Accessed October 14, 2010.

28. Burt CW, Hing E. Use of computerized clinical support systems in medical settings: United States, 2001–03. Advance Data. 2005;353:1–8.

29. Wager K, Schaffner M, Foulois B, Swanson Kazley A, Parker C, Walo H. Comparison of the quality and timeliness of vital signs data using three different data-entry devices. Comput Inform Nurs. 2010;28(4):205–212.

30. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120(2):135–142.

31. Smith DH, Perrin N, Feldstein A, et al. The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med. 2006;166(10):1098–1104.

32. Tierney W, Overhage M, Murray M, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005;40(2):477–498.

33. Trygstad TK, Christensen DB, Wegner SE, et al. Analysis of the North Carolina Long-Term Care Polypharmacy Initiative: a multiple-cohort approach using propensity-score matching for both evaluation and targeting. Clin Ther. 2009;31(9):2018–2037.