Can someone help me with this assignment?The Effects of CPOE on Clinical Decision SupportAfter read The Impact of CPOE and reviewing contents from Module 6, submit a final paper to discuss the effects

international journal of medical informatics 7 7(2008)365–376 journal homepage: www.intl.elsevierhealth.com/journals/ijmi Review The impact of computerized physician medication order entry in hospitalized patients—A systematic review Saeid Eslami ∗, Nicolette F. de Keizer, Ameen Abu-Hanna Department of Medical Informatics, Academic Medical Center, Universiteit van Amsterdam, Meibergdreef 15, 1105 AZ Amsterdam, The Netherlands article info Article history:

Received 30 January 2007 Received in revised form 26 September 2007 Accepted 3 October 2007 Keywords:

Medical order entry systems Computerized physician order entry (CPOE) Medication safety Evaluation Drug therapy Computerized abstract Objective:To identify all published studies evaluating computerized physician order entry (CPOE) in the inpatient setting and uniformly classify these studies on outcome measure and study design.

Data sources:All studies that evaluated the effect of CPOE on outcomes pertaining to the medication process in inpatients were electronically searched in MEDLINE ®(1966 to August 2006), EMBASE ®(1980 to August 2006) and the Cochrane library. In addition, the bibliogra- phies of retrieved articles were manually searched. Articles were selected if one of their main objectives was CPOE evaluation in an inpatient setting.

Review method:Identiﬁed titles and abstracts were independently screened by three review- ers to determine eligibility for further review.

Results:We found 67 articles, which included articles on CPOE evaluation on some outcome at the time of ordering. Most papers evaluated multiple outcome measures. The outcome measures were clustered in the following categories: adherence (n= 22); alerts and appro- priateness of alerts (n= 7); safety (n= 21); time (n= 7); costs and (organizational) efﬁciency (n= 23); and satisfaction, usage and usability (n= 10). Most studies used a before–after design (n= 35) followed by observational studies (n= 24) and randomized controlled trials (n= 8).

Conclusion:The impact of CPOE systems was especially positive in the category adherence to guidelines, but also to some extent in alerts and appropriateness of alerts; costs and organizational efﬁciency; and satisfaction and usability. Although on average, there seems to be a positive effect of CPOE on safety, studies tended to be non-randomized and were focused on medication error rates, not powered to detect a difference in adverse drug event rates.

Some recent studies suggested that errors, adverse drug events (ADEs) and even mortality increased after CPOE implementation. Only in the category time the impact has been shown to be negative, but this only refers to the physician’s time, not the net time. Except for safety, on the whole spectrum of outcomes, results of RCT studies were in line with non-RCT study results.

E-mail address:[email protected](S. Eslami).

doi:10.1016/j.ijmedinf.2007.10.001 366 international journal of medical informatics 7 7(2008)365–376 Contents 1. Introduction.................................................................................................................. 366 2. Methods...................................................................................................................... 368 3. Results........................................................................................................................ 368 3.1. CPOE and adherence to guidelines................................................................................... 369 3.2. CPOE and medication safety.......................................................................................... 369 3.3. CPOE, cost and (organizational) efﬁciency........................................................................... 369 3.4. CPOE, alerts and appropriateness of alerts.......................................................................... 371 3.5. CPOE and time........................................................................................................ 371 3.6. CPOE, user satisfaction and usability................................................................................ 371 4. Discussion.................................................................................................................... 371 4.1. Former reviews........................................................................................................ 372 4.2. CPOE and adherence to guidelines................................................................................... 372 4.3. CPOE and medication safety.......................................................................................... 372 4.4. CPOE, cost and (organizational) efﬁciency........................................................................... 372 4.5. CPOE, alerts and appropriateness of alerts.......................................................................... 372 4.6. CPOE and time........................................................................................................ 372 4.7. CPOE, user satisfaction and usability................................................................................ 373 4.8. Other evaluation measures........................................................................................... 373 4.9. Study designs.......................................................................................................... 373 5. Conclusion.................................................................................................................... 373 Appendix A. Supplementary data............................................................................................. 373 References.................................................................................................................... 373 1. Introduction Medication prescription plays a central role in health care. It concerns 65% of the US population and it annually accounts for 13% of health care expenditures[1]. Various studies show that adverse drug events (ADEs), many of which are pre- ventable, form a major problem in the US[2–5]. In the early 1990s, it was estimated that there are 3.7 adverse events per 100 admissions in the US[6]. Of these, 28% are medication related, half of which preventable. From an economic point of view, hospital costs of adverse drug events were estimated at $2 billion per year. Similar reports in other countries show that medication errors indeed have important impact on mortality, morbidity and cost of care[7].

Medication errors are usually the result of failures dur- ing the medication process. Errors can occur in any step of this process: taking history, ordering, pharmacy management, administration management or surveillance[8]. A medication error may or may not result in patient harm, but almost all medication errors are considered to be preventable. adverse drug events (ADEs) are usually considered to include both medication errors that result in harm (preventable ADEs), and adverse drug reactions (ADRs), which are considered unpreventable[9]. Although high workload[10]and failures in monitoring patients[11]have been reported as a possi- ble causes of medication errors, most medication errors and preventable ADEs are related to the medication process and mainly occur during the ordering step[3,12]. In addition to the inability of the average physician to memorize the ever increasing number of drugs, treatment regiments and side effects, prescribing the old fashioned way with pen and pad is prone to slips which are sometimes errors of inattention[13].The institute of medicine and other important stakehold- ers have identiﬁed computerized physician order entry (CPOE) or electronic prescription (EP) as the main opportunity to reduce medication errors and thereby improve safety[2,14] especially when decision support is provided. CPOE systems promise to have also effects on outcomes other than safety, such as medication and process costs. A CPOE system refers to a variety of computer-based systems that share the com- mon features of automating the medication ordering process and that ensure standardized, legible, and complete orders [15]. Electronic medical record systems which merely docu- ment medication orders and medication administration, after the time of ordering, are beyond the scope of this paper and are therefore excluded. In addition, we deﬁned a decision sup- port system (DSS) in this context as any system designed to aid a health professional in decision-making at the moment of ordering medication. A DSS can be an inherent part of CPOE or a separate system communicating with the CPOE system. The main objective of this review is to identify, uniformly charac- terize, and assess the reported CPOE impact in all published studies evaluating any aspect, safety and otherwise, associ- ated with the use of a CPOE system in the inpatient setting.

We excluded all studies in the outpatient setting since this is a completely different context with different challenges and a review on CPOE in the outpatient setting is published elsewhere[16]. The measured effects are summarized and associated with an evidence level. One should, however, real- ize that the assessment of effects of information technology in health care is inherently hard because the technology is only part of a much larger and complex social system[17].Nev- ertheless, by presenting and assessing the state-of-the-art in CPOE evaluation studies, this review contributes to a better international journal of medical informatics 7 7(2008)365–376 367 Fig. 1 – Keywords used in the two search strategies and the search ﬂow. In strategy 1, keywords and MeSH terms that are currently in use for referring to a CPOE system (part A) are combined with terms related to inpatient care (part B). In strategy 2, computer (C) and medication (D) related terms are combined to identify studies that address prescribing with computerized systems in an inpatient setting (B), for especially uncovering older studies. The results of these two strategies are combined by using the boolean operator “or”. 368 international journal of medical informatics 7 7(2008)365–376 understanding of the merits of CPOE systems in the inpatient setting, and identiﬁes lacunas in current research on CPOE evaluation.

2. Methods We searched for relevant english language articles, based on keywords in title, abstract and MeSH terms, using Ovid MEDLINE ®& MEDLINE ®in-process (1966 to August 2006), Embase ®(1980 to August 2006), and Cochrane library.Fig. 1 shows the two applied search strategies and the correspond- ing search ﬂowchart. In strategy 1, keywords and MeSH terms that are currently in use for referring to a CPOE system (part A) are combined with terms related to inpatient care (part B). In strategy 2, computer (C) and medication (D) related terms are combined to identify studies that address prescrib- ing with computerized systems in an inpatient setting (B), for especially uncovering older studies. The results of these two strategies are combined by using the boolean operator “or”.

Searching was supplemented by scanning bibliographies from identiﬁed review articles. The literature search was per- formed in August 2006.

Identiﬁed titles and abstracts were screened by three reviewers to determine eligibility for further review. Articles were selected if they reported original data from a study in which one of the main focuses was on evaluation of a CPOE system for medication ordering in an inpatient set- ting. All studies reporting on alerts, reminders, and DSS which are not part of or related to a CPOE system and which are not triggered during the medication order entry were excluded. Opinion papers, reviews, and letters were excluded.From the selected papers, the same three reviewers extracted data on the materials, study design, outcome mea- sures, and results. Discrepancies between reviewers were resolved by consensus. For the included papers all reported outcome measures have been extracted and clustered into homogeneous outcome groups.

To obtain insight into the heterogeneous nature of these evaluation studies, we classiﬁed them according to the hierar- chy of study designs developed by the University of California San Francisco Stanford evidence-based practice Center and implemented by Kaushal et al.[15]in their review (Table 1). 3. Results Searching the online databases resulted in 1,004 articles from Ovid MEDLINE ®, Ovid MEDLINE ®In-process, Ovid EMBASE ®, and the Cochrane library. After initial screening of titles and abstracts, 74 articles were considered for full text review. Six additional articles were identiﬁed by reviewing bibliographies, yielding a total of 80 articles. Based on the full text review, six articles were excluded because they turned out not to address a CPOE system according to our deﬁnition, and seven arti- cles have been excluded because evaluation was not a main objective of the study or evaluation was not performed on a speciﬁc CPOE system, leaving 67 articles for detailed anal- ysis (seeFig. 2). A summary of the study design, outcome measures and main results of the 67 studies is available as supplementary material. It is noteworthy to mention that the number of articles on CPOE evaluation has sharply increased in recent years (Fig. 2).

Most of evaluation studies on CPOE were done in USA (83%, 56 out of 67), only 11% (7 out of 67) in the European Union and the rest (6%, 4 out 67) in Brazil, Canada and Australia. Table 1 – Hierarchy of study designs [15] Level Study design Description I Randomized controlled rrial (RCT) A study in which people are allocated at random (by chance alone) to receive one of several clinical interventions. One of these interventions is the standard of comparison or control. The investigator controls the exposure to the intervention.

II Non-randomized controlled trial A study in which people are allocated to receive one of several clinical interventions. One of these interventions is the standard of comparison or control. The investigator controls the exposure to the intervention but allocation of people is not based on chance. It includes interrupted time series and before–after studies.

III Observational study with control A study in which individuals are observed or certain outcomes are measured without a speciﬁc attempt to affect the outcome (the investigator does not control the exposure to the intervention e.g. the use of a CPOE). The intent is to observe how exposure to risk factors (implemented CPOE) inﬂuences the outcome of interest. Includes cross-sectional studies to estimate the prevalence of the outcome of interest or the prevalence of exposure to intervention or both; cohort (longitudinal) studies with control in which individuals who are exposed to the intervention are followed for a deﬁned length of time and the effects of the intervention on the exposed group is compared to a group that was not exposed; and case control studies in which a comparison of exposure to the CPOE in a group of individuals with the outcome of interest (cases) is compared to those without the outcome of interest (controls).

IV Observational study without control Includes cohort studies without controls or case series. international journal of medical informatics 7 7(2008)365–376 369 Fig. 2 – Distribution of the number of published articles on CPOE evaluation over the years.

Benchmark leaders in CPOE evaluation are Brigham Women’s Hospital and Partners Heath Care in Boston (19%, 13 out of 67), Vanderbilt University Medical Center in Nashville (9%, 6 out of 67) and Regenstrief Institute (6%, 4 out of 67).

In all these studies, we categorized the main outcome mea- sures in six main groups: adherence to guidelines; medication safety; cost and (organizational) efﬁciency; alerts and appro- priateness of alerts; time; and satisfaction, usage and usability.

Most papers evaluated multiple outcome measures.

Table 2shows a summary of all reported measured effects of CPOE evaluation studies where results are reported sepa- rately by study design and by outcome group. The column on the right summarizes the effect of CPOE accompanied by the level of evidence.

Below, we describe the effects per outcome group for all included studies. Details about materials and results of each study are shown in thesupplementary material.

3.1. CPOE and adherence to guidelines Twenty-two articles were classiﬁed in this group. Two articles in this group described adherence to system alerts without using a control group[18,19]. One randomized clinical trial[20] (RCT) and one non-RCT[21]did not ﬁnd a signiﬁcant difference between the control and the intervention group with regard to adherence to the guideline. Seventeen other studies, four RCTs and thirteen non-RCTs, showed positive effects on adherence to guidelines[22–38]. Among these, four non-RCTs, showed positive effects on the adherence to order sets, which are pre- deﬁned prescriptions[32,36–38]. McAlearney et al.[39]showed that order set utilization and utilization trend varied by con- dition. Dexter et al.[40]in an RCT, showed that adherence to standing orders was signiﬁcantly better than adherence to reminders during order entry.

3.2. CPOE and medication safety In 21 controlled studies, medication errors, ADEs, and/or mor- tality were compared between an intervention group using the CPOE and a control group not using the CPOE. In all of these trials the controls were not randomly selected. In addition, ten articles described the effect of CPOE on safetyin an observational study[19,41–49]. Next to ADEs, various combinations of parameters were measured to indicate med- ication errors including wrong dose adjustment, route and interval errors, drug–drug interaction, drug allergy, wrong therapy and contraindications, formulary errors, illegibility, transcription errors, administration errors, and dispensing errors. There were 24 studies that mentioned the medica- tion error type. These included 18 studies (75%) reporting on dosing errors especially excessive dosage, 7 (29%) on wrong therapy/contraindication, 5 (21%) on drug interaction, 5 (21%) on illegibility, and 4 (17%) on drug allergy.

In 1998, Bates et al.[50]performed a prospective non- RCT and showed for the ﬁrst time that CPOE reduced serious medication error rate by 55%. Despite a positive trend the study was not powered to detect a signiﬁcant difference in the preventable ADE rate. To evaluate safety, ADE rate is more relevant than medication errors because errors do not nec- essarily result in adverse events. All 19 non-RCT studies that evaluated the effect of a CPOE on the number of errors showed positive effects on the reduction of the number of errors [22,34,50–66]. Shulman et al.[64]showed a signiﬁcant reduc- tion of medical errors in a non-RCT but they also reported two serious errors—those of deﬁnite clinical signiﬁcance with the potential to cause harm to the patient—caused by CPOE leading to adverse events and increase in length of stay, and another three potential ADEs attributed to CPOE. In the same year, another observational study showed that the number of medication errors remained at a high-level after CPOE imple- mentation[48]and three other recent observational studies described how new medication errors have been associated with CPOE systems[47,49,67].

Ten studies, all non-RCTs, assessed the effect of DS on the number of ADEs[23,50–54,57,58,62,65]. Among them, seven studies showed a reduction in the number of ADEs, while the remaining three studies did not show an effect[23,54,62].Two studies which showed reduction in the number of errors could not show a reduction in the number of ADEs[54,62]. Nebeker et al.[68]reported in an observational study that the number of ADEs remained at a high-level after CPOE implementation.

In 2005, a non-RCT, showed that after CPOE implementa- tion mortality increased signiﬁcantly in pediatric ICU[69].In 2006, another non-RCT showed a non-signiﬁcant reduction in mortality rate after CPOE implementation in pediatric ICU [70]. One observational study reported 55 incidents which were related to CPOE. However, few incidents resulted in patient harm varying between 12% for physiologic harm and 2% for psychological distress[46]. 3.3. CPOE, cost and (organizational) efﬁciency Twenty-three studies, four RCTs, ﬁfteen non-RCTs and four observational studies fell in this group[23–29,32–34,36,45,51, 53,54,57,59,61,71–75]. The main outcome measures were hos- pital cost, medication cost, cost of process time, and proxies for efﬁciency and costs such as length of stay (LOS) and deﬁned daily dose. Organizational efﬁciency was measured by changes in the distribution of workﬂow processes.

One RCT showed that a CPOE system did not change any related aspects to cost and efﬁciency[25]. One non-RCT study showed mixed effects, some positive and some negative, on 370 international journal of medical informatics 7 7(2008)365–376 Table 2 – Effects of CPOE on outcome categories subdivided by quantitative study designs Outcome categoryStudy designTotal Positive effect No effect a Negative effect Mixed effect b Overall conclusion based on level of evidence Demonstrated cStatistically sig.Mix d Demonstrated e Statistically sig.Mix f Adherence to guidelineRCT 5 1 3 – 1 – – – – Adherence to guideline or to computerized recommendation increase due to CPOE systems. Non-RCT 14 3 8 1 1 – – – 1 Obs. ctrl. – – – – – – – – – SafetyRCT – – – – – – – – –Prescribing errors decrease although there are some negative (observational) studies recently. There is no evidence on the effect of CPOE systems on ADEs. Non-RCT 21 7 10 – 1 – 1 – 2 Obs. ctrl. 1 1 – – – – – – – Cost and efﬁciencyRCT 4 – 2 1 1 – – – – Studies on cost and effectiveness showed mixed results. In addition, some important costs may not be accounted for. Non-RCT 16 6 2 4 2 – 1 – 1 Obs. ctrl. – – – – – – – – – AlertRCT 1 – 1 – – – – – –Quantitative studies show high adherence to alerts. However qualitative studies show many overridden alerts. Acceptance rate increase with the clinical importance of the alerts. Non-RCT 2 – 2 – – – – – – Obs. ctrl. 2 1 – – 1 – – – – TimeRCT 1 – – – – – 1 – – Direct order entry time increase. When indirect time is also measured the overall time did not change, or even decreased. Non-RCT 5 1 2 – – – 2 – – Obs. ctrl. – – – – – – – – – Satisfaction, usage and usabilityRCT – – – – – – – – – No conclusion due to lack of quantitative studies.

Non-RCT – – – – – – – – – Obs. ctrl. – – – – – – – – – TotalRCT 1 6 1 2 – 1 – – Non-RCT 17 24 5 4 – 4 – 4 Obs. ctrl. 2 – – – – – – – Outcome categories are ordered in decreasing level of evidence of the overall conclusion.aWhen reported as such by the authors, with or without statistical arguments.bMix of positive, absence of, and negative effects.cWhen the authors report a positive effect but without reporting statistical signiﬁcance.dMix of statistically signiﬁcant and demonstrated positive effects.eWhen the authors report a negative effect but without reporting statistical signiﬁcance.fMix of statistically signiﬁcant and demonstrated negative effects. international journal of medical informatics 7 7(2008)365–376 371 cost and efﬁciency[27]. Fifteen studies, three RCTs and twelve non-RCTs, showed that CPOE had a positive effect on at least one relevant aspect of cost and/or efﬁciency. While nine studies showed that CPOE decreased the hospital or phar- macy costs[26,28,29,51,53,54,57,59,71], four articles showed that there was no statistically signiﬁcant effect on hospital or pharmacy costs[23,25,36,61]. Mekhjian et al.[61]showed that CPOE did not change cost signiﬁcantly although the total cost per admission decreased signiﬁcantly in some selected ser- vices. The outcome measure LOS gave mixed results as ﬁve studies, two RCTs and three non-RCT, showed that CPOE has no statistically signiﬁcant effect on LOS[25,27,33,36,71]and even increase it, and three other studies, non-RCTs, showed a signiﬁcant reduction of LOS[23,51,61].

Kaushal et al.[74]considered in an observational study var- ious kinds of CPOE costs and beneﬁts and showed that over 10 years, 80% of CPOE implementation cost was reimbursed.

Finally, Cheng et al.[72]evaluated the effect of CPOE on disruption in the workﬂow process in an observational qual- itative study. They showed that policies designed to increase ﬂexibility and safety led to an increased coordination load on the health care team, and created new sources of errors.

A recent observational study showed that after CPOE imple- mentation, physicians and nurses worked in asynchronous mode, and left the coordination of their actions to the system.

The study also showed that orders were exhaustively docu- mented but that some data may be misinterpreted forming new sources of errors[75].

3.4. CPOE, alerts and appropriateness of alerts Three observational studies assessed the impact of CPOE on generated, accepted and ignored alerts[42,43,76]. Most of the alerts were ignored by the physicians. One study showed that when the alerts were classiﬁed in high-level and low-level groups, the high-level alerts were more often accepted than the low-level alerts (57% versus 8%)[42].

Three studies, one RCT, one non-RCT and one obser- vational study, showed that alerts given by pharmacists for solving prescription problems, and efforts for helping clinicians decreased signiﬁcantly after the implementation of CPOE[25,54,60]. Another non-RCT showed that the rate of alerts given by pharmacists was not different in the clinical areas with electronic prescription system com- pared to the clinical areas using hand-written prescription [44].

3.5. CPOE and time Seven articles focused on the effect of CPOE on time [56,59,61,71,77–79]. We considered ordering time and turn- around time and have not attempted to distinguish between patient care time and teaching time. Three studies, one RCT and two non-RCTs, showed that CPOE increased the ordering time[71,77,78]. These studies focused on physicians, and time savings for pharmacy and nursing have not been addressed.

Only a recent observational study showed that direct patient care time did not change signiﬁcantly[79].

Another outcome measure in this group was medication turn-around time, the total time from writing the medicationorder to the delivery of the medication to the ward. Three non- RCTs showed that a CPOE system decreased medication turn- around time signiﬁcantly[56,59,61]. 3.6. CPOE, user satisfaction and usability Ten observational studies[42,44,45,54,57,79–83]focused on user satisfaction and usability. Eight studies showed that after the introduction of the CPOE system, the majority of users were satisﬁed with the system and they believed that the sys- tem is usable. Although the environment, questionnaire, and target groups (nurses or physicians) were different in these studies, the majority of users believe that CPOE improved drug management and quality of care. One additional observational study compared two CPOE systems, one commercial and one self-developed, and showed that users were more satisﬁed with the self-developed system[82]. Only one (recent) paper found that electronic prescribing with DS was not feasible at the hospital studied[84]. Although feasibility forms a different concept than usability and satisfaction, it was taken here as a crude proxy for this category.

4. Discussion We identiﬁed and described the results of 67 papers on eval- uation of CPOE systems in hospitalized patients. The number of such evaluation studies shows a strong increase in recent years.

Our ﬁndings suggest that the impact of CPOE systems has been shown to be positive especially in Adherence to guide- lines, but also in Alerts and Appropriateness of alerts; costs and organizational efﬁciency, and satisfaction and usabil- ity. A major goal for the implementation of CPOE is safety improvement. Despite the generally positive effect of CPOE on safety shown in generally non-randomized studies, they were focused on medication error rates, not powered to detect a dif- ference in adverse drug event rates or mortality. In addition, in light to recently published case studies showing a negative effect on safety one should conclude that, by itself, the intro- duction of a CPOE system does not necessarily lead to better safety and that there are other factors at play[47,49,67,69].

Implementing information technology applications such as CPOE is a socio-technical activity, which often depends more on organizational context than on a speciﬁc technology[92].

Although in the category Time, the impact has been negative for physicians’ time, there was a positive effect on turn- around-time. Except for Safety, on the whole spectrum of outcomes, results of RCT studies were in line with non-RCT study results.

The inﬂuence of publication bias was not investigated in this review but one should be aware of its possible existence while interpreting the generally positive ﬁndings of the impact of CPOE systems. Ammenwerth et al.[92]showed in their anal- ysis of the contradictory results concerning the effect of CPOE systems on mortality rates in two pediatric intensive care units that implementing a CPOE system primarily concerns a socio-technical activity and that each setting is unique in its combination of sociological, technical, organizational and human factors. Because of this, it is hard to make absolute 372 international journal of medical informatics 7 7(2008)365–376 conclusions about the effect of CPOE on an outcome category, especially when there is a negative study.

Our literature search has been extensive in the sense that our search criteria were meant to capture different ways a CPOE system is referred to while covering search criteria used by other reviews. However, our review is restricted to CPOE systems i.e. a system that allows clinicians to enter medica- tion prescriptions for inpatients into a computer prospectively and not general EMR systems in which medication orders and medication administration are merely documented after paper-based prescription.

4.1. Former reviews To our knowledge ﬁve other reviews have been published on the evaluation of CPOE systems in inpatients[15,85–88].

The scope of these reviews includes CPOE systems that are sometimes embedded in a larger whole, and CPOE systems that may or may not focus on medication. Kaushal et al.[15] focused on medication safety and the review was restricted to studies evaluating the effect of CPOE on errors and ADEs.

In 2003, Oren et al.[85]published a second review article focusing on the impact of CPOE, bar-coding and computer- ized medication administration records on medication errors and ADEs. However, eight out of the eleven studies in their review did not concern medication errors or ADEs. In 2003, Kuperman et al.[86]also published a review article on CPOE but it did not focus on medication. In early 2005, another broad review article on the impact of health information technology on quality, efﬁciency, and cost of medical care has been published[87]. This review captured fewer stud- ies than ours in the ﬁeld of CPOE evaluation. Another recent review article focused on overriding drug safety alerts in CPOE which forms only one aspect of CPOE system evalua- tion[88].Gragetal.[89]recently published a review article about DS systems but, assumingly because it did not specif- ically focus on CPOE, it did not include any paper about CPOE. Compared to these ﬁve reviews, we have uncovered more studies in the same time frame; this is due to the more comprehensive strategy we adopted which included old and new CPOE related terms. Furthermore we extended the review time frame to also include the most recent stud- ies. We also provided a more extensive characterization of the selected studies including description of a studies evi- dence level, study design, methods, materials, and results.

Our study focuses on medication but the evaluation aspects are not limited to only safety although this is one of the most important outcome measures. We will now summarize and discuss the effects of CPOE on six categories of outcome measures.

4.2. CPOE and adherence to guidelines Almost all studies in our review that evaluated the effect of CPOE systems on adherence to guidelines showed a positive effect. This shows that clinicians generally accept the sys- tem’s suggestions. However, if the guidelines are not correctly implemented in the system or when patient information is not correctly recorded in the system, any problem in match- ing the guidelines with patient information can lead to seriouserrors and to ADEs as some recent papers[49,67]have indeed reported. 4.3. CPOE and medication safety Our review shows that the rates of the proximal outcomes, medication errors, fell due to CPOE introduction although the effect on ADEs, which is a more relevant clinical outcome, did not merit enough attention. One study even showed a nega- tive effect of CPOE on mortality. All of the reviewed studies on safety turned out to be non-RCTs although most are controlled ones. The great majority of studies have employed relatively little DS.

4.4. CPOE, cost and (organizational) efﬁciency There is some evidence about the positive effects of CPOE on hospital and pharmacy costs. But we believe that it would be beneﬁcial to include also the cost of software and its maintenance, of personnel’s time utilization, of adherence to guidelines, and of errors and ADEs, e.g. in terms of addi- tional treatment, and logistic and nursing cost, but we admit this is difﬁcult to operationalize. Only a recent single-center study assessed some of these aspects and showed that devel- oping, implementing and operating a CPOE system is costly ($11.8 million over 10 years) but considering the money saved due to the CPOE system the net effect was positive[74].

Because adherence to computerized suggestions based on guidelines is high, costs are expected to be affected accord- ingly: costs will tend to decrease when the guideline is geared towards cost reduction, and costs will tend to increase when the guideline entails more expensive treatment as, e.g.

Hulgan et al.[29]reported on decreasing medication costs after the implementation of an intravenous/oral conversion DSS.

4.5. CPOE, alerts and appropriateness of alerts All three articles concerning the acceptance of alerts showed that physicians did not accept most of the alerts. This should be considered from two points of view: system weakness and user response. We believe that a major system weakness is the provision of too many alerts leading to low user com- pliance. They also include patient non-speciﬁc advice and use incomplete patient information. Overloading physicians with irrelevant alerts may result in inattention. Alert systems should be redesigned to only show important patient-speciﬁc alerts[42,88].

4.6. CPOE and time All three studies which consider ordering time as an outcome measure, from 1993 until the most recent one in 2001, showed that CPOE increased ordering time for physicians. In addition this increase in time might lead to an increase in the clini- cian’s workload which can introduce new medical errors[11].

However there were no studies in our review which assessed the effect of a CPOE system on workload. CPOE time reduction for pharmacy and nursing was not studied. Recently, a study showed that direct-patient-care time did not change signiﬁ- international journal of medical informatics 7 7(2008)365–376 373 cantly[79]. As this study is an observational one, and the only one found on direct-patient-care, it is premature to arrive at strong conclusions.

4.7. CPOE, user satisfaction and usability Almost all studies which evaluated the effect of CPOE on user satisfaction and usability were observational studies and showed positive results. However, more studies with better evidence level are necessary especially for studying the impact on usability, usage and feasibility.

4.8. Other evaluation measures Interestingly, none of the studies in our review evaluated technical facets[90]such as user interface, data manage- ment, data security, technical functionality of a checking mechanism for dangerous drug dosage, ﬂexibility in chang- ing the knowledge and rules etc. Also the following medical facets were mainly missing: knowledge base completeness and accuracy; the extent of adverse drug reaction reporting and consistency with dealing with different medication trade names.

4.9. Study designs On the whole, the studies we found were heterogeneous in terms of their study design and most of them produced level-two evidence, non-RCTs such as before–after studies.

Twenty-six out of the 35 non-RCTs (74%) reported at least one statistically signiﬁcant positive difference, which is compara- ble to six out of eight RCTs (75%) reporting on at least one statistically signiﬁcant positive difference. The comparability among studies in this review is hard at least due to the lack of standardization methodology for CPOE evaluation. Moreover, the scope of the studies varies from a narrow domain such as speciﬁc drugs to a hospital-wide setting; studies also differ in the implementation environment of the CPOE, the approach to DS, and the nature and types of medication and treatment.

This heterogeneity may explain at least some differences in results such as effects on cost or LOS.

5. Conclusion In conclusion, CPOE evaluation studies vary in scope, aims and results, and one should not expect unequivocal judg- ment about their effects. One could perhaps argue for more RCT studies in the evaluation of CPOEs but they do have pro- hibitive costs. One fruitful way to proceed with is the use of controlled trials focusing on CPOE systems with more decision support for speciﬁc patient groups, high risk drugs, typical ADEs, using more powerful designs like interrupted time- series. Another fruitful direction is to recognize that while the standard RCT methodology is excellent for studying system or clinical performance, it is not well suited to answering ques- tions concerning whether systems will be used or how they will be used[91]. This calls upon a complementary evaluation methodology that considers the social context in which CPOE systems operate. Summary points What is already known:

•CPOE systems have been identiﬁed as key for improv- ing patient’s safety.

•CPOE systems can inﬂuence both positively and nega- tively physicians’ behavior and outcomes What this study added to our knowledge:

•The impact of CPOE systems was especially positive in adherence to guidelines, but also to some extent in alerts and appropriateness of alerts; costs and organi- zational efﬁciency; and satisfaction and usability.

•Studies on safety showed some positive effect of CPOE but tended to be non-randomized and were focused on medication error rates, not powered to detect a differ- ence in adverse drug event rates.

•CPOE had negative impact on the prescribing physi- cian’s time, however, there was a positive effect on the net turn-around-time.

•Except for safety, on the whole spectrum of outcomes, results of RCT studies were in line with non-RCT study results.

Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, atdoi:10.1016/j.ijmedinf.2007.10.001.

references [1] D.S. Bell, S. Cretin, R.S. Marken, et al., A conceptual framework for evaluating outpatient electronic prescribing system based on their functional capacity, J. Am. Med.

Inform. Assoc. 11 (2004) 60–70.

[2] L.T. Kohn, J.M. Corrigan, M.S. Donaldson, To Err is Human:

Building a Safer Health Syst., National Academy Press, Washington, DC, 1999.

[3] D.W. Bates, D.J. Cullen, N. Laird, et al., Incidence of adverse drug events and potential adverse drug events, JAMA 274 (1995) 29–34.

[4] D.W. Bates, A.M. Burrows, D.G. Grossman, et al., Top priority actions for preventing adverse drug events in hospitals:

recommendation of an expert panel, Am. J. Health-Syst. Ph.

53 (1996) 747–751.

[5] A. Oliven, I. Michalake, D. Zalman, et al., Prevention of prescription errors by computerized, on-line surveillance of drug order entry, Int. J. Med. Inform. 74 (2005) 377–386.

[6] T.A. Brennan, L.L. Leape, N.M. Laird, et al., Incidence of adverse events and negligence in hospitalized patients.

Results of Harvard medical practice study I, New Engl. J.

Med. 324 (1991) 370–376.

[7] C. Vincent, G. Neale, M. Woloshynowych, Adverse events in British hospitals: preliminary retrospective record review, BMJ 322 (2001) 517–519. 374 international journal of medical informatics 7 7(2008)365–376 [8] P. Kilbridge, D. Classen, et al., A process model of inpatient medication management and information technology interventions to improve patient safety, VHA’s research series (2001).

[9] B.D. Franklin, C. Vincent, M. Schachter, et al., The incidence of prescribing errors in hospital inpatients: an overview of the research methods, Drug Safety 28 (2005) 891– 900.

[10] R. Grifﬁth, H. Grifﬁths, S. Jordan, Administration of medicine.

part 1: the law and nursing, Nurs. Stand 18 (2003) 47–53.

[11] S. Jordan, C. Tunnicliffe, A. Sykes, Minimizing side-effects:

the clinical impact of nurse-administered ‘side effect’ checklist, J Adv. Nurs. 37 (2002) 155–165.

[12] T.S. Lesar, L.L. Briceland, D.S. Stein, Factors related to errors in medication prescribing, JAMA 277 (1997) 312–317.

[13] B. Dean, M. Schachter, C. Vincent, et al., Causes of prescribing errors in hospital inpatients: a prospective study, Lancet 359 (2002) 1373–1378.

[14] A. Milstein, R.S. Galvin, S.F. Delbanco, et al., Improving the safety of health care: the leapfrog initiative, Eff. Clin. Pract. 3 (2000) 313–316.

[15] R. Kaushal, K.G. Shojania, D.W. Bates, Effects of computerized physician order entry and clinical decision support systems on medication safety, Arch. Int. Med. 163 (2003) 1409–1416.

[16] S. Eslami, A. Abu-Hanna, N.F. de Keizer, Evaluation of outpatient computerized physician medication order entry systems; a systematic review, J. Am. Med. Inform. Assoc. 14 (2007) 400–406.

[17] J. Aarts, M. Berg, Same system, different outcomes:

comparing the implementation of computerized physician order entry in two Dutch hospitals, Method. Inform. Med. 45 (2006) 53–61.

[18] M.I. Oppenheim, C. Vidal, F.T. Velasco, et al., Impact of a computerized alert during physician order entry on medication dosing in patients with renal impairment, in:

Proceedings of AMIA Symposium, 2002, pp. 577–581.

[19] B. Dubeshter, J. Griggs, C. Angel, J. Loughner, Chemotherapy dose limits set users of a computer order entry system, Hosp. Pharm. 41 (2006) 136–142.

[20] J.M. Overhage, M. Tierney, C.J. McDonald, Computer reminders to implement preventive care guidelines for hospitalized patients, Arch. Int. Med. 156 (1996) 1551–1556.

[21] J. Butler, T. Speroff, P.G. Arbogast, et al., Improved compliance with quality measures at hospital discharge with a computerized physician order entry system, Am.

Heart J. 151 (2006) 643–653.

[22] J.M. Teich, P.R. Merchia, J.L. Schmiz, et al., Effects of computerized physician order entry on prescribing practices, Arch. Int. Med. 160 (2000) 2741–2747.

[23] G.M. Chertow, J. Lee, G.J. Kuperman, et al., Guided medication dosing for inpatients with renal insufﬁciency, JAMA 286 (2001) 2839–2844.

[24] N.E. Kawahara, F.M. Jordan, Inﬂuencing prescribing behavior by adapting computerized order-entry pathways, Am. J.

Hosp. Pharm. 46 (1989) 1798–1801.

[25] J.M. Overhage, W.M. Tierney, X. Zhou, et al., A randomized trial of “corollary orders” to prevent error of omission, J. Am.

Med. Inform. Assoc. 4 (1997) 364–375.

[26] K.G. Shojania, D. Yokoe, R. Platt, et al., Reducing vancomycin use utilizing a computer guideline, J. Am. Med. Inform.

Assoc. 5 (1998) 554–562.

[27] M.A. Fischer, D.H. Solomon, J.M. Teich, et al., Conversion from intravenous to oral medications, Arch. Int. Med. 163 (2003) 2585–2589.

[28] B. Bogugki, B.R. Jacobs, J. Hingle, et al., Computerized reminders reduce the use of medications during shortages, J. Am. Med. Inform. Assoc. 11 (2004) 278–280.[29] T. Hulgan, S.T. Rosenbloom, F. Hargrove, et al., Oral quinolones in hospitalized patients: an evaluation of a computerized decision support intervention, J. Int. Med. 256 (2004) 349–357.

[30] W.L. Galanter, A. Polikaitis, R.J. DiDomenico, A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry, J. Am. Med. Inform.

Assoc. 11 (2004) 270–277.

[31] P.R. Dexter, S. Perkins, M. Overhage, et al., A computerized reminder system to increase the use of preventive care for hospitalized patients, New Engl. J. Med. 345 (2001) 965–970.

[32] N.A. Ali, H.S. Mekhjian, P.L. Kuehn, et al., Speciﬁcity of computerized physician order entry has a signiﬁcant effect on the efﬁciency of workﬂow for critically ill patients, Crit.

Care Med. 33 (2005) 110–114.

[33] J.F. Peterson, G.J. Kuperman, C. Shek, et al., Guided prescription of psychotropic medications for geriatric inpatients, Arch. Int. Med. 165 (2005) 802–807.

[34] J. Manzo, M.J. Sinnett, F. Sosnowski, et al., Case study:

challenges, successes and lessons learned from implementing computerized physician order entry (CPOE) at two distinct health systems: implication of CPOE on pharmacy and medication-use process, Hosp. Pharm. 40 (2005) 420–429.

[35] J. Judge, T.S. Field, M. DeFlorio, et al., Prescribers’ responses to alerts during medication ordering in long term care setting, J. Am. Med. Inform. Assoc. 13 (2006) 385–390.

[36] D.J. Chisolm, A.S. McAlearney, S. Veneris, et al., The role of computerized order sets in pediatric inpatient asthma treatment, Pediatr. Allergy Immu. 17 (2006) 199–206.

[37] A. Ozdas, T. Speroff, R. Waitman, et al., Integrating “best of care” protocols into clinicians’ workﬂow via care provider order entry: impact on quality-of-care indications for acute myocardial infarction, J. Am. Med. Inform. Assoc. 13 (2006) 188–196.

[38] P.V. Asaro, A.L. Sheldahl, D.M. Char, Embedded guideline information without patient speciﬁcity in a commercial emergency department computerized order-entry system, Acad. Emerg. Med. 13 (2006) 452–458.

[39] A.S. McAlearney, D. Chisolm, S. Veneris, et al., Utilization of evidence-based computerized order sets in pediatrics, Int. J.

Med. Inform. 75 (2006) 501–512.

[40] P.R. Dexter, S.M. Perkins, K.S. Maharry, et al., Inpatient computer-based standing orders versusphysician reminders to increase inﬂuenza and pneumococcal vaccination rates, JAMA 292 (2004) 2366–2371.

[41] G. Del Fiol, B.H. Rocha, P. Nohama, Design, implementation and evaluation of a clinical decision support system to prevent adverse drug events, Stud. Health Technol. Inform.

77 (2000) 740–744.

[42] P.G. Nightingale, D. Adu, N.T. Richards, et al., Implementation of rules based computerized bedside prescribing and administration: intervention study, BMJ 320 (2000) 750–753.

[43] T.C. Hsieh, G.J. Kuperman, T. Jaggi, et al., Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry, J. Am. Med. Inform.

Assoc. 11 (2004) 482–491.

[44] D. Mitchell, J. Usher, S. Gray, et al., Evaluation and audit of a pilot of electronic prescribing and drug administration, J.

Inform. Technol. Healthc. 2 (2004) 19–29.

[45] M. Weiner, T. Gress, D.R. Thiemann, et al., Contrasting views of physician and nurses about an inpatient computer-based provider order-entry system, J. Am. Med. Inform. Assoc. 6 (1999) 234–244.

[46] D.A. Thompson, L. Duling, G.C. Holzmueller, et al., Computerized physician order entry, a factor in medication errors: descriptive analysis of events in the intensive care international journal of medical informatics 7 7(2008)365–376 375 unit safety reporting system, J. Clin. Out. Manag. 12 (2005) 407–412.

[47] J. Horsky, G.J. Kuperman, V.L. Patel, Comprehensive analysis of a medication dosing error related to CPOE, J. Am. Med.

Inform. Assoc. 12 (2005) 377–382.

[48] A. Mirco, L. Campos, F. Falcao, et al., Medication error in an internal medicine department. Evaluation of a computerized prescription system, Pharm World Sci. 27 (2005) 351– 352.

[49] S. Eslami, A. Abu-Hanna, N.F. de Keizer, E. de Jonge, Error associated with applying decision support by suggesting default doses for aminoglycosides, Drug Safety 29 (2006) 803–809.

[50] D.W. Bates, L.L. Leape, D.J. Cullen, et al., Effect of computerized physician order entry and a team intervention on prevention of serious medication errors, JAMA 280 (1998) 1311–1316.

[51] R.S. Evans, S.L. Pestotnik, D.C. Classen, et al., A computer-assisted management program for antibiotics and other antiinfective agents, New Engl. J. Med. 338 (1998) 232–238.

[52] D.W. Bates, J.M. Teich, J. Lee, et al., The impact of computrized physician order entry on medication error prevention, J. Am. Med. Inform. Assoc. 6 (1999) 313–321.

[53] R.S. Evans, S.L. Pestotnik, D.C. Classen, et al., Evaluation of computer-assisted antibiotic-dose monitor, Ann. Pharmaco.

(1999) 1026–1031.

[54] C.J. Mullett, R.S. Evans, J.C. Christenson, et al., Development and impact of a computerized pediatric antiinfective decision support program, Pediatrics 108 (2001) E75.

[55] C.A. Igboechi, C.L. Ng, C.S. Yang, et al., Impact of computerized prescriber order entry on medication errors at an acute tertiary care hospital, Hosp Pharm. 38 (2003), pp.

277–231.

[56] L. Cordero, L. Kuehn, R.R. Kumar, et al., Impact of computerized physician order entry on clinical practice in a newborn intensive care unit, J. Perinatol. 24 (2004) 88–93.

[57] M. Krampera, F. Venturini, F. Benedetti, et al., Computer-based drug management in a bone marrow transplant unit: a suitable tool for multiple prescriptions even in critical condition, Br. J. Hematol. 125 (2004) 50–57.

[58] A.L. Potts, F.E. Barr, D.F. Gregory, et al., Computerized physician order entry and medication errors in a pediatric critical care unit, Pediatrics 113 (2004) 59–63.

[59] R. Taylor, J. Manzo, M. Sinnett, Quantifying value for physician order entry systems: a balance of cost and quality, Healthc. Financ. Manage. 56 (2002) 44–48.

[60] K.E. Bizovi, B.E. Beckley, M.C. McDade, et al., The effect of computer-assisted prescription writing on emergency department prescribing errors, Acad. Emerg. Med. 9 (2002) 1168–1175.

[61] H.S. Mekhjian, R.R. Kumar, L. Kuehn, et al., Immediate beneﬁts realized following implementation of physician order entry at an academic medical center, J. Am. Med.

Inform. Assoc. 9 (2002) 529–539.

[62] W.J. King, N. Paice, J. Rangrej, et al., The effect of computized physician order entry on medication errors and adverse drug events in pediatric inpatients, Pediatrics 112 (2003) 506–509.

[63] W.L. Galanter, R.J. Didomenico, A. Polikaitis, A trial of automated decision support alerts for contraindicated medications using computerized physician order entry, J.

Am. Med. Inform. Assoc. 12 (2005) 269–274.

[64] R. Shulman, M. Singer, J. Goldstone, G. Bellingan, Medication errors: a prospective cohort study of hand-written and computerized physician order entry in the intensive care unit, Crit. Care 9 (2005), p. R516–21.[65] J.S. Upperman, P. Staley, K. Friend, et al., The impact of hospital-wide computerized physician order entry on medical errors in a pediatric hospital, J. Pediatr. Surg. 40 (2005) 50–59.

[66] G.R. Kim, A.R. Chen, R.J. Arceci, et al., Error reduction in pediatric chemotherapy, Arch. Pediatr. Adolesc. Med. 160 (2006) 495–498.

[67] R. Koppel, J.P. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in facilitating medication errors, JAMA 293 (2005) 1197–1203.

[68] J.R. Nebeker, J.M. Hoffman, C.R. Weir, et al., High rates of adverse drug events in a highly computerized hospital, Arch. Int. Med. 165 (2005) 1111–1116.

[69] Y.Y. Han, J.A. Carcillo, S.T. Venkataraman, et al., Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system, Pediatrics 116 (2006) 1506–1512.

[70] M.A. Del Beccaro, H.E. Jeffries, M.A. Eisenberg, E.D. Harry, Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit, Pediatrics 118 (2006) 290–295.

[71] W.M. Tierney, M.E. Miller, J.M. Overhage, et al., Physician inpatient order writing on microcomputer workstations:

effects on resource utilization, J. Am. Med. Inform. Assoc.

269 (1993) 379–383.

[72] C.H. Cheng, M.K. Goldstein, E. Geller, et al., The effects of CPOE on ICU workﬂow: an observational study, in:

Proceeding of AMIA, 2003, pp. 150–154.

[73] S.T. Rosenbloom, A.J. Geissbuhler, W.D. Dupont, et al., Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care, J.

Am. Med. Inform. Assoc. 12 (2005) 458–473.

[74] R. Kaushal, A.K. Jha, C. Franz, et al., Return on investment for a computerized physician order entry system, J. Am.

Med. Inform. Assoc. 13 (2006) 261–266.

[75] M.C. Beuscart-Zephir, S. Pelayo, F. Anceaux, et al., Impact of CPOE on doctor–nurse cooperation for the medication ordering and administration process, Int. J. Med. Inform. 74 (2005) 629–641.

[76] S.A. Abookire, J.M. Teich, H. Sandige, et al., Improving allergy alerting in a computerized physician order entry system, in:

Proceedings of the AMIA, 2000, pp. 2–6.

[77] D.W. Bates, D.L. Boyle, J.M. Teich, Impact of computerized physician order entry on physician time, in: Proceedings of the Annual Symposium Computer Application in Medical Care, 1994, p. 996.

[78] K. Shu, D. Boyle, C. Spurr, et al., Comparison of time spent writing orders on paper with computerized physician order entry, in: Proceedings of the Medical Informatics, 2001, pp.

1207–1211.

[79] G.A. Banet, D.B. Jeffe, J.A. Williams, P.V. Asaro, Effects of implementing computerized practitioner order entry and nursing documentation on nursing workﬂow in an emergency department, J. Healthc. Inform. Manage. 20 (2006) 45–54.

[80] A.L. Costa, M.M.B. de Oliveira, O. Machado Rde, An information system for drug prescription and distribution in public hospital, Int. J. Med. Inform. 73 (2004) 371–381.

[81] S.T. Rosenbloom, D. Talbert, D. Aronsky, Clinicians’ perceptions of clinical decision support integrated into computerized provider order entry, Int. J. Med. Inform. 73 (2004) 433–441.

[82] H. Murff, J. Kannry, Physician satisfaction with two order entry system, J. Am. Med. Inform. Assoc. 8 (2001) 499– 509.

[83] F. Lee, J.M. Teich, C.D. Spurr, D.W. Bates, Implementation of physician order entry: user satisfaction and self-reported usage patterns, J. Am. Med. Inform. Assoc. 3 (1996) 42–55. 376 international journal of medical informatics 7 7(2008)365–376 [84] D. Bomba, T. Land, The feasibility of implementing an electronic prescribing decision support system: a case study of an Australian public hospital, Aust. Health Rev. 30 (2006) 380–388.

[85] E. Oren, E.R. Shaffer, B.J. Guglielmo, Impact of emerging technologies on medication errors and adverse drug events, Am. J. Health-Syst. Pharm. 60 (2003) 1447–1458.

[86] G.I. Kuperman, R.F. Gibson, Computer physician order entry:

beneﬁts, costs, and issues, Ann. Int. Med. 139 (2003) 31–39.

[87] B. Chaudhry, J. Wang, S. Wu, et al., Systematic review: impact of health information technology on quality, efﬁciency, and cost of medical care, Ann. Int. Med. 144 (2006) E12–E22.

[88] H. Van der Sijs, J. Aarts, A. Vulto, et al., Overriding of drug safety alerts in computerized physician order entry, J. Am.

Med. Inform. Assoc. 13 (2006) 138–147.[89] A.X. Grag, N.K.J. Adhikari, H. McDonald, et al., Effect of computerized clinical decision support systems on practitioner performance and patient outcomes, JAMA 293 (2005) 1223–1238.

[90] J. Niinimaki, J. Forsstrom, Approaches for certiﬁcation of electronic prescription software, Int. J. Med. Inform. 47 (1997) 175–182.

[91] B. Kaplan, Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism, Int. J. Med. Inform. 64 (2001) 39–56.

[92] A. Ammenwerth, J. Talmon, J.S. Ash, et al., Impact of CPOE on mortality rates—contradictory ﬁndings, important messages, Method. Inform. Med. 45 (2006) 586– 594.