Knowledge Activity: Applied Data Analytics III The activity Recently, the local paper did an article on the nation's high rate of cesarean section deliveries. The journalist featured interviews with S

Received 3 March 2018 Revised 4 April 2018 Accepted 6 April 2018 For numbered affiliations see end of article.

Correspondence to Dr Joshua S Ng-Kamstra; josh. ng@ mail. utoronto. ca Abs TrACT Introduction The Lancet Commission on Global Surgery proposed the perioperative mortality rate (POMR) as one of the six key indicators of the strength of a country’s surgical system. Despite its widespread use in high-income settings, few studies have described procedure-specific POMR across low-income and middle-income countries (LMICs). We aimed to estimate POMR across a wide range of surgical procedures in LMICs. We also describe how POMR is defined and reported in the LMIC literature to provide recommendations for future monitoring in resource-constrained settings.

Methods W e did a systematic review of studies from LMICs published from 2009 to 2014 reporting POMR for any surgical procedure. We extracted select variables in duplicate from each included study and pooled estimates of POMR by type of procedure using random-effects meta- analysis of proportions and the Freeman-Tukey double arcsine transformation to stabilise variances.

r esults W e included 985 studies conducted across 83 LMICs, covering 191 types of surgical procedures performed on 1 020 869 patients. Pooled POMR ranged from less than 0.1% for appendectomy, cholecystectomy and caesarean delivery to 20%–27% for typhoid intestinal perforation, intracranial haemorrhage and operative head injury. We found no consistent associations between procedure-specific POMR and Human Development Index (HDI) or income-group apart from emergency peripartum hysterectomy POMR, which appeared higher in low-income countries. Inpatient mortality was the most commonly used definition, though only 46.2% of studies explicitly defined the time frame during which deaths accrued.

Conclusions Efforts to improve access to surgical care in LMICs should be accompanied by investment in improving the quality and safety of care. To improve the usefulness of POMR as a safety benchmark, standard reporting items should be included with any POMR estimate. Choosing a basket of procedures for which POMR is tracked may offer institutions and countries the standardisation required to meaningfully compare surgical outcomes across contexts and improve population health outcomes. InTrodu CTI on Over 260 million surgical procedures are per formed each year globally, but a further 143 million procedures are required to meet essential surgical needs in low-income and middle-income countries (LMICs). 1 In addi- tion to increasing surgical access in LMICs, efforts should also focus on improving the quality and safety of surgical care and reducing the risk of death in the periopera- tive period. Key questions What is already known?

► Previous systematic reviews of anaesthetic mortality and mortality in specific surgical populations have shown decreasing mortality trends over time and differences by world region.

► Geographical differences have similarly been report- ed in cohort studies such as the GlobalSurg I study, the European Surgical Outcomes Study and the African Surgical Outcomes Study. What are the new findings?

► This is the first systematic review to attempt broad baseline estimation of perioperative mortality rate (POMR) across procedures and describe how low-in- come and middle-income countries (LMICs) authors define POMR.

► We show here that POMR varies widely by procedure or diagnosis; further, we show significant variation in how POMR is reported, limiting comparisons across contexts. What do the new findings imply?

► POMR is widely used and reported in all contexts; to promote its utility as a standardised surgical safety indicator, greater specificity in the types of proce- dures assessed and the way in which data are col- lected, risk adjusted and reported is required. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 2 BMJ Global Health The perioperative mortality rate (POMR), defined as the number of deaths during or after surgery divided by the number of procedures performed, has been cham- pioned in the literature as a useful indicator to measure surgical safety at an institutional and national level. 2–4 The Lancet Commission on Global Surgery recom- mended the national POMR as one of six key indicators to measure the strength of a country’s surgical system. 2 However, despite its demonstrated utility in high-in- come settings, few studies have described POMR across LMICs and little research exists on how POMR is used and defined in resource-constrained settings. Bainbridge et al showed decreasing perioperative and anaesthet- ic-related mortality rates in LMICs since 1970, although procedure-specific rates were not studied or reported.

5 Uribe-Leitz and others quantified mortality after three common procedures in LMICs: caesarean delivery, appendectomies and groin hernia repair. 6 A prospective cohort study across 58 countries found that emergency abdominal surgery POMR was three times higher in low-Human Development Index (HDI) compared with high-HDI countries. Most recently, the African Surgical Outcomes Study found that despite being younger, with a lower surgical risk profile and undergoing less complex surgeries, patients in Africa are twice as likely to die after surgery when compared with outcomes at the global level. 7 Nonetheless, to our knowledge, no study has reported procedure-specific POMR across a wide range of surgical conditions across LMICs. To address this gap, we undertook a systematic review of the perioperative mortality literature for all surgical procedures in LMICs. We reviewed all studies on POMR published in LMICs over a 6-year period between 2009 and 2014. This covers the period roughly between the publication of the WHO Guidelines for Safe Surgery 2009 and The Lancet Commission on Global Surgery, providing a modern account of the POMR literature. 2 8 This study had several aims: (1) to describe POMR across a wide range of surgical procedures in LMICs, (2) to deter - mine whether these rates vary across contexts and (3) to determine how POMR is defined, reported and used in the LMIC literature, including how risk adjustment is under - taken. Finally, we provide recommendations for improving POMR reporting in resource-constrained settings.

Me TH ods s earch strategy and selection criteria The original study protocol was published alongside a preliminary abstract in 2015, 9 the final version of which is available in the online Supplementary mate- rial protocol. We conducted a systematic review and meta-analysis following Preferred Reporting Items for Systematic Reviews and Meta-Analyses and Meta-analysis Of Observational Studies in Epidemiology guidelines. 10 11 We included published studies primarily reporting facil- ity-based outcomes or mortality for patients who under - went surgery in LMICs (defined according to 2013 World Bank Income Groups). 12 All study designs (descriptive, case–control, cohort or trial) were eligible for inclusion. We included full-text articles published in English between 1 January 2009 and 31 December 2014. Final searches were performed on 10 January 2015. The perioperative period was defined as the period from entry into the operating theatre to either discharge or 30 days following a surgical procedure, whichever was later . However, studies explicitly discussing surgery-related mortality, but whose shortest reporting interval was 31–90 days after surger y were included.

Surgery was defined as any procedure performed in an operating theatre. A list of excluded procedure types is available in the attached review protocol. Only studies providing raw mortality data were eligible for inclusion; those in which the numerator (deaths) or denominator (patients or procedures) were estimated or modelled were excluded. We did not impose a large sample size requirement that would exclude literature published in smaller centres with lower surgical volumes. We searched PubMed, EMBASE, LILACS, Web of Science, African Index Medicus and the WHO Global Health Library. Search terms for all databases were devel- oped in consultation with a medical librarian. Variants of ‘surgery’, ‘operation’, ‘anaesthesia’, ‘intraoperative’, ‘perioperative’, ‘postoperative’ and ‘mortality’ were included in all searches. In addition, we also hand-searched the references of recently published reviews of specific procedures. 6 13–16 Stand-alone abstracts and unpublished studies were excluded from the review. Full inclusion and exclusion criteria, as well as database-specific searches, are provided in the attached review protocol. d ata extraction, outcome definition and procedure classification Titles and abstracts were reviewed independently in duplicate to evaluate for inclusion. Eligibility assessment based on full-text reviews and data abstraction were done by two clinicians. Selected variables including the surgical procedure or diagnosis under consideration, the periop- erative mortality rate, case urgency and the definition of POMR were extracted in duplicate. Disagreements in data extraction were resolved by a single physician coder (JNK). A data dictionary describing all variables, codes, assumptions and simplifications is provided in the online Supplementary data dictionary. The primary outcome of interest was the POMR, and secondary outcomes were the definition of perioperative mortality and the reporting and adjustment for selected preoperative risk factors. When the time frame relative to surgery during which deaths accrued was not clearly defined, it was assumed to be in-hospital mortality. If mortality was reported at multiple postoperative inter - vals, the longest (up to 30 days) was used.

T o describe each patient population accurately and consistently, coding was performed iteratively. First, a clinician identified the broadest procedure or diagnosis group in each study and assigned it a code. A list of such codes was developed and revised by a second clinician. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 3 BMJ Global Health Within each code, we identified studies performed in high-risk populations (eg, restricted to patients with comorbidities such as renal dysfunction or HIV). When possible, we also stratified patients by case urgency.

e conomic variables and risk of bias We obtained country lending classification data from the World Bank and Human Development Index data from the UN Development Programme. 17 18 Where data were unavailable for a given country for a given year, the closest available year to the midpoint year of data collection was used. Two potential sources of bias were assessed: selec- tion bias resulting from failure to report on all consec- utive cases and detection bias resulting from failure to provide complete follow-up data. The data collected were analysed as case-series outcomes (mortality rates), regardless of the underlying study design. s tatistical analysis In order to summarise POMR across procedures, we pooled estimates using random-effects meta-analysis of proportions and the Freeman-Tukey double arcsine transformation to stabilise variances.

19 This procedure allows for the inclusion of studies with a zero event rate.

Meta-analyses were weighted by the inverse variance of the transformed estimates, giving more weight to the more precise rates in the pooled estimate. We used the metaprop command in Stata/IC V.13. 19 In order to deter - mine whether there were differences in procedure-spe- cific POMR across study-country income groups and HDI categories, we used the non-parametric Kruskal-Wallis equality-of-populations test. 20 The role of the funder This study was funded in part by Boston Children’s Hospital, which had no role in the design, conduct, anal- ysis or writing of this study and did not influence the deci- sion to publish.

r esul T s After the removal of duplicate citations, we screened the titles and abstracts of 7701 unique citations. Of these, we requested 1595 full-text articles for further review. A total of 985 articles met our inclusion criteria (figure 1).

These studies were conducted across 83 LMICs. The country where the most POMR literature was published was Brazil (145 articles), followed by Nigeria (121), China (111), Pakistan (107) India (87), Turkey (65) and Iran (62) (figure 2; online Supplementary appendix 1 table S1). These studies covered a total of 191 different procedure or diagnostic groups (‘procedures’) in 13 surgical specialties and ranged from small case series of five patients to expansive registries with 152 110 surgical patients (median, 86 patients, IQR 36–234,S upplemen- tary appendix table S2). In total, the surgical outcomes of 1 020 869 patients were included (figure 2 ). Most studies were conducted in urban environments (n=884, 89.7%) and in academic centres (n=821, 83.4%). The majority of studies were descriptive (n=711, 72.2%) and retrospec- tive (n=685, 69.5%, table 1). Primary data are available in the online Supplementary appendix 2.

Across the 191 procedures identified, the most commonly reported were caesarean delivery (55 studies) followed by coronary artery bypass grafts (49 studies), Figure 1. Flow diagram. Pr eferred Reporting Items for Systematic Reviews and Meta-Analyses flow di\ agram. HIC, high- income country; ICU, intensive care unit. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 4 BMJ Global Health emergency peripartum hysterectomies (39 studies) and cardiac valve procedures (35 studies) (table 2). A total of 77 procedures identified were reported in only one study (online Supplementary table s3). Among proce- dures described in at least four studies (n=67), most demonstrated significant between-study heterogeneity in reported POMR (I 2>50%, n=55).

While acknowledging the significant between-study heterogeneity within procedures resulting from differ - ences in methodology, outcome definition and patient- level risk, we elected to create pooled POMR estimates as an approximate baseline to inform future research.

Procedure-specific POMR pooled by inverse-variance- weighted random-effects meta-analyses for the 34 most commonly reported procedures (ie, reported in at least eight studies) are shown in table 2. Given the high between-study heterogeneity, we also include the median study-level POMR for each procedure and the range across which studies varied.

POMR varied considerably between procedures. For example, pooled POMR ranged from less than 0.1% for cholecystectomy to 20%–27% for typhoid intestinal perforation, intracranial haemorrhage and operative head injury (table 2). While caesarean delivery POMR was a mere 0.05% (95% CI 0 to 0.13), small outlier studies reported rates of up to 16%. After emergency peripartum hysterectomy, however, the risk of dying was 7.81% (95% CI 5.81 to 10.04). Similarly, the risk of dying after appendectomy was 0.01% (95% CI 0 to 0.19) whereas, after surgery for a perforated hollow viscus (excluding typhoid), POMR was 11.85% (95% CI 8.35 to 15.83). Paediatric surgical procedures demonstrated alarmingly high mortality rates: surgery for oesophageal atresia/tracheo-oesophageal fistula carried a 24.41% mortality risk (95% CI 6.76 to 48.04), Hirschsprung’ s disease 10.65% (95% CI 0.42% to 29.11%), intestinal atresias 30.95% (95% CI 18.71 to 44.53) and gastroschisis 29.68 (95% CI 10.75 to 53.14). Mortality rates across all 191 identified procedures are shown in the online Supplementary table s3. As a sensi- tivity analysis, we excluded all studies performed in high- risk populations (such as studies restricted to patients with specific comorbidities); results are shown in the online Supplementary table s4. There was a little change in pooled POMR estimates after excluding these studies. We looked at whether procedure-specific POMR varied by study country income group or categories of HDI.

For this analysis, procedure-specific POMR estimates for high-income countries were identified through a purpo- sive search of the literature. We found a statistically signif- icant difference in POMR for emergency peripartum hysterectomy across income groups (p<0.05). This rela- tionship was not statistically significant for any other procedure types including caesarean section, appendec- tomy and colon resection. We found no consistent associ- ation between procedure-specific POMR and HDI. Over half of the studies (n=530, 53.8%, table 3) did not provide a clear POMR definition (ie, of the timeframe during which deaths accrued). In the other studies, a variety of timeframes for calculating POMR were employed. About 20% of studies reported 30-day mortality and a smaller number referred to variants thereof (such as ‘mortality within 30 days of surger y or during the index hospital stay’). Some obstetric surgery studies reported mortality at 42 days after surger y (n=6). Most studies used the number of patients undergoing surgery, rather than the number of procedures performed, as the denominator of POMR (n=969, 98.4%). We also found that studies performed in upper-middle income countries were more likely to provide a clear definition of the POMR described. Risk-adjustment methodology varied widely (table 4).

Some studies reported gross mortality rates without risk adjustment; by contrast, authors from Brazil and China used detailed registry data to develop sophisticated Table 1 Hospital and study descriptors Hospital descriptors, n(%) Academic hospital 821 (83.4%) District or community hospital 67 (6.8%) Mixed hospital types 74 (7.5%) Other 23 (2.4%) Hospital location Urban 884 (89.7%) Rural 25 (2.5%) Mixed locations 76 (7.7%) Study design Retrospective 685 (69.5%) Prospective 266 (27.0%) Ambispective 34 (3.5%) Audit 711 (72.2%) Non-randomised cohort 225 (22.8%) Case–control 24 (2.4%) Randomised controlled  trial 25 (2.5%) Urgency Planned 292 (29.6%) Emergent 415 (42.1%) Mixed 338 (34.3%) ‘Other’ hospital types include facilities run by Médecins Sans \ Frontières. ‘Planned’ and ‘Emergent’ rows include studies providing mortality stratified on urgency and therefore totals exceed 100%.

Figure 2. Distribution of the perioperative mortality rate (POMR) literatur e in low-income and middle-income countries. The number of papers presenting POMR data for each country. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 5 BMJ Global Health Table 2 Inverse-variance aggr egated perioperative mortality rate (POMR) across the 34 most commonly reported procedures or diagnoses described in the low-income and middle-income countries surgical outcomes literature, 2009–2014 Diagnostic or procedure code Description Number of studiesTotal number of deaths Total denominator Median POMR (range, %) Inverse-variance aggregated POMR (%, 95 CI)I-Squared (%) CAES Caesarean section 55 8053 66 5010.12 (0–15.61) 0.05 (0 to 0.13) 93.52 CABG Coronary artery bypass graft 49 58071 23 5133.60 (0–52.81) 4.38 (3.47 to 5.37) 97.32 EPH Emergency peripartum hysterectomy 39 196 2245 10.30 (0–31.03) 7.81 (5.81 to 10.04)64.27 VALVE Cardiac valve procedures 35 119730 3654.29 (0–15.07) 4.17 (3.05 to 5.45) 93.61 CARD Cardiac surgery, not otherwise specified 31 78471 23 0604.79 (0–23.08) 4.96 (3.81 to 6.25) 97.80 COLRES Colon resection, excluding resection for volvulus 27 35612 6361.86 (0–33.64) 2.83 (1.62 to 4.31) 92.35 APPY Appendicitis 2328 5237 0 (0–2.78) 0.01 (0 to 0.19)37.74 LUNGRES Pulmonary resection, excluding resection for tuberculosis 2396 5630 1.14 (0–13.76) 1.3 (0.48 to 2.41) 84.46 PERF Perforated hollow viscus, excluding perforations secondary to salmonella typhi infection 22 2642427 11.85 (0–40.00) 11.85 (8.35 to 15.83) 84.73 LIVRES Hepatic resection 2099 7243 1.38 (0–13.16) 1.04 (0.32 to 2.02) 77.08 MULTI Multispecialty patient population, usually institution-level surgical mortality 19 16451 77 2831.06 (0.16–7.36) 1.29 (0.77 to 1.94) 99.02 PCARD Paediatric cardiac procedures, excluding complex congenital heart disease and valve-specific procedures 19 538 6618 7.14 (0–24.24) 6.76 (4.99 to 8.75)81.36 RIM Resection of intracranial mass 19 20814 0 (0–5.88) 1.29 (0.41 to 2.51)0.00 GASTCA Gastric cancer 18 2678250 2.71 (0–18.97) 3.72 (1.92 to 6.01) 94.48 INGHERN Inguinal hernia 177111 1960 (0–9.73) 0.38 (0 to 1.22) 93.48 LAPAR Laparotomy, not meeting other abdominal surgery codes. Includes laparotomy performed for trauma 17 354 3064 11.11 (4.94–42.11) 12.53 (9.39 to 16.04)83.70 PAED Paediatric surgical procedures, not otherwise specified 17 35554 3893.57 (0–62.22) 6.16 (4.06 to 8.64) 98.64 CHOLE Cholecystectomy 1546088 0 (0–0.15) 0 (0 to 0)0.00 UTRUP Uterine rupture 1587 1169 8.22 (0–17.50) 7.36 (4.42 to 10.88) 71.74 ESOCA Esophageal carcinoma 13 1341802 5.81 (0–24.00) 5.4 (2.28 to 9.54) 89.16 CCHD Complex congenital heart disease 12 93596 14.65 (0–61.54) 14.94 (7.03 to 24.75) 83.32 Continued on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 6 BMJ Global Health Diagnostic or procedure code DescriptionNumber of studiesTotal number of deaths Total denominator Median POMR (range, %) Inverse-variance aggregated POMR (%, 95 CI)I-Squared (%) BOBS Bowel obstruction 10 1371158 8.79 (2.27–38.10) 12.32 (6.77 to 19.15) 89.13 ICH Intracranial haemorrhage 10 2241011 25.48 (3.77–62.22) 24.47 (15.88 to 34.16) 89.05 LIVTRAUM Hepatic trauma 10 133909 17.01 (6.80–61.11) 15.84 (10.31 to 22.16) 76.86 WHIP Whipple pancreaticoduodenectomy 10 902065 2.84 (0–9.92) 2.94 (1.61 to 4.57) 52.48 MIS Minimally invasive surgery, not otherwise specified 92 1314 0 (0–4.17) 0 (0 to 0.1)0.00 RECTAL Rectal resection 96 1032 0 (0–5.88) 0.07 (0 to 0.92)50.09 SPINE Spine surgery, excluding trauma 9 11518 0 (0–8.96) 0.77 (0 to 3.8)67.18 TIP Typhoid intestinal perforation 9134 662 20.73 (4.55–33.33) 20.09 (14.36 to 26.48) 71.46 AABDO Acute abdomen but not meeting other abdominal surgery codes 8228 2877 10.42 (4.90–34.88) 11.2 (7.42 to 15.62) 86.27 ACHI All-comer head injury 8377 1390 23.08 (10.00–54.58) 27.2 (14.98 to 41.39) 96.32 BILD Bile duct procedures, excluding Whipple procedure 851 714 2.30 (0–21.54) 4.08 (0.1 to 11.63) 91.49 INTUSS Intussusception 843 355 3.66 (0–33.70) 4.8 (0.03 to 14.28) 88.09 TA D Thoracic aortic disease 8775 4203 8.66 (0–20.30) 9.5 (3.96 to 16.74) 91.49 Table 2 Continued on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 7 BMJ Global Health population-specific scores to determine risk after cardiac surgery. Most studies reported median patient age (95.0%) and case urgency (74.1%), but only 7.5% reported the ASA status (American Society of Anesthe- siologists Physical Status Classification). About a third of studies (n=331) reported a clinical risk score, but only 14.3% performed risk adjustment or stratification based on such scores. A summary of the scores used by surgical specialty is included in the online Supplemen- tary appendix 1 table S5.

dI s C uss I on In LMICs, the POMR literature is as diverse as the insti- tutions and countries that produce it. It spans all surgical specialties and a wide variety of procedures and diagnoses both common and rare. To our knowledge, this is the first systematic review to report procedure-specific POMR across a wide range of surgical conditions in LMICs. This review included data from 985 studies conducted in 83 LMICs and covering 191 types of surgical procedures performed on a population of 1 020 869 patients.

POMR is used for many purposes. In the studies included here, authors used POMR data to argue for increased critical care resources, 21 quantify the particular surgical risk for the HIV-positive population, 22 assess the impact of delay in reaching care on outcomes, 23 raise alarm over high mortality rates in the paediatric popu- lation 24 25 and establish the relative safety of tradition- ally high-risk procedures in select LMIC environments, among other aims. 26 The utility of this metric at the insti- tution level is undeniable; with clear outcome definitions, a well-defined population, and robust analysis, perioper - ative mortality rates can be used to monitor and improve patient safety. To demonstrate patterns in mortality rates beyond the level of the institution requires some stan- dardisation of definitions, methods of data capture and patient-level risk assessment. The studies included here were too heterogeneous on these fronts to demonstrate stable relationships between POMR and macroeconomic variables such as HDI or income groups. By contrast, the GlobalSurg group demonstrated a clear inverse relationship between POMR and HDI for emergency abdominal surgery, and a previous systematic review by Bainbridge et al, showed similar findings for all-comer anaesthesia-related mortality. 5 27 Assuming this relationship is true, several potential reasons may explain why this study did not demonstrate it. First, unlike Bain- bridge et al, we report procedure-specific POMR. The small number of studies within each procedure group decreased the power to detect such relationships and prevented us from conducting meta-regression analyses.

Second, heterogeneity in POMR definitions across studies introduces significant noise. The impact of differences in POMR definition can be dramatic; in the GlobalSurg analysis, 24 hours mortality was 1.6%, underestimating all-cause 30-day mortality (5.4%) by 70%. 27 Similarly, in a New Zealand data set, in-hospital mortality under - estimated 30-day mortality by about one-third.

28 Third, most studies included here were retrospective, increasing the risk of information bias or incomplete reporting of mortality data, which may vary by income level. This information bias can be significant. A study in Uganda compared mortality from retrospective chart reviews, surgical logbooks and prospective data collection. Of the 16 deaths identified prospectively, retrospective chart reviews captured only six. Surgical logbooks performed Table 3 Definitions of perioperative mortality rate Number of papers (%) Numerator Clearly defined 455 (46.2) Inpatient/hospital mortality (assumed for all studies lacking clear definition) 703 (71.4) Inpatient/hospital mortality , within 30 days of pr ocedure 13 (1.3) 30-day mortality 202 (20.5) Mortality within 30 days or same hospitalisation 32 (3.3) 7-day mortality 3 (0.3) Intraoperative mortality 14 (1.4) 24 hours mortality 4 (0.4) Other 14 (1.4) Multiple 24 (2.4) Denominator Number of patients 969 (98.4) Number of pr ocedures 16 (1.6) Most studies did not explicitly state the timeframe during which deaths accrued. For those studies lacking clear time definitions, deaths were assumed to accrue during the index hospitalisation alone. Table 4 Risk factor r eporting and adjustment Number of studies reporting (n, %)Number of studies providing adjustment or stratification (n, %) Patient age 936 (95.0) 145 (14.7) Comorbidities 402 (40.8) 146 (14.8) ASA status 74 (7.5) 33 (3.4) Case urgency 730 (74.1) 693 (70.4) HIV status 45 (4.6) 25 (2.5) Clinical Risk Score 331 (33.6) 141 (14.3) Reporting of case urgency required presentation of the proportion of planned versus emergent cases, or a population consisting exclusively of either planned or emergent cases. The latter group was considered to have reported mortality ‘stratified’ on urgency.

‘Adjustment or stratification’ implies a statistical analysis of\ the risk factor in relation to mortality, or mortality provided for separate strata. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 8 BMJ Global Health better, capturing 99% of procedures and 15 out of the 16 deaths. 29 Fourth, individual patient risk was not assessed in this analysis; it is possible that higher-income countries reported on the outcomes of older or more comorbid surgical patients. Fifth, many studies had small sample sizes with few mortalities, resulting in large variances in estimated POMR. Finally, high-income countries were not included in this analysis, narrowing the economic spectrum across which POMR was assessed. We also note that the strength of the relationship between POMR and economic variables is likely to vary by procedure. For procedures with low baseline risk, it may be more diffi- cult to detect a meaningful difference in mortality by level of development. 27 A clear recommendation to arise from this review is that whenever perioperative mortality data are reported, metadata describing the definition used and the quality of reporting should accompany them (box 1). Because POMR is a proportion (though termed a ‘rate’ in the literature), both the numerator and denominator should be clearly described. The numerator should be described specifically in terms of the time during which deaths accrue, with a preference for all-cause 30 day deaths where possible. Inpatient mortality misses many deaths. 27 28 However, it is easier to collect than 30-day mortality, which requires a concerted effort to contact patients at 30 days following surger y. This challenge is the primary reason why The Lancet Commission on Global Surgery recommended inpatient perioperative mortality as a key surgical systems indicator. 2 However, the ubiquity of cellphone technology and its utility in follow-up for surgical site infections in austere environments make it a promising tool for use in collecting POMR data. 30 Thir - ty-day POMR seems to be a more robust indicator and it is less sensitive to the varying postsurgery discharge prac- tices around the world. Nonetheless, in some contexts, reporting inpatient mortality will be necessary. In this case, the average length of stay should be reported and authors should specify whether outpatient procedures were included in the denominator. Including outpa- tient procedures can deflate POMR, as by definition, the outcome (‘inpatient mortality’) is unlikely to occur in an outpatient population (being limited to intraoperative deaths or deaths in the recovery room). In this review, we found that the denominator was most often defined as the number of surgical patients. Alternatively, the denom- inator can be defined as the number of procedures or number of admissions including a surgical procedure; the impacts on POMR of such subtle changes have been described elsewhere. 28 Furthermore, the population under study should be clearly described, all consecutive patients included and the completeness of follow-up and reasons for any missing data described. As the scope of analysis of POMR expands from the institutional to the national level, so too does the impor - tance of precision in data collection and reporting. The gross national POMR has been advocated by several groups as a global indicator of surgical safety. 2–4 The goal of this indicator is to provide a waypost for the improve- ment of safety in surgical systems. More specifically, it should indicate modifiable operative and postoperative factors that determine mortality, while preoperative factors and data factors are controlled (table 5). Others have argued that while POMR varies with case mix, the operative experience of a country is unlikely to change from year to year; policymakers can therefore monitor POMR over time to assess improvements in surgical safety. This argument relies on two premises: first, that the initial country-level POMR reviewed by a policymaker can be reasonably interpreted to motivate investment in surgical safety, and second, that case mix, defini- tions and quality of reporting remain stable over time to permit interpretation of changes in reported rates.

An effort to collect POMR from ministries of health has been undertaken. 31 While many countries did provide POMR, these data were difficult to interpret, as they varied in definition and case mix. It was not possible to assess whether a country was performing well, or poorly, compared with others based on the gross national POMR data provided. Even when definitions and methods of data capture are held constant, careful reporting of case mix remains important: an analysis by region in Brazil showed higher all-procedure POMR in wealthier regions, Box 1 Critical elements for reporting perioperative mortality rates 1. Define the surgical population under study.

a.

Report all relevant inclusion/exclusion criteria and report if all consecutive patients meeting criteria were included.

b.

If multiple procedures are inc luded, report case mix.

2.

Report stud y design.

a.

Report stud y perspective (retrospective, prospective or ambispective). i.

If retrospective, describe how all procedures/patients were identified and whether surgical logbooks, registries, or elec- tronic medical records were used.

b.

Describe methodolog y for any between-group comparisons.

3.

Sta te the timeframe during which deaths accrued: in-hospital (dur - ing index hospital stay alone), 30-day mortality or other.

a.

If in-hospital mortality is used,i . Report the mean length of hospital stay , standard deviation and range.

ii. Report the proportion of inc luded procedures being per- formed with same-day discharge (‘day surgery’).

4.

Sta te the denominator used: surgical patients, surgical procedures or admissions with a surgical procedure.

5.

Report an y loss to follow-up.

a.

If the outcomes of all surgical procedures cannot be identified.i . Report the proportion of missing da ta.

ii. Report why the da ta are missing.

6.

Report common surgical risk factors inc luding age, comorbidities, functional status, urgency status and ASA.

a.

If a valida ted risk-scoring system is available for the procedure under study, consider using and reporting it.

b.

If risk-adjusted mortality is reported, also report crude rates and clearly describe adjustment methodology. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 9 BMJ Global Health Table 5 Known or potential factors influencing perioperative mortality rates Preoperative factors Operative factors Patient factors ComorbiditiesUrgency Planned Age Emergent Severity and nature of illness Health systems factors Prehospital transportSurgical approach Open Delay to presentation Minimally invasive Appropriate centre for condition Intrinsic procedure risk By specialty By procedure By complexity score Surgeon skill Specialist versus non-specialist surgeon Surgeon versus non-surgeon physician Physician versus non-physician surgeon Trainee versus fully-trained surgeon Inter-surgeon variation Anaesthetic modality General, regional, local Anaesthetist skill Specialist versus non-specialist anaesthetist Anaesthetist versus non- anaesthetist physician Physician versus non-physician provider Trainee versus fully-trained anaesthetist Inter-anaesthetist variation Postoperative factors Data factors Discharge pathway Outpatient surgeryDatabase type Administrative Same-day admission Retrospective collection Inpatient surgery Prospective collection Dedicated surgical outcomes registry Postoperative surveillance for complications Nursing availability and level of trainingRisk adjustment methods Provider: patient ratio Crude measures reported Frequency of physician assessments Risk-adjusted outcomes reported Availability of diagnostic testing Risk-stratified outcomes reported Ability to rescue after complications Availability of preoperative care Definition of POMR Intraoperative deaths Availability of intravenous antibiotics Deaths within 24 hours of sur gery) Availability of blood bank Inpatient deaths Availability of image-guided interventions 30 day deaths Availability of critical care beds >30 day deaths Availability of ventilators Deaths attributable to surgery versus all-cause deaths Availability of dialysis Patients, procedures, or surgical admissions as denominator Availability of cardiac interventions on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 10 BMJ Global Health but when caesarean section was analysed in isolation, wealthier regions had lower POMR. 32 We have shown here that POMR varies by orders of magnitude according to which procedures are being studied. Even within such broad categories as ‘emergency intraperitoneal surgery’ as studied by GlobalSurg, we have demonstrated that mortality rates vary widely according to which specific procedure or diagnosis is under study. Based on these findings, we advocate for greater spec- ificity in the standard definition of POMR to be used by hospitals and countries. A selection of indicator proce- dures might be chosen to cover the lifespan, such as surgery for gastroschisis, caesarean section, colon resec- tion and repair of hip fracture. Each of these procedures is performed at all levels of HDI, is studied widely and has excellent science on how risk adjustment should be performed. The tradeoff encountered by focusing on POMR indicator procedures is wider CIs due to a lower number of events (compared with all-patient nationwide POMR). Again, we argue that wider CIs around indica- tors that are meaningful to policymakers at face value are preferable to narrower CIs around a gross indicator that is agnostic to case mix. Yet, to abandon the impetus to track all postoperative deaths might be short sighted.

Robust surgical registries tracking all procedures have been established in low-income countries. 33 An expan- sion of such efforts to all hospitals would allow for tracking of the outcomes of select global index proce- dures and locally important procedures, in addition to improving nationally representative surgical volume esti- mates. Other recommendations for the use of POMR to advance surgical safety are included in box 2.

While POMR is useful for assessing the safety of opera- tive care, contextualising it within the broader suite of six surgical indicators proposed by The Lancet Commission on Global Surgery is critical. 2 Specifically, juxtaposing surgical volume with POMR allows for an assessment of the quantity of care delivered and the safety thereof.

Consider an individual with a typhoid intestinal perfo- ration: our analysis would estimate a POMR of 20% in LMICs. This is still vastly preferable to the reported 70% mortality with non-operative management. 34 Universal access to the operating room is a dominant determinant of public health, with surgical safety an important secondary determinant of outcome. This was made strikingly clear in the recently published African Surgical Outcomes Study. 7 A sweeping cohort-based study providing complication and mortality data representative at the level of the conti- nent, it showed that surgical patients in Africa have twice the in-hospital POMR of an international cohort despite a favourable risk profile. POMR was particularly elevated after an index complication. However, in the study, hospi- tals were only able to deliver 212 surgical procedures per 100 000 population per annum, well short of the 5000 per 100 000 target set by The Lancet Commission on Global Surgery. While working to improve safety, focus must be maintained on scaling up delivery and bolstering the surgical workforce in low-volume settings. An important caveat to this discussion is that POMR may be less useful for surgical conditions in which patients do not consistently require a trip to the oper - ating theatre. For example, numerous studies reporting deaths after trauma were excluded from this analysis as a specific operative numerator and denominator were not reported. Finally, measurement alone is not enough. 35 While measuring POMR is indeed the first step towards reducing mortality rates, clinicians and policymakers must insist on deploying the resources and best practices required to prevent complications and rescue patients who experience them. This study had limitations. First, given the high between-study heterogeneity within procedure types, the pooled POMR estimates should be interpreted with caution. Second, this review may be subject to publi- cation bias. Studies of particularly high mortality may not have been submitted for publication in the interest of protecting institutional or surgeon reputation. Box 2 r ecommendations for improving surgical safety Clinicians, care providers and hospitals 1. Promote the implementa tion of best practices, such as the WHO guidelines for safe surgery and other procedure-specific or con- text-specific evidence to decrease complications. Develop a local culture of safety with regular quality-of-care discussions.

2.

Develop quality improvement networks across settings to work col - lectively to identify and implement strategies to improve safety and decrease perioperative mortality rate (POMR).  3.

Invest in the technolog y and human resources required for the pro- spective collection and analysis of POMR data.

researchers Globally 1. Choose indica tor procedures that are commonly performed across all settings, have a significant mortality risk, are representative of the burden of surgical disease across the population and have good existing science for risk adjustment.

2.

Determine sample sizes for these procedures required to stably estima te gross and risk-adjusted procedure-specific POMR at the facility and country level.  3.

Stud y these procedures across settings to estimate how differences in data collection methods and the definition of POMR influence estimates.

4.

Conduct global studies to establish na tionally  representative esti- mates of POMR for each indicator procedure.  5.

Advoca te for investment in the technology and human resources required for reliable ongoing collection of POMR data.

Locally 1.

Stud y indicator procedures in depth to develop local solutions to patient safety problems that can be scaled regionally, nationally or globally.

Ministries of Health:  1. Provide facilities with the administrative and financial support re- quired to collect POMR data prospectively.

2.

When higher-than-expected POMR is brought to the attention of the ministry, mobilise additional clinical and financial resources to aug- ment the safety of operative and postoperative care. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 11 BMJ Global Health Conversely, studies of low mortality may not have been deemed worthy of publication. Funnel plots are prob- lematic in meta-analyses of proportions where the proportions are small. 36 We were unsurprised to find that in this group of highly heterogeneous studies, constructing funnel plots, whether conventional funnel plots or those using study size versus log odds, failed to shed light on the file-drawer problem. Further, these results are most representative of mortality at academic medical centres and may not reflect mortality at smaller, more rural sites. Finally, this analysis included only studies published in English over a 6-year period. Some conditions continue to cause high surgical mortality in LMICs, particularly in the paediatric popula- tion. Mortality data are commonly reported in the LMIC surgical literature but the quality of reporting varies widely: more than half of the studies did not provide a clear definition of the outcome. Given that mortality rates differ dramatically by the procedure or diagnosis under study, analysis of mortality rates for a select basket of surgical procedures might add validity to POMR and allow for constructive comparison of outcomes between sites and countries. Author affiliations1Department of Surgery, University of Toronto, Toronto, Ontario, Canada2Program in Global Surgery and Social Change, Harvard Medical School, Boston, Massachusetts, USA 3Department of Plastic and Oral Surgery, Boston Children’s Hospital, Boston, Massachusetts, USA 4Department of Medicine, University of Toronto, Toronto, Ontario, Canada5Division of General and Thoracic Surgery, Seattle Children’s Hospital, Seattle, Washington, USA 6Department of Pediatric General and Thoracic Surgery, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA 7Department of Global Health and Population, Harvard T.H. Chan school of Public Health, Boston, Massachusetts, USA 8Department of Surgery, Sahlgrenska Academy, Institute of Clinical Sciences, University of Gothenburg, Gothenburg, Sweden 9Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada10Division of Emergency Medicine, University of Toronto, Toronto, Ontario, Canada11Skin and Cancer Foundation, Melbourne, Victoria, Australia12Faculty of Medicine, Lund University, Lund, Sweden13Department of Otolaryngology, University of Toronto, Toronto, Ontario, Canada14Department of Surgery, University of Colorado, Denver, Colorado, USA15Imperial College London, London, UK16Department of Surgery, Medical College of Wisconsin, Milwaukee, Wisconsin, USA17Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada18Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA Acknowledgements W e thank The Lancet Global Health Commission on High Quality Health Systems in the SDG era for support in the conduct and pub\ lication of this work and Boston Children’s Hospital for providing research funding.

Contributors JNK, SLMG, MK and JGM conceived the study. JNK wrote the study protocol, reconciled duplicate data and ensured overall consistency in data extraction and drafted the manuscript. JNK, SLMG and MK developed the search strategy, completed searches and screened titles and abstracts. JNK, SA, AA, CF, AN, FYXL, CLP, TF, TM, SR, SS, IHM, AB and LP extracted data. JNK, CA, DL, RRY and MGS analysed the data. JNK, CA and RRY developed the figures and tables. All authors critically reviewed the manuscript and approve of its content.

Funding This stud y was funded in part by Boston Children’s Hospital.

Competing interests MS holds grant funding from the GE F oundation and the Damon Runyon Cancer Research Foundation, which had no bearing on this work. Patient consent Not required.

Pro venance and peer review Not commissioned; externally peer reviewed. d ata sharing statement W e include primary data alongside the manuscript as supplementary material. Additional data are available upon request; please contact the corresponding author. o pen access This is an open access artic le distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http:// crea tivecommons. org/ licenses/ by/ 4. 0/ © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted. Re FeReNC eS 1. W eiser TG, Haynes AB, Molina G, et al. Size and distribution of the global volume of surgery in 2012. Bull World Health Organ 2016;94:201–9.

2.

Meara JG, Leather AJ, Hagander L, et al. Global Surgery 2030:

evidence and solutions for achieving health, welfare, and economic development. Lancet 2015;386:569–624.

3.

W atters DA, Hollands MJ, Gruen RL, et al. Perioperative mortality rate (POMR): a global indicator of access to safe surgery and anaesthesia. World J Surg 2015;39:856–64.

4.

W eiser TG, Makary MA, Haynes AB, et al. Standardised metrics for global surgical surveillance. Lancet 2009;374:1113–7.

5.

Bainbridge D, Martin J, Arango M, et al. Perioperative and anaesthetic-related mortality in developed and developing countries:

a systematic review and meta-analysis. Lancet 2012;380:1075–81.

6.

Uribe-Leitz T , Jaramillo J, Maurer L, et al. Variability in mortality following caesarean delivery, appendectomy, and groin hernia repair in low- income and middle-income countries: a systematic review and analysis of published data. Lancet Glob Health 2016;4:e165–74.

7.

Madiba TE, Biccar d B. The African surgical outcomes study: a 7-day prospective observational cohort study. S Afr J Surg 2017;55:75.

8.

W orld Health Organization. WHO guidelines for safe surgery 2009:

Safe Surgery Saves Lives. Geneva: WHO, 2009.

9.

Ng-Kamstra JS, Gr eenberg SL, Kotagal M, et al. Use and definitions of perioperative mortality rates in low-income and middle-income countries: a systematic review. Lancet 2015;385:S29.

10.

Moher D, Liberati A, T etzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9.

11.

Str oup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting.

Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283:2008–12.

12.

The W orld Bank Group. Country and lending groups. 2018 http:// data. worldbank. org/ about/ country- and- lending- groups.

13.

Ekenze SO, Ajuzieogu OV , Nwomeh BC. Challenges of management and outcome of neonatal surgery in Africa: a systematic review.

Pediatr Surg Int 2016;32:291–9.

14.

Sobhy S, Zamora J, Dharmarajah K, et al. Anaesthesia-related maternal mortality in low-income and middle-income countries:

a systematic review and meta-analysis. Lancet Glob Health 2016;4:e320–7.

15.

van den Akker T , Brobbel C, Dekkers OM, et al. Prevalence, indications, risk indicators, and outcomes of emergency peripartum hysterectomy worldwide: a systematic review and meta-analysis.

Obstet Gynecol 2016;128:1281–94.

16.

Rickar d J. Surgery for peptic ulcer disease in sub-Saharan Africa: systematic review of published data. J Gastrointest Surg 2016;20:840–50.

17.

The W orld Bank. World Development Indicators. 2015 http:// data. worldbank. org/ indicator/ SH. XPD. PCAP.

18.

United Nations Development Pr ogramme. Human development data (1990–2015). 2016 http:// hdr. undp. org/ en/ data.

19.

Nyaga VN, Arbyn M, Aerts M. Metapr op: a Stata command to perform meta-analysis of binomial data. Arch Public Health 2014;72:39.

20.

Kruskal WH, W allis WA. Use of ranks in one-criterion variance analysis.

J Am Stat Assoc 1952;47:583–621.

21.

Abdur -Rahman LO, Adeniran JO, Taiwo JO, et al. Bowel resection in Nigerian children. Afr J Paediatr Surg 2009;6:85–7. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810 12 BMJ Global Health 22. Karpelowsky JS, Millar AJ, van der Graaf N, et al. Comparison of in-hospital morbidity and mortality in HIV-infected and uninfected children after surgery. Pediatr Surg Int 2012;28:1007–14.

23.

Surapaneni S, S R, Reddy A VB. The perforation-operation time interval: an important mortality indicator in peptic ulcer perforation. \ J Clin Diagn Res 2013;7:880–2.

24.

Osifo OD, Ogiemwonyi SO. Peritonitis in childr en: our experience in Benin City, Nigeria. Surg Infect 2011;12:127–30.

25. T arcă E, Aprodu SG. Gastroschisis treatment: what are the causes of high morbidity and mortality rates? Chirurgia 2013;108:516–20.

26.

Lakhey PJ, Bhandari RS, Ghimir e B, et al. Perioperative outcomes of pancreaticoduodenectomy: Nepalese experience. World J Surg 2010;34:1916–21.

27.

GlobalSur g Collaborative. Mortality of emergency abdominal surgery in high-, middle- and low-income countries. Br J Surg 2016;103:971–88.

28.

Ariyaratnam R, Palmqvist CL, Hider P , et al. Toward a standard approach to measurement and reporting of perioperative mortality rate as a global indicator for surgery. Surgery 2015;158:17–26.

29.

Anderson GA, Ilcisin L, Abesiga L, et al. Surgical volume and postoperative mortality rate at a referral hospital in Western Uganda:

Measuring the Lancet Commission on Global Surgery indicators in low-resource settings. Surgery 2017;161:1710–9. 30. Pathak A, Sharma S, Sharma M, et al. Feasibility of a mobile phone- based surveillance for surgical site infections in rural India. Telemed J E Health 2015;21:946–9. 31. Raykar NP , Ng-Kamstra JS, Bickler S, et al. New global surgical and anaesthesia indicators in the World Development Indicators dataset.

BMJ Glob Health 2017;2:e000265. 32. Massenbur g BB, Saluja S, Jenny HE, et al. Assessing the Brazilian surgical system with six surgical indicators: a descriptive and modelling study. BMJ Glob Health 2017;2:e000226. 33. Liu C, Kayima P , Riesel J, et al. Brief surgical procedure code lists for outcomes measurement and quality improvement in resource-limited settings. Surgery 2017;162:1163–76. 34. Butler T , Knight J, Nath SK, et al. Typhoid fever complicated by intestinal perforation: a persisting fatal disease requiring surgical management. Rev Infect Dis 1985;7:244–56. 35. Osbor ne NH, Nicholas LH, Ryan AM, et al. Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries. JAMA 2015;313:496–504. 36. Hunter JP , Saratzis A, Sutton AJ, et al. In meta-analyses of proportion studies, funnel plots were found to be an inaccurate method of assessing publication bias. J Clin Epidemiol 2014;67:897–903. on 24 July 2018 by guest. Protected by copyright.

http://gh.bmj.com/ BMJ Glob Health: first published as 10.1136/bmjgh-2018-000810 on 22 June\ 2018. Downloaded from ng-KamstrafiJs, et al. BMJ Glob Health 2018;3:e000810. doi:10.1136/bmjgh-2018-000810