HCA375: Continuous Quality Monitoring & Accreditation-Adverse Event Reporting
149 6 Measuring Performance Stefano Lunardi/iStock/Thinkstock Learning Objectives After reading this chapter, you should be able to do the following:
• Discover the roles of measurement in quality improvement.
• Compare three different types of quality measures in the Donabedian framework.
• Explain the validity and reliability of a quality measure.
• Summarize key considerations in selecting measures for quality improvement projects.
• Describe sampling strategies and explain when they should be employed. fin81226_06_c06_149-176.indd 149 10/30/14 7:32 PM What is Special Education? 1 iStockphoto/Thinkstock Pre-Test 1. You can use the terms disability and handicap interchangeably. T/F 2. The history of special education began in Europe. T/F 3. The first American legislation that protected students with disabilities was passed in the 1950s. T/F 4. All students with disabilities should be educated in special education classrooms. T/F 5. Special education law is constantly reinterpreted. T/F Answers can be found at the end of the chapter. Introduction Introduction You may have heard the adage, “If you can measure it, you can improve upon it.” Measurement certainly plays a central role in the process of quality improvement. Let’s take an example.
Summit Medical Group is a physician-owned primary care group with over 53 clinics operat - ing in 11 counties around Knoxville, Tennessee.
Since 2008, the physician group began benchmarking its diabetic care processes with the National Committee for Quality Assurance (NCQA) Diabetic Recognition Program (McBride & Hensley, 2013). The program focuses on the care provided to diabetic patients. Diabetes is a serious disease that can lead to many health complications. It’s important that diabetics control their blood sugar levels, as too high or low levels can lead to health complications that include problems with eyesight and even amputations when an infection occurs in the hands or feet that cannot heal. These complications also result in serious costs to the health - care system. Therefore, groups such as the National Committee for Quality Assurance (NCQA) focus on care given to diabetic patients, as does the government, with its quality measure that looks at how well the country’s community health centers help diabetics control their blood sugar levels and its measures that focus on diabetes care on the Physician Compare website (ht tps://data.medicare.gov/data/physician -compare ). But how does a medical practice judge whether its doctors are providing good care to their diabetic patients? What are the measures to determine that? Some of the measures that Sum - mit Medical Group focused on include whether patients have annual eye examinations, regu - lar foot exams, and screening for kidney disease to prevent possible health complications.
Given that diabetes is the seventh leading cause of death in the United States, one of the dis - eases that the NCQA has focused on is diabetes care. The NCQA has established Comprehen - sive Diabetes Care measures as part of its Healthcare Effectiveness Data and Information Set (HEDIS) measures used for health plan performance reporting. HEDIS is a tool to measure performance on care and service and its diabetes care measures assess whether patients with diabetes receive care as recommended by guidelines and achieve control levels for their blood sugar, cholesterol, and blood pressure.
Using various tools that include checklists and templates, Summit created a systematic approach focused on continuous quality improvement to ensure that all of the physicians in its clinics provide quality of care to diabetic patients. The processes it set up enable Summit to incorporate effective coordination of care and collaboration among medical professionals across the continuum of care and provide evidence-based, patient-centric guidelines focused on patient health.
Summit’s continuous quality improvement (CQI) team works with management at each clinic, individual physicians, and their staff members to identify gaps in clinical quality care from the standards Summit has set and establish processes to close those gaps.
Summit uses an approach called FOCUS (which stands for Find a problem, Organize a team, Clarify the problem, Understand the problem, Select an intervention) Rapid Cycle PDSA (or Plan-Do-Study-Act model, which was described in Chapter 5). Data collection is essential to CQI projects, with data providing the CQI team with feedback and support. Once Summit had established standards of care for its diabetic patients, it could look for inconsistencies fin81226_06_c06_149-176.indd 150 10/30/14 7:32 PM Introduction in diabetes care related processes and procedures and address those problem outcomes through its CQI process.
In 2008, most of Summit’s practice locations were still using paper medical charts and had not yet converted to an electronic health record. Therefore, Summit created a paper diabetic checklist. A care team member prepared for a diabetic patient’s visit by reviewing his or her chart and then prepared the patient for the doctor in the exam room. The completed diabetic checklist was scanned into the patient’s medical record, filed in the chart, or incorporated into the physician’s progress notes. The checklist included measures that were identified across the organization and ensured the scheduling of annual eye exams, foot exams, and screening for kidney disease.
As Summit’s physician practices moved to adopt electronic health records (EHR), a physician- led workgroup designed a template within the EHR to ensure diabetic foot exams. The orga - nization trained nurses to prepare patients for a foot exam prior to the physician entering the exam room; in some cases, the nurses did a preliminary exam, with follow-up by the doctor if any problems were seen. As a result, many physicians who had previously failed to perform and document a foot exam on diabetic patients were able to meet the CQI measure (McBride & Hensley, 2013).
The EHR also alerts doctors to the need for regular screenings, such as eye exams and kidney screening. Summit also created care guides that allow physicians to quickly place reminders and orders simultaneously. As laboratory tests on a patient flow back into the EHR, the system automatically updates reminders with a date for when the test needs to be performed again.
Members of Summit’s quality reporting and improvement division tracked whether physi - cians met the criteria and ordered the proper exams for diabetic patients. They worked with doctors whose scores were low, indicating they were not following the protocol. They imple - mented process improvements, including a standardized procedure that called for staff pre - paring a patient for the physician exam to review the patient’s chart, check for documentation that shows whether studies were previously ordered, and prepare for that day’s exam by the doctor. Doctors were given a score based on how well they followed the protocol. For exam - ple, one doctor improved from a score of 30 to a passing score of 90 in one year. The changes had minimal impact on physician workflow and actually allowed doctors to spend more time talking to the patient rather than reviewing the medical record and searching to see whether required test results and exams had been completed.
One important factor in looking at quality improvement is sustaining that improvement over time and learning from failures. Summit did just that. Its quality and improvement team saw one case where a physician regressed back to previous behavior after the quality improve - ment project was put in place. The provider and a staff nurse had performed very well as a team following implementation of the quality improvement project in 2011, with a score of 80 out of 100, indicating most diabetic patients received care as desired. However, the nurse left the organization and her knowledge was not passed along to her successor. Failure to train the new nurse in the protocol resulted in the physician again failing to follow the proper protocol, and in 2012 his score dropped to 55. The problem was the ineffective training of new nurses to follow the protocol. fin81226_06_c06_149-176.indd 151 10/30/14 7:32 PM Section 6.1 The Role of Measurement in Quality Improvement As this case demonstrates, it is important to have measures in place that allow an organiza - tion to actually see whether a quality improvement project is working. While it is important that organizations track data to monitor the effects of their quality improvement efforts and make adjustments as they go along, it typically takes three years of data (36 monthly data points or data sets) to show whether or not a program is working. This is standard in CQI pro - gramming and involves pre- and post-testing validity based on a statistical principle known as Central Limit Theorem and assumptions of parametric distribution of data.
Measurement is the process of describing a phenomenon such as healthcare quality. Fre - quently, measures are quantitative in nature. Many features of healthcare quality, such as length of hospital stay, can be expressed numerically. However, some measures of quality are not so straightforward and have to be looked at in the context of the complicated healthcare world. For example, it may seem straightforward to look at the number of deaths that occur in a hospital in a year as a measure of quality care. But is a higher number of patient deaths necessarily an indication of poor care? If a hospital treats complicated cases with many older or sicker patients who are at higher risk of death, its mortality rate may be much greater than a hospital that treats young, healthy patients or does not take on complex cases.
Consider what happened in 2013, when Consumer Reports added to the drive for greater transparency in healthcare quality by releasing surgery ratings for nearly 2,500 hospitals in the United States (“What’s Behind,” 2013). The ratings drew national attention and responses from many hospitals, some of which simultaneously applauded the effort to inform patients about quality of care and questioned the validity of the ratings. Some hospital officials ques - tioned various methodological aspects of the measures, including whether the assumption that extended hospital stays (i.e., longer than average) were a marker of patient complications.
Due to legitimate concerns about the accuracy and validity of various quality measures, it is important to understand types of quality measures, the role measurement plays in quality improvement efforts, and important operational considerations, such as how specific mea - sures and measurement strategies are selected.
6.1 The Role of Measurement in Quality Improvement Measurement plays a central role in the process of quality improvement. Broadly defined, measurement includes all efforts that relate to collecting data and manipulating it to under - stand its meaning. For example, a quality improvement project to reduce hospital readmis - sions for patients with pneumonia (a lung infection) might need to collect data on length of stay, patient age, other conditions a patient may have (such as diabetes), and other pieces of information.
When initiating a quality improvement project, an early step is to construct the measurement strategy. A measurement strategy describes a project’s information needs and how these will be met during the project’s execution (Hackbarth et al., 2012).
For example, in 2011 the Department of Health and Human Services (HHS) initiated the Part - nership for Patients, which included more than 3,700 participating hospitals. This effort fin81226_06_c06_149-176.indd 152 10/30/14 7:32 PM Section 6.1 The Role of Measurement in Quality Improvement focused on improving patient safety and transitions between care settings, such as a move between a hospital and nursing home or rehabilitation center (Centers for Medicare & Med - icaid Services, 2013h).
One component of the project was to track national progress by determining whether participating hospitals across the country were improving patient safety. How could this be measured?
One way to measure how well hospitals were doing was to estimate the reduc - tion in adverse patient events—occur - rences such as a bad reaction to a drug, a complication following surgery, or an infection—since the project’s initiation.
The project planners used three sources to come up with the specific measures they wanted to track: • Adverse events reported by the hospitals in the Medicare Patient Safety Monitoring System, a national surveillance project aimed at identifying the rates of specific adverse events • The National Healthcare Safety Network, the nation’s most widely used healthcare- associated infection tracking system, maintained by the CDC • Several Patient Safety Indicators, which provide information on hospital complica - tions and adverse events following surgeries, procedures, and childbirth For each component the planners wanted to measure, they identified a source that would provide that data, such as the Healthcare Cost and Utilization Project, the largest collection of nationwide and state-specific hospital care data in the United States.
Keep in mind that the Partnership for Patients is a national endeavor, so the strategy for mea - suring results is detailed and comprehensive. Measurement strategies for quality improve - ment projects undertaken in a healthcare organization will likely be far less comprehensive.
However, when planning for any quality improvement project, a number of key stakeholder groups (and their measurement needs) should be considered (Hackbarth et al., 2012) (see Table 6.1). Some of those stakeholder groups include: • Employees who work within the health system who need data to support their daily activities • Members of the quality improvement team who need measures designed to deter - mine the success or failure of any changes made in the organization • Project leaders who need to monitor the progress of the quality improvement effort LCLPhoto/iStock/Thinkstock Over 3,700 hospitals participate in the Partnership for Patients program, which focuses on improving patient safety and transitions between care settings. fin81226_06_c06_149-176.indd 153 10/30/14 7:32 PM Section 6.1 The Role of Measurement in Quality Improvement • External programs, such as public reporting entities or insurance companies with quality-based reimbursement mechanisms, that may seek data regarding perfor - mance at the provider, unit, or hospital level • Public stakeholders who want to understand whether specific quality indicators are showing improvement • Organizational leadership who will want progress reports in order to communicate the impact of the quality improvement effort Table 6.1: Key constituencies to consider when planning measurement strategy for quality improvement Constituents Information need Example(s) Employees Provide care to patients or sup - port that care Monitor vital signs for fever or other symptoms of systemic infection Quality improvement team members Determine whether change in care procedures resulted in desired effect Number of patients with central lines developing a bloodstream infection Central line bundle adherence by providers Quality improvement project leaders Progress of quality improvement effort Overall change in number of central- line-associated bloodstream infections (CLABSI) External programs Hospital adherence to perfor - mance measures Number of risk-adjusted hospital- acquired infections Number of deaths from sepsis Public stakeholders (i.e., media) What is the quality of care at a hospital? Is care for the commu - nity improving? Estimated number of lives saved Number of infections prevented Organization leadership Impact of quality improvement effort Lives saved Infections prevented Costs reduced Source: Adapted from Hackbarth, A. D. (2012). Improvement Concepts and Methods Lecture Series, UCLA . 3. Improvement Project Measurement Design.
As an example, consider a project to decrease central-line-associated bloodstream infections (CLABSIs). A central line is a small tube inserted directly into one of the major blood vessels of the body. It is used to deliver fluids and other medications. One of the major risks of cen - tral lines is an infection of the bloodstream, which can be lethal. In 2009, more than 40,000 patients experienced this serious infection, and thousands will die from such infections each year (CDC, 2012b). (More information is available on CLABSIs and their prevention on the CDC website at ht tp://w w w.cdc.gov/HAI/bsi/bsi.html ). Therefore, hospitals are trying to prevent these infections from occurring in their patients. One of The Joint Commission’s National Patient Safety Goals for hospitals is the prevention of CLABSIs.
Hospitals are also motivated to prevent hospital-associated infections for financial reasons.
The Medicare program stopped payment for some hospital-acquired conditions—including vascular catheter-associated infections—starting in 2008, following provisions in the Medi - care Modernization Act of 2003 and the Deficit Reduction Act of 2005. The policy penalizes fin81226_06_c06_149-176.indd 154 10/30/14 7:32 PM Section 6.1 The Role of Measurement in Quality Improvement hospitals if Medicare patients acquire any of more than a dozen listed conditions during their stay in the hospital if those conditions were not present when they were admitted (CMS, 2013g). In other words, the government does not want to pay for additional care needed because a hospital made a patient sicker. A final rule released by CMS in 2010 also requires that hospitals that accept Medicare patients report CLABSIs to CMS through the CDC’s National Healthcare Safety Network (HCPro, 2010).
Suppose that as part of a quality improvement project to decrease healthcare–associated infections in your hospital, you are asked to lead an improvement project in the medical- surgical intensive care unit (ICU). The physicians and nurses will need to know if a patient with a central line has evidence of an infection. Typically, a CLABSI results in a variety of symptoms, potentially including fever, low blood pressure, or an increase in the number of infection-fighting cells in the patient’s bloodstream. Thus, providers will need to specifically monitor and measure a patient’s symptoms in order to detect and promptly treat infection in patients with a central line.
As a focus of the quality improvement effort, your team decides to create a central line “bun - dle.” It includes all of the recommended equipment for inserting a central line safely and a checklist for evidence-based steps to minimize infection risk, such as ensuring that the doc - tors and nurses properly wash their hands before starting the procedure and ensuring they have cleaned the patient’s skin with an antiseptic so they do not allow germs into the site that can cause an infection.
In order to determine whether use of the bundle has meaningfully changed outcomes, your quality improvement team will need to know whether a patient has a CLABSI and track the number of patients with these infections. As the project leader, you want to know about the CLABSI rate, but you may also want to measure compliance with the central line bundle. In other words, you want to know whether the doctors and nurses who need to insert a central line are correctly performing each of the evidence-based steps to prevent infection. If doctors and nurses follow the recommended steps, the number of patients with infections should decrease.
If the CLABSI rate does not change, knowing whether doctors, nurses, and other healthcare professionals are complying with the steps outlined in the bundle will help you determine why the infection rate has not dropped. Are they appropriately washing their hands before inserting the line? Are they using sterile gloves? Are they applying a sterile dressing after the procedure? If not, do the providers need more education? Are there other interventions you need to implement to address other risk factors? Are staff properly disinfecting equipment and the patient’s room to prevent patient infections? Are family members being educated about the need to maintain a clean environment in the patient’s room to prevent infection?
As part of a statewide effort to reduce healthcare-associated infections and resulting mor - tality, your state health department is collecting data from every hospital about infections.
(And, as mentioned above, CMS requires all hospitals that accept Medicare patients to report CLABSIs.) Your state health department is specifically seeking information about when treat - ment was initiated, since timely initiation of antibiotic therapy reduces the risk of death from bloodstream infection. Therefore, your hospital will need to collect data not only regarding whether patients with central lines develop bloodstream infections, but also how rapidly antibiotic therapy was initiated for these patients. The hospital media relations team may fin81226_06_c06_149-176.indd 155 10/30/14 7:32 PM Section 6.1 The Role of Measurement in Quality Improvement also want to share information with the community about progress on reducing infection and will therefore need some measurement of improvement.
Finally, the chief medical officer and other hospital leaders involved in patient safety will want to learn about reduction in the infection rate. It may also be important to translate the findings to show the impact the reduction in infections has had. Can your team estimate the number of lives saved from the intervention? Can the team determine how many ICU days were saved due to fewer severe infections? Entities that pay for healthcare may include incen - tives (positive or negative) for meeting certain quality-of-care metrics like reduced infections.
Thus, it may be possible to convert progress in preventing CLABSI into dollars saved. You will need to have collected the data that allows you to determine these specific improvements.
For instance, by looking at the percentage of patients who developed CLABSIs prior to imple - mentation of the bundle versus the percentage after implementation of the quality improve - ment, you will know approximately how many patients have benefited from the intervention.
By determining how many patients typically die from such infections, you can estimate the number of lives saved. Then you can look at how many days the typical patient with a CLABSI spent in the ICU and multiply that by the number of patients saved from an infection to cal - culate those savings. You can also calculate the reimbursement dollars that would have been lost if patients developed infections.
The take-home lesson is that it is important to consider the information needs of key stake - holders at the beginning of the project in order to gather information more efficiently while the quality improvement effort progresses.
It’s also true that the best way to increase the odds of solving a problem or implementing an improvement is to take into account the perspectives of the many people affected by the issue at hand. So in a nursing home, stakeholders will include department heads and supervisors, as well as direct care staff, residents, and their family members, and a quality improvement team should consider all of their needs, preferences, and opinions. It’s also important to con - sider all levels of staff in a facility, as well as different shifts and departments that may be affected, and include them in the quality improvement process. Questions to Consider 1. In your opinion, which stakeholder group(s) has the most important information needs?
Why? 2. Suppose you are part of an infection-control team tasked with improving hand wash - ing in an ICU. For each of the six key stakeholder groups, identify potential measures to address their information needs related to this project. fin81226_06_c06_149-176.indd 156 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care 6.2 Measuring Quality of Care There are several key players in CQI. These individuals include: W. Edwards Deming —He is regarded by many as the “father” of quality improvement, and his work evolved into one of the best-known quality improvement models: the Plan-Do- Study-Act (PDSA) cycle.
Joseph M. Juran —A Romanian-born American engineer and quality management expert, Juran, a noted author, is considered the “father” of modern day quality management.
Juran is widely credited with adding the human dimension to quality management, as he pushed for the education and training of managers. Working independently of Deming, who focused on the use of statistical process control, Juran also took his ideas to Japan after World War II and focused on managing for quality.
Avedis Donabedian —a physician and creator of what’s known as the Donabedian model of care. He built on the foundation of Deming and Juran and defined quality of healthcare services with his model that categorized potential quality measures into three related groups: structure, process, and outcome. He also described the seven “pillars of quality,” or attributes of healthcare that define its quality. In his seminal work, Avedis Donabedian (1980) categorized potential quality measures into three related groups: structure, process, and outcome. Information about quality of care can be drawn from these three sources.
Structure metrics relate to the characteristics and resources available in the setting where care takes place. Structure describes the context in which healthcare is delivered and can include hospital buildings, staff, financing, and equipment. For example, is an electronic health record available in the emergency department? Does the hospital have intensive care specialists available in the hospital 24 hours a day?
Process metrics relate to the manner in which care is delivered. In other words, do all patients receive appropriate medication to prevent blood clots forming in their legs after sur - gery? Do patients undergoing a surgical procedure receive appropriate antibiotics to prevent infection?
Outcome metrics typically include measures that access the state of the patient after a healthcare intervention. Was the cancer removed in its entirety at the conclusion of the sur - gery? Did the patient survive the surgical procedure to bypass blocked arteries and restore blood flow to the heart? For the purposes of quality improvement, process and outcome met - rics will likely be the most useful measures to drive change.
The model is most often represented by a chain of three boxes containing the terms structure , process , and outcome connected by unidirectional arrows in that order, which can be seen in Figure 6.1. fin81226_06_c06_149-176.indd 157 10/30/14 7:32 PM Structures of Care Structures of care include care personnel, facilities and buildings, financial resources, supplies, and record keeping systems. Processes of Care All care-related activities are included in processes of care. Outcome The outcome is the result of these structures and processes of care. Section 6.2 Measuring Quality of Care The framework can be used to change structures and processes within a healthcare orga - nization. For example, suppose that leaders in a small physician practice want to improve their treatment coordination process by creating a better procedure to communicate labora - tory results to physicians and streamline patient care. How can the exchange of information from the lab to the attending physician be improved? The leaders may decide to purchase an information technology solution that allows pop-up alerts for lab results that is incorporated into the electronic health record and allows the physician to receive results and complete a diagnosis on a patient.
The physician practice could then modify its process by changing the protocol that deter - mines how and when an alert is released by the lab and who is responsible for receiving and reacting to the alert. The outcomes that leaders will use to evaluate how well the quality improvement solution works might include patient satisfaction, timeliness of a diagnosis, or clinical outcomes.
Let’s look more closely at structure, process, and outcome measures.
Process Measures in Quality Improvement Process measures are commonly used in quality improvement endeavors, since they most directly assess how providers deliver care to patients. Process measures used for qual - ity improvement should ideally be strongly related to the desired outcomes of care (i.e., Figure 6.1: Donabedian quality framework The Donabedian quality framework classifies quality metrics in terms of structures, processes, and outcomes of care. Structures of Care Structures of care include care personnel, facilities and buildings, financial resources, supplies, and record keeping systems. Processes of Care All care-related activities are included in processes of care. Outcome The outcome is the result of these structures and processes of care. fin81226_06_c06_149-176.indd 158 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care improved patient survival). Four criteria have been proposed for process measures that will form the backbone of improvement efforts (Chassin, Loeb, Schmaltz, & Wachter, 2010): • proposed measure has a strong evidence base supporting improved outcomes as a result of the care process; • proposed measure is valid for determining whether the care process was actually provided to the patient; • care process is proximate to outcome of interest; and • there is a low risk of unintended or adverse effects from implementing the care process. First, a process measure should have strong evidence to suggest that it will lead to better patient outcomes. The more well-designed studies that have shown that a care process has resulted in better outcomes, the stronger the evidence. Multiple studies are better than just one or two (Chassin et al., 2010). For instance, what’s the best treatment for people with post - traumatic stress disorder (PTSD)? Evidence-based treatments are those for which numerous research studies have shown them to work. In addition, it has been suggested that a proposed measure should determine whether healthcare personnel actually delivered the care as intended to the patient (Chassin et al., 2010). For example, many healthcare systems empha - size counseling patients about quitting smoking and track this measure through the use of a checkbox on a form when they see a provider. However, the presence of a checkmark does not measure the thoroughness of cessation counseling or whether the patient received appropri - ate resources to stop using tobacco.
In addition, it is very important that the care process be closely related to the outcome you want to see. For example, whether a patient receives a dose of antibiotics before surgery to prevent infection is closely related to whether the patient develops a surgical site infec - tion. Other process measures, such as a bone scan at the time of diagnosis for asymptomatic (screening-detected) prostate cancer, are much more remote from the outcome of interest (patient survival). Thus, in order to determine what measures helped a patient survive pros - tate cancer, you might look at a number of intervening steps, such as the timeliness of therapy and the selection of optimal therapy. In other words, the patient starting treatment right away and the physician making sure the patient receives the best type of treatment could have a stronger influence on whether the patient will survive than a diagnostic test (the bone scan) performed at the initial stages of the disease.
Finally, implementing the care process measure should result in a low risk of adverse events (Chassin et al., 2010). For example, consider a process measure that encourages providers to administer antibiotics within four hours of when patients present symptoms of community- acquired pneumonia, a lung infection that can cause serious illness or death. Evidence sug - gests that timely administration of antibiotics to patients with pneumonia improves patient outcomes: they recover because they receive medication before their pneumonia worsens.
However, once this quality measure became widely used, some patients without pneumonia began to receive antibiotics unnecessarily, simply to comply with the quality indicator (Kan - war, Brar, Khatib, & Fakih, 2007; Wachter, Flanders, Fee, & Pronovost, 2008). Unnecessary anti - biotic use promotes antibiotic resistance, which ultimately harms patients. Therefore, imple - mentation of quality metrics can have unintended and potentially harmful consequences. fin81226_06_c06_149-176.indd 159 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care Well-intentioned ideas can have bad outcomes. Take a case in which a nursing home’s housekeeping department decided to try a new floor wax because it was easier to apply.
They soon discovered that the new wax caused the floors to be slippery and resulted in more resident falls.
Outcome Measures in Quality Improvement Ultimately, every healthcare organization wants to provide high quality care and achieve good patient outcomes. Several considerations exist when using outcome measures for quality improvement. Outcome measures have an important signaling function in quality improve - ment: They help identify areas for improvement (NQMC, n.d.). However, many factors can contribute to a specific outcome, which means that while outcome data point to a specific problem, they are less helpful in terms of identifying what specific intervention may be required to improve the outcome. Consider a surgical unit with a higher-than-expected pro - portion of patients who develop a deep vein thrombosis (DVT) after their surgical procedure.
Blood clots often form in the legs after surgery due to a number of factors, including immobi - lization or decreased mobility. As discussed in Chapter 4, DVT is an important surgical com - plication because these clots can travel to the lungs and cause severe illness or death.
Surgical unit leaders want to decrease the number of patients who experience DVT during their surgical recovery. Several potential factors could contribute to patients developing DVT.
Surgeons may not prescribe appropriate prophylactic interventions following surgery. There may be a delay in administering medications. Get - ting patients up and walking is an important com - ponent of a DVT prevention program, but nurs - ing staff may not have time to help patients walk because of other clinical care demands or high patient-to-nurse ratios. As this example shows, out - come measures help identify the “what” but do not necessarily show the “how” of fixing the problem.
It’s important to consider all the factors before drawing conclusions about outcomes measures.
Comparing outcomes among different entities, including doctors, units, hospitals, or other types of healthcare organizations can also be problematic.
The health status of patients cared for by different entities can vary in ways that influence the proba - bility of a specific outcome (NQMC, n.d.). For exam - ple, patients who have blocked arteries that supply blood to the heart often undergo coronary artery bypass graft (CABG) surgery to restore blood flow to the heart muscle. Risks of the surgery are relatively low but can include serious complications, such as stroke, kidney failure, or death. Patients with cer - tain risk factors, such as diabetes, poor heart func - tion, or advanced age (more than 70 years) are at higher risk for these adverse outcomes. Therefore, Stockbyte/Thinkstock Nurses may not have time to help patients walk, despite it being an important component of a DVT prevention program, due to other clinical care demands or high patient- to-nurse ratios. fin81226_06_c06_149-176.indd 160 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care a facility caring for older patients in poorer health may have less desirable outcomes than another facility caring for a different population.
Let us consider an example. Suppose the quality department at a hospital is comparing the outcomes for two cardiac surgeons, Dr. Beatright and Dr. Sickheart. Beatright usually operates on younger, healthier patients than Sickheart. Not surprisingly, Beatright has a lower rate of postoperative stroke, kidney failure, and death among his patients than Sickheart does.
Simply comparing the mortality rate between the two surgeons is not entirely accurate, since patients who are older or sicker may still benefit from CABG surgery. In order to compare these outcomes accurately, risk adjustment must be used to account for the differences in the patient population treated by each surgeon. Risk adjustment is a statistical procedure that seeks to balance differences in patient characteristics, such as age or health conditions, when comparing outcomes between facilities or providers. So, as adjusting for the differences in patients between two cardiac surgeons, similar methods should be used when compar - ing outcomes between different hospitals. The issue of risk adjustment means that outcomes must be carefully used to inform quality improvement and accountability measures.
Risk adjustment statistically factors out or accounts for differences in patients. As another example, risk adjustment minimizes the possibility that differences in outcomes between one home health agency and another are due to factors other than the care they provide. A home health agency may have more patients that require hospitalization than others in its state or even compared to the national rate. But that doesn’t mean the agency provides inferior care.
Risk adjustment helps ensure a fair comparison of outcomes by taking into consideration patient characteristics at the start of a home care visit that may affect the likelihood of spe - cific outcomes during the visit. The agency may have patients whose age is much higher or a higher proportion of patients with a cognitive impairment. The patients may be frailer or less able to understand or remember the instructions of the home health nurse. The patient may be more susceptible to falls or not take his or her medication as instructed and end up back in the hospital. Therefore, it is understandable, and perhaps even expected, that the agency’s hospitalization rate would be higher than other facilities’; the characteristics of its patients are very different.
A key to risk adjustment is determining which risk factors truly influence a given outcome.
Which risk factors have a clinically or statistically meaningful influence on a particular out - come? A patient taking many medications, especially high-risk medications such as antico - agulants or insulin, might be expected to be at higher risk for hospitalization. Patients who are depressed or have physical limitations that limit their ability to participate in their own care might not respond as well to a home health visit.
The final concern when employing outcomes, particularly for comparative purposes, is the issue of sample size. Sample size refers to the number of observed opportunities for an out - come to occur. When the sample size is small, the rate of outcomes is less statistically reliable than when the sample size is large (NQMC, n.d.). Therefore, it will be hard to draw conclusions about a surgeon who only operates on 20 patients in a year’s time.
This takes into account the statistical concept of the central limit theorem, which essentially says as a sample size increases, just about any distribution, normal or non-normal, will tend to behave normally. fin81226_06_c06_149-176.indd 161 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care Validity and reliability are important concepts when it comes to quality measures. Measures should have several key characteristics, including responsiveness, reliability, and validity.
Responsiveness refers to the concept that when the phenomenon being measured changes (e.g., patient satisfaction), the measure will also change. Reliability refers to the concept that if a phenomenon or process does not change, then the measure will not change, either. There are two basic types of reliability.
Inter-rater reliability refers to the psychometric property that when two different individu - als observe or measure a process, the reported measure is the same (again assuming a stable process) (Aday & Cornelius, 2006). If two nurses use the same instrument to rate the same thing, their ratings should be similar. Take, for example, the use of a pain assessment instru - ment used in a nursing home to measure pain in residents. Intra-rater reliability refers to the same results achieved by the same researcher when measuring and then re-measuring a process. So the same test is performed by the same person and should duplicate the results.
Validity means that the indicator actually measures the intended construct. For example, patient ratings of quality of care may actually represent satisfaction with other experiences, such as the quality of food, rather than the actual quality of care they received.
To understand the effect of small sample size on statistical reliability, imagine the results of a simple coin toss. Assuming an evenly balanced coin, the probability of heads is 50%. Although theoretically a series of four coin tosses should result in two heads and two tails, often four consecutive tosses result in heads, or four consecutive tosses result in three tails and a head.
However, as the number of tosses increases (that is, the sample size increases), the results will be closer to the true probability of 50%. This is known as the law of large numbers— that is, as the number of events or trials increases, the average of all the outcomes should approach the true probability (in this case, 50% chance of heads).
Returning to our two cardiac surgeons, it is important to understand whether Beatright oper - ates on 10 or 100 patients each year. The complication rate will be more precise if Beatright’s operative volume (i.e., the sample size) is 100 patients.
Given the statistical issues of risk adjustment and sample size, and the lack of information about underlying factors contributing to observed outcomes, quality improvement efforts tend to favor process measures for operational aspects. For instance, a physician practice will track the length of time it takes for a patient to receive an appointment or measure patient satisfaction with the care received. Or a surgery center will track how long it takes to get a test done or a service performed.
Project leaders will often identify a small “test of change”—a small-scale test to help deter - mine whether a change can result in a sustainable improvement. They will test a given pro - cess and then observe either the downstream effect on process measures or a measure of the ultimate outcome in order to determine the success of the intervention. Testing the change on a small scale allows an organization to learn from experience before trying to implement a change broadly throughout a facility. After the test or trial phase, project leaders can ana - lyze the results and act on what is learned. If the test is successful, how will the organization spread the change? What preparations are needed? Do staff need training to implement the improvement? If the test reveals the change didn’t work, you may need to adjust your plan. Or perhaps start again and test a new improvement. fin81226_06_c06_149-176.indd 162 10/30/14 7:32 PM Section 6.2 Measuring Quality of Care Returning to our example of DVT prophylaxis, quality improvement leaders could set up a computer prompt to remind surgeons to order medication for each patient. In order to assess the success of this intervention, they can then compare the rate whereby physicians ordered the medications (or the number of patients who received it) before and after the intervention. In addition, project leaders could also compare what they observe in terms of outcomes (although this could be clouded by the aforementioned issues of sample size or risk adjustment).
This small-scale testing can follow the Plan-Do-Study-Act (PDSA) cycle for improvement:
planning the test, doing it on a small scale, studying the results, and then acting on what is learned. Four steps for testing an improvement: Plan: Plan the test or observation. Who will be involved? What do you want to accom - plish? What data needs to be collected?
Do: Try out the test on a small scale. Record the results. Study: Analyze the results. Determine what was learned. Do the results show what you anticipated? Were there any surprises?
Act: Use what is learned to refine the test. Determine the next step. Along with small-scale testing, organizations may want to make their improvements in steps, especially for more complicated projects. A project may involve a series of improvements and the organization may want to test each step through the PDSA process rather than trying to test the whole project at once.
For example, a nursing home want to improve its dining services so that it can ensure all of its residents are eating better. The facility decides to change its system of providing meals from a tray delivery system to having steam tables on each unit of the nursing home. This will create more flexible dining hours allowing residents to eat when they want to and give them greater variety of meal choices.
The nursing home might first change the tray line to steam tables. Then it might experiment with flexible hours, and then try different menu options. With each step, the staff will assess whether the change is an improvement. There are many factors to measure in order to deter - mine whether the change to steam tables is an improvement. Are residents eating more? One measure to access the dining program is to look at the number of residents who have an unexpected weight loss. Are there some residents who are eating less now that a tray is not delivered to their room? Another measure might be the use of food supplements. If residents enjoy eating more, there may be less need for supplements after the change. But there must be data to compare the use of supplements before and after the steam tables were added. Are residents happier with the new dining arrangements? Are family members satisfied that their loved one is eating better? The QI team can ask residents and family members how the resi - dents feel about the food and mealtime experience. In addition to verbal feedback, they could conduct a survey. What do dining staff think about the change? What about the care staff on the units? Feedback from staff is also important. Is there an effect on quality of life? Do more residents come out of their rooms to eat their meals in the dining area on each unit and have more interaction with others? Or do some residents prefer to eat in their own rooms? fin81226_06_c06_149-176.indd 163 10/30/14 7:32 PM Section 6.3 Validity of Quality Measures Questions to Consider 1. Why are process measures the most common category of metrics used for quality improvement? 2. In your experience, what is an example situation in which implementation of a rule or guideline had unintended or adverse consequences? 6.3 Validity of Quality Measures Not all quality measures are created equally. The National Quality Measures Clearinghouse (NQMC) (n.d.) has identified several properties that determine the validity—measuring what you intend to measure—of a quality measure: • strong scientific evidence should exist to support the measure; • each patient should be eligible to receive the recommended care; • the entity or individual should reasonably be able to influence whether the quality measure is met; • the measure should adequately capture the event of interest; • the measure should permit fair comparison of individuals, entities, or populations; and • the measure should permit exclusion of patients with characteristics that prevent performance of the measure. We will examine each one of these factors.
First and foremost, strong scientific evidence should exist to support the measure (NQMC, n.d.). Ideally this would be in the form of multiple, well-designed studies linking the quality measure to improved patient outcomes. Often quality measures are incorporated into clinical guidelines or best practice statements. The American College of Chest Physicians produces a comprehensive review of the evidence for the best ways to prevent of DVT titled Guidelines for Diagnosis and Management of DVT/PE/VTE (Guyatt, Akl, Crowther, Gutterman, & Schuun - eman, 2012). Other best practices might focus on how nursing homes can prevent residents from developing painful pressure sores from lying in bed or sitting in a chair for long periods of time or the best method for an outpatient clinic to assess patients’ medication and allergy lists for accuracy.
Second, each patient should be eligible to receive the recommended care (NQMC, n.d.). For example, therapy with medications to prevent blood clot formation is typically used for patients after surgery in order to prevent DVT formation in the legs. However, some patients should not receive these medications, such as those with an adverse reaction or those who have ongoing bleeding. These patients should be excluded, since they are not eligible for the recommended therapy.
Third, it is important that when a quality measure is applied, the healthcare organization should reasonably be able to influence whether it can in fact the meet the quality measure fin81226_06_c06_149-176.indd 164 10/30/14 7:32 PM Section 6.3 Validity of Quality Measures (NQMC, n.d.). For instance, it doesn’t make sense to measure a health plan based on how many of its patients have asthma. The prevalence rate is quite meaningless. The health plan cannot influence how many of its patients have asthma, which is a condition caused by a number of factors, some genetic and some environmental. It is reasonable, however, to measure a health plan on how well it cares for the patients who do have asthma.
For example, the NCQA HEDIS measures that determine the effectiveness of care for respi - ratory conditions include measures for the care of patients with asthma. Health plans are evaluated on how well providers prescribe appropriate medications for asthma patients, the percentage of patients who remain on an asthma controller medication, and an asthma medi - cation ratio.
Fourth, the measure should adequately capture the event of interest. For example, whether an inpatient took a specific medication is relatively easy to accurately capture, because nurses document exactly when they administer each medication to individual patients. On the other hand, a checkbox on a form that documents whether the patient was counseled regarding weight loss following cardiac surgery may be less reliable, since it is unclear to what extent or how effective delivery of the counseling was in each instance. The exact manner in which a measure will be operationalized (that is, how a measure is actually put into use—definition, data gathering, etc.) is an important consideration for any potential measure in quality improvement activity.
The fifth validity criterion employed by the NQMC is that the quality marker should permit fair comparisons across various populations. As previously noted, two hospitals’ patient popula - tions may differ in important char - acteristics that relate to the ability to meet a given quality metric. For exam - ple, Medicare introduced financial incentives for hospitals to decrease the number of patients readmitted for heart failure within 30 days of dis - charge. However, some studies suggest that patients’ socioeconomic status is strongly related to the likelihood of readmission. Thus, if readmission metrics are used to compare hospitals treating patient populations from dif - ferent economic backgrounds, the hos - pitals that treat low-income patients will be disadvantaged in the compari - son (Joynt & Jha, 2013).
Finally, a valid quality measure should appropriately permit exclusion of patients with char - acteristics that prevent performance of the measure. For instance, if a patient refuses a ther - apy, the measure should allow the individual to be excluded from assessment (NQMC, n.d.). Associated Press/Marcio Jose Sanchez If readmission metrics for heart failure patients are used to compare hospitals treating patient populations from different economic backgrounds, the hospitals that treat patients from a low-income population will be disadvantaged in the comparison. fin81226_06_c06_149-176.indd 165 10/30/14 7:32 PM Section 6.4 Selecting Measures for Clinical Quality Improvement The NCQA’s HEDIS measures often allow for patients to be excluded. For example, one of the HEDIS measures for prevention and screening of patients is an assessment of the body mass index in all adults. However, the measure excludes patients who are pregnant, since they are expected to gain weight because of the baby. When it comes to the HEDIS measure for child - hood immunizations and the vaccines that must be completed on or before a child’s second birthday, it excludes children who have a contraindication for a specific vaccine, such as if a child would have an anaphylactic reaction to the vaccine or its components. While another HEDIS measure requires adults ages 50 to 75 to be screened for colorectal cancer, patients who already have a diagnosis of colorectal cancer or a total colectomy are excluded. Questions to Consider 1. In your opinion, are some of the properties to support the validity of a quality measure more important than others? Why? 2. Can any of the properties be eliminated without the validity of the measure coming into question? Why or why not? 6.4 Selecting Measures for Clinical Quality Improvement A number of considerations come into play when selecting measures for a clinical quality improvement project. When creating the strategy for measuring the quality improvement project, leaders should address numerous questions (see Table 6.2). Those questions need to address a number of considerations, which include factors such as the performance of the measure, the effort needed to analyze the results, any time lags, and sensitivity to failure.
Organizations also need to address those questions in the context of each of the key constitu - encies—such as employees, quality improvement team members, organization leaders—for the specific quality improvement effort, as outlined in Table 6.2. Many of these considerations relate to the feasibility of using an individual measure. Including these key considerations when designing a measurement strategy for quality improvement provides a strong founda - tion to successfully optimize care delivery. fin81226_06_c06_149-176.indd 166 10/30/14 7:32 PM Section 6.4 Selecting Measures for Clinical Quality Improvement Table 6.2: Considerations when selecting a measure for quality improvement Consideration Question to ask Performance How well does the measure capture the process or outcome of interest? Has the measure been validated? Existing versus new measures Is data already collected? Does a similar measure already exist? Analytic effort How much effort is required to make sense of the measure? Time lag How frequently is the measure generated? What is the time difference between the event and measure reporting? Sensitive to failures Does the measure identify when an error or quality lapse occurs? Actionable information Does the measure allow an organization to make a change in process or methods? Balancing measures What measures will detect unanticipated or adverse consequences? Regulatory compliance Does measure include identifying information about patients?
In what context will data be shared or used? “Parsimonious” measure selection (selecting the best measures that will support the improvement project) Which measures constitute the “vital few” measures? Source: Adapted from Hackbarth, A. D., Munier, W. B., Eldridge, N., Jordan, J., Richards, C., Brennan, N. J., . . . McGann, P. (2012). An overview of measurement activities in the Partnership for Patients. Journal of Patient Safety , 8. It is also important to note that healthcare providers are also required to report clinical qual - ity measures in order to demonstrate what is called meaningful use under the Medicare and Medicaid Electronic Health Record (EHR) Incentive Program. You can find the criteria for reporting those measures in the 2014 Clinical Quality Measures Tipsheet ( http://www.cms .gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Clinical QualityMeasuresTipsheet.pdf ). Let’s take a closer look at each of the considerations a healthcare organization should make when selecting a measure for quality improvement.
Performance The first consideration is typically to identify how well the measure captures the process or event of interest. If the measure does not reliably capture the outcome the organization desires, it does not particularly matter how easy the measure is to collect. For instance, a health plan wants to calculate the patients who should have received beta blockers that actu - ally were given a prescription by their physician. The NCQA’s HEDIS measure defines very precisely how to calculate that rate, so the health plan can determine its rate and how it com - pares to other plans.
However, a measure need not be absolutely perfect, particularly if existing data systems already collect the information. In addition, the manner in which the organization will use fin81226_06_c06_149-176.indd 167 10/30/14 7:32 PM Section 6.4 Selecting Measures for Clinical Quality Improvement the data will influence the choice of measures; it is more important to use validated, well- established measures when measuring the impact of a project or when reporting to outside stakeholders than when performing small tests of change within a nursing unit, for example.
Existing Versus New Measures and Analytic Effort Clinical quality improvement leaders often face tradeoffs when choosing between existing quality measures and new quality measures. Existing quality measures are those already collected within the healthcare organization. Examples of these existing measures include administrative claims data, vital signs (i.e., blood pressure), and laboratory test results that are routinely generated in the process of providing and billing for clinical care. If one of these preexisting measures is good enough for the purposes of the specific quality improvement project and the information needs of the stakeholders, then the organization should use exist - ing data.
A new quality measure is any piece of data that is not already being collected in the care envi - ronment. When they are in the process of selecting a measure, organizations should not over - look the effort it will take to analyze data. Some measures may require extensive statistical analysis or risk adjustment, which will increase the time and personnel, as well as computer resources, required to generate useable information after the data have been collected. When an organization needs to collect new data, the project leaders should carefully consider the effort and cost involved in creating, validating, collecting, and analyzing a new measure (par - ticularly one that has only slight differences from data that is preexisting or is more easily analyzed).
Time Lag Another important consideration in selecting mea - sures for quality improvement is that of time lag . Time lag is essentially how rapidly the information from the measure becomes available for the orga - nization to use. Clinical workers such as nurses and physicians typically need information on a daily—or more frequent—basis in order to make patient care decisions. Team members for a qual - ity improvement project may need information less frequently—for example, on a weekly basis. A mea - sure that will be used to communicate the impact of efforts to hospital leadership or external stake - holders may only need to be collected or reported quarterly or annually.
Another factor to consider is the time it takes from determining an outcome versus the need to change a care process. For example, some organiza - tions report hospital mortality rates on a quarterly basis. However, when a hospital’s mortality rate worsens, it can be difficult to reconstruct why the change occurred and what care processes may need Jupiterimage/liquidlibrary/Thinkstock Clinical workers typically need information on a daily basis in order to make patient care decisions, while team members for a quality improvement project may need information less frequently. fin81226_06_c06_149-176.indd 168 10/30/14 7:32 PM Section 6.4 Selecting Measures for Clinical Quality Improvement to change, given the potentially long time interval (up to three months) between a patient’s death and the quality metric report. A hospital that sees a dangerous trend in its quality of care needs to act right away to determine why this has happened and to correct any problems.
Failure of Care and Actionable Information Two other important operational considerations are the ability of a quality measure to detect a failure in care delivery and generate information that the quality improvement team can act on. Useful process or outcome measures should be able to identify when a lapse of care occurred. For example, an organization could measure compliance with DVT prophylaxis by auditing its computerized physician order entry (preexisting data), by auditing medica - tion administration (preexisting data), or direct observation of medication administration (time- and personnel-intensive data). Monitoring only physician orders will provide a fairly good approximation of compliance but may not allow the organization to see when a patient missed their medication dose for various reasons (i.e., a patient is off the nursing unit for an extended period to receive a diagnostic test).
Monitoring existing records that show when a specific medication is administered to a patient will likely prevent this oversight. The other operational aspect of a quality measure is that an organization can take action in response. Simply knowing that the DVT prophylaxis rate is low does not provide any information about where lapses in care may be occurring. Thus, it may be difficult to target interventions toward the parts of the care process responsible for the failure of patients to get their medication. Is the problem with the delivery by the nurses on the floor? Is it a problem in the hospital pharmacy? The ability to generate actionable information is one rea - son why process measures are the most frequently used data for clinical quality improvement.
Balancing Measures Another important consideration in quality improvement is when changes in the care pro - cess create unintended, and sometimes adverse, consequences. Balancing measures are the data collected specifically to detect unanticipated consequences of modifications to the care process (Institute for Healthcare Improvement, 2011). These measures examine the changes from “different directions/dimensions” within the healthcare system (Institute for Health - care Improvement, 2011).
Consider a project to reduce length of stay for patients hospitalized for symptoms of heart failure. One potential unintended consequence could be an increase in hospital readmissions because of further heart problems. In other words, if a hospital discharges a patient with heart failure too soon, he or she may require readmission within a few days if problems reoc - cur or were not corrected.
Regulatory Compliance Regulatory compliance is another critical issue when considering how measures will be used in a clinical quality improvement effort. A major consideration is protecting patient identities and the confidentiality of their information. The other consideration occurs when the orga - nization shares information as part of a research project. In that case, the organization must comply with ethical principles and regulations to protect research subjects. fin81226_06_c06_149-176.indd 169 10/30/14 7:32 PM Section 6.4 Selecting Measures for Clinical Quality Improvement The Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule which took effect in 2003 requires that healthcare organizations must guard a patient’s protected health information. It must also not release any information that can identify patients. The HIPAA regulation requires organizations that share information to remove or “de-identify” informa - tion, such as names, addresses, telephone numbers, and Social Security numbers, that could identify a patient. Intentional or unintentional disclosure of identifying information can result in civil penalties as well as criminal prosecution if the information is used for financial gain.
However, measures that may be used by caregivers, such as nurses or physicians, must include patient-specific identifiers in order to modify care plans for individual patients. If a patient fails to receive an antibiotic dose, the failure of care delivery cannot be corrected if the patient’s identity is unknown. However, if the end user of the information is a nursing unit manager, the information needed may only be the overall rate of missed doses in the unit in order to serve as a broad indicator of quality of care.
Parsimony Finally, when selecting measures, clinical quality improvement managers should pay close attention to parsimony , or the use of the fewest resources or measures necessary to achieve a given aim for a quality improvement project. That is, quality improvement managers will want to select the “vital few” measures that will support the improvement endeavor. In con - trast to research projects, where the goal is often to collect as much information as possible, even if it is not necessarily within the immediate goals of the project, data collection for qual - ity improvement should focus on the minimum number of measures to support the informa - tion needs of the project as laid out in the measurement strategy.
Quality management expert J. M. Juran coined the phrase “the vital few and the trivial many” to refer to the concept that 20% of the errors in a production pathway will account for 80% of the overall number of errors (Juran, 1951). The “80–20 rule,” or the Pareto principle , extends to many phenomena in life (Juran, 1951). In the context of selecting measures for quality improvement, this means that the vital 20% of all potential measures will provide 80% of the information needed to execute the project. Therefore, project leaders need to carefully weigh the effort required to collect data beyond the “vital few” measures against the value of the additional information produced by increasingly trivial data points. Questions to Consider Suppose you are leading a quality improvement project to improve flu vaccination rates among adults at an ambulatory clinic. You and your team members are creating the measurement strategy for the project.
1. What are the key constituencies (from Table 6.1) and their information needs for this project? 2. For each constituency, what are one or two key considerations for selecting a quality measure to serve the information need of the group? fin81226_06_c06_149-176.indd 170 10/30/14 7:32 PM Section 6.5 Sampling for Clinical Quality Improvement 6.5 Sampling for Clinical Quality Improvement Sampling is an important strategy to reduce the burden of information collection as part of a quality improvement project. Sampling means that data are only collected on a subset of a population, rather than on the entire population. For instance, in order to understand the main reasons for hospital readmissions, it may only be necessary to sample a subset of all hospital readmissions that occur within a given time period.
Sampling for research usually assumes a fixed population with a specified distribution of the variables of interest (Perla, Provost, & Murray, 2013). Using an anticipated therapeutic effect (for example, a 10% decrease in mortality for a new medication), investigators can project how many patients would need to be included in a study to reasonably determine whether a new medication worked as anticipated. In contrast, measurement for quality improvement occurs in the context of a dynamic, ongoing process in an uncontrolled environment (Perla et al., 2013).
Consider a quality improvement project to reduce the time patients wait to be seen by their doctor when they have an appointment at a family practice clinic. Project leaders could decide to collect appointment wait times for 50 patients every week, rather than for all of the hun - dreds of patients that come into the clinic each week. They will then continue to track wait times for 50 patients per week for a month. They could use a variety of sampling strategies.
For example, clinic leaders may decide to track patient visits that are spread out over each day of the week from Monday through Friday. Perhaps they will find backups in patient appoint - ments occur most often on one day. Then they can look at why that occurs. Is the clinic short a physician that day? Is more staff needed? Or they might decide to track data on both morning and afternoon visits. Are appointment delays worse at one time of the day versus another?
Do patients wait longer as the day goes by and physicians get backed up? Is the clinic busiest in the morning and early evenings when people try to schedule appointments before or after working hours? Or if there are several providers in the clinic, it may be necessary to increase the sample size in order to track patients who see each doctor, so leaders can make reliable comparisons between physicians.
Project leaders may choose to employ probability-based sampling schemes, which focus on a specified proportion of visits. One probability-based sampling strategy might be to collect wait times for 20% of the patient visits to each provider in a clinic. Are there certain provid - ers who get behind so that patients are waiting longer? Probability-based schemes, when appropriately designed, provide more representative data. Another sampling technique is convenience-based sampling. These strategies can simplify and expedite the data collection process. An example of a convenience sample might be to collect wait times from all patients who are seen in a clinic on Tuesday and Thursday, regardless of which provider their appoint - ment is with. All of these considerations should be included in the project’s measurement strategy. fin81226_06_c06_149-176.indd 171 10/30/14 7:32 PM Summary & Resources Questions to Consider 1. What are some of the factors that go into determining sample size? 2. What problems can you foresee if a sample size is too small? Summary & Resources Chapter Summary Measuring performance is a central activity for assessing and improving healthcare qual - ity. When undertaking a clinical quality improvement activity, a well-developed measure - ment strategy is one of the first steps toward success. In particular, a measurement strategy for quality improvement should consider the needs of a variety of stakeholders within the health system, such as employees, quality improvement leaders, and public stakeholders.
Once the needs of key stakeholders are clear, project leaders must select appropriate mea - sures for use in project management.
Measures of healthcare quality are broadly categorized as structure, process, or outcome measures. Process measures are most commonly used in quality improvement, since they often track how care is actually delivered. Outcome measures are most useful for moni - toring improvement (whether the patient was better off as a consequence of a change in process), as well as monitoring for unforeseen consequences (i.e., a balancing measure to detect unanticipated or adverse consequences). Guidelines exist for identifying valid clinical quality measures, and therefore project leaders should carefully evaluate potential metrics against these guidelines during the planning phase of a clinical quality improvement effort.
Finally, a number of key considerations pertain to selecting specific measures for a quality improvement project. Performance (that is, how well a measure captures a process of care) is important, as is the choice between using existing data versus initiating a new data col - lection process. Ideally, metrics reliably identify any failures in care and provide healthcare providers and quality assurance leaders with information that is timely and that they can act upon. Regulatory compliance and public reporting are also key considerations in plan - ning a measurement strategy. A well-planned, well-executed measurement strategy is the scaffolding upon which effective clinical quality improvement occurs.
Mini Case Study On an icy morning in 2008, the body of an 89-year-old woman was found on the roof of a Pittsburgh hospital. The elderly woman, who had a history of dementia, died after she wan - dered away from her hospital room and onto the facility’s roof through an unlocked door on a night when temperatures dropped into the 20s. Her death sparked a review by the state health department, resulted in a lawsuit brought by her son, and prompted major changes in hospital procedures.
In addition, the incident was a wake-up call to other hospitals regarding the need to improve the safety of vulnerable patients. Lessons learned in Pittsburgh could help other hospitals fin81226_06_c06_149-176.indd 172 10/30/14 7:32 PM Summary & Resources struggling to keep patients with Alzheimer’s disease and dementia, who sometimes have a habit of wandering, safe in their facilities. About 25% of older adults who are hospital - ized have some sort of dementia, and hospitalization brings a change of environment that often leads to increased confusion, agitation, and behavioral problems in that population (Alzheimer’s Association, 2007).
What can a hospital do to prevent such a tragedy from happening? This problem requires a team approach. In fact, a quality improvement team should include leaders from the secu - rity and safety departments, as well as clinical staff. Team members must be knowledgeable about the problem and able to come up with solutions. Important stakeholders should not be excluded, but if there are too many team members, it is difficult for the committee to complete its job.
The quality team can use hospital data to determine what percentage of patients is elderly and, of those, how many have Alzheimer’s and dementia. They can ask nursing staff if wan - dering patients are a problem on hospital floors and ask the security department if staff have responded to reports of wandering patients who have gotten lost in the facility.
What can the hospital do to keep these patients safe? The quality improvement team can focus on a number of possible actions to help protect dementia patients. First, clinical staff members must complete patient assessments to determine if a patient is at risk for wan - dering. If the patient came from a nursing home, did those staff note that he or she had a habit of wandering? Perhaps geriatric or psychiatric nurses can assess patients in question to determine who is most at risk. Then clinical staff must create a plan of care for these patients and decide what interventions are appropriate. The assessment is the responsibil - ity of the nursing department.
The hospital can also educate staff members to report any cases of wandering to supervi - sors and document instances in the patient’s health record. Nursing staff can create a plan for monitoring patients at risk and take additional measures to ensure their safety. When admitted to a new setting, dementia patients may not recognize where they are. They may attempt to escape from the facility or try to go home. They can get lost inside or outside the hospital. Once lost, they are in danger of injury or death from falls, accidents, and exposure.
The hospital can create a plan to provide more supervision to these patients. All employees interacting with a patient should be aware of the risk. Staff members can conduct more fre - quent checks on these patients and, when possible, place the patient in a room that allows for maximum staff surveillance, such as near the nurses’ station. The hospital may consider hiring sitters, employees who are paid to stay outside the rooms of wandering patients.
Wandering patients are also a security issue, and security staff members may do a patient watch in an emergency until a permanent sitter is found. Staff members must be educated to not leave patients at risk alone, such as while waiting for x-rays or tests.
Hospitals can also create lost-person protocols, so they have a search process to rapidly locate missing patients. The quality committee may decide to create an emergency code that can be announced throughout a facility, alerting every available staff member, including security, nursing, maintenance, and housekeeping, to assist in the search. Security should notify police and designate someone to notify the family of a missing patient. fin81226_06_c06_149-176.indd 173 10/30/14 7:32 PM Summary & Resources Some hospitals have adopted electronic monitoring systems for dementia patients, combat - ing wandering by placing an electronic bracelet on them. Hospitals can also provide a differ - ent color hospital gown for patients are risk for wandering, making them easier to identify if they leave the hospital unit.
Facility maintenance and safety also play a role in protecting these patients. Doors should be locked and secured at all times and broken locks must be fixed immediately. Security and maintenance personnel can inspect door locks leading to mechanical areas and roofs on a regular basis. Emergency plans and procedures for handling missing patients should be up to date.
If a patient goes missing, the quality team must critique what went wrong. Where are the security gaps? Did the lost-person protocol work as planned?
Discussion Questions 1. Which key stakeholders should be included in the quality improvement project to protect dementia patients from wandering? What is the measurement strategy for this project? 2. Are there any important considerations that have been excluded from the plans for improvement? If so, please describe. 3. What additional quality measures would be useful for this project? Classify each as a structure, process, outcome, or balancing measure. Key Terms balancing measures Measures intended to identify an unintended consequence or harm that results from a change in care delivery.
existing quality measures Data that are already routinely generated in the process of patient care. Examples include administra - tive claims data or clinical data (i.e., labora - tory testing results).
inter-rater reliability An assessment tool that does not change when employed by different individuals, assuming that the measured process or phenomenon does not change.
intra-rater reliability refers to the same results achieved by the same researcher when measuring and re-measuring a process. measurement The process of quantifying or otherwise describing a phenomenon; it refers to describing (usually quantitatively) healthcare delivery and outcomes.
measurement strategy A plan that describes the information needs of a project and how those needs will be met during the project’s execution.
new quality measures Data that are not currently collected in the process of care delivery but instead require additional effort to specifically identify and record outcomes or processes of care.
outcome metrics Relating to the change in or final state of health of a patient as a result of a healthcare intervention. fin81226_06_c06_149-176.indd 174 10/30/14 7:32 PM Summary & Resources Critical Thinking Questions 1. Describe key constituencies to consider when designing the measurement strategy for a quality improvement project. 2. Quality improvement leaders must balance different and potentially competing needs for information when planning an improvement project. How might collecting information for quality improvement impact the cost of care delivery in a clinic or hospital? 3. Describe the concept of risk adjustment. When is risk adjustment important in assessing quality of care? 4. Patients evaluated in the emergency department are often admitted to the hospital for continuing care of their ailments. Often there is a significant delay between when patients are admitted and when they arrive in their hospital rooms. You are tasked with designing a sampling strategy for the project. Describe factors (e.g., time of day) that would influence your sampling strategy. Suggested Websites • Commonwealth Fund:
ht tp://w w w.commonwealthfund.org The Commonwealth Fund is a private foundation that promotes high-performing healthcare systems. Pareto principle Refers to the phenom - enon that in quality improvement projects, 80% of necessary information is often iden - tified through a small proportion (20%) of the available measures.
parsimony Use of the fewest resources or measures necessary to achieve a given aim for a quality improvement project.
process metrics Measurements relating to the activities of delivering healthcare or the interactions between patients and providers.
risk adjustment A statistical procedure that seeks to balance differences in patient characteristics (i.e., age or health conditions) when comparing outcomes between health - care facilities or providers.
reliability How stable a measure is when the item being measured does not change. responsiveness How a measure changes with an underlying change in the item being measured.
sample size The number of observa - tions (i.e., patients) used to measure a phenomenon.
sampling A process to increase the effi - ciency of measurement by observing only a portion of all cases to measure a phenomenon.
structure metrics Measures that relate to the characteristics of the setting where care occurs (e.g., resources and facilities).
time lag The time between when an event is measured and when the information becomes available to inform a constituency.
validity Whether an indicator actually mea - sures what it purports to measure. fin81226_06_c06_149-176.indd 175 10/30/14 7:32 PM Summary & Resources • Institute for Healthcare Improvement – Measures:
ht tp://w w w.ihi.org/knowledge/Pages/Measures/default.aspx Links to measurement resources created by the Institute for Healthcare Improvement. • Leapfrog Group:
ht tp://w w w.leapfroggroup.org A large group of employers that have organized to help improve healthcare quality. • National Quality Measures Clearinghouse:
ht tp://w w w.qualit ymeasures.ahrq.gov/index.aspx A central resource for quality measures sponsored by the Agency for Healthcare Research and Quality. • Why Not the Best?:
ht tp://w w w.whynot thebest.org A resource for tracking performance on measures of healthcare quality that is spon - sored by the Commonwealth Fund. fin81226_06_c06_149-176.indd 176 10/30/14 7:32 PM
