One month a provider could rate in the 80th percentile and the next in the 30th, showing that patient experience data is often unreliable.
Hospital emergency departments are working with insufficient patient experience data that does not accurately portray all patient sentiments, according to a recent data analysis out of the George Washington University School of Medicine & Health Sciences.
Patient experience data has become a boon in healthcare, and for good reason. Healthcare professionals are working to provide a positive experience to create better customer satisfaction and to ensure their market share in the medical space.
For hospital administrators, patient experience data is essential to set organizational improvement efforts. If patient experience scores show that patients are unhappy with emergency department wait times, organization leaders might implement a more streamlined admissions process, for example.
But none of those efforts can be truly fruitful if healthcare professionals are working with subpar data, an issue recently identified by Jesse Pines, MD, MBA, director of the Center for Healthcare Innovation and Policy Research and professor of emergency medicine at the GW School of Medicine and Health Sciences.
“The concept of measuring patient experience and rewarding providers who deliver a better experience is absolutely right on,” Pines said in a statement. “No one argues with that. Yet what we found is that the data currently being gathered is not particularly reliable nor valid.”
Pines and his research team assessed Press Ganey patient experience surveys from a national emergency medicine group from between 2012 and 2015 to analyze how valid this data was. Overall, the team looked at 42 medical institutions’ and 242 clinicians’ patient satisfaction data.
The team found notable variability amongst both individual provider scores and overall facility scores, although the variability was more pronounced in individual provider scores.
From month-to-month, about 32 percent of providers changed at least 20 percentile points. This means in one month a physician could be in the 50th percentile and the following the physician could be rated in the 70th percentile.
For facilities, about 41 percent experienced month-to-month variability of 10 percentile points.
The scores should have been more consistent than they were, said senior author Arvind Venkat, MD, chair of research at US Acute Care Solutions.
“Presumably, if a physician produces a particular experience for his or her patients, then scores should be relatively stable over time,” Venkat explained. “But from month-to-month, physician scores bounced around tremendously. In some cases, a physician was rated in the 20th percentile one month, then 80th percentile the next month, then in the 30th percentile. Facility scores also bounced around, but less so.”
The research team concluded that low response rate was partly to blame for the variable patient experience data. Most hospitals only see between a 3 and 16 percent response rate, and most of those responses are usually either very enthusiastic or very critical.
“Imagine you conduct a survey, and only the very happy and very unhappy return their surveys,” Venkat offered. “What you get is a very biased sample. That makes it difficult to come to any meaningful conclusions from the data.”
Despite the score variability, the researchers were able to glean some common predictors of a good patient experience score for hospitals. Departments with residency programs, higher volume of older, white, male patients not on Medicaid, lower overall patient volume, lower need for physician night coverage, and shorter lengths of stay usually resulted in better experience scores.
Individual clinicians saw better scores when they were younger, had participated in patient satisfaction training, treated more commercially insured patients, had higher CT/MRI use, worked during lower volume times, and worked fewer night shifts.
The very fact that the researchers could determine some commonalities amongst top performers did confirm at least some data validity in patient satisfaction scores. However, more work needs to be done to make patient experience surveying more inclusive and to draw a more complete data set.
“The voice of the patient is increasingly important in healthcare, particularly today with rising costs of care and increasing out-of-pocket costs for our patients,” Pines concluded. “What is clear from our study is there needs to be a better process to measure, capture, and report patient experience data.”
Source: Thinkstock
UPDATED 12/21: In response to the report, Press Ganey has issued the following statement via email:
According to Press Ganey, the variability is a function of the volume of data used in the analyses for monthly estimates and monthly provider estimates of performance. “Roughly half of all patients enter the hospital through the Emergency Department. It’s a critical setting to collect deep patient experience and evaluate care,” said Deirdre Mylod, Senior Vice President of Analytics/Solutions for Press Ganey and Executive Director of the Institute for Innovation. “We understand the challenges associated with capturing patient experience data in the ED. However, without adequate sample sizes, variability like that observed by the researchers is likely to occur. For this reason, both CMS and Press Ganey have minimum standards for numbers of surveys, which were not considered in this research. This study demonstrates why capturing the voice of every patient is critical in order to gain a true picture of the patient experience.”
