HSS Study: Doctor Rating Sites Are Inconsistent

A new study from Hospital for Special Surgery (HSS) in New York, recently presented at the American Academy of Orthopaedic Surgeons Annual Meeting, has found that while it pays to shop around, you might not be able to trust the results. The researchers, led by Anil Ranawat, M.D., senior investigator and sports medicine surgeon at HSS, determined that when comparing three physician rating websites, the results were inconsistent when it came to individual sports medicine doctors.

“Consumer-driven healthcare and an increasing emphasis on quality metrics have encouraged patient engagement in the rating of healthcare. As such, online physician rating websites have become mainstream and may play a potential role in future healthcare policy,” said co-author Benedict Nwachukwu, M.D., M.B.A., in the March 14, 2017 news release.

“Although it is debatable whether these websites in their current form truly capture patient satisfaction and objectively evaluate the delivery of care, they represent a potential tool for both payors and healthcare systems to gauge how surgeons are assessed by their patients,” said Anil Ranawat, M.D., senior investigator and sports medicine surgeon at HSS. “Historically, three key qualities—affability, availability and ability, known as the ‘three As’—have been suggested to promote a successful surgical career and favorable interactions with patients.”

Using the online member directory of The American Orthopaedic Society for Sports Medicine (AOSSM), the researchers compiled data on years in practice, location, academic affiliation and ratings for each sports medicine surgeon on the three websites. Patients’ written comments were categorized as relating to surgeon competence, affability and the process of care delivery.

According to the press release, “Investigators discovered a low degree of correlation in ratings for individual surgeons on the different websites, an important finding that had not been previously demonstrated, according to Dr. Nwachukwu…being female was a significant predictor of higher ratings on Healthgrades. Surgeons with an academic affiliation were also more likely to receive higher overall ratings…A physician’s online and social media presence, including Facebook, Twitter and possession of a personalized website, did not influence the strength of ratings on Healthgrades, Vitals or Ratemds. Surprisingly, across all three websites, an increased number of years in practice generally led to lower ratings.”

“Surgeons with the highest and lowest ratings were significantly more likely to receive comments about their competence or affability,” said Dr. Nwachukwu. “As such, it appears that even in the modern era, and with the adoption of online rating mechanisms, the traditional three As of ‘availability, affability and ability’ still hold sway.”

“The variation in ratings among the sites may be explained by an insufficiency of reviews necessary to appropriately grade a surgeon, according to the researchers, who noted that the number of reviewers needed to improve reliability is unclear and may warrant further investigation. The researchers also indicated that the study was somewhat limited by the data available—online surgeon reviews may not represent an accurate appraisal of surgeon quality. In addition, it’s not possible to determine if the ratings were posted by actual patients. It was beyond the scope of the study to determine the veracity of the reviews.”

Dr. Ranawat told OTW, “There have been studies in other journals showing that physician rating websites have a lot of inaccuracies and limited scientific validity. We wanted to put these websites up to the scientific process and show, as most orthopedics surgeons assumed, that there is limited scientific and social credibility to them.”