Do you take user ratings into consideration when booking a hotel or shopping online? Many people use ratings and comments on websites like Tripadvisor or Amazon for deciding whether to book or buy something. The reasoning behind this is that such ratings are reliable and accurate in a “wisdom of the crowd” like manner. However, recent research now questions this notion.
Lev Muchnik from the Hebrew University of Jerusalem and Sinan Aral and Sean J. Taylor from NYU Stern School of Business conducted an experimental study on a website on which users could give positive and negative ratings to and comment on articles on a social news aggregation website. They randomly assigned 101,281 comments submitted on the site to three different groups: up-vote, down-vote, or no vote (control).
It turned out that negative ratings were likely to be corrected. However, positively rated comments were more likely to get positive votes and received better final ratings on average than comments without a vote. This herding effect was dependent on the topic (only affected the topics politics, culture and society, and business, but not economics, IT, fun and general news) and the relationship between the voters (friends tended to up-vote their friends’ positive ratings and correct negative ones).
Thus, in positive ratings, it seems to be hard to distinguish between the influence of high quality of the item rated on the one hand and social influence bias on the other hand. A positive rating could thus result from the high quality of the item rated, but just as well from past positive votes. So, as an article in the NewYork Times on this study states it: “Is something popular because it is actually good, or is it popular just because it is popular?”
The original article is in the August volume of the journal Science. There is also a short video explaining the study:
What does this mean? Negative ratings are likely to be accurate once there are a certain number of votes. However, when evaluating positive ratings, we need to be very critical and try to find more information. If there are written reviews of items, it might make sense to not only look at the positive, but also at the negative evaluations and compare both. Also take the relationship you have with the other voters into account. If they are people you like, be even more critical with your own evaluation. Be aware of the social influence bias!