Today at the office, we had a discussion about the distribution of grades on review websites.
You see, common knowledge is that the distribution of reviews take the shape of the letter U. That means: many people give negative reviews, many people give positive reviews, and less people give reviews in between. That's the distribution you usually see when you go to places like the App Store. Lots of 1:s and 5:s. Some people love your app, some people hate it. You've probably seen the pattern yourself.
However, I felt this agreed-upon knowledge would be false when it comes to social recommendation sites, such as Tripbirds. Somehow, I just felt people on a social recommendation sites would behave differently.
To get some data to support my case I made a quick analysis of all the reviews from my Swedish restaurant review site, Restaurangkartan. 13.000 reviews on a scale of one to five stars. And then - a very interesting pattern emerged.
The dark red line is people who just gave one review. That's the people who stop by the site once, write a review, and then never return. For these people, there was a clear U-shape distribution.
Then I looked at people who gave more than one review. That's when things got interesting. Quickly, a very super-clear pattern emerged. There's a straight correlation between the number of reviews someone's written and the distribution of those reviews.
As a matter of fact for people who write many reviews, the distribution is inverted. The shape looks not like a U, but rather like an A.
The more reviews people write, the less likely those reviews are to be either 1:s or 5:s.
So what's the conclusion? Well, it's about trust. On my restaurant site, when someone has written more than 10 reviews... that's when I start trusting them. If someone has written fewer, chances are that they are not really trustworthy. The worst example is when you will find restaurant owners reviewing their own places, and writing bad reviews for their competitors. That's actually quite common. But there are also many other reasons why you can't really trust people with few reviews.
The real power of social recommendations is trust. You might not always agree with your friends but at least you can trust their intentions. This is actually the reason I wanted to leave the "web 2.0" and "crowdsourcing" mentality behind, and focus on social. It's much more powerful. That's why I'm so excited about Tripbirds.
And let's hope the U-shape distribution was just a historical parenthesis on the web. :)
Posts
6 comments :
Based on a quick look, it seems like the people that give more grade tend to give a higher grade on avarage, this ought be very interesting for buisnes owners.
Have you concidered weighting the ratings of more experienced users higher? A kind of meritocracy that will also tend to minimize the impact of scammers voting only for their personla benefits without any commitment to to the community.
Another hypothesis (pushing things futher):
The U-shaped rating is an reflection of a rigged system where votes are casted based on slef-interest/ rather than giving honest recommendations?
@Kristoffer. On my restaurant site I actually already have a pretty advanced algorithm that weighs different reviews / reviewers. But that's not really what it's about.
It's more about WHY people with few reviews show such a different pattern. To some extent, it's about votes in their self-interest.
But I also think there's ALSO something else in play.
What information are you interested in sharing with STRANGERS? Usually it's when you have a strong opinion about something.
But if you're sharing with FRIENDS, or if you're trying to build a "reputation" or a "profile", you have an incentive to give a much more balanced view. Which is actually also closer to the truth.
The interesting thing with social recommendations is you already HAVE a reputation among your friends, and there's a much bigger incentive to give balanced reviews.
That's an interesting point. It's very true that we adapt our communication to whom we are sending it. And the approach is very different when it's sent to someone we don't know (think trolling)
Another, not quite as exciting factor, I think, is that it requiers a pretty strong opinon in order to find a new webpage and register and vote.
A thing I would be very curious about is to weather the people who now represent the long-term members were following the same pattern when they started voting aswell. Based on the current hypothesis, that would answer the question weather people are intrinsically longterm users or if they are converted down the line.
Intressant. TechCrunch och YouTube skrev lite om detta för några år sedan http://techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realization-its-ratings-are-useless/
Actually, I don't think it's about social at all. It's about personal investment.
If someone hates or loves something, they are not less likely to give extreme ratings just because their friends can see them. Rather the opposite, I would say.
However, when someone has invested a lot of time in the rating system they have a more clear view of what the different ratings imply and they also feel a responsibility that the grades gives a good impression of the actual quality, thus taking more time to adjust their own ratings based on much more than just one situation.
Post a Comment