Here's something I was thinking about vote-based sites like StackOverflow or Digg: can we consider the aggregate sum of votes an accurate representation of people's opinions?
I think that we can't, for a few reasons:
- Different people use rating systems differently: As was proposed in the Netflix contest earlier this year, some users may reserve a "5 star" rating only for all-time classics whereas some other may rate any funny movie a "5 star". Likewise, some may upvote anything that doesn't offend them, whereas others may only upvote things with which they strongly connect with (by personal experience, for example).
- Collective thought bias: most people are more likely to agree with the majority than not. This can extend beyond the scope of a single voting decision: the characteristics of future user-generated content are also usually affected by the outcome of past elections. This shift can cause users with differing opinions to avoid participation or leave a site altogether.
- The incentives are usually not balanced: there's usually a strong preset bias towards taking action on one of the options (i.e. "upvote if this speaks to you" vs. "if you're not compelled/interested by this, you have to choose between acting (downvoting) or not acting (not voting)").
- Voting systems are often subject to momentum: this is particularly true of news sites - users are encouraged to vote as soon as possible (often before they are able to grasp the full concept, or before they are able to figure out that a certain source is misleading). Voting after a certain time threshold often has little or no social value.
I'm not saying voting-based systems are useless in websites - they can provide a quick insight into whether something is likable or useful. What I'm pondering is this: what's the point of being a sheep?