Research pair suggest crowd sourcing could improve social science experiment results

(Phys.org)—Social science experiments do not have the best track records—oftentimes, studies are conducted, followed by wild claims which are then either contested or outright refuted by colleagues. There might be a better way Raphael Silberzahn and Eric Uhlmann suggest in an essay piece published in the journal Nature—crowdsourcing.

The two social scientists have experience in this area, together they published the results of a study they conducted a couple of years ago where they found that German people with royal sounding last names, tended to become business managers. A colleague asked for their data and after conducting an analysis of his own, subsequently refuted the claims made by the pair. This led the pair and some of their colleagues to look for a way to improve social science experiment results—they came up with crowd sourcing.

To test their idea, the researchers picked a topic—do soccer refs give red cards to dark skinned people more often than light skinned people? To find out, they consulted with a sports data gathering company that pulled statistics from actual matches—the data was then given to 29 teams of researchers who volunteered to give their time to the project. Each team was asked to use the same data given to them to make a determination, with no other constraints. Each team sent back their results along with the methods they used to obtain them. Each of the research team was then sent a document outlining all the methodologies used by all the teams, but without the results and were asked to rate them and to provide feedback. Each research team received all feedback and ratings and a chance to change their initial findings. Then, finally, all of the teams were invited to participate in an e-mail exchange to discus methodologies and results. Teams were then given another chance to modify their own results and then a final document was drawn up that gave a group consensus on the original question.

The result was that 20 of the research teams found that darker skin color did indeed lead to more red cards, but the researchers varied widely on how prevalent it was—the group consensus was that it was 1.3 times more likely. And while that number is significant all by itself, what is really important Silberzahn and Uhlmann write, is that the experiment revealed that using crowd sourcing resulted in a more hesitant response from researchers, which they believe led to a reduced incentive for publishing flashy results.

© 2015 Phys.org