Go to the EdMedia 2023 Vienna Conference »

The Effort Educational Researchers Should Take to Make Their Studies Replicable

Posted by Jordan H. Reiter on April 3 2015 at 7:03 p.m.

Replication - a rarity in educational research. Can open and social software increase transparency?

Replication is important to science. It helps make science a self-correcting system. Any time a result is surprising, researchers will try to replicate it, to see if the phenomenon is dependable or just random occurrence.

Among different fields in social science, education is the one that is most deeply troubled by the rarity of replication. Only around 0.13 percent of education studies published in top 100 educational journals are replications, according to a research conducted by Matthew Makel and Jonathan Plucker at Duke University in 2012. Some educational researchers may argue that education is an exception in which replication doesn’t play an important role. However, it would be impossible to tell which educational theory is built upon foundation of stone, or which upon the foundation of sand.

The lack of replication does not only make it difficult to tell dependable phenomenon from random occurrence, but also makes it impossible to identify fraud. Jonathan Plucker said education research had not a single fraud accusation in many years, simply because educational researchers failed to replicate each others’ research.

What can we do

The rarity of replication in educational research is partially due to “publish or perish” culture in academics. Many editors of high-end journals stress novelty over replicability, though some of them realize this is a problem. Fortunately, some new journals have been set up to encourage the submission of replication studies very recently.

The other reason for such rarity is complexity of educational research. Educational studies often involve in some unspoken assumption and unlisted important factors. Difference between participants, learning and teaching settings, and other uncontrollable factors all contribute to the difficulty of replication.

For a researcher, to make the study as transparent as possible will definitely help with replication. Detailed description of the study context and participants helps a lot, but the efforts can always go a little bit further. The increasing popularity of social media use and online version control tools can help increase the transparency of many research.

Text data with unified hashtag on social media, like Twitter or Facebook can be easily accessed and collected by any interested researchers. GitHub (one popular online version control tool) makes it easier to share the analysis script and refined data online. GitHub page can even easily generate a professional website for the promotion of your research project. Many educational studies involves activities that require participants to read and write, like studies on cognition, reflection, self-regulated learning or problem-based learning. Many of these activities can be easily migrated from classroom context to online context. For instance, this study is an ongoing research project investigating the effect of a proposed intervention on college students’ goal setting. Participant’s tweets with unified hashtag “edit4020” provide researchers an easy way to access all the text data. Each participant’s public profile can give rich information about the participant him or her self. Different tools that help data collection and text analysis have been developed and applied in other fields, like political science and psychology. For example, this set of scripts focus on data collection from Twitter.

To migrate studies from classroom environment to online environment does not only make the data collection easier, but also make the studies more transparent. The transparency increases the credibility of your finding, and lowers the difficulty of replication.

The post The Effort Educational Researchers Should Take to Make Their Studies Replicable appear

Original Post: http://blog.aace.org/2015/04/03/the-effort-educational-researchers-should-try-to-make-their-studies-replicable/

Comments

Log in to post a a comment in this discussion.