Lessons Learned the Hard Way (Spring 2021)

Introduction: The only thing you are guaranteed to learn is a new way experiments can go wrong

by: Christopher B. Mann, section editor

Things go wrong in experiments. But you wouldn’t know it from reading published articles that all report successfully implemented experiments. And the absence of sharing is a costly problem for the discipline: different scholars have to learn the same lessons the hard way, and thus waste time, energy, and scarce resources that could otherwise move research forward. These lessons are more basic than issues with null results, unexpected results, or deviations from pre-registered designs. They include design mistakes and bad luck in planning or implementation leading to experiments that fail. These situations can be politely called severe internal validity problems, although that is a rather grand way of describing what are simply screw ups. Not substantive or theoretical failure, but “oh, crap” failure.

This new (and hopefully ongoing) section of the newsletter is intended as a forum to share stories of these screwups and the lessons learned by those experiencing them. The primary purpose is to help avoid the costly waste of multiple people learning lessons the hard way. It may also help reassure scholars new (and not so new) to experiments that it is normal for things to go wrong. Indeed, the title of this introduction is what my graduate advisor, Alan Gerber, said to console me when one of my early experiments went off-track.

Having conducted dozens of field and survey experiments over the last fifteen years, I have made a number of embarrassing mistakes. Numerous conversations with colleagues suggest that I am far from alone. Tell a story of an experiment gone wrong to a group of experimental political scientists, and you are very likely to get a response of “if you think that is bad, here is what happened in my experiment.” When I solicited “Lessons Learned the Hard Way” from Experimental Research section leaders, each responded that the hardest part would be deciding what snafu to write about among many.  

The first contributions (links below) were solicited from past and present leaders of the Experimental Research section and the editorial team of the Journal of Experimental Political Science. They are eminent scholars who have conducted hundreds of lab, survey, and field experiments and written about and taught many courses on experimental methodology. And they have each had many experiments go off track, in new and different ways. We can all benefit from their willingness to share the lessons they learned the hard way. We hope you’ll be inspired to submit your own lessons learned the hard way for future issues.

Our plan is to have this section be a part of each edition of the newsletter. To facilitate submission, we have created a simple Google Form (https://forms.gle/hqKPYudg38BBVAu6A). The guidelines are simple: describe the lesson you learned in less than 750 words. Good storytelling and even a dash of humor is a plus. You must identify yourself when you submit, but we will publish contributions under pseudonyms to encourage the sharing of truly embarrassing mistakes. Submissions will be selected for inclusion based on the import of the lesson learned, clarity of writing, and an eye to sharing lessons about all types of experiments. Since the newsletter has few space constraints, we will seek to share as many lessons as possible. 

Lessons Learned:

Spring 2021