When Implementing Field Experiments, 90% of Life is Just Showing Up

Don Green, Columbia University
for Lessons Learned the Hard Way in The Experimental Political Scientist, Spring 2021

I have long wondered whether it is possible to craft an intervention of some sort that would increase support for civil liberties, such as free speech or the right to assemble.  I therefore jumped at the chance to evaluate the effectiveness of the Bill of Rights Institute’s high school curriculum focusing on civil liberties.  The central hypothesis: a curriculum that imparts a strong understanding of constitutional rights (expression, assembly, due process, etc.) increases support for these rights among high school students.  This hypothesis is suggested by abundant correlational evidence indicating that those who are most knowledgeable about constitutional rights are also the most supportive of protecting civil liberties and extending constitutional protections to unpopular groups.

So all we had to do is recruit high school teachers and convince them to implement the Bill of Rights Institute’s curriculum in a random subset of their civics classes.  Teachers who were teaching two civics classes over the course of a year, for example, would allow us to flip a coin and put one of the classes in the treatment group; the other they would teach as usual.  Easy, right?

The two greatest practical challenges in conducting field experiments are noncompliance and attrition.  In this case, noncompliance occurs when classes assigned to the treatment group do not in fact administer the treatment.  Attrition occurs when we are unable to obtain outcome measures for students in some of the classes.  It took three attempts to overcome debilitating problems of noncompliance and attrition.

The first study derailed when it became apparent that teachers who agreed to teach the experimental curriculum were not in fact doing so.  Foolishly, we neglected to monitor each of the classrooms and assumed that teachers would slot in the experimental curriculum during the semester.  Although teachers sometimes gave us their assurances that things were on track, without observers dropping in from time to time to monitor classrooms, we were never entirely sure whether the treatment was administered in full.  The more we talked with teachers and students about their use of the curriculum, the more skeptical we became.  So we pulled the plug on that study and tried again.

The second time we were careful to have observers periodically drop into treatment and control classrooms.  Our presence clearly helped ensure that that the curricula in the two groups was markedly different in terms of its focus on the Bill of Rights.  Unfortunately, at the end of the semester, we dropped off our end-line surveys with teachers and asked them to have students complete the questionnaire in class.  That was a mistake.  Teachers often neglected to distribute our surveys, and, when they did, the number of surveys that were returned was far smaller than the number of students in the class.  We again pulled the plug on the study and tried again.

This time The Three Little Pigs decided to make their house out of bricks so that it would withstand both noncompliance and attrition.  We hired a charismatic and highly organized former teacher to run the study, and she attracted the cooperation of a new cohort of teachers.  She oversaw a team of observers who regularly visited treatment and control classrooms and (blind to the hypothesis) took notes on the pedagogy.  She oversaw a different team of blinded survey enumerators who, on the appointed day, administered pencil-and-paper surveys in all classes.  She even returned a year later to conduct another round of interviews with students to gauge the curriculum’s long-term effects.  The methodological lesson?  Do not expect a field experiment to operate on auto-pilot.  If you want things to run smoothly (or at all), stay in constant contact with your research partner and be present at key moments of implementation.  The substantive lesson: a curriculum focused on the Bill of Rights indeed increased students’ knowledge about the provisions of the Constitution; it did not, however, increase their support for civil liberties.[1]

[1] Green, Donald P., Peter M. Aronow, Daniel E. Bergan, Pamela Greene, Celia Paris, and Beth I. Weinberger. “Does knowledge of constitutional principles increase support for civil liberties? Results from a randomized field experiment.” The Journal of Politics 73, no. 2 (2011): 463-476.