You Cannot Have Too Many Checks

Kevin Arceneaux, Temple University
for Lessons Learned the Hard Way in The Experimental Political Scientist, Spring 2021

When I inaugurated the Behavioral Foundations Lab with its initial study a few years back, I had learned a lot from talking to other researchers with labs. Their recurring advice was, “have a lab protocol.” So, I developed one — based in part on my intuitions and based in part on what others had done. One of the lessons that I did not fully take to heart, though, was the need to have daily checks embedded in the protocol along with the need to fully plan everything from the design phase all the way to the data analysis phase. I hope these pieces of advice seem obvious to everyone else. I wish they had been to me.

All experiments have many moving parts — be they in the lab, online, or in the field. Once a protocol is put into place, it is easy to slide into a kind of autopilot. And you do want some degree of autopilot. Many of the studies that we run in the Behavioral Foundations Lab collect physiological data. Before a study begins, we practice multiple times placing electrodes on participants and going through the steps of the study until it becomes muscle memory. If a participant coughs or someone inadvertently slams a door while we are collecting physiological data, we immediately note this in the log book and connect the note to the data we collect for this subject. But even this care to detail cannot prevent a cable for coming loose, causing us to lose physiological data, as we learned in one study after failing to collect the skin conductance data for nearly a dozen participants before we caught the mistake. Now, we check the connectivity of wires before we begin the study for each participant. We also now regularly check data output, as a programming bug may slip passed detection in the initial set of participants only to emerge as a problem later. We learned this the hard way, too. Thankfully, we were able to recover the structure of the data through other redundancies in the programming code, but it showed that we cannot assume that just because something appears to work once or twice that it will always work. Murphy’s law is true when running experiments. Whatever can go wrong, will. It is not possible to think of every possible thing that could go wrong, but building in regular checks will help you catch and correct them before they turn from a small hiccup into a massive headache.

In addition to developing and testing a protocol for the experiment itself, it is also important to develop a protocol for the data analysis portion as well. I must admit that until I began the habit preregistering studies, I was not very good at doing this. I would sketch out the structure of the data and data analysis in my mind. I have plenty of stories about how this didn’t always turn out well: that experiment with a missing control group; that moderator that I forgot to measure; obvious data analyses that did not occur to me until I started working with the data. These are bad habits that preregistration helps us avoid, but it does not fully solve the problem either, as I have learned.

My first attempts at writing a pre-analysis plan were like my first attempts at riding a bike: comically awkward and not very effective. Part of the problem is that I did not devote enough time to the endeavor. I did better than mentally sketch out the data analysis, but honestly not that much better. Once I collected the data, I sat before my computer saying to myself, “Oh, why didn’t I think to pre-register X, Y, or Z?” Some of these ideas were not obvious at the start — that’s always going to be the case because no human being can anticipate everything — but some of them were obvious and they would have occurred to me if I had spent more time crafting the pre-analysis plan. After some trial and error, I now write out the empirical model for my analyses, simulate the data, and go through the act of coding up data and “running” models. I have caught mistakes in my thinking, in the measures, and in the design by going through this approach. Software applications, like Declare Design (https://declaredesign.org/), can help you think along.