The “Exact Instructions” challenge

A hands-on introduction to challenges of practical reproducibility.

The Exact Instructions challenge is inspired by the hilarious video by Josh Darnit and his two kids, Johnna and Evan, who take turns trying to help their father put together a peanut butter and jelly sandwich. Josh follows instructions to the letter, diligently.

Purpose and aims

The purpose of the exercise to have participants experience the difference between what they express and what others understand, a fundamental tenet of reproducibility. It is a great introduction to issues of reproducibility for complete beginners and an awesome ice-breaker!

Participants take part either individually or in groups. They write down a set of instructions to accomplish a task, like putting together a local desert. Instructions are then read and enacted literally by the moderator who attempts to accomplish the task. The moderator can either be the trainer or a different group. Participants will mess up, and have to go through several rounds. At the end of the session, the trainer can moderate a discussion about the practice of scientific communication.

Learning outcomes

At the end of the session, participants will be able to:

• Identify, describe and illustrate issues of reproducibility related to the communication of scientific findings;
• Reflect on and improve their own practice of communication;
• … hopefully discover, cook and enjoy the local cuisine from a distanced country!

Preparation and what to do, on the day

1. Choose a task: something simple can be reproducing a drawing from written instructions, but it can be more fun to put together a (simple!) local delicatessen, like a sandwich.
2. Decide who gets to enact the instructions; it can either be you as a Trainer, or another group of participants.
3. The moderator has the task of going through written instructions literally, until it is jointly agreed that the task has been successful.
4. The convenor can then seize the opportunity to moderate a discussion about the similarities and challenges in scientific reporting.

Recommended reading

• Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, et al. ‘Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams’. Nature, 20 May 2020, 1–7. https://doi.org/10/ggwrvt.
• Carp, Joshua. ‘The Secret Lives of Experiments: Methods Reporting in the FMRI Literature’. NeuroImage 63, no. 1 (15 October 2012): 289–300. https://doi.org/10.1016/j.neuroimage.2012.07.004.
• Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. ‘False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant’. Psychological Science 22, no. 11 (1 November 2011): 1359–66. https://doi.org/10/bxbw3c.