One of the cardinal points of open science is collaboration; effectively working together is the engine that open science derives speed, accuracy and impact from beyond what we achieve working alone. But collaboration contains several moving parts; things like trust, rapport, and communication. Without these, no collaboration can succeed. How then do we build these things around collaborating on scientific coding? Code review not only gives us a vehicle to discuss our code and build collaborative rapport, but a framework to bolster code’s quality and most importantly, the quality of the science it supports.

Most researchers are unfamiliar with code review; mechanically, it is as it sounds – examining code to ensure it meets standards of quality and relevance to the project at hand. But just by asking what standards to review against, we already have to start supplying clear answers to some key questions: what, in our opinions, makes a piece of code ‘good’? What are the goals of the coding project in question, how can we communicate those goals to our collaborators, and how can they provide code that answers those goals in a way we find clear, compelling and useful? All critique requires goal posts, and where we define and locate those goal posts uncovers something much deeper: our values. Our fundamental value for scientific code has to be accuracy; but we can also build in reproducibility, reuse, and reliability, by setting our standards well.

But the question remains: what will it take to introduce code review in the sciences, when it isn’t part of the current conversation? Earlier this year, the Science Lab participated in the second round of a study on code review for scientists, lead by Marian Petre of the Open University. The results from both rounds of the study are summarized in this paper, and the differences were remarkable.

To summarize very briefly: in the first study, code was reviewed after the studies it was written for were complete, by reviewers who had never met the team that created it. In the second, code was reviewed as it was being written, by a coder-scientist in the same or cognate discipline as the scientists producing the code, with regular conversations between code reviewers and creators.

The differences were stark. In the post-hoc review, reviewers felt disconnected from the process of creating the code; they had little rapport with the creators’ intentions, often lacked the data or context to be able to run the code and see it in operation, and were as a result only able to perform very superficial reviews – the equivalent of checking the spelling on a document whose meaning is out of reach. But when reviews were conducted regularly and in conversation with the creators as the code developed, the impact was large and immediate; by creating a custom where participants were able to build a relationship and a deeper understanding of the project, its goals, and the standards being applied to it, the second study found that “[m]any of the scientists reported benefits even during the study: code improvements, better use of facilities provided in development tools, and better documentation practices spreading in the lab.” The simple act of discussing one’s code with someone prepared to build a rapport with the author and their project improved its quality, made the author a more deft practitioner and radiated clearer communication practices beyond the project at hand.

The seven recommendations Petre makes at the end of the study all speak to this power of rapport building; she encourages practices like having clear goals, conducting regular and frank conversations about the project, and maintaining clear and open access to the project – all ingredients in promoting understanding and clarity of thought among peers. This work has given us a clear picture of the ‘when’ (right now & regularly) and ‘who’ (you & your collaborators) of code review, building on the natural ‘why’ of seeking more accurate, reusable code. What I’d like to address next, is the ‘how’ – the study concludes with some action items for further work, and first among them is the construction of a curriculum for scientists who would like to do code review.

Work began on this project a couple of months ago, but I’d like to round out 2014 by completing a first cut at walking scientists through the process of setting up a formal code review process, from defining their expectations for a project, to circumscribing contributions, to analyzing and discussing those contributions in a way that builds rapport, reinforces quality, and doesn’t overburden our already busy research schedules. I look forward to your comments there, and to exploring this material with you all in the new year.

Image Credit: Cropped from Valery Kenski [CC BY 2.0]