A month ago, we launched a second pilot study of code review for scientific software as a follow-on to the first study reported here in November. Where the first round had professional software developers review the software associated with papers published in PLOS Computational Biology, this round paired experienced scientific programmers with small groups of less-experienced ones to explore ways of transferring the practice itself.
Ten groups ranging in size from a couple of people to half a dozen are taking part in this study. In interviews and early discussions, they identified four main reasons why they think they ought to do code review. The first is rigor: scientists want to get the right answer, and they want other scientists and their future selves to be able to reproduce that answer. Most recognize that correctness and reproducibility aren’t the same thing, i.e., that it’s possible for a wrong answer to be 100% reproducible. If given the choice, most would probably choose the former over the latter, but the goal is to achieve both, reliably.
Many are also acutely aware of how their understanding of their own code dissipates over time and of the cost downstream of either recovering understanding or re-coding. As once said, “one month later, it’s too much work to impose order… six months later, you just throw it away and start from scratch”. They may not know how code review will help with that, but they hope that it will.
The second motivation is reusability. Scientists suspect that they spend a lot of time reinventing the wheel, and that their wheels are continually being reinvented by colleagues. They believe (or at least hope) that doing code reviews will help them create software that others will be able to use. Some hope that, as a result, results may be aggregated more easily. However, they also recognize that packaging and distribution for re-use will be relatively easy, compared to shifting their colleagues to a culture of re-use. Part of what they hope to learn through this study is what the actual costs and benefits of making something reusable are, so that they can tell whether that effort will be worthwhile.
A third reason for taking part in this study is that it’s an opportunity for collaboration: they can use code review as an excuse for talking about code with their colleagues. More than one has said that these conversations simply don’t happen right now, and that as a result, a lot of sharing doesn’t happen either. Some believe that coding as part of a collaborative team is a “better way to work”, both for the code and for the science, because the collaboration will foster better testing, will encourage scientists to produce code that is easier to understand and share, and will make team members more aware of each other’s research. It’s not enough to develop rigorous and reusable code — achieving the full benefits of re-use requires good social mechanisms for promoting awareness and sharing.
And finally, many participants are using the study as an opportunity to learn about more than just code review, e.g., “I have a lot to learn about better coding”. They all feel that there must be a better way to build programs than what they’re doing right now; taking part in this study is, for them, a way to get mentoring from someone who can answer questions they don’t know to ask. They also want to “share best practice with other scientists”.
One thing that we didn’t hear in our interviews is people saying that they wanted to do code reviews in order to learn about the science embodied in that code. Rightly or wrongly, scientists seem to feel that “what it does” can be learned in others ways. We’ll talk about a few other surprises in our next posts, which will look at what mentors think the payoff from code review is.
