Chapter 4 Software Peer Review, Why? What?
This chapter contains a general intro to our software peer review system for packages, reasons to submit a package, reasons to volunteer as a reviewer, why our reviews are open, and acknowledgements of actors of the system.
If you use our standards/checklists etc. when reviewing software elsewhere, do tell the recipients (e.g. journal editors, students, internal code review) that they came from rOpenSci, and tell us in our public forum, or privately by email.
4.1 What is rOpenSci Software Peer Review?
rOpenSci’s suite of packages is partly contributed by staff members and partly contributed by community members, which means the suite stems from a great diversity of skills and experience of developers. How to ensure quality for the whole set? That’s where software peer review comes into play: packages contributed by the community undergo a transparent, constructive, non adversarial and open review process. For that process relying mostly on volunteer work, associate editors manage the incoming flow and ensure progress of submissions; authors create, submit and improve their package; reviewers, two per submission, examine the software code and user experience. This blog post written by rOpenSci editors is a good introduction to rOpenSci software peer review Other blog posts about review itself and reviewed packages can be find via the “software-peer-review” tag on rOpenSci blog.
You can recognize rOpenSci packages that have been peer-reviewed via a green “peer-reviewed” badge in their README, linking to their reviews (cf this example); and via a blue comment icon near their description on rOpenSci packages page, also linking to the reviews.
Technically, we make the most of GitHub infrastructure: each package review process is an issue in the ropensci/software-review GitHub repository. For instance, click here to read the review thread of the
ropenaq package: the process is an ongoing conversation until acceptance of the package, with two external reviews as important milestones. Furthermore, we use GitHub features such as the use of issue templates (as submission templates), and labelling which we use to track progress of submissions (from editor checks to approval).
4.2 Why submit your package to rOpenSci?
- First, and foremost, we hope you submit your package for review because you value the feedback. We aim to provide useful feedback to package authors and for our review process to be open, non-adversarial, and focused on improving software quality.
- Once aboard, your package will continue to receive support from rOpenSci members. You’ll retain ownership and control of your package, but we can help with ongoing maintenance issues such as those associated with updates to R and dependencies and CRAN policies.
- rOpenSci will promote your package through our webpage, blog, and social media. Packages in our suite also get a documentation website that is automatically built and deployed after each push.
- rOpenSci packages can be cross-listed with other repositories such as CRAN and BioConductor.
- rOpenSci packages that are in scope for the Journal of Open-Source Software and add the necessary accompanying short paper, would, at the discretion of JOSS editors, benefit from a fast-tracked review process.
- If you write one, rOpenSci will promote gitbooks related to your package: the source of such books can be transferred to the
ropensci-booksGitHub organisation for books to be listed at books.ropensci.org.
4.3 Why review packages for rOpenSci?
- As in any peer-review process, we hope you choose to review to give back to the rOpenSci and scientific communities. Our mission to expand access to scientific data and promote a culture of reproducible research is only possible through the volunteer efforts of community members like you.
- Review is a two-way conversation. By reviewing packages, you’ll have the chance to continue to learn development practices from authors and other reviewers.
- The open nature of our review process allows you to network and meet colleagues and collaborators through the review process. Our community is friendly and filled with supportive members expert in R development and many other areas of science and scientific computing.
- To volunteer to be one of our reviewers, fill out this short form providing your contact information and areas of expertise. We are always looking for more reviewers with both general package-writing experience and domain expertise in the fields where packages are used.
4.4 Why are reviews open?
Our reviewing threads are public. Authors, reviewers, and editors all know each other’s identities. The broader community can view or even participate in the conversation as it happens. This provides an incentive to be thorough and provide non-adversarial, constructive reviews. Both authors and reviewers report that they enjoy and learn more from this open and direct exchange. It also has the benefit of building a community. Participants have the opportunity to meaningfully network with new peers, and new collaborations have emerged via ideas spawned during the review process.
We are aware that open systems can have drawbacks. For instance, in traditional academic review, double-blind peer review can increase representation of female authors, suggesting bias in non-blind reviews. It is also possible reviewers are less critical in open review. However, we posit that the openness of the review conversation provides a check on review quality and bias; it’s harder to inject unsupported or subjective comments in public and without the cover of anonymity. Ultimately, we believe that having direct and public communication between authors and reviewers improves quality and fairness of reviews.
Furthermore, authors and reviewers have the ability to contact privately the editors if they have any doubt or question.
4.5 How will users know a package has been reviewed?
- Your package README will feature a peer-review badge linking to the software review thread.
- Your package will get a
docs.ropensci.orgdocs website that you can link from DESCRIPTION.
- Your package repo will be transferred to the rOpenSci organization.
- If reviewers agree to be listed in DESCRIPTION, their metadata will mention the review.
4.6 Editors and reviewers
4.6.1 Associate editors
rOpenSci’s Software Peer Review process is run by:
- Noam Ross, EcoHealth Alliance
- Karthik Ram, rOpenSci
- Maëlle Salmon, rOpenSci
- Anna Krystalli, University of Sheffield RSE
- Melina Vidoni, RMIT University (School of Science)
- Brooke Anderson, Colorado State University
- Mauro Lepore, 2 Degrees Investing Initiative
- Laura DeCicco, USGS
- Julia Gustavsen, SOPHiA GENETICS
We are grateful to the following individuals who have offered up their time and expertise to review packages submitted to rOpenSci.
Sam Albers · Toph Allen · Alison Appling · Zebulun Arendsee · Taylor Arnold · Dean Attali · Mara Averick · Suzan Baert · James Balamuta · Vikram Baliga · David Bapst · Joëlle Barido-Sottani · Cale Basaraba · John Baumgartner · Marcus Beck · Gabe Becker · Jason Becker · Dom Bennett · Kenneth Benoit · Aaron Berdanier · Fred Boehm · Carl Boettiger · Ben Bond-Lamberty · Anne-Sophie Bonnet-Lebrun · Alison Boyer · François Briatte · Eric Brown · Julien Brun · Jenny Bryan · Lorenzo Busetto · Scott Chamberlain · Jennifer Chang · Jorge Cimentada · Chase Clark · Jon Clayden · Will Cornwell · Enrico Crema · Ildiko Czeller · Christophe Dervieux · Amanda Dobbyn · Jasmine Dumas · Remko Duursma · Mark Edmondson · Paul Egeler · Evan Eskew · Manuel Fernandez · Kim Fitter · Rich FitzJohn · Robert Flight · Zachary Foster · Auriel Fournier · Carl Ganz · Duncan Garmonsway · Sharla Gelfand · Monica Gerber · Duncan Gillespie · David Gohel · Laura Graham · Charles Gray · Corinna Gries · Hugo Gruson · Julia Gustavsen · W Kyle Hamilton · Ivan Hanigan · Jeff Hanson · Rayna Harris · Ted Hart · Nujcharee Haswell · Verena Haunschmid · Andrew Heiss · Max Held · Rafael Pilliard Hellwig · Bea Hernandez · Jim Hester · Peter Hickey · Roel M. Hogervorst · Jeff Hollister · Kelly Hondula · Sean Hughes · Brandon Hurr · Najko Jahn · Tamora D James · Max Joseph · Krunoslav Juraic · Soumya Kalra · Zhian N. Kamvar · Michael Kane · Andee Kaplan · Hazel Kavılı · Os Keyes · Michael Koontz · Bianca Kramer · Will Landau · Erin LeDell · Thomas Leeper · Sam Levin · Stephanie Locke · Robin Lovelace · Julia Stewart Lowndes · Tim Lucas · Andrew MacDonald · Jesse Maegan · Tristan Mahr · Ben Marwick · Miles McBain · Lucy D’Agostino McGowan · Amelia McNamara · Elaine McVey · Bryce Mecum · Francois Michonneau · Helen Rodgers Miller · Jessica Minnier · Priscilla Minotti · Nichole Monhait · Paula Moraga · Ross Mounce · Lincoln Mullen · Matt Mulvahill · Maria Victoria Munafó · Dillon Niederhut · Rory Nolan · Kari Norman · Jakub Nowosad · Daniel Nüst · Paul Oldham · Samantha Oliver · Jeroen Ooms · Philipp Ottolinger · Mark Padgham · Marina Papadopoulou · Edzer Pebesma · Thomas Lin Pedersen · Lindsay Platt · Nicholas Potter · Etienne Racine · Nistara Randhawa · David Ranzolin · Neal Richardson · Emily Riederer · tyler rinker · Emily Robinson · Julia Romanowska · Xavier Rotllan-Puig · Bob Rudis · Edgar Ruiz · Kent Russel · Francisco Rodriguez Sanchez · Alicia Schep · Clemens Schmid · Marco Sciaini · Heidi Seibold · Julia Silge · Margaret Siple · Margaret Siple · Peter Slaughter · Mike Smith · Frank Soboczenski · Tuija Sonkkila · Gaurav Sood · Adam Sparks · Joseph Stachelek · Christine Stawitz · Irene Steves · Michael Sumner · Chung-Kai Sun · Sarah Supp · Jason Taylor · Filipe Teixeira · Andy Teucher · Jennifer Thompson · Joe Thorley · Tiffany Timbers · Tim Trice · Ted Underwood · Kevin Ushey · Josef Uyeda · Frans van Dunné · Mauricio Vargas · Remi Vergnon · Claudia Vitolo · Jake Wagner · Ben Ward · Elin Waring · Rachel Warnock · Leah Wasser · Lukas Weber · Karissa Whiting · Stefan Widgren · Anna Willoughby · Luke Winslow · David Winter · Kara Woo · Bruna Wundervald · Lauren Yamane · Taras Zakharko · Hao Zhu · Chava Zibman · Naupaka Zimmerman · Jake Zwart · Lluís · NA · NA · NA
We are also grateful to the following individuals who have served as guest editors.