This chapter contains a general intro to our software peer review system for packages, reasons to submit a package, reasons to volunteer as a reviewer, why our reviews are open, and acknowledgements of actors of the system.
Our system has recently been expanded to statistical software peer-review.
If you use our standards/checklists etc. when reviewing software elsewhere, do tell the recipients (e.g. journal editors, students, internal code review) that they came from rOpenSci, and tell us in our public forum, or privately by email.
rOpenSci’s suite of packages is partly contributed by staff members and partly contributed by community members, which means the suite stems from a great diversity of skills and experience of developers. How to ensure quality for the whole set? That’s where software peer review comes into play: packages contributed by the community undergo a transparent, constructive, non adversarial and open review process. For that process relying mostly on volunteer work, associate editors manage the incoming flow and ensure progress of submissions; authors create, submit and improve their package; reviewers, two per submission, examine the software code and user experience. This blog post written by rOpenSci editors is a good introduction to rOpenSci software peer review Other blog posts about review itself and reviewed packages can be find via the “software-peer-review” tag on rOpenSci blog.
You can recognize rOpenSci packages that have been peer-reviewed via a green “peer-reviewed” badge in their README, linking to their reviews (cf this example); and via a blue comment icon near their description on rOpenSci packages page, also linking to the reviews.
Technically, we make the most of GitHub infrastructure: each package review process is an issue in the ropensci/software-review GitHub repository. For instance, click here to read the review thread of the
ropenaq package: the process is an ongoing conversation until acceptance of the package, with two external reviews as important milestones. Furthermore, we use GitHub features such as the use of issue templates (as submission templates), and labelling which we use to track progress of submissions (from editor checks to approval).
- First, and foremost, we hope you submit your package for review because you value the feedback. We aim to provide useful feedback to package authors and for our review process to be open, non-adversarial, and focused on improving software quality.
- Once aboard, your package will continue to receive support from rOpenSci members. You’ll retain ownership and control of your package, but we can help with ongoing maintenance issues such as those associated with updates to R and dependencies and CRAN policies.
- rOpenSci will promote your package through our webpage, blog, and social media. Packages in our suite also get a documentation website that is automatically built and deployed after each push.
- rOpenSci packages can be cross-listed with other repositories such as CRAN and BioConductor.
- rOpenSci packages that are in scope for the Journal of Open-Source Software and add the necessary accompanying short paper, would, at the discretion of JOSS editors, benefit from a fast-tracked review process.
- If you write one, rOpenSci will promote gitbooks related to your package: the source of such books can be transferred to the
ropensci-booksGitHub organisation for books to be listed at books.ropensci.org.
- As in any peer-review process, we hope you choose to review to give back to the rOpenSci and scientific communities. Our mission to expand access to scientific data and promote a culture of reproducible research is only possible through the volunteer efforts of community members like you.
- Review is a two-way conversation. By reviewing packages, you’ll have the chance to continue to learn development practices from authors and other reviewers.
- The open nature of our review process allows you to network and meet colleagues and collaborators through the review process. Our community is friendly and filled with supportive members expert in R development and many other areas of science and scientific computing.
- To volunteer to be one of our reviewers, fill out this short form providing your contact information and areas of expertise. We are always looking for more reviewers with both general package-writing experience and domain expertise in the fields where packages are used.
Our reviewing threads are public. Authors, reviewers, and editors all know each other’s identities. The broader community can view or even participate in the conversation as it happens. This provides an incentive to be thorough and provide non-adversarial, constructive reviews. Both authors and reviewers report that they enjoy and learn more from this open and direct exchange. It also has the benefit of building a community. Participants have the opportunity to meaningfully network with new peers, and new collaborations have emerged via ideas spawned during the review process.
We are aware that open systems can have drawbacks. For instance, in traditional academic review, double-blind peer review can increase representation of female authors, suggesting bias in non-blind reviews. It is also possible reviewers are less critical in open review. However, we posit that the openness of the review conversation provides a check on review quality and bias; it’s harder to inject unsupported or subjective comments in public and without the cover of anonymity. Ultimately, we believe that having direct and public communication between authors and reviewers improves quality and fairness of reviews.
Furthermore, authors and reviewers have the ability to contact privately the editors if they have any doubt or question.
- Your package README will feature a peer-review badge linking to the software review thread.
- Your package will get a
docs.ropensci.orgdocs website that you can link from DESCRIPTION.
- Your package repo will be transferred to the rOpenSci organization.
- If reviewers agree to be listed in DESCRIPTION, their metadata will mention the review.
rOpenSci’s Software Peer Review process is run by:
- Noam Ross, EcoHealth Alliance
- Karthik Ram, rOpenSci
- Maëlle Salmon, rOpenSci
- Mark Padgham, rOpenSci
- Anna Krystalli, University of Sheffield RSE
- Melina Vidoni, RMIT University (School of Science)
- Mauro Lepore, 2 Degrees Investing Initiative
- Laura DeCicco, USGS
- Julia Gustavsen, Agroscope
- Emily Riederer, Capital One
- Adam Sparks, Department of Primary Industries and Regional Development
- Jeff Hollister, US Environmental Protection Agency
We are grateful to the following individuals who have offered up their time and expertise to review packages submitted to rOpenSci.
Sam Albers · Toph Allen · Kaique dos S. Alves · Brooke Anderson · Alison Appling · Zebulun Arendsee · Taylor Arnold · Al-Ahmadgaid B. Asaad · Dean Attali · Mara Averick · Suzan Baert · James Balamuta · Vikram Baliga · David Bapst · Joëlle Barido-Sottani · Allison Barner · Cale Basaraba · John Baumgartner · Marcus Beck · Gabriel Becker · Jason Becker · Dom Bennett · Ken Benoit · Aaron Berdanier · Fred Boehm · Carl Boettiger · Will Bolton · Ben Bond-Lamberty · Anne-Sophie Bonnet-Lebrun · Alison Boyer · Abby Bratt · François Briatte · Eric Brown · Julien Brun · Jenny Bryan · Lukas Burk · Lorenzo Busetto · Mario Gavidia Calderón · Brad Cannell · Kevin Cazelles · Scott Chamberlain · Cathy Chamberlin · Jennifer Chang · Pierre Chausse · Jorge Cimentada · Nicholas Clark · Chase Clark · Jon Clayden · Will Cornwell · Nic Crane · Enrico Crema · Ildiko Czeller · Kauê de Sousa · Christophe Dervieux · Amanda Dobbyn · Jasmine Dumas · Remko Duursma · Mark Edmondson · Paul Egeler · Evan Eskew · Salvador Fernandez · Alexander Fischer · Kim Fitter · Robert M Flight · Sydney Foks · Zachary Stephen Longiaru Foster · Auriel Fournier · Carl Ganz · Duncan Garmonsway · Sharla Gelfand · Monica Gerber · Duncan Gillespie · David Gohel · Rohit Goswami · Laura Graham · Charles Gray · Matthias Grenié · Corinna Gries · Hugo Gruson · W Kyle Hamilton · Ivan Hanigan · Jeffrey Hanson · Rayna Harris · Ted Hart · Nujcharee Haswell · Verena Haunschmid · Stephanie Hazlitt · Andrew Heiss · Max Held · Anna Hepworth · Bea Hernandez · Jim Hester · Peter Hickey · Roel Hogervorst · Kelly Hondula · Allison Horst · Sean Hughes · James Hunter · Brandon Hurr · Ger Inberg · Christopher Jackson · Najko Jahn · Tamora D James · Mike Johnson · Will Jones · Max Joseph · Krunoslav Juraic · Soumya Kalra · Zhian N. Kamvar · Michael Kane · Andee Kaplan · Hazel Kavılı · Jonathan Keane · Os Keyes · Aaron A. King · Michael Koontz · Bianca Kramer · Will Landau · Erin LeDell · Thomas Leeper · Sam Levin · Stephanie Locke · Marion Louveaux · Robin Lovelace · Julia Stewart Lowndes · Tim Lucas · Andrew MacDonald · Jesse Maegan · Tristan Mahr · Paula Andrea Martinez · Joao Martins · Ben Marwick · Claire Mason · Miles McBain · Lucy D’Agostino McGowan · Amelia McNamara · Elaine McVey · Bryce Mecum · François Michonneau · Mario Miguel · Helen Miller · Beatriz Milz · Jessica Minnier · Priscilla Minotti · Nichole Monhait · Kelsey Montgomery · Paula Moraga · Ross Mounce · Athanasia Monika Mowinckel · Lincoln Mullen · Matt Mulvahill · Maria Victoria Munafó · David Neuzerling · Dillon Niederhut · Rory Nolan · Kari Norman · Jakub Nowosad · Matt Nunes · Daniel Nüst · Joseph O’Brien · Paul Oldham · Samantha Oliver · Dan Olner · Jeroen Ooms · Luis Osorio · Philipp Ottolinger · Mark Padgham · Marina Papadopoulou · Edzer Pebesma · Thomas Lin Pedersen · Marcelo S. Perlin · Rafael Pilliard-Hellwig · Rodrigo Neto Pires · Lindsay Platt · Nicholas Potter · Etienne Racine · Manuel Ramon · Nistara Randhawa · David Ranzolin · Quentin Read · Neal Richardson · tyler rinker · Emily Robinson · David Robinson · Francisco Rodriguez-Sanchez · Julia Romanowska · Xavier Rotllan-Puig · Bob Rudis · Edgar Ruiz · Kent Russel · Alicia Schep · Klaus Schliep · Clemens Schmid · Patrick Schratz · Marco Sciaini · Heidi Seibold · Julia Silge · Margaret Siple · Peter Slaughter · Mike Smith · Tuija Sonkkila · Jemma Stachelek · Christine Stawitz · Irene Steves · Kelly Street · Alex Stringer · Michael Sumner · Chung-Kai Sun · Sarah Supp · Jason Taylor · Filipe Teixeira · Andy Teucher · Jennifer Thompson · Joe Thorley · Tiffany Timbers · Tan Tran · Tim Trice · Kyle Ueyama · Ted Underwood · Adithi R. Upadhya · Kevin Ushey · Josef Uyeda · Frans van Dunné · Mauricio Vargas · Remi Vergnon · Jake Wagner · Ben Ward · Elin Waring · Rachel Warnock · Leah Wasser · Lukas Weber · Marc Weber · Karissa Whiting · Stefan Widgren · Anna Willoughby · Saras Windecker · Luke Winslow · David Winter · Witold Wolski · Kara Woo · Marvin N. Wright · Bruna Wundervald · Lauren Yamane · Emily Zabor · Taras Zakharko · Hao Zhu · Chava Zibman · Naupaka Zimmerman · Jake Zwart · Flury · Pachá · Rich · Claudia · Jasmine · Lluís · gaurav
We are also grateful to the following individuals who have served as guest editors.