Skip to main content

rOpenSci Packages: Development, Maintenance, and Peer Review

8 Guide for Editors

Software Peer Review at rOpenSci is managed by a team of editors. We are piloting a system of a rotating Editor-in-Chief (EiC).

This chapter presents the responsabilities of the Editor-in-Chief, of any editor in charge of a submission, how to respond to an out-of-scope submission and how to manage a dev guide release.

If you’re a guest editor, thanks for helping! Please contact the editor who invited you to handle a submission for any question you might have.

8.1 Handling Editor’s Checklist

8.1.1 Upon submission:

  • Tag issue with 1/editor-checks tag and assign a main editor. Please strive to finish the checks and start looking for reviewers within 5 working days.
  • Use the editor template to guide initial checks and record your response to the submission. You can also streamline your editor checks by using the pkgreviewr package created by associate editor Anna Krystalli
  • Check that template has been properly filled out.
  • Check against policies for fit and overlap. Initiate discussion via Slack #software-review channel if needed for edge cases that haven’t been caught by previous checks by the EiC. If reject, see this section about how to respond.
  • Check that mandatory parts of template are complete. If not, direct authors toward appropriate instructions.
  • Run automated tests: spelling::spell_check_package(), goodpractice::gp() (most exceptions will need to be justified by the author in the particular context of their package.), devtools::spell_check(). Run covr::package_coverage() using NOT_CRAN if needed, as well. Check that documentation is generated using roxygen2, not by hand (this isn’t part of automatic tests yet). Report relevant outputs in the issue thread.
  • For packages needing continuous integration on multiple platforms (cf criteria in this section of the CI chapter) make sure the package gets tested on multiple platforms (having the package built on several operating systems via GitHub Actions for instance).
  • Wherever possible when asking for changes, direct authors to automatic tools such as usethis and styler, and to online resources (sections of this guide, sections of the R packages book) to make your feedback easier to use. Example of editor’s checks.
  • Ideally, the remarks you make should be tackled before reviewers start reviewing.
  • If initial checks show major gaps, request changes before assigning reviewers. If the author mentions changes might take time, apply the holding label.
  • If the package raises a new issue for rOpenSci policy, start a conversation in Slack or open a discussion on the rOpenSci forum to discuss it with other editors (example of policy discussion).

8.1.2 Look for and assign reviewers:

8.1.2.1 Tasks

  • Switch numbered tag to 2/seeking-reviewers.
  • Ask author to add a rOpenSci review badge to their README, via rodev::use_review_badge(), rodev::use_review_badge(<issue_number>). Badge URL is https://badges.ropensci.org/<issue_id>_status.svg. Full link should be:
[![](https://badges.ropensci.org/<issue_id>_status.svg)](https://github.com/ropensci/software-review/issues/<issue_id>)

  • Also ask the authors to add a NEWS file to their package if there’s none yet.

  • Use the email template if needed for inviting reviewers

    • When inviting reviewers, include something like “if I don’t hear from you in a week, I’ll assume you are unable to review,” so as to give a clear deadline when you’ll move on to looking for someone else.

  • Assign a due date 3 weeks after all reviewers have been found.

  • Once two or more reviewers are found, assign reviewers by tagging in the issue with the following format:

Reviewer: @githubname1 
Reviewer: @githubname2
Due date: YYYY-MM-DD
  • Switch numbered tag to 3/reviewers-assigned once reviewers are assigned.
  • Update Airtable database: Add one of the review record numbers associated with the package to the Reviews field of each reviewer’s record. Insert new Reviewers record if required.

8.1.2.2 How to look for reviewers

8.1.2.2.1 Where to look for reviewers?

As a (guest) editor, use

  • the potential suggestions made by the submitter(s), (although submitters may have a narrow view of the types of expertise needed. We suggest not using more than one of suggested reviewers)
  • the Airtable database of reviewers and volunteers
  • and the authors of rOpenSci packages.

When these sources of information are not enough,

  • ping other editors in Slack for ideas,
  • look for users of the package or of the data source/upstream service the package connects to (via their opening issues in the repository, starring it, citing it in papers, talking about it on Twitter).
  • You can also search for authors of related packages on r-pkg.org.
  • R-Ladies has a directory specifying skills and interests of people listed.

8.1.2.3 Criteria for choosing a reviewer

Here are criteria to keep in mind when choosing a reviewer. You might need to piece this information together by searching CRAN and the potential reviewer’s GitHub page and general online presence (personal website, Twitter).

  • Has not reviewed a package for us within the last 6 months.
  • Some package development experience.
  • Some domain experience in the field of the package or data source
  • No conflicts of interest.
  • Try to balance your sense of the potential reviewer’s experience against the complexity of the package.
  • Diversity - with two reviewers both shouldn’t be cis white males.
  • Some evidence that they are interested in openness or R community activities, although blind emailing is fine.

Each submission should be reviewed by two package reviewers. Although it is fine for one of them to have less package development experience and more domain knowledge, the review should not be split in two. Both reviewers need to review the package comprehensively, though from their particular perspective. In general, at least one reviewer should have prior reviewing experience, and of course inviting one new reviewer expands our pool of reviewers.

8.1.3 During review:

  • Check in with reviewers and authors occasionally. Offer clarification and help as needed.
  • In general aim for 3 weeks for review, 2 weeks for subsequent changes, and 1 week for reviewer approval of changes.
  • Upon all reviews being submitted, change the review status tag to 4/review-in-awaiting-changes.
  • Update Airtable database: Add the link to the review comment to the review_url field and the number of review hours to the review_hours field of each review record.
  • If the author stops responding, refer to the policies and/or ping the other editors in the Slack channel for discussion. Importantly, if a reviewer was assigned to a closed issue, contact them when closing the issue to explain the decision, thank them once again for their work, and make a note in our database to assign them to a submission with high chances of smooth software review next time (e.g. a package author who has already submitted packages to us).
  • Upon changes being made, change the review status tag to 5/awaiting-reviewer-response, and request that reviewers indicate approval with the reviewer approval template.

8.1.4 After review:

  • Change the status tag to 6/approved.
  • You can use the comment template.
  • Add review/er information to the review database.
  • If authors intend to submit to CRAN, direct them to the section about CRAN gotchas and offer to provide support through this process.
  • Ask authors to migrate to ropensci
    • Create a two-person team in rOpenSci’s “ropensci” GitHub organization, named for the package, with yourself and the package author as members.
    • Have the author transfer the repository to ropensci
    • Go to the repository settings in rOpenSci’s “ropensci” GitHub organization and give the author “Admin” access to the repository.
  • If the original repository owner opposes transfer, add a line with its address to this repos list to ensure the package gets included in rOpenSci package registry.
  • Ask author to:
    • Add a CodeMeta file by running codemetar::write_codemeta() (codemetar GitHub repo)
    • Change any needed links, such those for CI badges
    • Re-activate CI services
      • For AppVeyor, tell the author to update the GitHub link in their badge, but do not transfer the project: AppVeyor projects should remain under the authors’ account. The badge is [![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/ropensci/pkgname?branch=master&svg=true)](https://ci.appveyor.com/project/individualaccount/pkgname).
      • For Codecov, the webhook may need to be reset by the author.
  • Nominate a package to be featured in an rOpenSci blog post or tech note if you think it might be of high interest. Please note in the software review issue one or two things the author could highlight, and tag @ropensci/blog-editors for follow-up.
  • If authors maintain a gitbook that is at least partly about their package, contact an rOpenSci staff member so they might contact the authors about transfer to the ropensci-books GitHub organisation.
  • Add a “peer-reviewed” topic to the repo.
  • Close the software-review issue.

8.1.5 Package promotion:

8.2 EiC Responsibilities

The EiC serves for 3 months or a time agreed to by all members of the editorial board. The EiC is entitled to taking scope and overlap decisions as independently as possible (but can still request help/advice). In details, the EiC plays the following roles

  • Watches all issues posted to the software-review repo.
  • Assigns package submissions to other editors, including self, to handle. Mostly this just rotates among editors, unless the EiC thinks an editor is particularly suited to a package, or an editor declines handling the submission due to being too busy or because of conflicting interests.
  • Monitors pace of review process and reminds other editors to move packages along as needed.
  • Responds to issues posted to the software-review-meta repo
  • Makes decisions on scope/overlap for pre-submission inquiries, referrals from JOSS or other publication partners, and submissions if they see an ambiguous case (This last case may also be done by handling editors (see below)). To initiate discussion, this is posted to the rOpenSci Slack editors-only channel along with a small summary of what the (pre-)submitted/referred submission is about, what doubts the EiC has i.e. digesting information a bit. If after one day or two the EiC feels they haven’t received enough answers, they can ping all editors.
  • Any editor should feel free to step in on these. See this section about how to respond to out-of-scope (pre-) submissions.

8.2.1 Asking for more details

In some cases online documentation is sparse. Minimal README, no pkgdown website make assessment harder. In that case please ask for more details: even if the package is deemed out-of-scope, the package docs will have gotten better so we are fine asking for these efforts.

Example text

Hello <username> and many thanks for your submission.

We are discussing whether the package is in scope and need a bit more information.

Would you mind adding more details and context to the README?
After reading it someone with little domain knowledge should have been informed about the aim, goals and functionality of the package.

<optional>
If a package has overlapping functionality with other packages, we require it to demonstrate in the documentation [how it is best in class](https://devguide.ropensci.org/policies.html#overlap). Could you add a more detailed comparison to the packages you mention in the README so we can evaluate?
</optional>

8.2.2 Inviting a guest editor

After discussion with other editors the EiC might invite a guest editor to handle a submission (e.g. if all editors have a conflict of interest).

When inviting a guest editor,

If the person said yes (yay!),

  • Make sure they enabled 2FA for their GitHub account,
  • Give them write access to the software-review repository,
  • Once they’ve accepted this repo invitation, assign the issue to them,
  • Ensure they’re (already) invited to rOpenSci Slack workspace,
  • Add their name to the Airtable guest-editor table (so their names might appear in this book and in the software-review README).

After the review process is finished (package approved, issue closed),

  • Thank the guest editor again,
  • Remove their write access from the software-review repository.

8.3 Responding to out-of-scope submissions

Thank authors for their submission, explain the reasons for the decision, and direct them to other publication venues if relevant, and to the rOpenSci discussion forum. Use wording from Aims and scope in particular regarding the evolution of scope over time, and the overlap and differences between unconf/staff/software-review development.

Examples of out-of-scope submissions and responses.

8.4 Managing a dev guide release

If you are in charge of managing a release of the very book you are reading, use the book release guidance as an issue template to be posted in the dev guide issue tracker, and do not hesitate to ask questions to other editors.

8.4.1 Dev guide governance

For very small amendments to the dev guide, no PR review is needed. For larger amendments, request review from at least a few editors (if none participated in the discussion related to the amendment, request a review from all of them on GitHub, and in the absence of any reaction merge after a week).

Two weeks before a dev guide release, once the PR from dev to master and the release blog post are ready for review, all editors should be pinged by GitHub (“review request” on the PR from dev to master) and Slack, but the release doesn’t need all of them to explicitly approve the release.

8.4.2 Blog post about a release

The blog post about a release will be reviewed by editors, and one of @ropensci/blog-editors.

8.4.2.1 Content

Refer to the general rOpenSci blogging guidance, and the more specific guidance below.

First example of such a post; second example.

The blog post should mention all important items from the changelog organized in (sub)sections: e.g. a section about big change A, another one about big change B, and one about smaller changes lumped together. Mention the most important changes first.

For each change made by an external contributor, thank them explicitly using the information from the changelog. E.g. [Matt Fidler](https://github.com/mattfidler/) amended our section on Console messages [ropensci/dev_guide#178](https://github.com/ropensci/dev_guide/pull/178)..

At the end of the post, mention upcoming changes by linking to open issues in the issue tracker, and invite readers to contribute to the dev guide by opening issues and participating in open discussions. Conclusion template:

In this post we summarized the changes incorporated into our book ["rOpenSci Packages: Development, Maintenance, and Peer Review"](https://devguide.ropensci.org/) over the last X months. 
We are grateful for all contributions that made this release possible. 
We are already working on updates for our next version, such as ISSUE1, ISSUE2. 
Check out the [the issue tracker](https://github.com/ropensci/dev_guide/issues/) if you'd like to contribute.

8.4.2.2 Authorship

The editor writing the post is first author, other editors are listed by alphabetical order.