17-step code review checklist
An interactive checklist for the SORTEE Code Club on code peer review

Access the workshop materials on OSF: https://osf.io/kncw6/ 

The four R questions (reported, run, reliable, reproducible) are taken from a recent paper by Ed Ivimey-Cook et al. (2023) in Journal of Evolutionary Biology: https://doi.org/10.1111/jeb.14230 
Sign in to Google to save your progress. Learn more
Please select the paper and code which you are reviewing
 *
1. Reported
Is the code as reported? Code should be checked for consistency when presented with the corresponding paper.
Does the code match the methods as described in the corresponding article, metadata, and/or documentation?
Clear selection
If you answered "no" above (i.e., the code does not match the documentation), what deviations have you noticed?
2. Run
Is the code executable?  Even if code matches the methods reported in the paper, this does not mean that the code is executable. Programmatic and syntactic errors can make the code fail to run.
Does the code run in its entirety and without issue?
Clear selection
If the answer above is "no" (i.e. the code does not run), provide details on the issues you encountered. For example, missing packages/libraries, deprecated functions, absolute file paths, broken/incomplete scripts, etc. Provide suggestions for correcting these issues where possible.
3. Reliable
Is the code reliable? Errors can propagate through code that runs and produces an output. That is, code can produce incorrect results in a reproducible manner by, for example, selecting a wrong column in a dataset.
Is the code reliable?
Clear selection
If you answered "no" above (i.e., the code is not reliable), what issues have you observed? How would you recommend they be addressed?
Does the code include "unit tests" or other checks that verify the code is working as intended? For example, following a bit of data wrangling or transformation, is there code that checks whether the transformation has been accurately implemented?
Clear selection
4. Reproducible
Can the code produce the output and support the conclusions given in the paper?
Are the results and conclusions reproducible from the code as provided?
Clear selection
If you answered "no" above, which results are not reproducible?
Does the code specify which versions of packages/libraries were used?
Clear selection
Does the code include clear documentation that detail the rationale behind, e.g., data wrangling decisions and analytical approaches?
Clear selection
Is the whole workflow code/script-based? Is it self contained (e.g., the code does not involve steps outside the script or pipeline, such as manual manipulation in other software like Excel)?
Clear selection
5. Organisation & Structure
Good readability of code is very important in enabling effective code review.
Is the code readable? For example, is the code annotated? Does the code apply appropriate indentation? Does the code include headers and sections?
Clear selection
Is the code consistently formatted?
Clear selection
Does the code use informative names for objects?
Clear selection
6. Other, opiniated considerations
Beyond the 4Rs, "good" scripts often share some common features.
Is the code efficient? For example, does it implement functional programming (e.g., does it use generalised, custom functions or for loops to repeat processes)?
Clear selection
Does the code avoid hard-coding (i.e., does the code include sections that assign fixed values or data directly rather than using variables)?
Clear selection
Does the code explicitly calls the package namespace when calling functions (i.e., `package::function()`)?
Clear selection
7. Consideration for public code sharing
These checks are only relevant for code that has been published or archived (e.g., as part of a scientific publication).
Has a license (e.g., MIT, Apache License 2.0, or equivalent) been applied to the code ?
Clear selection
Does the documentation (metadata, README) include clear citations for related materials like data and the article? This may not be applicable in all instances.
Clear selection
Does the code have a persistent identifier (e.g., DOI)?
Clear selection
Feedback
What do you think about the checklist? How straightforward is it to fill in the checklist? Are there any questions that are unclear or ambiguous? 
Please share your thoughts and ideas about our checklist with us!
Submit
Clear form
Never submit passwords through Google Forms.
This content is neither created nor endorsed by Google. - Terms of Service - Privacy Policy

Does this form look suspicious? Report

Google Forms