Title of the study: Studying the Impact of Risk Assessment Analytics on Risk Awareness and Code Review Performance
Principal Investigator: Dr. Shane McIntosh, School of Computer Science, University of Waterloo. Email:
shane.mcintosh@uwaterloo.ca Co-Principal Investigator: Dr. Michael Godfrey, School of Computer Science, University of Waterloo. Email:
migod@uwaterloo.ca Student Investigator: Xueyao Yu, MMath Student, School of Computer Science, University of Waterloo. Email:
xueyao.yu@uwaterloo.ca Co-Investigator: Dr. Filipe R. Cogo, Huawei Technologies Canada Co., Ltd. Email:
filipe.roseiro.cogo1@huawei.com To help you make an informed decision regarding your participation, this letter will explain what the study is about, the possible risks and benefits, and your rights as a research participant. If you do not understand something in the letter, please ask one of the investigators prior to consenting to the study. Please print/save a copy of this letter for your records.
What is the study about?
You are invited to participate in a research study about the impact of risk assessment analytics on code review. The purpose of the study is to examine the extent to which risk assessment analytics impact developer awareness of the riskiness of code changes, as well as reviewer effectiveness and efficiency in defect detection during code review.
I. Your responsibilities as a participant
What does participation involve?
Participation in the study will consist of at most three sessions.
(1) You will first be asked to fill out a short pre-experiment questionnaire where we will ask about your development and code review background and confirm your eligibility. If you are eligible to the study, we will then collect your development experience and code review preferences. This questionnaire will take you no more than 3 minutes to complete.
(2) Then we will send you a confirmation email with a link to initiate an experiment. During the experiment, you will be asked to complete two tasks and will be provided with three code changes for each task. The tasks include a) assessing the risk level of each code change and b) identifying the defects in each code change. At the end of the experiment, we will follow up with a questionnaire, which asks about your perceptions during the experiment. The experiment will take approximately 30-60 minutes.
(3) In some rare cases, we may contact you through your provided email address and invite you to a semi-structured follow-up interview via Zoom, in which you will be asked for feedback on the provided risk assessment tool and how it can benefit the code review process. The interview will take no more than 30 minutes and will be audio-recorded. As zoom records video by default, we will delete the video recording files once the interview is over. Your response may be quoted in the write-up of the study. The quotation will be anonymous.
II. Your rights as a participant
Is participation in the study voluntary?
Your participation in this study is voluntary. You may decide to end your participation in the study at any time by not submitting your responses in the questionnaire/experiment or communicating this to the researcher during the interview. In the event that you withdraw from the study, all associated data collected will be immediately destroyed. Once you have finished the study, you can also request your data be removed from the study within 30 days as it is not possible to withdraw your data once papers and publications have been submitted to publishers.
What are the possible benefits of the study?
We will integrate our risk assessment analytics technique with a code review platform. If we can confirm the effectiveness of risk assessment analytics to improve code review, practitioners can adopt our tool in their daily code review activities.
What are the risks associated with the study?
There are no known or anticipated risks associated with participation in this study. If a question, or the discussion, makes you uncomfortable, you can choose not to answer. See above for more details on voluntary participation.
Privacy, Data Retention and Storage
The pre-experiment survey will be conducted over Google Forms. When information is transmitted over the internet privacy cannot be guaranteed. There is always a risk your responses may be intercepted by a third party (e.g., government agencies, hackers). University of Waterloo researchers will not collect or use internet protocol (IP) addresses or other information which could link your participation to your computer or electronic device without first informing you.
The experiment will be conducted over a web application developed by the researchers. The researchers have implemented technical, administrative, and physical safeguards to protect the information provided via the application from loss, misuse, and unauthorized access.
The interview will be conducted over an online platform, Zoom. Zoom has implemented technical, administrative, and physical safeguards to protect the information provided via the Services from loss, misuse, and unauthorized access, disclosure, alteration, or destruction. However, no Internet transmission is ever fully secure or error free.
We will not collect any of your identifiable information except an email address, which will be used for further contact regarding the study. To maximize the anonymity, you can create a specific email address for this study. The data you share will be kept confidential by assigning an ID code so that your identifiable information is not associated with the data and will not appear in any report or publication of the research. Your information will be safely stored in a locked facility in a secured UW server and only accessible by the research team. Your identity will be confidential and the company you work for will not be provided with any participant information. The dataset without identifiers may be shared publicly. Further, research data will be retained for a minimum of seven years at which time it will be deleted.
III. Conflict of Interest
The Co-I (Dr. Filipe Cogo) is an employee of the company (Huawei Technologies Canada Co., Ltd) funding our study and he will have access to the study data. In our study, we will recruit some participants from Huawei. This may be perceived as a potential conflict of interest even though Filipe has no direct professional relationship with participants and his participation in the research stems only from the Huawei-UoW collaboration. To mitigate this potential conflict of interest that may affect study participants and impede voluntary participation, restrict participant autonomy, or prevent informed choices being made, we will not invite employees under Filipe’s supervision to participate in our study. Moreover, we will anonymize the raw data and Filipe will not have access to the data with participants’ identifiable information. Additionally, the PI (Dr. Shane McIntosh) is the PI of another grant that is funded by the partner organization (Huawei Technologies Canada Co., Ltd). However, the funding is not in any way tied to the outcome of this research and we will not have any financial interests from this study.
IV. Questions, comments, or concerns
Who is sponsoring/funding this study?
This study is funded by Huawei Technologies Canada Co., Ltd. Our sponsor (Huawei) will not receive any extra information other than the ones we want to share publicly or publish on an academic paper.
Has the study received ethics clearance?
This study has been reviewed and received ethics clearance through a University of Waterloo Research Ethics Board (REB 44022). If you have questions for the Board contact the Office of Research Ethics, at 1-519-888-4567 ext. 36005 or
reb@uwaterloo.ca Who should I contact if I have questions regarding my participation in the study?
If you have any questions regarding this study or would like additional information to assist you in reaching a decision about participation, please contact Xueyao Yu or the Principal Investigator, Dr. Shane McIntosh, at the contact information listed below.
Xueyao Yu, MMath Student
School of Computer Science, University of Waterloo
xueyao.yu@uwaterloo.ca Dr. Shane McIntosh
School of Computer Science, University of Waterloo
shane.mcintosh@uwaterloo.caLegal Rights and Signatures:
By providing your consent, you are not waiving your legal rights or releasing the investigator(s) or involved institution(s) from their legal and professional responsibilities.
