Context: Various approaches aim to support program comprehension by automatically detecting algorithms in source code. However, no empirical evaluations of their helpfulness have been performed. Objective: To empirically evaluate how algorithm labels - which include the algorithm's name and additional information - impact program comprehension in terms of correctness and time.
Objective: To empirically evaluate how algorithm labels - which include the algorithm's name and additional information - impact program comprehension in terms of correctness and time.
Method: We conducted a controlled experiment with 56 participants, where the experimental group received code with labeled algorithms. The groups completed exercises designed to measure program comprehension as well as a post-questionnaire on label helpfulness, use cases for algorithm recognition, and reasons for self-implementation of algorithms in practice.
Results: Annotating source code with algorithm labels significantly improves program comprehension (p=0.040), with a median improvement of 6 points (~23%), but does not affect completion times (p=0.991). Qualitative analysis revealed that a majority of participants perceived the labels as helpful, especially for recognizing the codes intent. Participants also proposed use cases such as error detection, optimization, and library replacement. Reasons for self-implementing algorithms included library inadequacies, performance needs and avoiding dependencies or licensing costs.
Conclusion: This study shows that algorithm labels improve program comprehension, especially for developers with medium programming experience. Our qualitative analysis also sheds light on how participants benefit from the labels, further use cases for algorithm recognition and motivations behind self-implementing algorithms.
A preprint is available on arxiv: