Maartje de Jonge, Eelco Visser. Automated Evaluation of Syntax Error Recovery . Technical Report TUD-SERG-2012-035, Delft University of Technology, Software Engineering Research Group, Delft, The Netherlands, 2012. [doi]
Evaluation of parse error recovery techniques is an open problem. The community lacks objective standards and methods to measure the quality of recovery results. This paper provides an automated technique for recovery evaluation. The technique offers a solution for two main problems in this area. First, a representative testset is generated by a mutation based fuzzing technique that applies knowledge about common syntax errors. Secondly, the quality of the recovery results is automatically measured using an oracle-based evaluation technique. We evaluate the validity of our approach by comparing the results obtained by automated evaluation with the results obtained by manual inspection. The evaluation shows a clear correspondence between our quality metric and human judgement.