Towards better ML-based software services: an investigation of source code engineering impact

In recent years, the development of machine learning-based solutions for software services, particularly for source code, has grown rapidly. It is witnessed that many machine learning models for software services require the input of source code snippets in a desired form of abstract syntax tree (AST), which is mostly generated from an external tool. However, such data pre-processing tasks could be done by different engineering tools, and the impact of these tools towards final models is often neglected. In this work, we aim to investigate the source code engineering impacts towards machine learning-based software services. Three different types of parsing tools are identified, which are parser generator, parsing library and parser developed for a certain purpose. They are thoroughly evaluated towards the impacts on the prediction model of Code2Vec for the prediction task of the method name in Java language. The collective result on the Java-small dataset shows that the generated ASTs differ a lot in terms of source code structures and contents when using different parsing tools. The difference could influence the performance of the trained model significantly. Our result suggests that when machine learning models are implemented for software services, especially for code-related tasks, the selection of parsing tools should be thoroughly considered during the data pre-processing stage. While there are some interesting findings on Java-med and Java-small, we anticipate this work could provide some insights for better ML-based software service solutions from the perspective of source code engineering.

Automated summary

In recent years, there has been rapid development in machine learning-based software service solutions, specifically for source code. However, the impact of source code engineering on these models is often overlooked. This study evaluates different parsing tools for their impact on a Code2Vec model's prediction task for method names in Java language. The results show that ASTs generated by different parsing tools vary significantly in terms of source code structures and contents, which can significantly affect the model's performance. Therefore, the selection of appropriate parsing tools during data pre-processing is crucial for machine learning models implemented in software services.