Survey of Protocol Reverse Engineering Algorithms: Decomposition of Tools for Static Traffic Analysis

Knowledge about a network protocol to understand the communication between entities is necessary for vulnerability research, penetration testing, malware analysis, network reconnaissance, and network modeling. Traffic analysis is one approach to infer a protocol, and this approach has specific challenges, tasks, methods, and solutions. In this survey, we collect tools presented by prior research in the field of protocol reverse engineering by static traffic trace analysis. We dissect each tool to discern the individual mechanisms and the algorithms on which they are based, then categorize and contrast the mechanisms and algorithms used in static traffic trace analysis to discuss how successfully they were applied in each case. To structure our discussion about the tools, we compared classification schemes for protocol reverse engineering. We present and discuss an explicit process model for static traffic trace analysis to reveal the common structure of the decomposed tools and frameworks from previous research. Via discussions of the algorithms applied within each tool, we show relations between tools, methods, and the process for each process task. We validate our model by applying it to each of the tools, then provide an outline of the utility of protocol reverse engineering. Beginning with the process description, we deduce which solutions and algorithms have already been investigated and where challenges remain to determine how new solutions may be researched in the future. Across the entire field of protocol reverse engineering, few implementations of tools and frameworks are publicly available, which remains a prevalent problem.