A deep learning algorithm for simulating autonomous driving considering prior knowledge and temporal information

Autonomous vehicle (AV) stakeholders continue to seek assurance of the safety performance of this new technology through AV testing on in-service roads, AV-dedicated road networks, and AV test tracks. However, recent AV-related fatalities on in-service roads have exacerbated public skepticism and eroded some public trust in the safety of AV operations. Further, test tracks are unable to characterize adequately the real-world driving environment. For this reason, driving simulators continue to serve as an attractive means of AV testing. However, in most AV driving simulators, the AV operation is based on commands external to the vehicle and embedded in the code for the driving environment. To address the simulation shortfalls associated with this approach, this paper develops a deep convolutional neural network-long short-term memory (CNN-LSTM) algorithm for self-driving simulation. This algorithm observes and characterizes the AV's driving environment, and controls the AV movement in the driving simulation. The CNN part extracts features that use transfer learning to introduce human prior knowledge, and the LSTM part uses temporal information to process the extracted features, and incorporates temporal dynamics to predict driving decisions. The AV may also use an external server with a database containing road environment data as an additional source of information. It is acknowledged that different driving simulators differ in their functions and their capabilities to access driving-environment data. Therefore, to make it sufficiently flexible to facilitate replication by other researchers that use driving simulators, the algorithm has been designed and demonstrated using only image data of the driving environment as input. This is because roadway image data are easily and readily accessible from the screen of any driving simulator. The proposed algorithm was tested using the open racing car simulator test track platform and was found to be able to mimic human driving decisions with a high degree of accuracy.