A Latent Variable Model State Estimation System for Image Sequences

Self-driving cars need to be able to assess and understand the state of their surroundings. To achieve this goal, it is necessary to construct a model which holds information about the state of the environment based on sensor measurements. In common state estimation systems like Kalman filters, it is necessary to explicitly model state transitions and the observation process. These models have to match the internal dynamics of the observed system as closely as possible to yield reliable estimation results. In this work, we propose a method that can learn an approximation of the internal dynamics of a system, without the need to explicitly model these processes. Our system even works on highly complex data like frames of a video sequence. The approach is based on a latent variable model with a continuous hidden state space. To deal with the fact that the estimated processes are sequential, we use recurrent neural networks. As an example to show the potential of this system, resulting predicted future frames of short video sequences are shown. The proposed system shows a general approach for state estimation without the need for any knowledge about the underlying state transition or observation processes.