Real-Time Application for Generating Multiple Experiences from 360 degrees Panoramic Video by Tracking Arbitrary Objects and Viewer's Orientations

Featured Application The proposed system specifically targets applications related to 360 degrees multimedia content in virtual reality. Overall, this platform supports the generation of multiple interesting experiences or multimedia contents from a single 360 degrees video, an easy to use authoring system for the 360 degrees content providers, the efficient transferring of the generated 360 degrees experiences to the prospective viewers in virtual reality, and enabling the viewers to share their personal experiences of the 360 degrees multimedia content in virtual reality with the other viewers. Abstract We propose a novel authoring and viewing system for generating multiple experiences with a single 360 degrees video and efficiently transferring these experiences to the user. An immersive video contains much more interesting information within the 360 degrees environment than normal videos. There can be multiple interesting areas within a 360 degrees frame at the same time. Due to the narrow field of view in virtual reality head-mounted displays, a user can only view a limited area of a 360 degrees video. Hence, our system is aimed at generating multiple experiences based on interesting information in different regions of a 360 degrees video and efficient transferring of these experiences to prospective users. The proposed system generates experiences by using two approaches: (1) Recording of the user's experience when the user watches a panoramic video using a virtual reality head-mounted display, and (2) tracking of an arbitrary interesting object in a 360 degrees video selected by the user. For tracking of an arbitrary interesting object, we have developed a pipeline around an existing simple object tracker to adapt it for 360 degrees videos. This tracking algorithm was performed in real time on a CPU with high precision. Moreover, to the best of our knowledge, there is no such existing system that can generate a variety of different experiences from a single 360 degrees video and enable the viewer to watch one 360 degrees visual content from various interesting perspectives in immersive virtual reality. Furthermore, we have provided an adaptive focus assistance technique for efficient transferring of the generated experiences to other users in virtual reality. In this study, technical evaluation of the system along with a detailed user study has been performed to assess the system's application. Findings from evaluation of the system showed that a single 360 degrees multimedia content has the capability of generating multiple experiences and transfers among users. Moreover, sharing of the 360 degrees experiences enabled viewers to watch multiple interesting contents with less effort.