期刊
IEEE ROBOTICS AND AUTOMATION LETTERS
卷 8, 期 10, 页码 6091-6098出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/LRA.2023.3300252
关键词
Optical and scene flow; multimodal fusion; self- and cross-attention
类别
This study proposes FusionRAFT, a novel deep neural network approach that enables early-stage information fusion between RGB and depth modalities for estimating optical and scene flow. By incorporating self- and cross-attention layers at different network levels, FusionRAFT constructs informative features that leverage the strengths of both modalities. Comparative experiments demonstrate that our approach outperforms recent methods on the FlyingThings3D synthetic dataset and shows better generalization on the real-world KITTI dataset. Furthermore, our approach exhibits improved robustness in the presence of noise and low-lighting conditions affecting RGB images.
This letter presents an investigation into the estimation of optical and scene flow using RGBD information in scenarios where the RGB modality is affected by noise or captured in dark environments. Existing methods typically rely solely on RGB images or fuse the modalities at later stages, which can result in lower accuracy when the RGB information is unreliable. To address this issue, we propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities (RGB and depth). Our approach incorporates self- and cross-attention layers at different network levels to construct informative features that leverage the strengths of both modalities. Through comparative experiments, we demonstrate that our approach outperforms recent methods in terms of performance on the synthetic dataset FlyingThings3D, as well as the generalization on the real-world dataset KITTI. We illustrate that our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据