ERRA: An Embodied Representation and Reasoning Architecture for Long-Horizon Language-Conditioned Manipulation Tasks

This letter introduces ERRA, an embodied learning architecture that enables robots to jointly obtain three fundamental capabilities (reasoning, planning, and interaction) for solving long-horizon language-conditioned manipulation tasks. ERRA is based on tightly-coupled probabilistic inferences at two granularity levels. Coarse-resolution inference is formulated as sequence generation through a large language model, which infers action language from natural language instruction and environment state. The robot then zooms to the fine-resolution inference part to perform the concrete action corresponding to the action language. Fine-resolution inference is constructed as a Markov decision process, which takes action language and environmental sensing as observations and outputs the action. The results of action execution in environments provide feedback for subsequent coarse-resolution reasoning. Such coarse-to-fine inference allows the robot to decompose and achieve long-horizon tasks interactively. In extensive experiments, we show that ERRA can complete various long-horizon manipulation tasks specified by abstract language instructions. We also demonstrate successful generalization to the novel but similar natural language instructions.

Automated summary

This letter presents ERRA, an embodied learning architecture that enables robots to jointly acquire reasoning, planning, and interaction abilities for solving long-horizon language-conditioned manipulation tasks. ERRA integrates coarse-resolution inference using a large language model to infer action language from natural language instruction and environment state, and fine-resolution inference using a Markov decision process to perform concrete actions based on action language. The robot decomposes and accomplishes long-horizon tasks interactively through the coarse-to-fine inference. Extensive experiments demonstrate that ERRA can successfully complete various long-horizon manipulation tasks specified by abstract language instructions and generalize to novel but similar natural language instructions.