4.6 Article

Video driven adaptive grasp planning of virtual hand using deep reinforcement learning

Journal

MULTIMEDIA TOOLS AND APPLICATIONS
Volume 82, Issue 11, Pages 16301-16322

Publisher

SPRINGER
DOI: 10.1007/s11042-022-14190-3

Keywords

Virtual hand; Grasp planning; Motion generation; Deep reinforcement learning; Monocular 3D hand pose estimation

Ask authors/readers for more resources

This paper proposes a fast and easy framework for the design of grasping controllers using abundant and flexible videos of desired grasps and kinematic algorithms based on monocular 3D hand pose estimation and deep reinforcement learning. With this framework, natural and stable grasps can be easily generated from monocular video demonstrations, and adaptive abilities can be added to objects of different shapes and sizes in the target object library.
Data-driven grasp planning can generate anthropopathic grasps, providing controllers with robust and natural responses to environmental changes or morphological discrepancies. Mocap data, which is the widely used source of motion data, can provide high-fidelity dynamic motions. However, it is challenging for non-professionals to quickly get start and collect sufficient mocap data for grasp training. Furthermore, current grasp planning approaches suffer from limited adaptive abilities, and thus cannot be applied to objects of different shapes and sizes directly. In this paper, we propose the first framework, to the best of our knowledge, for fast and easy design of grasping controller with kinematic algorithms based on monocular 3D hand pose estimation and deep reinforcement learning, leveraging abundant and flexible videos of desired grasps. Specially, we first get original grasping sequences through 3D hand pose estimation from given monocular video fragments. Then, we reconstruct the motion sequences using data smoothing based on the peek clipping filter, and further optimize them using the CMA-ES (Covariance Matrix Adaptation Evolution Strategy). Finally, we integrate the reference motion with the adaptive grasping controller through deep reinforcement learning. Quantitative and qualitative results demonstrate that our framework is able to generate natural and stable grasps easily from monocular video demonstrations, added the adaptive ability to primitive objects of different shapes and sizes in the target object library.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available